[Pkg-xen-devel] Bug#666135: Multiple "Domain-0", slow libvirt

Wed Mar 28 23:39:12 UTC 2012

Package: xen-hypervisor-4.0-amd64
Version: 4.0.1-4
Severity: normal

Hi all,

I have a problem with libvirt, it works slow. For example, "virsh
list" takes 6..7 seconds.

I ask xen-users list with no luck. According to google, this problem
is not wide spreaded, so maybe it's debian specific. If not, please
hint where I should go next.

I found this in libvirt log:

...
11:38:12.765: debug : virEventDispatchHandles:477 : Dispatch n=8 f=56
w=9 e=1 0x25ce830
11:38:12.765: debug : virEventUpdateHandleImpl:147 : Update handle w=8 e=3
11:38:12.765: debug : virEventInterruptLocked:668 : Interrupting
11:38:12.766: debug : xenStoreDomainIntroduced:1421 : Some domains
were missing, trying again
11:38:12.867: debug : xenStoreDomainIntroduced:1421 : Some domains
were missing, trying again
11:38:12.968: debug : xenStoreDomainIntroduced:1421 : Some domains
were missing, trying again
(about 20 such entries)
11:38:14.681: debug : xenStoreDomainIntroduced:1421 : Some domains
were missing, trying again
11:38:14.782: debug : virEventDispatchHandles:464 : i=9 w=10
11:38:14.782: debug : virEventCleanupTimeouts:495 : Cleanup 3
11:38:14.782: debug : virEventCleanupHandles:536 : Cleanupo 10
...

At least 2 seconds spent here. And there is a missing domains
somewhere in configs.

Here they are (xenstored config via script from
http://wiki.xensource.com/xenwiki/XenStore):

/vm/00000000-0000-0000-0000-000000000000/on_xend_stop=ignore
/vm/00000000-0000-0000-0000-000000000000/shadow_memory=0
/vm/00000000-0000-0000-0000-000000000000/uuid=00000000-0000-0000-0000-000000000000
/vm/00000000-0000-0000-0000-000000000000/on_reboot=restart
/vm/00000000-0000-0000-0000-000000000000/image/ostype=linux
/vm/00000000-0000-0000-0000-000000000000/image/kernel=
/vm/00000000-0000-0000-0000-000000000000/image/cmdline=
/vm/00000000-0000-0000-0000-000000000000/image/ramdisk=
/vm/00000000-0000-0000-0000-000000000000/on_poweroff=destroy
/vm/00000000-0000-0000-0000-000000000000/bootloader_args=
/vm/00000000-0000-0000-0000-000000000000/on_xend_start=ignore
/vm/00000000-0000-0000-0000-000000000000/on_crash=restart
/vm/00000000-0000-0000-0000-000000000000/xend/restart_count=0
/vm/00000000-0000-0000-0000-000000000000/vcpus=4
/vm/00000000-0000-0000-0000-000000000000/vcpu_avail=15
/vm/00000000-0000-0000-0000-000000000000/bootloader=
/vm/00000000-0000-0000-0000-000000000000/name=Domain-0
/vm/00000000-0000-0000-0000-000000000000-1/on_xend_stop=ignore
/vm/00000000-0000-0000-0000-000000000000-1/shadow_memory=0
/vm/00000000-0000-0000-0000-000000000000-1/uuid=00000000-0000-0000-0000-000000000000
/vm/00000000-0000-0000-0000-000000000000-1/on_reboot=restart
/vm/00000000-0000-0000-0000-000000000000-1/image/ostype=linux
/vm/00000000-0000-0000-0000-000000000000-1/image/kernel=
/vm/00000000-0000-0000-0000-000000000000-1/image/cmdline=
/vm/00000000-0000-0000-0000-000000000000-1/image/ramdisk=
/vm/00000000-0000-0000-0000-000000000000-1/on_poweroff=destroy
/vm/00000000-0000-0000-0000-000000000000-1/bootloader_args=
/vm/00000000-0000-0000-0000-000000000000-1/on_xend_start=ignore
/vm/00000000-0000-0000-0000-000000000000-1/on_crash=restart
/vm/00000000-0000-0000-0000-000000000000-1/xend/restart_count=0
/vm/00000000-0000-0000-0000-000000000000-1/vcpus=4
/vm/00000000-0000-0000-0000-000000000000-1/vcpu_avail=15
/vm/00000000-0000-0000-0000-000000000000-1/bootloader=
/vm/00000000-0000-0000-0000-000000000000-1/name=Domain-0
/local/domain/0/vm=/vm/00000000-0000-0000-0000-000000000000-1
/local/domain/0/device=
...

I have two (maybe more) "Domain-0" in "vm" section, and every reboot
adds a new one.

I run "xenstore-rm /vm/00000000-0000-0000-0000-000000000000",
"xenstore-rm /vm/00000000-0000-0000-0000-000000000000-1" and so on for
every entry. After reboot I have one "Domain-0" and "virsh list" takes
0,4 second until next reboot.

Part of xend.log (reboot):

[2012-03-21 12:17:51 1768] INFO (SrvDaemon:220) Xend exited with status 0.
[2012-03-21 12:20:53 1791] INFO (SrvDaemon:332) Xend Daemon started
[2012-03-21 12:20:53 1791] INFO (SrvDaemon:336) Xend changeset: unavailable.
[2012-03-21 12:20:55 1791] DEBUG (XendDomainInfo:149)
XendDomainInfo.recreate({'max_vcpu_id': 3, 'cpu_time': 10554784758L,
'ssidref': 0, 'hvm': 0, 'shutdown_reason': 255, 'dying': 0,
'online_vcpus': 4, 'domid': 0, 'paused': 0, 'crashed': 0, 'running':
1, 'maxmem_kb': 17179869180L, 'shutdown': 0, 'mem_kb': 3400960L,
'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'blocked':
0, 'name': 'Domain-0'})
[2012-03-21 12:20:55 1791] INFO (XendDomainInfo:167) Recreating domain
0, UUID 00000000-0000-0000-0000-000000000000. at /local/domain/0
[2012-03-21 12:20:55 1791] DEBUG (XendDomainInfo:3400) Storing VM
details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid':
'00000000-0000-0000-0000-000000000000', 'on_reboot': 'restart',
'image': '(linux (kernel ) (superpages 0) (tsc_mode 0) (nomigrate
0))', 'on_poweroff': 'destroy', 'bootloader_args': '',
'on_xend_start': 'ignore', 'on_crash': 'restart',
'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15',
'bootloader': '', 'name': 'Domain-0'}
[2012-03-21 12:20:55 1791] DEBUG (XendDomainInfo:1804) Storing domain
details: {'cpu/3/availability': 'online', 'description': '',
'console/limit': '1048576', 'memory/target': '3400960',
'cpu/2/availability': 'online', 'vm':
'/vm/00000000-0000-0000-0000-000000000000-1', 'domid': '0',
'cpu/0/availability': 'online', 'cpu/1/availability': 'online',
'control/platform-feature-multiprocessor-suspend': '1',
'console/type': 'xenconsoled', 'name': 'Domain-0'}
[2012-03-21 12:20:55 1791] DEBUG (XendDomain:464) Adding Domain: 0
[2012-03-21 12:20:55 1791] DEBUG (XendDomain:398) number of vcpus to use is 0
[2012-03-21 12:20:55 1791] DEBUG (XendDomainInfo:1891)
XendDomainInfo.handleShutdownWatch
[2012-03-21 12:20:55 1791] INFO (SrvServer:177) unix
path=/var/lib/xend/xend-socket
...

Xen creates a new vm "Domain-0" instead of using an existing one. I
have no idea how I can tune it, so I suppose this is a bug.

My software:
root at sl03500008180:~# dpkg -l | grep xen
ii  libxenstore3.0                      4.0.1-4
Xenstore communications library for Xen
ii  linux-image-2.6.32-5-xen-amd64      2.6.32-41
Linux 2.6.32 for 64-bit PCs, Xen dom0 support
ii  xen-hypervisor-4.0-amd64            4.0.1-4
The Xen Hypervisor on AMD64
ii  xen-linux-system-2.6-xen-amd64      2.6.32+29
Xen system with Linux 2.6 for 64-bit PCs (meta-package)
ii  xen-linux-system-2.6.32-5-xen-amd64 2.6.32-41
Xen system with Linux 2.6.32 on 64-bit PCs (meta-package)
ii  xen-qemu-dm-4.0                     4.0.1-2
Xen Qemu Device Model virtual machine hardware emulator
ii  xen-utils-4.0                       4.0.1-4
XEN administrative tools
ii  xen-utils-common                    4.0.0-1
XEN administrative tools - common files
ii  xenstore-utils                      4.0.1-4
Xenstore utilities for Xen

Regards,
Dmitry

-- System Information:
Debian Release: 6.0.4
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.32-5-xen-amd64 (SMP w/4 CPU cores)
Locale: LANG=ru_RU.UTF-8, LC_CTYPE=ru_RU.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

xen-hypervisor-4.0-amd64 depends on no packages.

Versions of packages xen-hypervisor-4.0-amd64 recommends:
ii  xen-utils-4.0                 4.0.1-4    XEN administrative tools

Versions of packages xen-hypervisor-4.0-amd64 suggests:
pn  xen-docs-4.0                  <none>     (no description available)

-- no debconf information