[Pkg-libvirt-maintainers] Bug#910857: libvirt-daemon: race condition on reboot with spice monitor

Axel Kittenberger axkibe at gmail.com
Fri Oct 12 13:16:42 BST 2018


Package: libvirt-daemon
Version: 3.0.0-4+deb9u3
Severity: normal

Dear Maintainer,

* What led up to the situation?

Tonight I had a powerfailure on our virtual machine host, when I came into the
office in the morning I noticed, when it came back up all virtual machines that
had Spice configured as display had failed to start, the ones that had VNC were
running tough (I didn't care about this setting, and the default template
changed sometime)

* What exactly did you do (or not do) that was effective (or ineffective)?

I could manually start the virtual machines in the morning without any issue.
I now changed them all the VNC as display so the issue shouldn't bother me if
another unexpected reboot should happen. It should IMO be fixed tough.

This is a production system I cannot retest rebooting this for no reason /
informing users etc.

I manually had rebooted the host 2 weeks ago after a kernel upgrade, then
everything started without issue. Thus I suppose this is simply a boot race
condition from systemd, libvirtd tries to get things running before something
else is ready?

Kind regards, Axel


********* /var/log/syslog *******
....
Oct 12 01:39:49 vmhost kernel: [  102.736770] br0: port 2(vnet0) entered disabled state
Oct 12 01:39:49 vmhost kernel: [  102.738711] device vnet0 left promiscuous mode
Oct 12 01:39:49 vmhost kernel: [  102.738714] br0: port 2(vnet0) entered disabled state
Oct 12 01:39:50 vmhost libvirtd[1181]: 2018-10-11 23:39:50.253+0000: 1448: info : libvirt version: 3.0.0, package: 4+deb9u3 (Guido Günther <agx at sigxcpu.org> Mon, 12 Mar 2018 19:11:51 +0100)
Oct 12 01:39:50 vmhost libvirtd[1181]: 2018-10-11 23:39:50.253+0000: 1448: info : hostname: vmhost
Oct 12 01:39:50 vmhost libvirtd[1181]: 2018-10-11 23:39:50.253+0000: 1448: error : qemuMonitorOpenUnix:376 : failed to connect to monitor socket: No such process
Oct 12 01:39:50 vmhost libvirtd[1181]: 2018-10-11 23:39:50.254+0000: 1448: error : qemuProcessReportLogError:1792 : internal error: process exited while connecting to monitor: el1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
Oct 12 01:39:50 vmhost libvirtd[1181]: ((null):1898): Spice-Warning **: reds.c:2489:reds_init_socket: getaddrinfo(127.0.0.1,5900): Address family for hostname not supported
Oct 12 01:39:50 vmhost libvirtd[1181]: 2018-10-11T23:39:49.367704Z qemu-system-x86_64: failed to
Oct 12 01:39:50 vmhost virtlogd[1870]: 2018-10-11 23:39:50.254+0000: 1870: info : libvirt version: 3.0.0, package: 4+deb9u3 (Guido Günther <agx at sigxcpu.org> Mon, 12 Mar 2018 19:11:51 +0100)
Oct 12 01:39:50 vmhost virtlogd[1870]: 2018-10-11 23:39:50.254+0000: 1870: info : hostname: vmhost
Oct 12 01:39:50 vmhost virtlogd[1870]: 2018-10-11 23:39:50.254+0000: 1870: error : virNetSocketReadWire:1801 : End of file while reading data: Input/output error
Oct 12 01:39:50 vmhost virtlogd[1870]: 2018-10-11 23:39:50.311+0000: 1870: error : virNetSocketReadWire:1801 : End of file while reading data: Input/output error
Oct 12 01:39:50 vmhost libvirtd[1181]: 2018-10-11 23:39:50.328+0000: 1448: error : qemuAutostartDomain:297 : internal error: Failed to autostart VM 'mymachine': internal error: process exited while connecting to monitor: el1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
Oct 12 01:39:50 vmhost libvirtd[1181]: ((null):1898): Spice-Warning **: reds.c:2489:reds_init_socket: getaddrinfo(127.0.0.1,5900): Address family for hostname not supported
Oct 12 01:39:50 vmhost libvirtd[1181]: 2018
Oct 12 01:39:50 vmhost kernel: [  103.792593] br0: port 2(vnet0) entered blocking state
Oct 12 01:39:50 vmhost kernel: [  103.792595] br0: port 2(vnet0) entered disabled state
....
************************
This is repeated for all virtual machines that failed to autostart.


Log for example /var/log/libvirt/qemu/mymachine.log
****************************************
2018-10-11 23:39:48.171+0000: starting up libvirt version: 3.0.0, package: 4+deb9u3 (Guido Günther <agx at sigxcpu.org> Mon, 12 Mar 2018 19:11:51 +0100), qemu version: 2.8.1(Debian 1:2.8+dfsg-6+deb9u4), hostname: vmhost
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=spice /usr/bin/kvm -name guest=mymachine,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-mymachine/master-key.aes -machine pc-i440fx-2.8,accel=kvm,usb=off,vmport=off,dump-guest-core=off -cpu Broadwell,+vme,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+dca,+osxsave,+f16c,+rdrand,+arat,+tsc_adjust,+xsaveopt,+pdpe1gb,+abm -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 9b712a4a-1d34-481c-b934-509bbe9cd77e -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-mymachine/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive if=none,id=drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/vm/mymachine.img,format=qcow2,if=none,id=drive-ide0-0-1 -device ide-hd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1,bootindex=1 -netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:38:d4:1e,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-1-mymachine/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
char device redirected to /dev/pts/0 (label charserial0)
((null):1898): Spice-Warning **: reds.c:2489:reds_init_socket: getaddrinfo(127.0.0.1,5900): Address family for hostname not supported
2018-10-11T23:39:49.367704Z qemu-system-x86_64: failed to initialize spice server
2018-10-11 23:39:50.254+0000: shutting down, reason=failed

*****************************************

-- System Information:
Debian Release: 9.5
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.9.0-8-amd64 (SMP w/32 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages libvirt-daemon depends on:
ii  libapparmor1        2.11.0-3+deb9u2
ii  libaudit1           1:2.6.7-2
ii  libavahi-client3    0.6.32-2
ii  libavahi-common3    0.6.32-2
ii  libblkid1           2.29.2-1+deb9u1
ii  libc6               2.24-11+deb9u3
ii  libcap-ng0          0.7.7-3+b1
ii  libdbus-1-3         1.10.26-0+deb9u1
ii  libdevmapper1.02.1  2:1.02.137-2
ii  libfuse2            2.9.7-1+deb9u1
ii  libgnutls30         3.5.8-5+deb9u3
ii  libnetcf1           1:0.2.8-1+b2
ii  libnl-3-200         3.2.27-2
ii  libnl-route-3-200   3.2.27-2
ii  libnuma1            2.0.11-2.1
ii  libparted2          3.2-17
ii  libpcap0.8          1.8.1-3
ii  libpciaccess0       0.13.4-1+b2
ii  librados2           10.2.5-7.2
ii  librbd1             10.2.5-7.2
ii  libsasl2-2          2.1.27~101-g0780600+dfsg-3
ii  libselinux1         2.6-3+b3
ii  libssh2-1           1.7.0-1
ii  libudev1            232-25+deb9u4
ii  libvirt0            3.0.0-4+deb9u3
ii  libxen-4.8          4.8.4+xsa273+shim4.10.1+xsa273-1+deb9u10
ii  libxenstore3.0      4.8.4+xsa273+shim4.10.1+xsa273-1+deb9u10
ii  libxml2             2.9.4+dfsg1-2.2+deb9u2
ii  libyajl2            2.1.0-2+b3

Versions of packages libvirt-daemon recommends:
ii  libxml2-utils   2.9.4+dfsg1-2.2+deb9u2
ii  netcat-openbsd  1.130-3
ii  qemu-kvm        1:2.8+dfsg-6+deb9u4

Versions of packages libvirt-daemon suggests:
ii  libvirt-daemon-system  3.0.0-4+deb9u3
pn  numad                  <none>

-- no debconf information


More information about the Pkg-libvirt-maintainers mailing list