[Pkg-libvirt-maintainers] Bug#910857: libvirt-daemon: race condition on reboot with spice monitor
Axel Kittenberger
axkibe at gmail.com
Fri Oct 12 13:16:42 BST 2018
Package: libvirt-daemon
Version: 3.0.0-4+deb9u3
Severity: normal
Dear Maintainer,
* What led up to the situation?
Tonight I had a powerfailure on our virtual machine host, when I came into the
office in the morning I noticed, when it came back up all virtual machines that
had Spice configured as display had failed to start, the ones that had VNC were
running tough (I didn't care about this setting, and the default template
changed sometime)
* What exactly did you do (or not do) that was effective (or ineffective)?
I could manually start the virtual machines in the morning without any issue.
I now changed them all the VNC as display so the issue shouldn't bother me if
another unexpected reboot should happen. It should IMO be fixed tough.
This is a production system I cannot retest rebooting this for no reason /
informing users etc.
I manually had rebooted the host 2 weeks ago after a kernel upgrade, then
everything started without issue. Thus I suppose this is simply a boot race
condition from systemd, libvirtd tries to get things running before something
else is ready?
Kind regards, Axel
********* /var/log/syslog *******
....
Oct 12 01:39:49 vmhost kernel: [ 102.736770] br0: port 2(vnet0) entered disabled state
Oct 12 01:39:49 vmhost kernel: [ 102.738711] device vnet0 left promiscuous mode
Oct 12 01:39:49 vmhost kernel: [ 102.738714] br0: port 2(vnet0) entered disabled state
Oct 12 01:39:50 vmhost libvirtd[1181]: 2018-10-11 23:39:50.253+0000: 1448: info : libvirt version: 3.0.0, package: 4+deb9u3 (Guido Günther <agx at sigxcpu.org> Mon, 12 Mar 2018 19:11:51 +0100)
Oct 12 01:39:50 vmhost libvirtd[1181]: 2018-10-11 23:39:50.253+0000: 1448: info : hostname: vmhost
Oct 12 01:39:50 vmhost libvirtd[1181]: 2018-10-11 23:39:50.253+0000: 1448: error : qemuMonitorOpenUnix:376 : failed to connect to monitor socket: No such process
Oct 12 01:39:50 vmhost libvirtd[1181]: 2018-10-11 23:39:50.254+0000: 1448: error : qemuProcessReportLogError:1792 : internal error: process exited while connecting to monitor: el1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
Oct 12 01:39:50 vmhost libvirtd[1181]: ((null):1898): Spice-Warning **: reds.c:2489:reds_init_socket: getaddrinfo(127.0.0.1,5900): Address family for hostname not supported
Oct 12 01:39:50 vmhost libvirtd[1181]: 2018-10-11T23:39:49.367704Z qemu-system-x86_64: failed to
Oct 12 01:39:50 vmhost virtlogd[1870]: 2018-10-11 23:39:50.254+0000: 1870: info : libvirt version: 3.0.0, package: 4+deb9u3 (Guido Günther <agx at sigxcpu.org> Mon, 12 Mar 2018 19:11:51 +0100)
Oct 12 01:39:50 vmhost virtlogd[1870]: 2018-10-11 23:39:50.254+0000: 1870: info : hostname: vmhost
Oct 12 01:39:50 vmhost virtlogd[1870]: 2018-10-11 23:39:50.254+0000: 1870: error : virNetSocketReadWire:1801 : End of file while reading data: Input/output error
Oct 12 01:39:50 vmhost virtlogd[1870]: 2018-10-11 23:39:50.311+0000: 1870: error : virNetSocketReadWire:1801 : End of file while reading data: Input/output error
Oct 12 01:39:50 vmhost libvirtd[1181]: 2018-10-11 23:39:50.328+0000: 1448: error : qemuAutostartDomain:297 : internal error: Failed to autostart VM 'mymachine': internal error: process exited while connecting to monitor: el1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
Oct 12 01:39:50 vmhost libvirtd[1181]: ((null):1898): Spice-Warning **: reds.c:2489:reds_init_socket: getaddrinfo(127.0.0.1,5900): Address family for hostname not supported
Oct 12 01:39:50 vmhost libvirtd[1181]: 2018
Oct 12 01:39:50 vmhost kernel: [ 103.792593] br0: port 2(vnet0) entered blocking state
Oct 12 01:39:50 vmhost kernel: [ 103.792595] br0: port 2(vnet0) entered disabled state
....
************************
This is repeated for all virtual machines that failed to autostart.
Log for example /var/log/libvirt/qemu/mymachine.log
****************************************
2018-10-11 23:39:48.171+0000: starting up libvirt version: 3.0.0, package: 4+deb9u3 (Guido Günther <agx at sigxcpu.org> Mon, 12 Mar 2018 19:11:51 +0100), qemu version: 2.8.1(Debian 1:2.8+dfsg-6+deb9u4), hostname: vmhost
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=spice /usr/bin/kvm -name guest=mymachine,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-mymachine/master-key.aes -machine pc-i440fx-2.8,accel=kvm,usb=off,vmport=off,dump-guest-core=off -cpu Broadwell,+vme,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+dca,+osxsave,+f16c,+rdrand,+arat,+tsc_adjust,+xsaveopt,+pdpe1gb,+abm -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 9b712a4a-1d34-481c-b934-509bbe9cd77e -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-mymachine/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive if=none,id=drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/vm/mymachine.img,format=qcow2,if=none,id=drive-ide0-0-1 -device ide-hd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1,bootindex=1 -netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:38:d4:1e,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-1-mymachine/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
char device redirected to /dev/pts/0 (label charserial0)
((null):1898): Spice-Warning **: reds.c:2489:reds_init_socket: getaddrinfo(127.0.0.1,5900): Address family for hostname not supported
2018-10-11T23:39:49.367704Z qemu-system-x86_64: failed to initialize spice server
2018-10-11 23:39:50.254+0000: shutting down, reason=failed
*****************************************
-- System Information:
Debian Release: 9.5
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 4.9.0-8-amd64 (SMP w/32 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages libvirt-daemon depends on:
ii libapparmor1 2.11.0-3+deb9u2
ii libaudit1 1:2.6.7-2
ii libavahi-client3 0.6.32-2
ii libavahi-common3 0.6.32-2
ii libblkid1 2.29.2-1+deb9u1
ii libc6 2.24-11+deb9u3
ii libcap-ng0 0.7.7-3+b1
ii libdbus-1-3 1.10.26-0+deb9u1
ii libdevmapper1.02.1 2:1.02.137-2
ii libfuse2 2.9.7-1+deb9u1
ii libgnutls30 3.5.8-5+deb9u3
ii libnetcf1 1:0.2.8-1+b2
ii libnl-3-200 3.2.27-2
ii libnl-route-3-200 3.2.27-2
ii libnuma1 2.0.11-2.1
ii libparted2 3.2-17
ii libpcap0.8 1.8.1-3
ii libpciaccess0 0.13.4-1+b2
ii librados2 10.2.5-7.2
ii librbd1 10.2.5-7.2
ii libsasl2-2 2.1.27~101-g0780600+dfsg-3
ii libselinux1 2.6-3+b3
ii libssh2-1 1.7.0-1
ii libudev1 232-25+deb9u4
ii libvirt0 3.0.0-4+deb9u3
ii libxen-4.8 4.8.4+xsa273+shim4.10.1+xsa273-1+deb9u10
ii libxenstore3.0 4.8.4+xsa273+shim4.10.1+xsa273-1+deb9u10
ii libxml2 2.9.4+dfsg1-2.2+deb9u2
ii libyajl2 2.1.0-2+b3
Versions of packages libvirt-daemon recommends:
ii libxml2-utils 2.9.4+dfsg1-2.2+deb9u2
ii netcat-openbsd 1.130-3
ii qemu-kvm 1:2.8+dfsg-6+deb9u4
Versions of packages libvirt-daemon suggests:
ii libvirt-daemon-system 3.0.0-4+deb9u3
pn numad <none>
-- no debconf information
More information about the Pkg-libvirt-maintainers
mailing list