<div dir="ltr">Hi,<div>I think we can forget my former suggestion for isolation at least for now.</div><div>Thanks for closing 848319 btw that explanation gave me the confidence to continue debugging this case.</div><div><br></div><div>Now I found that I seem to be "able to" run into this issue on x86 as well - so not arch dependent at all.</div><div>Still it is weird, in a dep8 environment I sem to run into this 100% while I never do when I redo the same steps on my system.</div><div><br></div><div>I logged into the dep8 KVM guest and checked how reproducible it is.</div><div>It turns out that after the FIRST restart the lxc-guest is killed and listed as shut-down then.</div><div>I lists an error like this:</div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">Dec 19 14:40:19 autopkgtest libvirtd[4500]: </span><span style="font-weight:bold;color:rgb(255,84,84)">internal error: No valid cgroup for machine sl</span><span style="color:rgb(0,0,0)">
</span><br>Dec 19 14:40:19 autopkgtest libvirtd[4500]: <span style="font-weight:bold;color:rgb(255,84,84)">End of file while reading data: Input/output error</span><br></span></div><div><br></div><div>That is somewhat familiar if you know <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=774237">https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=774237</a></div><div><br></div><div>But then we are clearly above that systemd level.</div><div>Now what makes this even more interesting is that AFTER the issue happened it seems to be ok.</div><div><br></div><div>So afterwards doing</div><div>export ...</div><div>virsh start sl</div><div># now I can restart libvirt without affecting guest "sl"</div><div><br></div><div>The following cleanup and re-define gets me back to the situation where a following restart will destroy the guest and throw the error listed above to the log:</div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">virsh destroy sl; virsh undefine sl; rm -rf /etc/libvirt; apt-get r</span>emove --purge libvirt-daemon-system libvirt-clients libxml2-utils; apt-get install libvirt-daemon-system libvirt-clients libxml2-utils; virsh define smoke-lxc.xml; virsh start sl; virsh list --all</span></div><div># now trigger the fail with </div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">/etc/init.d/libvirtd restart</span><br></span></div><div><br></div><div>After this I have again libvirt running, but not really:</div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">systemctl status libvirtd
</span><br><span style="font-weight:bold;color:rgb(84,255,84)">●</span><span style="color:rgb(0,0,0)"> libvirtd.service - Virtualization daemon
</span><br> Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
<br> Active: <span style="font-weight:bold;color:rgb(84,255,84)">active (running)</span><span style="color:rgb(0,0,0)"> since Mon 2016-12-19 14:52:51 CET; 14s ago
</span><br> Docs: man:libvirtd(8)
<br> <a href="http://libvirt.org">http://libvirt.org</a>
<br> Main PID: 7608 (libvirtd)
<br> Tasks: 18
<br> CGroup: /system.slice/libvirtd.service
<br> ├─2035 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leases
<br> ├─2036 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leases
<br> └─7608 /usr/sbin/libvirtd
<br>
<br>Dec 19 14:52:51 autopkgtest systemd[1]: Starting Virtualization daemon...
<br>Dec 19 14:52:51 autopkgtest systemd[1]: Started Virtualization daemon.
<br>Dec 19 14:52:52 autopkgtest dnsmasq[2035]: read /etc/hosts - 8 addresses
<br>Dec 19 14:52:52 autopkgtest dnsmasq[2035]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
<br>Dec 19 14:52:52 autopkgtest dnsmasq-dhcp[2035]: read /var/lib/libvirt/dnsmasq/default.hostsfile
<br>Dec 19 14:52:52 autopkgtest libvirtd[7608]: libvirt version: 2.5.0, package: 1ubuntu1~ppa3 (Christian Ehrhardt <<a href="mailto:christian.ehrhardt@canonical.com">christian.ehrhardt@canonical.com</a>> <br>Dec 19 14:52:52 autopkgtest libvirtd[7608]: hostname: autopkgtest.localdomain
<br>Dec 19 14:52:52 autopkgtest libvirtd[7608]: <span style="font-weight:bold;color:rgb(255,84,84)">internal error: No valid cgroup for machine sl</span><span style="color:rgb(0,0,0)">
</span><br>Dec 19 14:52:52 autopkgtest libvirtd[7608]: <span style="font-weight:bold;color:rgb(255,84,84)">End of file while reading data: Input/output error</span><br></span></div><div><span style="font-family:monospace"><span style="font-weight:bold;color:rgb(255,84,84)"><br></span></span></div><div><div>I say "not really" even if systemd says active here because a virsh list now looks the following way:</div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">$ virsh list --all
</span><br> Id Name State
<br>----------------------------------------------------<br>
<br></span></div><div><span style="font-family:monospace">That is it, the guest is completely gone.</span></div><div><span style="font-family:monospace">Restarting the service again lets it start normally and the guest returns - although in stopped state:</span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">$ /etc/init.d/libvirtd restart</span><br></span></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">$ virsh list --all </span><br> Id Name State
<br>----------------------------------------------------
<br> - sl shut off<br></span></div><div><span style="font-family:monospace"><br></span></div><div>I can start the guest again now, and from this point on it is resilient against restarts.</div><div>$ virsh start sl</div><div><span style="color:rgb(0,0,0);font-family:monospace">$ /etc/init.d/libvirtd restart</span><br></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">$ virsh list --all </span><br> Id Name State
<br>----------------------------------------------------
<br> 9020 sl running<br></span></div><div><br></div><div><br></div><div>So much for now trying to gather some extra debug data next ...</div>
</div></div>