[Pkg-xen-devel] Recent hypervisor update on Debian Wheezy breaks domU networking

[Gavin]

On 18 February 2013 12:50, Ian Campbell <ijc at hellion.org.uk> wrote:

> On Mon, 2013-02-18 at 12:04 +0200, Gavin wrote:
>
>
> > Firstly I apologise for the cross-post,
>
> I've added xen-users since you also bounced this there.
>

Thanks. :-/  Thanks too for your quick reply.

>
> >  however I don't expect to get as quick a response from the package
> > maintainers as I do from the Debian community, and this issue affects
> > a service that I've got scheduled to go live at midnight this
> > evening. :(
> >
> >
> > A recent update from xen-hypervisor-4.1-amd64 version 4.1.3-7, to
> > version 4.1.3-8 on Debian Wheezy has caused all vm's on this host to
> > not receive their arp replies anymore and as such they cannot reach
> > their gateways and are now isolated from the network.
> >
> >
> > There was a more recent update as well (4.1.4-2) which I have now
> > since applied however this particular issue persists.
>
> Networking level stuff is all done by the dom0 (or driver domain) kernel
> rather than the hypervisor so it is far more likely that a kernel level
> change rather than a hypervisor change would be responsible. What kernel
> version are you running? Did it also change?
>

This makes sense, although when I did the apt-get upgrade, there was no
kernel update, however there may have been packages/drivers that required a
kernel mod.

Here is the apt history which details what was upgraded when this broke:-

Upgrade: dnsmasq-base:amd64 (2.62-3, 2.62-3+deb7u1), tasksel-data:amd64
(3.14, 3.14+nmu1), xen-hypervisor-4.1-amd64:amd64 (4.1.3-7, 4.1.3-8),
xen-utils-common:amd64 (4.1.3-7, 4.1.3-8), perl:amd64 (5.14.2-16,
5.14.2-17), firmware-linux-free:amd64 (3.1, 3.2), perl-base:amd64
(5.14.2-16, 5.14.2-17), xen-utils-4.1:amd64 (4.1.3-7, 4.1.3-8),
libgnutls26:amd64 (2.12.20-2, 2.12.20-4), perl-modules:amd64 (5.14.2-16,
5.14.2-17), psmisc:amd64 (22.19-1, 22.19-1+deb7u1), python2.6:amd64
(2.6.8-0.2, 2.6.8-1.1), libxenstore3.0:amd64 (4.1.3-7, 4.1.3-8),
python2.6-minimal:amd64 (2.6.8-0.2, 2.6.8-1.1), coreutils:amd64 (8.13-3.4,
8.13-3.5), libvirt0:amd64 (0.9.12-5, 0.9.12-6), libcurl3:amd64 (7.26.0-1,
7.26.0-1+wheezy1), manpages:amd64 (3.42-1, 3.44-1), tasksel:amd64 (3.14,
3.14+nmu1), libperl5.14:amd64 (5.14.2-16, 5.14.2-17),
libsystemd-login0:amd64 (44-7, 44-8), libxen-4.1:amd64 (4.1.3-7, 4.1.3-8),
libcurl3-gnutls:amd64 (7.26.0-1, 7.26.0-1+wheezy1), host:amd64
(9.8.4.dfsg.P1-1, 9.8.4.dfsg.P1-4), libvirt-bin:amd64 (0.9.12-5, 0.9.12-6),
rinse:amd64 (2.0-1, 2.0.1-1), ca-certificates:amd64 (20120623, 20130119),
xenstore-utils:amd64 (4.1.3-7, 4.1.3-8)

The kernel I am using is: 3.2.0-2-amd64, also tried 3.2.0-4-amd64 on
another host with no success.

Would the upgrade above of xen-hypervisor-4.1-amd64 on this Debian system
not cause the Dom0 kernel to be changed in any way ??


> > The arp replies are received by the host and passed all the way up to
> > the bridge (br200) being used by Xen, however they are not seen on the
> > vif (vif2.0) created for the particular vm.
>
> Do you have any firewall or ebfilter entries which might have either
> been discarded or reintroduced by the reboot? (i.e. a manual settings
> modification which wasn't propagated to the startup scripts). Or perhaps
> sysctl tweaks?
>

Nope, not using iptables/ebtables, this was working 100% until the apt
upgrade above.

After this broke I did try and add the following to /etc/sysctl.conf but it
made no difference:-

net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

It did add iptables rules but made no difference to this issue.


>
> > 1) Please let me know if I should roll-back this particular xen
> > update, kernel and all, and what those steps may be, or if this is a
> > known issue with a particular workaround that I can apply.
>
> I'd certainly be tempted to try the older kernel, assuming that was also
> upgraded. It may even still be installed and in your grub menu already.
>

The problem is now we are using grub2 and it appears that on boot grub
loads a Linux menu, then the Xen Menu with configs in /etc/grub.d/ so I'm
battling to figure out how to do this.

I also do not have physical access to this host at the moment so need to
set the boot order 'correctly' prior to a reboot.

>
> > 2) Would moving to openvswitch be another possible workaround?
>
> Without knowing what the underlying issue is it is hard to predict
> whether it will also affect ovs.
>

Agreed.

>
> > My config:-
>
> Looks correct to me.
>
> Ian.
>

Thanks Ian.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-xen-devel/attachments/20130218/a0a2298a/attachment-0001.html>