[Pkg-xen-devel] Reliability issues

[Hans van Kranenburg]

Hi Casper,

On 10/12/26 5:58 PM, Casper Langemeijer wrote:
> 
> I've been experiencing reliability issues running bookworm with the
> recent xen with security updates. I've downgraded to the original
> packages (4.17.5+72-g01140da4e8-1) and that solved my issues
> immediately.

Ok, at least a 'good' thing here is that it apparently relatively easy
to find out if you have something running with the problem or without.

Are you able to free up some hardware to have a test environment for this?

> One of the machines had been upgraded to Trixie, the same
> issues there. In fact, reliability was not good on this Trixie machine
> before, which prompted me to stop upgrades on other server. I
> 'downgraded' this machine with a bookworm sources.list and installed the
> bookworm xen packages and its dependencies. I have not tried the
> original 4.20 trixie packages without security patches.

Aha, what you see happening with the Trixie 4.20 packages very much the
same problems as with 4.17, you think?

> I'm experiencing rebooting virtual machines, some have rebooted 60-ish
> times a day. The reboots seem to happen when there is a lof of disk
> activity in the VM. I see this on about 10 servers, this is not a
> hardware problem.
> 
> I'm running pretty old hardware, the most recent server having this
> issue is a dual Intel(R) Xeon(R) CPU E5-2640 v4 with 128GB of memory.
> This machine has been running since 2017. My 2022 and newer servers
> are AMD EPYC's and seem unaffected. I have no servers between 2017 and
> 2022 running xen.

Yeah, it doesn't really matter that it's a bit old, this is very
commonly used hardware.

> Virtual machines are PV using pygrub. All debian, most bookworm. All
> software packages are current. Dom0 and DomU's are running the 6.1.187-1
> kernel.
> 
> What can I do to help further investigate this problem?
First of all, trying to get more logging output, I'd guess.

When you have a situation where a domU is crashing/rebooting, look at xl
dmesg output in dom0, and look at the console output (xl console
<domU-name>), if needed maybe even in combination with
on_crash="preserve" for the domU.

Hopefully this shows something, that at least shows if the domU wants to
crash and disappear itself, or that it's e.g. Xen which maybe wants to
crash it and get rid of it because it's misbehaving in some way.

If nothing shows up already, try options like loglvl=all and
guest_loglvl=all on the Xen hypervisor boot command line.

If there is some clear related logging, we have something that we can
take to the upstream Xen team, to ask if this in combination with the
behaviour that you notice (lots of IO activity) rings a bell related to
one of the changes that was done in 4.17 between those two packages.

If we then get some advise to try taking out this or that patch, or try
something else, I can help with building a few test packages for you, if
you don't have the setup to do that yourself.

Also, (but I understand you probably rather do not want to immediately
go change other things right now...) really do look into running your
virtual machines as PVH type and using pvgrub2, instead of PV and pygrub.

Groet,
Hans