[Pkg-xen-devel] Bug#908771: HP Proliant iLO Virtual Serial Port gets stuck during Xen boot at "Brought up "

[Hans van Kranenburg]

Package: src:xen
Version: 4.11.1~pre.20180911.5acdd26fdc+dfsg-1~exp1

HP Proliant servers feature a virtual serial port which can be accessed
by setting up an ssh session to the management interface (ilo) and then
doing vsp:

---- >8 ----

-$ ssh luser at box-ilo
luser at box-ilo's password:
User:luser logged-in to box-ilo

iLO Advanced 2.61 at  Jul 27 2018
Server Name: box (CZXXXXXXXX)
Server Power: On

</>hpiLO-> vsp

Virtual Serial Port Active: COM2

Starting virtual serial port.
Press 'ESC (' to return to the CLI Session.

Debian GNU/Linux 9 box hvc0

box login:

---- >8 ----

Now, if I e.g. use this...

xen command line: [...] com2=115200,8n1 console=com2,vga
linux command line: [...] console=tty0 console=hvc0

...then I can access all xen and linux serial console functionality
remotely.

However, during boot, there's a chance (it feels like ~40% chance) that
while booting Xen, this serial console gets stuck. When it does it's
*always* at exactly the same point, which is right after "Brought up "

So, console output is for example:

---- >8 ----

[...]
(XEN)  - Virtual Interrupt Delivery
(XEN)  - Posted Interrupt Processing
(XEN)  - VMCS shadowing
(XEN)  - VM Functions
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) Brought up

---- >8 ----

And then it hangs. Input is also not echoed back.

When doing a reboot of the server again, the queued output starts
appearing during shutdown, but quickly gets killed off when the hardware
reboots and the screen empties.

---- >8 ----

20 CPUs
(XEN) Dom0 has maximum 840 PIRQs
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x266a000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   000000480c000000->0000004810000000 (1025803 pages
to be allocated)
(XEN)  Init. ramdisk: 000000487e70b000->000000487ffff4af
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff8266a000
(XEN)  Init. ramdisk: 0000000000000000->0000000000000000
(XEN)  Phys-Mach map: 0000008000000000->0000008000800000
(XEN)  Start info:    ffffffff8266a000->ffffffff8266a4b4
(XEN)  Xenstore ring: 0000000000000000->0000000000000000
(XEN)  Console ring:  0000000000000000->0000000000000000
(XEN)  Page tables:   ffffffff8266b000->ffffffff82682000
(XEN)  Boot stack:    ffffffff82682000->ffffffff82683000
(XEN)  TOTAL:         ffffffff80000000->ffffffff82800000
(XEN)  ENTRY ADDRESS: ffffffff82433180
(XEN) Dom0 has maximum 4 VCPUs
[...]

---- >8 ----

I've seen this happen on all HP Proliant dl360 gear we have, g7, gen8
and gen9 (and we only have dl360). I think I first saw it with Xen 4.4
in Jessie long ago, but my memory may be failing, and it might as well
also have happened with 4.1 in Wheezy already.

Since I want a working serial console (the alternative is some windows
program to show the VGA, in which I can't copy paste etc), currently we
always check the vsp after a reboot and reboot again until it doesn't
hang. This is obviously a big annoyance and waste of time.

I'd like to find out if this is a bug in Xen, or if not, if it can be
worked around. It feels like some kind of race condition thing.

-- 
Hans van Kranenburg