<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_quote"><div dir="ltr">Op wo 7 nov. 2018 om 14:30 schreef Hans van Kranenburg <<a href="mailto:hans@knorrie.org">hans@knorrie.org</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
On 11/7/18 12:48 PM, Roalt Zijlstra | webpower wrote:<br>
> <br>
> Op di 6 nov. 2018 om 18:54 schreef Hans van Kranenburg <<a href="mailto:hans@knorrie.org" target="_blank">hans@knorrie.org</a><br>
> <mailto:<a href="mailto:hans@knorrie.org" target="_blank">hans@knorrie.org</a>>>:<br>
> <br>
> Hi,<br>
> <br>
> On 11/5/18 12:37 PM, Roalt Zijlstra wrote:<br>
> > Package: src:xen<br>
> > Version: 4.8.4+xsa273+shim4.10.1+xsa273-1+deb9u10<br>
> > Severity: important<br>
> ><br>
> > Updating Xen to the latest 4.8 version from the security repo<br>
> makes servers unstable.<br>
> <br>
> Can you confirm that this is the only change that you made between the<br>
> before/after scenario? I mean, if you downgrade the packages, or you<br>
> drop the old hypervisor xen-x.y-amd64.gz in /boot again, it's stable<br>
> again?<br>
> <br>
> <br>
> We have several servers running the previous versions and those are<br>
> still stable. The servers that we upgraded using 'apt-get update;<br>
> apt-get upgrade' were rock solid before the upgrade.<br>
<br>
Yes, that's why I was asking. Did that apt-get upgrade also upgrade your<br>
dom0 kernel? You can look back in /var/log/dpkg.log* about what<br>
happened. This is very relevant information.<br></blockquote><div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">Two servers have been installed at 2018-04-24 and then upgraded:</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><div class="gmail_default">2018-10-08 19:40:57 upgrade xen-hypervisor-4.8-amd64:amd64 4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u9 4.8.4+xsa273+shim4.10.1+xsa273-1+deb9u10</div><div class="gmail_default">2018-10-08 19:41:14 status installed xen-hypervisor-4.8-amd64:amd64 4.8.4+xsa273+shim4.10.1+xsa273-1+deb9u10</div><div class="gmail_default">2018-07-31 18:50:01 upgrade xen-hypervisor-4.8-amd64:amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u9</div><div class="gmail_default">2018-07-31 18:50:45 status installed xen-hypervisor-4.8-amd64:amd64 4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u9</div><div class="gmail_default">2018-04-24 16:22:56 install xen-hypervisor-4.8-amd64:amd64 <none> 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5</div><div class="gmail_default">2018-04-24 16:23:05 status installed xen-hypervisor-4.8-amd64:amd64 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5</div><div class="gmail_default"><br></div><div class="gmail_default">The two other servers ran Cent OS first and were converted to Debian for other reasons and so are fresh installs:</div><div class="gmail_default"><div class="gmail_default">2018-09-26 22:01:34 install xen-hypervisor-4.8-amd64:amd64 <none> 4.8.4+xsa273+shim4.10.1+xsa273-1+deb9u10</div><div class="gmail_default">2018-09-26 22:01:57 status installed xen-hypervisor-4.8-amd64:amd64 4.8.4+xsa273+shim4.10.1+xsa273-1+deb9u10</div></div></div></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> I did prepare a downgrade script if needed, but atm. the crash interval<br>
> in days seems to be higher then before. We did have servers crashing<br>
> every 2 days or even one crashing twice a day.<br>
<br>
> > The servers randomly reset without any logs.<br>
> <br>
> Do you have the noreboot option set on the Xen hypervisor command line?<br>
> <br>
> <br>
> For now one busy servers runs an older 4.9.0-4-amd64 kernel with a 3.16<br>
> kernel DomU with MySQL server on it. The second busy server runs all<br>
> domUs with 4.9 (backport) kernels on the lastest 4.9.0-8-amd64 kernel<br>
> for the Dom0. Currently we are awaiting any crash. <br>
<br>
In Debian, 4.9.0-8-amd64 is in the name of the package, but the real<br>
kernel version is in the version of that package.<br>
<br>
So, if you have linux-image-4.9.0-8-amd64, you should always also<br>
mention the real version, which is now e.g. 4.9.110-3+deb9u6. This means<br>
it's based on 4.9.110 upstream.<br>
<br>
The kernel team follows the 4.9 LTS releases, but only if the changes<br>
have to break the ABI (so custom modules have to be rebuilt), they up<br>
the number in the package name to trigger that process.<br></blockquote><div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">Right I completely missed that detail:</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">Two heavy used servers run kernels: </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">4.9.65-3+deb9u1 with one Jessie DomU kernel: 3.16.57-2</div><div class="gmail_default"><font face="arial, helvetica, sans-serif">4.9.110-3+deb9u6 with a few Jessie DomU kernels: 4.9.110-3+deb9u5~deb8u1</font><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">Two less used servers run: </div><div class="gmail_default"><div class="gmail_default"><font face="arial, helvetica, sans-serif">4.9.110-3+deb9u5 with one Jessie DomU kernel: 3.16.59-1 </font></div></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">4.9.110-3+deb9u5 with a few mixed Jessie DomU kernels: 3.16.59-1 and 3.16.57-2</font><br></div><div class="gmail_default"> <br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> The last mentioned server was rebooted with the noreboot option, so we<br>
> could eventually check the console for errors once it crashes. <br>
> The remain two servers are our fall-back servers and are not that busy.<br>
> We have seen them crashtoo, but we noticed that the less busy servers<br>
> did not crash that often. But once they were busy they crashed as<br>
> quickly as the master servers.<br>
<br>
Ok, that's interesting extra data.<br>
<br>
> Are you able to configure and capture output from serial console?<br>
> <br>
> <br>
> Oh wow.. Using old technology for debugging :-) I will need to see how<br>
> that configuration is done. We could connect up physical serial cables<br>
> between different machines.<br>
<br>
Well... old... It's the best way to capture text after everything<br>
crashes. On a vga display it scrolls away and you can't copy paste.<br>
<br>
If you're using recent Dell hardware, then I guess your drac provides an<br>
extra emulated serial console. I use HP hardware, there it's the ilo<br>
virtual serial port.<br></blockquote><div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">I will get into this, never used it before as most crashes so far, did log errors </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">before things stop to work.</div></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> First interesting thing to know is if it's the Dom0 that crashes, or if<br>
> it's the hypervisor itself, and the logging will tell you that.<br>
> <br>
> > We have serveral Debian Stretch servers running Xen 4.8 and only<br>
> the ones updated to the 4.8.4+xsa273+shim4.10.1+xsa273-1+deb9u10<br>
> > version tend to crash ranging from 'twice a day' to 'once every<br>
> two weeks'. We have already ruled out if hardware was an<br>
> > issue, since we have 4 individual servers which are different in<br>
> hardware setup and also were bought at different times.<br>
> > And these servers ran stable with the previsous version<br>
> 4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u9.<br>
> > These servers are acting exactly the same. Every thing works as it<br>
> should, but without any logs it crashes and resets at<br>
> > a certain point.<br>
> ><br>
> > It looks like it could have something to do with DomUs running<br>
> older (3.16) Linux kernels. As a test we applied 4.9 kernels to<br>
> > all Jessie DomU servers and so far it runs for 13 days (but this<br>
> server did crash twice on a day).<br>
> > We have seen this behaviour with Xen on CentOS6 and 7 too, but the<br>
> trouble seems to be fixed after some more updates.<br>
> <br>
> It can be frustrating that there's not much response on the mailing<br>
> lists. But, these kinds of problems can be really hard to debug and<br>
> solve. Unless there's a clear reproduction scenario and debug output,<br>
> there's often noone who can help you remotely.<br>
> <br>
> <br>
> Well we have been having the issues since february this year with<br>
> unstable Xen servers crashing once in a months or so. The first issues<br>
> were on fresh Cent OS 7 servers, but then we also got them with updated<br>
> Cent OS 6 servers. We then decided to use Debian Stretch and the first<br>
> tests were pretty stable. We did install a new R740 with it (Xen<br>
> 4.8.4-pre) and that ran for 110 days pretty well.<br>
<br>
I know this feeling. I've been debugging similar kinds of issues this<br>
year that appeared "every few weeks".<br>
<br>
> > As said.. I cannot provide logs since it simply resets without notice.<br>
> <br>
> It's still the best starting point...<br>
> <br>
> <br>
> Well hopefully the 'noreboot' provided server crashes soon for some<br>
> logs. I will check if we can do any serial console tricks.<br>
<br>
Yes.<br></blockquote><div><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">Oh and before I forget.. Thanks for all the feedback/help!</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small">Roalt</div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>