[Qa-jenkins-dev] Bug#875990: reproducible: i/o issues with profitbricks-build2-i386 since stretch upgrade

[Vagrant Cascadian]

Package: jenkins.debian.org
Severity: normal

It looks like after the upgrade to stretch (late june/early july), two
of the i386 builders, profitbricks-build2-i386 and
profitbricks-build12-i386 suddenly developed large i/o issues.

You can see this on the munin graphs for the year, where the blue i/o
wait spikes:

  https://jenkins.debian.net/munin/debian.net/profitbricks-build2-i386.debian.net/cpu.html
  https://jenkins.debian.net/munin/debian.net/profitbricks-build12-i386.debian.net/cpu.html

Comparing this to the other i386 builders, where there is no huge
spike in i/o wait:

  https://jenkins.debian.net/munin/debian.net/profitbricks-build6-i386.debian.net/cpu.html
  https://jenkins.debian.net/munin/debian.net/profitbricks-build16-i386.debian.net/cpu.html

I suspect this is reducing the i386 builds per day significantly,
averaging only ~1200 in the last 3 months.


My *hunch* is that build2 and build12 are running a PAE kernel with more
than 8GB of ram, and affected by this kernel bug (introduced in linux
~4.2, possibly):

  https://bugzilla.kernel.org/show_bug.cgi?id=196157
  https://bugs.launchpad.net/ubuntu/+source/linux-hwe/+bug/1698118


Reducing the ram of the affected builders to 8GB and having more PAE
builders with lighter workloads might be a workaround that would get
better performance... while still testing 32/64-bit kernel
variation.

Alternately, switching to only amd64 kernels might also fix the issue,
though wouldn't test 32/64-bit kernel variations.

Running a linux 4.1 kernel from snapshot.debian.org might be another way
to test the issue, even if not running long-term.


live well,
  vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/qa-jenkins-dev/attachments/20170916/985f0b8f/attachment.sig>