[Debian-med-packaging] Bug#1053509: shasta: autopkgtest regression on arm64: is memory suddenly not enough?

[Paul Gevers]

Source: shasta
Version: 0.11.1-1
Severity: important
User: debian-ci at lists.debian.org
Usertags: regression

Dear maintainer(s),

Your package has an autopkgtest, great. However, it recently started 
failing on arm64 in unstable, testing *and* stable. Until mid September 
2023, most runs passed (there are some historical rare failures where we 
don't have logs anymore), but somewhere between 2023-09-17 and 
2023-10-01 the tests started failing (although, in testing we had some 
passes on 2023-09-28 and 2023-10-04, interestingly these runs too *much* 
longer than normal (factor ~20)). I first suspected that this regression 
aligned with my fix for bug 1050256 where I switched the kernel of our 
hosts to backports, but there are multiple good runs with that kernel. 
Today I learned that "Killed" in the log is often coming from the kernel 
when a program is allocating too much memory and the kernel kills the 
process. We have some striking logs of failures [1] where the shasta 
test actually bails out itself for lack of memory.

So, there's a couple of things (weirdness and options):
1) why does it now suddenly start to (nearly always) fail across the 
board on arm64 (in Debian, Ubuntu still seems fine), without changes to 
the infrastructure that I know of?
2) do you also believe this is related to memory consumption?
3) If 2 == yes, what are the memory requirements for the test? The test 
*could* test for that before it starts and bail out (restriction: 
skippable with exit code 77 [2]) if the amount of memory available is 
too small.
4) just stop testing on arm64 (but again, in Ubuntu the test is still 
running fine)
5) the recent glibc security update also came to my mind, but that's not 
*yet* installed on our hosts, nor is it installed in the stable testbed.

The release team has announced [3] that failing autopkgtest on amd64 and 
arm64 are considered RC in testing. However, due to nature of the 
symptoms, I've filed this as important for now.

Paul

[1] 
https://ci.debian.net/data/autopkgtest/testing/arm64/s/shasta/38544284/log.gz
  48s 2023-Oct-04 12:01:05.603191 Memory allocation failure  during 
mremap call for MemoryMapped::Vector.
  48s This assembly requires more memory than available.
  48s Rerun on a larger machine.
[2] 
https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst
[3] https://lists.debian.org/debian-devel-announce/2019/07/msg00002.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <http://alioth-lists.debian.net/pipermail/debian-med-packaging/attachments/20231005/33b2d365/attachment.sig>