[Reproducible-builds] Bug#813052: diffoscope takes more than an hour on foreign arch libc6

[Helmut Grohne]

Package: diffoscope
Version: 47
User: helmutg at debian.org
Usertags: rebootstrap

Hi reproducible team,

I wish I could give you a detailed bug report, but instead you get a
very vague one. Sorry about that.

Since the 24th I am observing that all rebootstrap_*_diffoscope jobs on
jenkins.d.n fail no later than running diffoscope on libc6. Given the
date, we can be pretty sure that the regression was introduced precisely
in version 47.

The actual failure is rather unspecific. diffoscope is killed by timeout
with SIGKILL after running for more than an hour. You can see an example
failure log at
https://jenkins.debian.net/job/rebootstrap_arm64_gcc5_supported_diffoscope/27/console
One also sees that it works just fine on libc6-pic for instance.

So I tried to copy one of those packages off the host
profitbricks-build4-amd64.debian.net to run a local diffoscope and
inspect the issue. Instead the very same diffoscope in a very similar
amd64 chroot was done analyzing within just 2 minutes. In reality, I
couldn't reproduce the issue at all, but it still persists on that
jenkins slave.

What else could I tell? It's using the --text output mode. It's always
run on an amd64 system and always on non-amd64 binary packages.
Recommends are turned off, but binutils-multiarch and vim-common are
explicitly installed.

I also found looking into munin a bit interesting:
https://jenkins.debian.net/munin/debian.net/profitbricks-build4-amd64.debian.net/index.html
Since precisely the 24th, it started swapping a lot more and there are
large spikes in the resident memory usage. There is a chance that we can
attribute this to diffoscope.

Then I looked through the commits between 46 and 47 and the only thing
that caught my eye was the automatic use detached debug symbols. In my
own tests, I was never able to make that work (according to --debug
output), so maybe the feature works on
profitbricks-build4-amd64.debian.net?

Even though I cannot reproduce the issue at hand, I think that the code
adding automatic debug symbols looks fishy to me. It appears to recurse
over /tmp here and that looks very wrong to me. /tmp is small here, but
on the jenkins slave, /tmp is big.

Any further ideas for debugging this?

Helmut