Bug#991059: diffoscope: out-of-memory

[Roland Clobus]

On 15/07/2021 18:41, Mattia Rizzolo wrote:
> On Wed, Jul 14, 2021 at 08:31:48PM +0200, Roland Clobus wrote:

>> The difference between these two files is located inside a squashfs
>> file, which is inside the iso file.
>> During the invocation of diffoscope, diffoscope needs lots of memory
>> (>32GB) and free space on /tmp (>32GB but <48GB).

I've got the 2 ISO files that were used in a Jenkins run now, each 2.6GB.

In attempting to reproduce the case, I've mounted /tmp to a file instead
of using tmpfs and I've booted with 'single'. My computer has 32GB
memory, about 200MB is in use by the OS.

When I'm running diffoscope as root, I get the following metrics
(obtained by polling every 5 seconds):

21935816 1k blocks on /tmp (about 22GB)
1010 MiB memory at the peak
Needed time: 120 minutes +/ 10

However, when I run the same command as a regular user, I get an OOM
after about 5 minutes. At that time, the first squashfs image (2.3GB) is
completely decompressed to disc (8.0GB), and xxd is running.

The output of diffoscope with --debug:

2021-08-11 09:15:48 D: diffoscope.comparators.utils.file: Instantiating
a squashfs.SquashfsContainer for live/filesystem.squashfs
2021-08-11 09:15:48 D: diffoscope.comparators.squashfs: Extracting
/tmp/diffoscope_wzbylzvt_/tmpz60gl4slLibarchiveContainerWithFilelist/0/887.squashfs
to /tmp/diffoscope_wzbylzvt_/tmpg2r8zivtsquashfs
2021-08-11 09:15:48 D: diffoscope.comparators.utils.command: Calling
external command: unsquashfs -n -f -no -li -d .
/tmp/diffoscope_wzbylzvt_/tmpz60gl4slLibarchiveContainerWithFilelist/0/887.squashfs
2021-08-11 09:16:55 D: diffoscope.comparators.utils.command: Executing
xxd {}
2021-08-11 09:18:40 D: diffoscope.comparators.utils.command: Executing
xxd {}
Killed

So it appears to me that different code is activated for regular users
and root.
I hope this report helps in finding/fixing the issue.

With kind regards,
Roland Clobus

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://alioth-lists.debian.net/pipermail/reproducible-builds/attachments/20210811/572e066d/attachment.sig>