Memory error with unsquashfs

Jeremiah C. Foster jeremiah.foster at puri.sm
Wed Sep 25 18:02:25 BST 2019 

Hi,

While this needs a bit more investigation on my end, I thought I would
share this issue with this list since there may be folks who've seen it
before.

When I run this command;

/usr/bin/diffoscope --max-report-size=0 --html /var/www/html/pureos-
9.0-images.html /srv/artifacts/1111/pureos-9.0-oem_20190925-
amd64.hybrid.iso  /srv/artifacts/0000/pureos-9.0-oem_20190925-
amd64.hybrid.iso

I get this output;

Traceback (most recent call last):
  File "/usr/lib/python3/dist-
packages/diffoscope/comparators/utils/file.py", line 432, in compare
    difference = self._compare_using_details(other, source)
  File "/usr/lib/python3/dist-
packages/diffoscope/comparators/utils/file.py", line 380, in
_compare_using_details
    other.as_container, no_recurse=no_recurse
  File "/usr/lib/python3/dist-
packages/diffoscope/comparators/utils/container.py", line 131, in
comparisons
    my_members = OrderedDict(self.get_adjusted_members_sizes())
  File "/usr/lib/python3/dist-
packages/diffoscope/comparators/utils/container.py", line 123, in
get_adjusted_members_sizes
    for name, member in self.get_adjusted_members():
  File "/usr/lib/python3/dist-
packages/diffoscope/comparators/utils/container.py", line 79, in
get_filtered_members
    for name in filter_excludes(self.get_member_names()):
  File "/usr/lib/python3/dist-
packages/diffoscope/comparators/squashfs.py", line 250, in
get_member_names
    self.ensure_unpacked()
  File "/usr/lib/python3/dist-
packages/diffoscope/comparators/squashfs.py", line 274, in
ensure_unpacked
    cwd=self._temp_dir,
  File "/usr/lib/python3.7/subprocess.py", line 395, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.7/subprocess.py", line 487, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '('unsquashfs', '-n', '-f', '-
no', '-li', '-d', '.',
'/tmp/diffoscope_p22_7rig/tmpaqdov1d4/0/18.squashfs')' returned non-
zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 732,
in main
    sys.exit(run_diffoscope(parsed_args))
  File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 685,
in run_diffoscope
    difference = compare_root_paths(path1, path2)
  File "/usr/lib/python3/dist-
packages/diffoscope/comparators/utils/compare.py", line 72, in
compare_root_paths
    difference = compare_files(file1, file2)
  File "/usr/lib/python3/dist-
packages/diffoscope/comparators/utils/compare.py", line 124, in
compare_files
    return file1.compare(file2, source)
  File "/usr/lib/python3/dist-
packages/diffoscope/comparators/utils/file.py", line 432, in compare
    difference = self._compare_using_details(other, source)
  File "/usr/lib/python3/dist-
packages/diffoscope/comparators/utils/file.py", line 380, in
_compare_using_details
    other.as_container, no_recurse=no_recurse
  File "/usr/lib/python3/dist-
packages/diffoscope/comparators/utils/container.py", line 195, in
compare_pair
    file1, file2, source=None, diff_content_only=no_recurse
  File "/usr/lib/python3/dist-
packages/diffoscope/comparators/utils/compare.py", line 124, in
compare_files
    return file1.compare(file2, source)
  File "/usr/lib/python3/dist-
packages/diffoscope/comparators/utils/file.py", line 453, in compare
    difference = self.compare_bytes(other, source=source)
  File "/usr/lib/python3/dist-
packages/diffoscope/comparators/utils/file.py", line 331, in
compare_bytes
    return compare_binary_files(self, other, source)
  File "/usr/lib/python3/dist-
packages/diffoscope/comparators/utils/compare.py", line 146, in
compare_binary_files
    has_internal_linenos=True,
  File "/usr/lib/python3/dist-packages/diffoscope/difference.py", line
255, in from_command
    klass, path1, path2, *args, **kwargs
  File "/usr/lib/python3/dist-packages/diffoscope/difference.py", line
287, in from_command_exc
    feeder1, feeder2, path1, path2, *args, **kwargs
  File "/usr/lib/python3/dist-packages/diffoscope/difference.py", line
212, in from_feeder
    unified_diff = diff(feeder1, feeder2)
  File "/usr/lib/python3/dist-packages/diffoscope/diff.py", line 333,
in diff
    return run_diff(fifo1_path, fifo2_path, fifo1.end_nl_q,
fifo2.end_nl_q)
  File "/usr/lib/python3/dist-packages/diffoscope/diff.py", line 218,
in __exit__
    self.join()
  File "/usr/lib/python3/dist-packages/diffoscope/diff.py", line 251,
in join
    raise self._exception
  File "/usr/lib/python3/dist-packages/diffoscope/diff.py", line 242,
in run
    end_nl = self.feeder(fifo)
  File "/usr/lib/python3/dist-packages/diffoscope/feeders.py", line 91,
in feeder
    feeder = from_raw_reader(command.stdout, command.filter)
  File "/usr/lib/python3/dist-
packages/diffoscope/comparators/utils/command.py", line 112, in stdout
    return self._process.stdout.splitlines(True)
MemoryError

Running with the debug flat produces;

2019-09-25 12:47:16 D: diffoscope.presenters.formats: Will generate the
following formats: html
2019-09-25 12:47:16 D: diffoscope.main: Starting diffoscope 125
2019-09-25 12:47:16 D: diffoscope.locale: Normalising locale, timezone,
etc.
2019-09-25 16:47:16 D: diffoscope.main: Starting comparison
2019-09-25 16:47:16 D: diffoscope.comparators: Loaded 72 comparator
classes
2019-09-25 16:47:16 D: diffoscope.comparators.utils.specialize: Using
diffoscope.comparators.iso9660.Iso9660File for
/srv/artifacts/1111/pureos-9.0-oem_20190925-amd64.hybrid.iso
2019-09-25 16:47:16 D: diffoscope.comparators.utils.specialize: Using
diffoscope.comparators.iso9660.Iso9660File for
/srv/artifacts/0000/pureos-9.0-oem_20190925-amd64.hybrid.iso

<snip>

2019-09-25 16:47:18 D: diffoscope.comparators.utils.compare:
has_same_content_as returned True; skipping further comparisons
2019-09-25 16:47:18 D: diffoscope.comparators.utils.compare: Comparing
casper/filesystem.size (LibarchiveMember) and casper/filesystem.size
(LibarchiveMember)
2019-09-25 16:47:18 D: diffoscope.comparators.utils.file:
File.has_same_content: <<class
'diffoscope.comparators.utils.libarchive.LibarchiveMember'>
casper/filesystem.size> <<class
'diffoscope.comparators.utils.libarchive.LibarchiveMember'>
casper/filesystem.size>
2019-09-25 16:47:18 D: diffoscope.comparators.utils.specialize: Using
diffoscope.comparators.text.TextFile for casper/filesystem.size
2019-09-25 16:47:18 D: diffoscope.comparators.utils.specialize: Using
diffoscope.comparators.text.TextFile for casper/filesystem.size
2019-09-25 16:47:18 D: diffoscope.diff: Running diff -aU7
/tmp/diffoscope_c45v90xv/tmpnlp_7oz5/fifo1
/tmp/diffoscope_c45v90xv/tmpnlp_7oz5/fifo2
2019-09-25 16:47:18 D: diffoscope.diff: diff -aU7
/tmp/diffoscope_c45v90xv/tmpnlp_7oz5/fifo1
/tmp/diffoscope_c45v90xv/tmpnlp_7oz5/fifo2: returncode 1, parsed True
2019-09-25 16:47:18 D: diffoscope.comparators.utils.compare: Comparing
casper/filesystem.squashfs (LibarchiveMember) and
casper/filesystem.squashfs (LibarchiveMember)
2019-09-25 16:47:18 D: diffoscope.comparators.utils.file:
File.has_same_content: <<class
'diffoscope.comparators.utils.libarchive.LibarchiveMember'>
casper/filesystem.squashfs> <<class
'diffoscope.comparators.utils.libarchive.LibarchiveMember'>
casper/filesystem.squashfs>
2019-09-25 16:47:18 D: diffoscope.comparators.utils.specialize: Using
diffoscope.comparators.squashfs.SquashfsFile for
casper/filesystem.squashfs
2019-09-25 16:47:18 D: diffoscope.comparators.utils.specialize: Using
diffoscope.comparators.squashfs.SquashfsFile for
casper/filesystem.squashfs
2019-09-25 16:47:18 D: diffoscope.comparators.utils.command: Executing
unsquashfs -s /tmp/diffoscope_c45v90xv/tmp5h26j1jy/0/18.squashfs
2019-09-25 16:47:18 D: diffoscope.comparators.utils.command: Executing
unsquashfs -s /tmp/diffoscope_c45v90xv/tmplswrk2b9/0/18.squashfs
2019-09-25 16:47:18 D: diffoscope.diff: Running diff -aU7
/tmp/diffoscope_c45v90xv/tmpkvpi9ewa/fifo1
/tmp/diffoscope_c45v90xv/tmpkvpi9ewa/fifo2
2019-09-25 16:47:18 D: diffoscope.diff: diff -aU7
/tmp/diffoscope_c45v90xv/tmpkvpi9ewa/fifo1
/tmp/diffoscope_c45v90xv/tmpkvpi9ewa/fifo2: returncode 1, parsed True
2019-09-25 16:47:18 D: diffoscope.comparators.utils.command: Executing
unsquashfs -d '' -lls
/tmp/diffoscope_c45v90xv/tmp5h26j1jy/0/18.squashfs
2019-09-25 16:47:18 D: diffoscope.comparators.utils.command: Executing
unsquashfs -d '' -lls
/tmp/diffoscope_c45v90xv/tmplswrk2b9/0/18.squashfs
2019-09-25 16:47:19 D: diffoscope.diff: Running diff -aU7
/tmp/diffoscope_c45v90xv/tmpu0wye3cv/fifo1
/tmp/diffoscope_c45v90xv/tmpu0wye3cv/fifo2
2019-09-25 16:47:19 D: diffoscope.diff: diff -aU7
/tmp/diffoscope_c45v90xv/tmpu0wye3cv/fifo1
/tmp/diffoscope_c45v90xv/tmpu0wye3cv/fifo2: returncode 1, parsed True
2019-09-25 16:47:19 D: diffoscope.comparators.utils.file: Instantiating
a diffoscope.comparators.squashfs.SquashfsContainer for
casper/filesystem.squashfs
2019-09-25 16:47:19 D: diffoscope.comparators.utils.file: Returning a
diffoscope.comparators.squashfs.SquashfsContainer for
casper/filesystem.squashfs
2019-09-25 16:47:19 D: diffoscope.comparators.utils.file: Returning a
diffoscope.comparators.squashfs.SquashfsContainer for
casper/filesystem.squashfs
2019-09-25 16:47:19 D: diffoscope.comparators.utils.file: Returning a
diffoscope.comparators.squashfs.SquashfsContainer for
casper/filesystem.squashfs
2019-09-25 16:47:19 D: diffoscope.comparators.utils.file: Instantiating
a diffoscope.comparators.squashfs.SquashfsContainer for
casper/filesystem.squashfs
2019-09-25 16:47:19 D: diffoscope.comparators.utils.file: Returning a
diffoscope.comparators.squashfs.SquashfsContainer for
casper/filesystem.squashfs
2019-09-25 16:47:19 D: diffoscope.comparators.squashfs: Extracting
/tmp/diffoscope_c45v90xv/tmp5h26j1jy/0/18.squashfs to
/tmp/diffoscope_c45v90xv/tmpjaqzp36l
2019-09-25 16:47:31 D: diffoscope.comparators.utils.command: Executing
xxd /tmp/diffoscope_c45v90xv/tmp5h26j1jy/0/18.squashfs
2019-09-25 16:47:57 D: diffoscope.comparators.utils.command: Executing
xxd /tmp/diffoscope_c45v90xv/tmplswrk2b9/0/18.squashfs
2019-09-25 16:48:24 D: diffoscope.diff: Running diff -aU7
/tmp/diffoscope_c45v90xv/tmpfcnkfldo/fifo1
/tmp/diffoscope_c45v90xv/tmpfcnkfldo/fifo2
Killed

Running the diff command against the files in tmp, which are two named
pipes (fifo1, fifo2) just hangs.

I think I'm running out of memory? The machine has about 4 gigs.

Regards,

Jeremiah
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/reproducible-builds/attachments/20190925/6824386d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: This is a digitally signed message part
URL: <http://alioth-lists.debian.net/pipermail/reproducible-builds/attachments/20190925/6824386d/attachment.sig>

More information about the Reproducible-builds mailing list