diffoscope timing out.

Wed Nov 15 14:15:23 GMT 2023

On 2023-11-15 4:29 am, Holger Levsen wrote:
> On Mon, Nov 06, 2023 at 10:42:05AM +0000, Chris Lamb wrote:
>> Christos Zoulas wrote:
>> 
>> > For weeks now there is no useful output from diffoscope. Is there anything I
>> > can do to help debug the issue?
>> >
>> > tests.reproducible-builds.org        [eeB]
>> > Wed 1 Nov 23:35:09 UTC 2023 - diffoscope 251 produced no output for
>> > x86_64-amd64/amd64/binary/sets/comp.tar.xz and was killed after running into
>> > timeout after 30m...
> 
> these issues seem to have been resolved,
> https://jenkins.debian.net/job/reproducible_netbsd/1029/console
> contains no such timeouts.
> 
>> We'd actually love some help debugging this issue. As you can see from
>> the list archives, we have indeed been seeing this for a little while:
>>   
>> https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20230828/thread.html
>> … as well as some threads in later weeks.
> 
> I'm not sure those issues are related, maybe or maybe not.
> 
>> However, are unsure whether this an issue with:
>> 
>> a) The system running tests.reproducible-builds.org (which, I believe,
>> was upgraded around the time this issue appeared?)
>> 
>> b) The code in diffoscope itself (which is somewhat unlikely, as there
>> have not been meaningful changes that might cause crashes or
>> timeouts.)
> 
> https://jenkins.debian.net/job/reproducible_netbsd/1029/console
> contains this, which IMO should not happen and is an issue in
> diffoscope:
> 
> Traceback (most recent call last):
>   File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 767, 
> in main
>     sys.exit(run_diffoscope(parsed_args))
>              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 718,
> in run_diffoscope
>     difference = compare_root_paths(path1, path2)
>                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File
> "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py",
> line 69, in compare_root_paths
>     difference = compare_files(file1, file2)
>                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File
> "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py",
> line 149, in compare_files
>     return file1.compare(file2, source)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File
> "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py",
> line 532, in compare
>     difference = self._compare_using_details(other, source)
>                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File
> "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py",
> line 467, in _compare_using_details
>     details.extend(
>   File
> "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/container.py",
> line 197, in compare_pair
>     difference = compare_files(
>                  ^^^^^^^^^^^^^^
>   File
> "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py",
> line 149, in compare_files
>     return file1.compare(file2, source)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File
> "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py",
> line 616, in compare
>     return self.compare_bytes(other, source)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File
> "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py",
> line 412, in compare_bytes
>     return compare_binary_files(self, other, source)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File
> "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py",
> line 176, in compare_binary_files
>     return Difference.from_operation(
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3/dist-packages/diffoscope/difference.py", line
> 267, in from_operation
>     return Difference.from_operation_exc(
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3/dist-packages/diffoscope/difference.py", line
> 288, in from_operation_exc
>     feeder1, operation1, excluded1 = operation_and_feeder(path1)
>                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3/dist-packages/diffoscope/difference.py", line
> 285, in operation_and_feeder
>     operation.start()
>   File
> "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/command.py",
> line 45, in start
>     self._process = subprocess.run(
>                     ^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.11/subprocess.py", line 550, in run
>     stdout, stderr = process.communicate(input, timeout=timeout)
>                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.11/subprocess.py", line 1209, in communicate
>     stdout, stderr = self._communicate(input, endtime, timeout)
>                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.11/subprocess.py", line 2128, in _communicate
>     data = os.read(key.fd, 32768)
>            ^^^^^^^^^^^^^^^^^^^^^^
> MemoryError
> Fri 10 Nov 22:41:22 UTC 2023 - diffoscope 251 had trouble comparing
> the two builds. Please investigate
> x86_64-amd64/images/NetBSD-10.99.10-amd64-bios-install.img.gz
> Traceback (most recent call last):
>   File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 767, 
> in main
>     sys.exit(run_diffoscope(parsed_args))
>              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 718,
> in run_diffoscope
>     difference = compare_root_paths(path1, path2)
>                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File
> "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py",
> line 69, in compare_root_paths
>     difference = compare_files(file1, file2)
>                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File
> "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py",
> line 149, in compare_files
>     return file1.compare(file2, source)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File
> "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py",
> line 532, in compare
>     difference = self._compare_using_details(other, source)
>                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File
> "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py",
> line 467, in _compare_using_details
>     details.extend(
>   File
> "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/container.py",
> line 197, in compare_pair
>     difference = compare_files(
>                  ^^^^^^^^^^^^^^
>   File
> "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py",
> line 149, in compare_files
>     return file1.compare(file2, source)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File
> "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py",
> line 616, in compare
>     return self.compare_bytes(other, source)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File
> "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py",
> line 412, in compare_bytes
>     return compare_binary_files(self, other, source)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File
> "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py",
> line 176, in compare_binary_files
>     return Difference.from_operation(
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3/dist-packages/diffoscope/difference.py", line
> 267, in from_operation
>     return Difference.from_operation_exc(
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3/dist-packages/diffoscope/difference.py", line
> 288, in from_operation_exc
>     feeder1, operation1, excluded1 = operation_and_feeder(path1)
>                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3/dist-packages/diffoscope/difference.py", line
> 285, in operation_and_feeder
>     operation.start()
>   File
> "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/command.py",
> line 45, in start
>     self._process = subprocess.run(
>                     ^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.11/subprocess.py", line 550, in run
>     stdout, stderr = process.communicate(input, timeout=timeout)
>                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.11/subprocess.py", line 1209, in communicate
>     stdout, stderr = self._communicate(input, endtime, timeout)
>                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.11/subprocess.py", line 2128, in _communicate
>     data = os.read(key.fd, 32768)
>            ^^^^^^^^^^^^^^^^^^^^^^
> MemoryError
> Fri 10 Nov 22:43:45 UTC 2023 - diffoscope 251 had trouble comparing
> the two builds. Please investigate
> x86_64-amd64/images/NetBSD-10.99.10-amd64-install.img.gz
> 

Thanks for looking into this Holger. It seems to me that some of the 
issues
were resolved because we fixed the non-reproducible parts of the build 
for
those files. From the trace above, it seems that the subprocess that
diffoscope was running became unresponsive which is probably the 
original
reason we got timeouts.

Best,

christos