Bug#1040189: miscomputes SHA1 sums on IMX53 (armhf)

[Simon McVittie]

On Mon, 03 Jul 2023 at 09:12:08 +0300, Dmitry Baryshkov wrote:
> I tried building the quake shareware packages on my fresh iMX53 install
> (Debian bookworm, armhf port). However I faced an issue with
> game-data-packager being unable to identify any source file, because it
> miscomputes sha1 sums. As you can see in the following fragment, md5 and
> sha256 sums match, but sha1 doesn't:
> 
> $ game-data-packager quake
> INFO:game_data_packager.download:downloading ftp://ftp.fu-berlin.de/pc/msdos/games/idgames/idstuff/quake/quake106.zip
> WARNING:game_data_packager.build:found quake106.zip but it is not one of the expected versions:
>     file:   /tmp/gdptmp.0aow0955/quake106.zip
>     size:   9094045 bytes
>     md5:    8cee4d03ee092909fdb6a4f84f0c1357
>     sha1:   e24e7b1c5030bf38d6caaa71dcd54469087aef0f
>     sha256: ec6c9d34b1ae0252ac0066045b6611a7919c2a0d78a3a66d9387a8f597553239
> expected:
>   quake106.zip:
>     size:   9094045 bytes
>     md5:    8cee4d03ee092909fdb6a4f84f0c1357
>     sha1:   f8a1a509b094ccdbed3c54b96f7d9b351c0898f5
>     sha256: ec6c9d34b1ae0252ac0066045b6611a7919c2a0d78a3a66d9387a8f597553239

If I'm understanding you correctly, your use of Quake and quake106.zip
is just an example, and you get similar results (md5 and sha256 correct,
sha1 wrong) for other files and other games? Is that true?

Is this a general problem with Python on this hardware? game-data-packager
doesn't do anything particularly special to calculate hashes, it just uses
the Python standard library to do it.

Possible steps to reproduce (for simplicity this just reads the whole file
into RAM, so it will briefly require a few MB):

$ wget ftp://ftp.fu-berlin.de/pc/msdos/games/idgames/idstuff/quake/quake106.zip
...
$ sha1sum quake106.zip
f8a1a509b094ccdbed3c54b96f7d9b351c0898f5  quake106.zip
$ python3
>>> import hashlib
>>> hasher = hashlib.new('sha1')
>>> with open('quake106.zip', 'rb') as reader:
...     hasher.update(reader.read())
...
>>> hasher.hexdigest()
'f8a1a509b094ccdbed3c54b96f7d9b351c0898f5'

If this gives a different answer for you on the affected hardware, then
this is a general problem with hashlib and should be reassigned to python3.

Or if that code reports f8a1... for you too, the next thing to try would be
reading it a block at a time.

The fact that gdp is reporting the correct md5 and sha256 suggests that
hashlib (and gdp's code that interfaces with hashlib) is basically working
correctly, but the sha1 implementation might be doing something that is
only valid for newer CPUs or something like that?

Thanks,
    smcv