[Filesystems-devel] Bug#781894: squashfs 4.[23]: two different compressors, same discrepancy in large file

[Steve Newcomb]

Package: squashfs-tools
Version: 4.3

Version 4.2 also has this bug!

I use mksquashfs frequently for archiving purposes.  No recent version 
has failed me, and I usually verify the squashfs output against the 
input.  But one verification cycle recently failed. I'm pretty sure I've 
found what appears to be a compressor-independent bug somewhere in 
mksquashfs, versions 4.3 (2014/05/12) (built from source I just 
downloaded from SourceForge) and 4.2-git (2013/04/07) which is installed 
by default in Ubuntu 14.04.

To verify the bug (I have verified it several times now), you may 
download the uncompressed original file (big!) from:

  http://snewcomb.net/mksquashfs_discrepancy/mksqtest//Crawl-Anywhere-4.0.0-rc-Appliance-Ubuntu-Desktop-12.04-LTS/Crawl-Anywhere-4.0.0-rc-Appliance-Ubuntu-Desktop-12.04-LTS-disk1.vmdk

..and the same file in a v4.3 squashfs (xz compression) context at:

http://snewcomb.net/mksquashfs_discrepancy/mksqtest.sqfs

It doesn't matter whether I use gzip or xz compression. Either way, when 
I compare the original to the unsquashed retrieved version, the same 
discrepancy appears.

A report about the first 128K discrepant bytes can be seen at:

http://snewcomb.net/mksquashfs_discrepancy/diff_128K.txt

(The "cmp" tool verifies that there is a discrepancy, so the problem is 
not in my verification program.) The discrepancies exist regardless of 
whether I use unsquashfs or I mount the sqfs filesystem and use the 
kernel to decompress.

The tests were made on a host machine whose uname -a command says:

Linux daat 3.13.0-46-generic #76-Ubuntu SMP Thu Feb 26 18:52:13 UTC 2015 
x86_64 x86_64 x86_64 GNU/Linux

It is a fully up-to-date Ubuntu Server 14.04 installation. The hardware 
is quite stable. It's an Intel i7-930 with 24 Gb RAM and not heavily 
loaded. I have exactly reproduced the problem several times, and I'm 
pretty sure you will be able to reproduce it faithfully, given the files 
at the URLs given above.

  Steve Newcomb
srn at coolheads.com