Bug#634089: jh_manifest causes CRC error

Wookey wookey at wookware.org
Fri Jan 6 04:14:16 UTC 2012


This seems to me to be a serious problem blocking all
java-helper-using packages

So, I took a look at what's going on and found the following:

The short version is that Archive::Zip is misbehaving as jh_manifest
uses it and making corrupt jars. A patch to workaround that is
attached. It may be that a fix in Archive::Zip is a better solution. 


As it took me a while to get there I'll leave my working in in case it
helps anyone else working on the issue:

If I build the package in squeeze everything is fine.
If I build it in wheezy then running it:
 terraintool
gives CRC error.

Looking a little more closely. This is OK:
 java -jar /usr/share/terraintool/terraintool.jar

It seems that what the binfmt support does is run this:
 jarwrapper /usr/share/terraintool/terraintool.jar
That give CRC error.

jarwrapper is just a shell script. The bit that dies is this:
 fastjar xf /usr/share/terraintool/terraintool.jar META-INF/MANIFEST.MF

Niels Thykier suggested seeing if LD_PREloading 
it was sugested to see if replacing memcpy with memmove using
LD_PRELoAD fixed the situation. I found that you do that like this:


Create a file named "memcpy_hider.c" containing

    #include <string.h>
    void* memcpy(void *dst, const void *src, size_t size)
    {
      memmove(dst,src,size);
    }


Then compile it with
    gcc -fPIC -O2 -c memcpy_hider.c
    ld -G memcpy_hider.o -o memcpy_hider.so


Finally, to test it out, run your browser like

    LD_PRELOAD=/abolute/path/to/memcpy_hider.so program

So that built OK. I ran:
LD_PRELOAD=/home/wookey/debian/wheezy/memcpy_hider.so fastjar xf /usr/share/terraintool/terraintool.jar META-INF/MANIFEST.MF
and it still gives a CRC error :-(

So either I've done something wrong or the memcpy thing is not the
problem here?

This suggests that the _hider lib is being used
LD_PRELOAD=/home/wookey/debian/wheezy/memcpy_hider.so ldd /usr/bin/fastjar
linux-vdso.so.1 =>  (0x00007fffc55ff000)
/home/wookey/debian/wheezy/memcpy_hider.so (0x00007f8819ebf000)
libz.so.1 => /lib/libz.so.1 (0x00007f8819c81000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f88198fc000)
/lib64/ld-linux-x86-64.so.2 (0x00007f881a0c2000)

stracing the fastjar command shows it falling over as soon as it tries
to write some output when unpacking the jar
open("/usr/share/terraintool/terraintool.jar", O_RDONLY) = 3
read(3, "PK\3\4", 4)                    = 4
read(3, "\24\0\0\0\0\0S\217%@\0\0\0\0\2\0\0\0\0\0\0\0\t\0\4\0", 26) = 26
read(3, "META-INF/", 9)                 = 9
lseek(3, 4, SEEK_CUR)                   = 43
read(3, "PK", 2)                        = 2
write(2, "Error! CRCs do not match! Got 1a"..., 51Error! CRCs do not match! Got 1a6cd7b3, expected 0 ) = 51


So it's opened the jar, found the PKZIp header, found the META-INF
directory entry, skipped 4 bytes, found PK again and died.

Some perusal of the PKZIP header spec:
http://www.pkware.com/documents/casestudies/APPNOTE.TXT
od -tx1 -c /usr/share/terraintool/terraintool.jar |less
 
show that that header contians:

 local file header signature     4 bytes  (0x04034b50)
        version needed to extract       2 bytes
        general purpose bit flag        2 bytes
        compression method              2 bytes
        last mod file time              2 bytes
        last mod file date              2 bytes
        crc-32                          4 bytes
        compressed size                 4 bytes
        uncompressed size               4 bytes
        file name length                2 bytes
        extra field length              2 bytes

        file name (variable size)
        extra field (variable size)

  B.  File data

      Immediately following the local header for a file
      is the compressed or stored data for the file. 
      
0000000  50  4b  03  04  14  00  00  00  00  00  53  8f  25  40  00  00
          P   K 003 004 024  \0  \0  \0  \0  \0   S 217   %   @  \0  \0
0000020  00  00  02  00  00  00  00  00  00  00  09  00  04  00  4d  45
         \0  \0 002  \0  \0  \0  \0  \0  \0  \0  \t  \0 004  \0   M   E
0000040  54  41  2d  49  4e  46  2f  fe  ca  00  00  50  4b  03  04  14
          T   A   -   I   N   F   / 376 312  \0  \0   P   K 003 004 024

This seems to say that this entry has uncompressed size 0 and
compressed size 2 - is that right?

and this is a header for another directory:
                                 50  4b  03  04  0a  00  00  00  00  00
                                  P   K 003 004  \n  \0  \0  \0  \0  \0
0002120  4e  8f  25  40  00  00  00  00  00  00  00  00  00  00  00  00
          N 217   %   @  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
0002140  08  00  00  00  6d  63  63  6f  6d  62  65  2f  50  4b  03  04
         \b  \0  \0  \0   m   c   c   o   m   b   e   /  

this has size 0 for both compressed and uncompressed. 

doing  unzip -t /usr/share/terraintool/terraintool.jar 
Archive:  /usr/share/terraintool/terraintool.jar
META-INF/:  ucsize 0 <> csize 2 for STORED entry
         continuing with "compressed" size value
     testing: META-INF/                bad CRC 1a6cd7b3 (should be 00000000)
     testing: Tester.class             OK
     testing: mccombe/                 OK

hmm, so the jar is corrupt. with the wrong size for the compressed
file. So fastjar reads 2 extra bytes after the file entry and gets
'PK'. And the crc of "PK" is 1a6cd7b3 which is why you always get that
number no matter what .jar is used. 

OK, we're getting somewhere now. And it's nothing to do with
memcpy/memmove - that was a red herring. 

Earlier mails say jh_manifest is at fault. That's a perl script that
is responsible for making this jar (zip archive). It uses
libarchive-zip-perl to do the zipifying. 

So why is it now making corrupt archives when it wasn't before?

The version in squeeze is 1.30-3, and in wheezy 1.30.4. But perl
itself has gone from 5.10.1-17 to 5.14.2-6. Could that be it?
Javahelper has gone from 0.32 to 0.37 and jh_manfest has changed a
little but not obviously in areas that matter. 

jar if jh_manifest not run:
0000000  50  4b  03  04  14  00  08  00  08  00  c1  18  26  40  00  00
          P   K 003 004 024  \0  \b  \0  \b  \0 301 030   &   @  \0  \0
0000020  00  00  00  00  00  00  00  00  00  00  09  00  04  00  4d  45
         \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \t  \0 004  \0   M   E
0000040  54  41  2d  49  4e  46  2f  fe  ca  00  00  03  00  50  4b  07
          T   A   -   I   N   F   / 376 312  \0  \0 003  \0
	  
jar after jh_manifest is run:
0000000  50  4b  03  04  14  00  00  00  00  00  c1  18  26  40  00  00
          P   K 003 004 024  \0  \0  \0  \0  \0 301 030   &   @  \0  \0
0000020  00  00  02  00  00  00  00  00  00  00  09  00  04  00  4d  45
         \0  \0 002  \0  \0  \0  \0  \0  \0  \0  \t  \0 004  \0   M   E
0000040  54  41  2d  49  4e  46  2f  fe  ca  00  00  50  4b  03  04  0a
          T   A   -   I   N   F   / 376 312  \0  \0 

The relevant code in jh_manifest is the end of 'update_jar':
        verbose_print("Updating manifest in $jar");
        $zip->removeMember( 'META-INF/MANIFEST.MF' ) unless($new_manifest);
        $mem = $zip->addString($var, 'META-INF/MANIFEST.MF');
        $mem->desiredCompressionMethod(COMPRESSION_DEFLATED);
        # This on the other hand may fail.
        $zip->overwrite() == AZ_OK or error("Writing modified jar ($jar) failed$

so the issue, which may well actually be a bug in Archive::Zip given
that this presumably used to work is that nothing explicit is done
with the META-INF/ directory entry member in the archive, but by
creating a new 'META-INF/MANIFEST.MF' member is gets corrupted.
We can fix it by explicitly removing and recreating it:

        verbose_print("Updating manifest in $jar");
        $zip->removeMember( 'META-INF/MANIFEST.MF' ) unless($new_manifest);
        $zip->removeMember( 'META-INF/' );
        $mem = $zip->addString($var, 'META-INF/MANIFEST.MF');
        $mem->desiredCompressionMethod(COMPRESSION_DEFLATED);
        $zip->addDirectory( 'META-INF/' );
        # This on the other hand may fail.
        $zip->overwrite() == AZ_OK or error("Writing modified jar ($jar) failed$

which I hope is an acceptable fix. Patch attached.

Wookey
-- 
Principal hats:  Linaro, Emdebian, Wookware, Balloonboard, ARM
http://wookware.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: javatool-0.37-manifest-zipfix.patch
Type: text/x-diff
Size: 628 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-java-maintainers/attachments/20120106/4ae6bbc5/attachment.patch>


More information about the pkg-java-maintainers mailing list