[Pkg-tigervnc-devel] Bug#768369: Acknowledgement ([libjpeg62-turbo] [DOS] Stack smashing)

[DRC]

I can readily reproduce the failure with the supplied test case, but 
what I'm tripping on right now is understanding why a Huffman-encoded 
block can grow so much larger than the size of the source block (128 
bytes.)  While this test case is very unusual, there may be others out 
there, and I want to understand what the worst case is for Huffman 
encoding.  That would determine the appropriate value for BUFSIZE. 
Generally speaking, libjpeg-turbo will only need to use the local 
Huffman buffer when the buffer supplied in the destination manager is 
nearly exhausted-- that is, when libjpeg-turbo feels like the size of 
the encoded Huffman data for a given block would overrun the destination 
manager's buffer.  But we don't want to make the local Huffman buffer 
too big, else it might affect performance (since it introduces an extra 
memcpy() for all of the bytes that are encoded into the local buffer.) 
Hence the desire to understand exactly how big a Huffman-encoded block 
can grow in theory.


On 11/22/14 12:43 AM, Bernhard Übelacker wrote:
> Hello,
> I created a minimal test case in around 200 lines.
>
> It uses a file with the intercepted scanlines of the calls to jpeg_write_scanlines.
>
> Also the Exif marker is read from such a file.
> (And without this Exif marker the stack smash does not happen...)
>
> The partial output file is byte equal to that generated by imagemagick before it crashes.
>
> The number of calls to encode_mcu_huff and the stack seem also to be the same.
>
> Kind regards,
> Bernhard
>
>
>
> Place all three files in the same directory and open a shell there:
> (I just created the breakpoint to see how often it is called.)
>
>
> $ bunzip2 jpeg_write_marker.bin.bz2
> $ bunzip2 jpeg_write_scanlines.bin.bz2
> $ gcc -g -O0 -fstack-protector-all test-768369.c -ljpeg
> $ gdb --args ./a.out
> ...
> (gdb) b encode_mcu_huff
> Breakpoint 1 (encode_mcu_huff) pending.
> (gdb) ignore 1 10000
> Will ignore next 10000 crossings of breakpoint 1.
> (gdb) run
> Starting program: /home/bernhard/data/entwicklung/2014/debian/libjpegturbo/a.out
> *** stack smashing detected ***: /home/bernhard/data/entwicklung/2014/debian/libjpegturbo/a.out terminated
> ...
> (gdb) info break
> Num     Type           Disp Enb Address            What
> 1       breakpoint     keep y   0x00007ffff7b8c190 in encode_mcu_huff at jchuff.c:593
>          breakpoint already hit 9842 times
>          ignore next 158 hits
>
> (gdb) bt
> #0  0x00007ffff7811107 in __GI_raise (sig=sig at entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1  0x00007ffff78124e8 in __GI_abort () at abort.c:89
> #2  0x00007ffff784f044 in __libc_message (do_abort=do_abort at entry=2, fmt=fmt at entry=0x7ffff793f6ab "*** %s ***: %s terminated\n") at ../sysdeps/posix/libc_fatal.c:175
> #3  0x00007ffff78d2147 in __GI___fortify_fail (msg=msg at entry=0x7ffff793f693 "stack smashing detected") at fortify_fail.c:31
> #4  0x00007ffff78d2110 in __stack_chk_fail () at stack_chk_fail.c:28
> #5  0x00007ffff7b96553 in encode_mcu_huff (cinfo=0x7fffffffdd70, MCU_data=0x602720) at jchuff.c:641
> #6  0x00007ffff7b89717 in compress_output (cinfo=0x7fffffffdd70, input_buf=<optimized out>) at jccoefct.c:381
> #7  0x00007ffff7b89006 in jpeg_finish_compress (cinfo=0x7fffffffdd70) at jcapimin.c:183
> #8  0x0000000000401196 in main () at test-768369.c:205
>
>
>
>