[Pkg-tigervnc-devel] Bug#768369: Acknowledgement ([libjpeg62-turbo] [DOS] Stack smashing)

[DRC]

I find that by setting the AC coefficients to alternating values of 
32767 and -32768 in the JPEG scanning order (1, 8, 16, 9, 2, 3, etc.), I 
can make the Huffman encoder exceed 200 bytes/block every single time. 
So that further confirms that 256 is the worst case.  I've checked in an 
upstream patch to subversion trunk, as well as branches/1.4.x and 
branches/1.3.x.


On 11/22/14 2:06 PM, roucaries bastien wrote:
> On Sat, Nov 22, 2014 at 6:58 PM, DRC <dcommander at users.sourceforge.net> wrote:
>> I can readily reproduce the failure with the supplied test case, but what
>> I'm tripping on right now is understanding why a Huffman-encoded block can
>> grow so much larger than the size of the source block (128 bytes.)  While
>> this test case is very unusual, there may be others out there, and I want to
>> understand what the worst case is for Huffman encoding.  That would
>> determine the appropriate value for BUFSIZE. Generally speaking,
>> libjpeg-turbo will only need to use the local Huffman buffer when the buffer
>> supplied in the destination manager is nearly exhausted-- that is, when
>> libjpeg-turbo feels like the size of the encoded Huffman data for a given
>> block would overrun the destination manager's buffer.  But we don't want to
>> make the local Huffman buffer too big, else it might affect performance
>> (since it introduces an extra memcpy() for all of the bytes that are encoded
>> into the local buffer.) Hence the desire to understand exactly how big a
>> Huffman-encoded block can grow in theory.
>
> Could you exactly describe that you are doing (mathematically) ?
>
> Bastien
>
>>
>>
>> On 11/22/14 12:43 AM, Bernhard Übelacker wrote:
>>>
>>> Hello,
>>> I created a minimal test case in around 200 lines.
>>>
>>> It uses a file with the intercepted scanlines of the calls to
>>> jpeg_write_scanlines.
>>>
>>> Also the Exif marker is read from such a file.
>>> (And without this Exif marker the stack smash does not happen...)
>>>
>>> The partial output file is byte equal to that generated by imagemagick
>>> before it crashes.
>>>
>>> The number of calls to encode_mcu_huff and the stack seem also to be the
>>> same.
>>>
>>> Kind regards,
>>> Bernhard
>>>
>>>
>>>
>>> Place all three files in the same directory and open a shell there:
>>> (I just created the breakpoint to see how often it is called.)
>>>
>>>
>>> $ bunzip2 jpeg_write_marker.bin.bz2
>>> $ bunzip2 jpeg_write_scanlines.bin.bz2
>>> $ gcc -g -O0 -fstack-protector-all test-768369.c -ljpeg
>>> $ gdb --args ./a.out
>>> ...
>>> (gdb) b encode_mcu_huff
>>> Breakpoint 1 (encode_mcu_huff) pending.
>>> (gdb) ignore 1 10000
>>> Will ignore next 10000 crossings of breakpoint 1.
>>> (gdb) run
>>> Starting program:
>>> /home/bernhard/data/entwicklung/2014/debian/libjpegturbo/a.out
>>> *** stack smashing detected ***:
>>> /home/bernhard/data/entwicklung/2014/debian/libjpegturbo/a.out terminated
>>> ...
>>> (gdb) info break
>>> Num     Type           Disp Enb Address            What
>>> 1       breakpoint     keep y   0x00007ffff7b8c190 in encode_mcu_huff at
>>> jchuff.c:593
>>>           breakpoint already hit 9842 times
>>>           ignore next 158 hits
>>>
>>> (gdb) bt
>>> #0  0x00007ffff7811107 in __GI_raise (sig=sig at entry=6) at
>>> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
>>> #1  0x00007ffff78124e8 in __GI_abort () at abort.c:89
>>> #2  0x00007ffff784f044 in __libc_message (do_abort=do_abort at entry=2,
>>> fmt=fmt at entry=0x7ffff793f6ab "*** %s ***: %s terminated\n") at
>>> ../sysdeps/posix/libc_fatal.c:175
>>> #3  0x00007ffff78d2147 in __GI___fortify_fail
>>> (msg=msg at entry=0x7ffff793f693 "stack smashing detected") at
>>> fortify_fail.c:31
>>> #4  0x00007ffff78d2110 in __stack_chk_fail () at stack_chk_fail.c:28
>>> #5  0x00007ffff7b96553 in encode_mcu_huff (cinfo=0x7fffffffdd70,
>>> MCU_data=0x602720) at jchuff.c:641
>>> #6  0x00007ffff7b89717 in compress_output (cinfo=0x7fffffffdd70,
>>> input_buf=<optimized out>) at jccoefct.c:381
>>> #7  0x00007ffff7b89006 in jpeg_finish_compress (cinfo=0x7fffffffdd70) at
>>> jcapimin.c:183
>>> #8  0x0000000000401196 in main () at test-768369.c:205
>>>
>>>
>>>
>>>
>>