[Pkg-tigervnc-devel] Bug#768369: Acknowledgement ([libjpeg62-turbo] [DOS] Stack smashing)

[roucaries bastien]

We have investigated a little bit (thansk y dlemstra )

was able to reproduce the problem under Windows that uses libturbo
1.3.0. It looks like the problem is inside the STORE_BUFFER macro at
line 421 of jchuff.c:

Code: Select all

buffer = _buffer; \
while (bytes > 0) { \
  bytestocopy = min(bytes, state->free_in_buffer); \
  MEMCOPY(state->next_output_byte, buffer, bytestocopy); \
  state->next_output_byte += bytestocopy; \
  buffer += bytestocopy; \
  state->free_in_buffer -= bytestocopy; \
  if (state->free_in_buffer == 0) \
    if (! dump_buffer(state)) return FALSE; \
  bytes -= bytestocopy; \
} \


The size of _buffer is BUF_SIZE = (DCTSIZE2 * 2) = (64 * 2) = 128. The
following is happening at *block == -591:

1. bytes = 171, state->free_in_buffer = 14 and bytestocopy becomes 14
2. memcopy is called and the buffer is moved to index 14
4. state->free_in_buffer becomes 0, dump_buffer is called and returns true
5. bytes = 157, state->free_in_buffer = 16384 and bytestocopy becomes 157

(14 + 157) > 128 and this causes a buffer overflow.

Not sure why bytes becomes more then 128 but this info should probably
give the maintainers of libjpegturbo a bit more information to
investigate this issue.

On Fri, Nov 7, 2014 at 9:33 PM, DRC <dcommander at users.sourceforge.net> wrote:
> I don't know why you would expect me to have tried that, given that there
> was no reasonable expectation that I would know any of that information
> prior to now.
>
> But I did just try it, and it works fine.
>
> I will re-iterate:
> You need to produce a test case that demonstrates this failure using only
> libjpeg-turbo.  You cannot expect me to spend any time on this until it is
> proven that it is my bug and not someone else's.  Sometimes issues like this
> can be caused by the overlying application, even though the bug manifests in
> the underlying library.
>
>
>
> On 11/7/14 12:47 PM, roucaries bastien wrote:
>>
>> On Fri, Nov 7, 2014 at 6:36 PM, DRC <dcommander at users.sourceforge.net>
>> wrote:
>>>
>>> I want exactly what I asked for:  a way to reproduce this issue.
>>> Currently I
>>> cannot.  A backtrace from your machine is not helpful, as it does not
>>> tell
>>> me anything regarding how the library is being used by ImageMagick.
>>
>>
>> Did you try to compile libjpeg-turbo with -fstack-protector-all ggc
>> flags. Debian do it and thus detect stack overflow (valgrind is not at
>> help here).
>>
>> BTW could you nevertheless get a glimpse at the last backtrace. I but
>> a watch point on the canary (I tried but because this function is
>> called a lot of time I may be missing something) using method [1]. It
>> seems the code that smash the code is at encode_one_block line 543:
>>   kloop(59);  kloop(52);  kloop(45);  kloop(38);  kloop(31);  kloop(39);