Bug#801745: vlc: Segfault when watching WebM video with increased speed

[Andreas Cadhalpun]

Control: tag -1 confirmed

Hi Michael,

On 14.10.2015 07:31, Michael Weghorn wrote:
> when watching particular WebM videos with increased speed, VLC often crashes
> due to a segmentation fault. 

Thanks for reporting this bug.

> When watching the same video multiple times, the segmentation fault does not
> always occur and if so, it may occur at different points in time.

Indeed.

> The following command can be used to reproduce the problem and examine it in GDB.
> It plays a video in VLC repeatedly, until the segmentation fault occurs (which is often in the
> first run already for me):
> 
> while true; do gdb -ex run -ex quit --args vlc -vvv --play-and-exit --rate 2.5 http://download.media.tagesschau.de/video/2015/1012/TV-20151012-2022-5601.webm.webm; done

Many thanks for providing this test case.
Using it, I could reproduce the problem.
However, the crash happens only once every 5-10 runs here.

It is sufficient to first download the video and the play it, saving bandwith.
The crash happens in the vp8 decoder, so it's reproducible without
audio stream. It's also reproducible without '--rate 2.5'.

I couldn't reproduce this problem directly with ffplay/ffmpeg.

> I am attaching the VLC output and the GDB backtrace of the segmentation fault.
> The packages "vlc-dbg" and "ffmpeg-dbg" are installed on my system.
> 
> As far as I understand, the segmentation fault itself occurs in the ffmpeg library "libavcodec-ffmpeg56".
> The crash happens with a similar backtrace on Debian jessie, where libav is used instead of ffmpeg.

The crash happens in hand-written assembler optimizations for mmx.
Disassembling the relevant function shows the exact location of the crash:
(gdb) disassemble
Dump of assembler code for function ff_emu_edge_vfix3_mmx.body_loop:
=> 0x00007fefa6b90d77 <+0>:	mov    (%rdx),%eax
   0x00007fefa6b90d79 <+2>:	mov    %ax,(%rdi)
   0x00007fefa6b90d7c <+5>:	shr    $0x10,%eax
   0x00007fefa6b90d7f <+8>:	mov    %al,0x2(%rdi)
   0x00007fefa6b90d82 <+11>:	add    %rsi,%rdi
   0x00007fefa6b90d85 <+14>:	add    %rcx,%rdx
   0x00007fefa6b90d88 <+17>:	dec    %r9
   0x00007fefa6b90d8b <+20>:	jne    0x7fefa6b90d77 <ff_emu_edge_vfix3_mmx.body_loop>
End of assembler dump.
(gdb) info registers rdx
rdx            0x7fefa00c4ffd	140667159072765

The source code of this can be found in libavcodec/x86/videodsp.asm [1]:
    ; copy body pixels
.body_loop:                                     ; do {
    READ_NUM_BYTES  body, %%n                   ;   $variable_regs = read($n)
    WRITE_NUM_BYTES body, %%n                   ;   write($variable_regs, $n)
    add            dstq, dst_strideq            ;   dst += dst_stride
    add            srcq, src_strideq            ;   src += src_stride
    dec          end_yq                         ; } while (--end_y)
    jnz .body_loop

The documentation of the READ_NUM_BYTES macro states:
; macro to read/write a horizontal number of pixels (%2) to/from registers
; on sse, - fills xmm0-15 for consecutive sets of 16 pixels
;         - if (%2 & 8)  fills 8 bytes into xmm$next
;         - if (%2 & 4)  fills 4 bytes into xmm$next
;         - if (%2 & 3)  fills 1, 2 or 4 bytes in eax
; on mmx, - fills mm0-7 for consecutive sets of 8 pixels
;         - if (%2 & 4)  fills 4 bytes into mm$next
;         - if (%2 & 3)  fills 1, 2 or 4 bytes in eax
; writing data out is in the same way
%macro READ_NUM_BYTES 2

When the crash happens, n = 3 and end_y = 1, so this should read the last
3 bytes of the frame data plane.
However, the first mov instruction of READ_NUM_BYTES, reads 4 bytes into eax.
Now, according to vlc's debug output ("enabling direct rendering") it provides
the frame data buffers itself [2], instead of letting libavcodec allocate them.
It seems the buffer provided by vlc is exactly the size of the frame data plane, while
buffers created by libavcodec are always a bit larger [3].
With enough bad luck, reading the byte after the buffer (at 0x7fefa00c5000 in this case)
causes a segmentation fault.

So one way to fix this would be for vlc to use larger buffers, though I'm not
sure how to implement that.
Another way would be to change the assembler optimizations to not read more
bytes than absolutely necessary, but I doubt that's feasible, as it'd probably
make the code slower.
A simpler fix would be to remove AV_CODEC_CAP_DR1 from ff_vp8_decoder's capabilities,
forcing vlc to let libavcodec allocate the buffers for this decoder by disabling
direct rendering. But that doesn't seem desirable.

Thus I'm not sure what's the best way to fix this problem.

Best regards,
Andreas


1: https://sources.debian.net/src/ffmpeg/7:2.8.1-1/libavcodec/x86/videodsp.asm/?hl=315#L315
2: https://sources.debian.net/src/vlc/2.2.1-4/modules/codec/avcodec/video.c/?hl=1127#L1116
3: https://sources.debian.net/src/ffmpeg/7:2.8.1-1/libavcodec/utils.c/?hl=552#L552