Bug#599329: Found cause of vainfo segfaults

[Anthony DeRobertis]

reassign 599329 vdpau-va-driver
forcemerge 611974 599329
thanks

Sorry for the mess here, this is from combining three emails...

I'm seeing the same segfault here, and rebuild with nostrip to get a
more useful backtrace:

anthony at Zia:~$ gdb vainfo
GNU gdb (GDB) 7.0.1-debian
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/vainfo...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/bin/vainfo 
[Thread debugging using libthread_db enabled]
libva: libva version 0.31.1
Xlib:  extension "XFree86-DRI" missing on display ":0.0".
libva: va_getDriverName() returns 0
libva: Trying to open /usr/lib/dri/nvidia_drv_video.so

Program received signal SIGSEGV, Segmentation fault.
XDisplayString (dpy=0x0) at ../../src/Macros.c:118
118     ../../src/Macros.c: No such file or directory.
        in ../../src/Macros.c
(gdb) bt
#0  XDisplayString (dpy=0x0) at ../../src/Macros.c:118
#1  0x00007ffff5e92e62 in vdpau_common_Initialize (driver_data=0x611b90) at vdpau_driver.c:181
#2  0x00007ffff5e937c3 in vdpau_Initialize_0_31_1 (ctx=0x60eb40) at vdpau_driver_template.h:534
#3  __vaDriverInit_0_31 (ctx=0x60eb40) at vdpau_driver.c:291
#4  0x00007ffff7bba242 in vaInitialize () from /usr/lib/libva.so.1
#5  0x0000000000400a92 in ?? ()
#6  0x00007ffff7330c4d in __libc_start_main (main=<value optimized out>, argc=<value optimized out>, 
    ubp_av=<value optimized out>, init=<value optimized out>, fini=<value optimized out>, 
    rtld_fini=<value optimized out>, stack_end=0x7fffffffe0d8) at libc-start.c:228
#7  0x0000000000400959 in ?? ()
#8  0x00007fffffffe0d8 in ?? ()
#9  0x000000000000001c in ?? ()
#10 0x0000000000000001 in ?? ()
#11 0x00007fffffffe435 in ?? ()
#12 0x0000000000000000 in ?? ()





After rebuilding with noopt as well, I found this:

(gdb) up
#3  0x00007ffff5e8f5a0 in __vaDriverInit_0_31 (ctx=0x60eb40) at vdpau_driver.c:291
291             return vdpau_Initialize_0_31_1(ctx);
(gdb) l
286
287         /* Assume a NULL display implies VA-API 0.31.1 struct with the
288            vtable_tpi field placed just after the vtable, thus replacing
289            original native_dpy field */
290         if (!ctx0->native_dpy)
291             return vdpau_Initialize_0_31_1(ctx);
292
293         return vdpau_Initialize_0_31_0(ctx);
294     }
295     #endif

That comment looks damning. It is calling vdpau_Initialize_0_31_1, with
the null native_dpy (as apparently expected), which is then ultimately
being passed to XDisplayString, leading to a segfault.

So, it seems there is an API mismatch.




Finally, list of all the related (hopefully) packages I have installed:

anthony at Zia:~$ dpkg --get-selections | egrep 'nvidia|vdpau|libva' | awk '{print $1}' | xargs dpkg-query -W
libgl1-nvidia-alternatives      260.19.44-1
libgl1-nvidia-glx       260.19.44-1
libglx-nvidia-alternatives      260.19.44-1
libva-dev       1.0.8-3
libva-glx1      1.0.8-3
libva-tpi1      1.0.8-3
libva-x11-1     1.0.8-3
libva1  1.0.8-3
libvamp-hostsdk3        2.1-1
libvariable-magic-perl  0.43-1
libvdpau-dev    0.4.1-2
libvdpau1       0.4.1-2
nvidia-common   20110213+1
nvidia-glx      260.19.44-1
nvidia-kernel-common    20100522+1
nvidia-kernel-dkms      260.19.44-1
nvidia-settings 195.36.24-1
nvidia-vdpau-driver     260.19.44-1
nvidia-xconfig  195.36.31-1
vdpau-va-driver 0.7.2-1
vdpau-video-build-deps  1.0