Bug#660268: vlc: Dumps core when coverting to MP3

[Reinhard Tartler]

On Fri, Feb 17, 2012 at 9:48 PM, Nigel Horne <njh at bandsman.co.uk> wrote:
> Package: vlc
> Version: 1.1.13-1+b1
> Severity: normal
>
> Dear Maintainer,
> *** Please consider answering these questions, where appropriate ***
>
>   * What led up to the situation?
> Attempted to conver to MP3
>   * What exactly did you do (or not do) that was effective (or
>     ineffective)?
> Media->Convert/Save->Add File->Convert/Save->Fill in destination file->select MP3 from pull down list->Press Start
>   * What was the outcome of this action?
>   vlc: /build/buildd-libav-extra_0.8.0.1+b1-amd64-PoqmIt/libav-extra-0.8.0.1+b1/libav/libavutil/mathematics.c:79: av_rescale_rnd: Assertion `c > 0' failed.
>   Aborted (core dumped)
>
>   * What outcome did you expect instead?
> Audio extracted into MP3 file

That function (av_rescale_rnd) has not been touched since 2005, so the
problem is most probably in its caller. The function itself starts
like this (docstring copied from header):

/**
 * Rescale a 64-bit integer with specified rounding.
 * A simple a*b/c isn't possible as it can overflow.
 */
int64_t av_rescale_rnd(int64_t a, int64_t b, int64_t c, enum AVRounding rnd){
    int64_t r=0;
    assert(c > 0);
    assert(b >=0);
    assert((unsigned)rnd<=5 && rnd!=4);


Remi, unless I have a backtrace that shows that this is a problem
within some libav library, there is little we can do about it. TBH, I
rahter suspect the bug to be in the calling application.

Looking in launchpad, I see this (possibly the same) crash:
https://bugs.launchpad.net/vlc/+bug/927050

Unfortunately, the backtrace does not show the offending caller:

#0  0xb7798424 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb74f2c8f in __GI_raise (sig=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
        resultvar = <optimized out>
        resultvar = <optimized out>
        pid = -1218183180
        selftid = 18107
#2  0xb74f62b5 in __GI_abort () at abort.c:92
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0xd, sa_sigaction =
0xd}, sa_mask = {__val = {3076784116, 3076789248, 128, 3076784116,
3076789248, 128, 127, 3075716637, 3076784116, 127, 3076784116, 127,
3075651409, 178484352, 178484352, 127, 178483592, 0, 178484352,
4222451712, 178484352, 178484453, 178484352, 178484352, 178484479,
178484652, 178484352, 178484652, 0, 0, 0, 0}}, sa_flags = 0,
sa_restorer = 0}
        sigs = {__val = {32, 0 <repeats 31 times>}}
#3  0xb74eb826 in __GI___assert_fail (assertion=0xb01031e4 "c > 0",
file=0xb01031a4
"/build/buildd/libav-extra-0.8.0.1/libav/libavutil/mathematics.c",
line=79, function=0xb0103520 "av_rescale_rnd") at assert.c:81
        buf = 0xaa375b0 "vlc:
/build/buildd/libav-extra-0.8.0.1/libav/libavutil/mathematics.c:79:
av_rescale_rnd: Zusicherung \302\273c > 0\302\253 nicht
erf\303\274llt.\n"
#4  0xb00f73b5 in ?? ()
No symbol table info available.


Nigel, can you reproduce the crash and can ideally try to retrieve a
proper stacktrace?

-- 
regards,
    Reinhard