Select provider of libav* libraries

[Andreas Cadhalpun]

Hi Reinhard,

On 04.06.2015 02:32, Reinhard Tartler wrote:
> On Sun, May 31, 2015 at 7:21 AM, Andreas Cadhalpun
> <andreas.cadhalpun at googlemail.com> wrote:
> 
>>> and would rather propose to just rename the current "libav" source
>>> package to "ffmpeg" and package the latest ffmpeg source tarball.
>>
>> Then that ffmpeg source package would take over the binary package
>> names as well. If you think it's not necessary to preserve the
>> development packages from src:libav, my proposal becomes even simpler:
>> Just rename lib*-ffmeg-dev to lib*-dev in the current ffmpeg source
>> package. That'd work, because src:ffmpeg has a higher epoch.
> 
> That's not what I'm proposing.

It would certainly help if you would explain in more detail what you
are proposing.

>>> I'd rather keep the packaging of the package that is currently called
>>> "libav", because on many architectures, it compiles multiple "flavors"
>>> with hardware optimized flavors (on i386 for instance, there is a
>>> flavor without latest SEE, etc).
>>
>> I had considered producing optimized flavors, but I came to the conclusion
>> that it would complicate debian/rules without much benefit, because
>> the flavors are mostly unnecessary:
> 
> That's not only because of optimization, but also for licensing
> issues: The lib*-extra-* packages are licensed with GPLv3+, which is
> not acceptable for the majority of applications in debian.
> 
> Note that it does not matter whether or not there are GPLv3+
> applications in the archive. These -extra- package do provide
> additional functionality that users do appreciate. However, we cannot
> provide that in the "standard" libav* packages because we do have
> GPLv2-only applications that we must not link against a GPLv3
> libavcodec.

The current implementation of the extra flavor doesn't prevent using it
with a GPLv2-only program and thus is effectively nearly as bad as just
enabling the GPLv3 code in the standard build.
Adding a mechanism to prevent this would make the extra flavor even
more complex.

>> the only additional functionality of the extra variant is limited
>> to an AMR encoder [3]. Thus I don't think this justifies the additional
>> complexity of the extra variant.
>
> I disagree with this assessment,

Well, you could file a bug report against src:ffmpeg. If enough users
agree that having the AMR encoder available is important, I might reconsider
my position.

> and can easily imagine future functionality that requires a GPLv3 build.

If that happens, we can reevaluate the situation.

>>  * On i386 all the SSE etc. optimizations are guarded by runtime
>>    CPU detection, so that even when compiled with SSE the libraries work
>>    fine on CPUs without SSE.
>>    The only optimization were this doesn't apply are the i686 optimizations,
>>    which currently amount to a dozen assembler lines. So I don't think
>>    this justifies shipping two flavors.
> 
> Have a look at #688384 - the conclusion was to compile for i586, which
> seems absurd to me for most users, even if most optimizations are
> indeed the runtime selected hand-written assembler parts. The libav
> packaging solves this with multiple compilation passes.

The SIGILL in bug #688384 was caused by an MMX instruction (emms), which
should have been guarded by runtime CPU detection. Since the bug reporter
provided no additional information, it is impossible to say what the
actual problem was.
Anyway, this problem doesn't exist in the current ffmpeg package. At least
I couldn't produce any SIGILL crash with 'qemu-system-i386 -cpu pentium,-mmx',
which I think is the lowest supported CPU for i386.

>>  * The situation for arm*, where the neon/vfp optimizations are runtime
>>    guarded, is similar.
> 
> I don't think so, have a look at #657885, which shows a SIGILL in
> cmdutils.c. This can hardly be caught at runtime.

Thanks for pointing me to bug #657885, which is an excellent example for
the problems that can be caused by building multiple flavors.

The problem here was the the ffmpeg binary from the build with ARMv7
compiler optimizations was installed instead of the one with only ARMv4
instructions. This does not happen in the current ffmpeg package, which
only builds the ARMv4 variant of the ffmpeg binary, while still having
optimized code in the libraries, guarded by runtime CPU detection.

The only SIGILL crash I could find with 'qemu-system-arm -M versatilepb'
is caused by a bug in the have_setend detection. I've sent a patch
fixing this upstream [1].

>>  * This leaves powerpc, where configure would enable '-maltivec'
>>    together with the hand-written altivec optimizations, which are guarded
>>    by runtime CPU detection. This causes gcc to add some altivec
>>    instructions, which are not guarded and thus cause SIGILL.
>>    Looking into fixing this in the configure script is on my TODO list.
> 
> It is impossible to fix (or rather not worth the effort to fix
> properly). Either you disable altivec completely, or you need need to
> pass -maltivec which allows you to use the altivec optimizations and
> break on non-altivec machines. So you either make all multimedia
> package unusably slow for altivec machines because altivec is not
> enabled, or you exclude all uses of non-altivec machines.

It should be possible to only use '-maltivec' for the files actually
containing hand-written altivec optimizations. But maybe that's not
worth the effort.

>> Are there good reasons to produce these optimized flavors?
> 
> Yes, there are, at least on i386, arm and powerpc. I would consider
> not having them a serious regression.

As explained above, I don't think there are good reasons to provide
flavors on i386 or arm. For powerpc it might be the simplest option.

>>> Andreas, would that approach be acceptable to you?
>>
>> I'd rather continue with the packaging of the current ffmpeg source
>> package, because the one from src:libav uses a pre-dh7 debian/rules,
>> and a quite complicated one at that.
> 
> I'm happy to simplify or refactor the packaging, but I fear I have to
> insist to continue using the current src:libav packaging.

Why would you want to simplify/refactor the old-style debian/rules file
although the ffmpeg package already has a new-style debian/rules file?
I don't think this makes much sense.

> It also seems easier to conduct the resulting transition if the
> package names of what application packages depend on don't change.

Do you mean libav-tools should contain ffmpeg etc.?
If so I strongly disagree, because that would be just wrong.

> Please let's not overcomplicate these things.

Then why do you insist on the complicated extra flavor or needlessly
rebasing the current ffmpeg package on the libav packaging?

>>> If so, then I'd propose to get a test package ready in Git, and I'd
>>> then do a mass-rebuild on my amd64 box to see how many packages fail
>>> to build.
>>
>> I already did a mass-rebuild for my proposal and the results can be
>> found in my mail [1] mentioned above.
>> The blender and paraview FTBFS bugs have been resolved and the new mpd
>> version is now in sid, so unless new, unrelated FTBFS bugs were
>> introduced, only 7 packages would fail to build.
>> Three of those FTBFS already (dvswitch has a patch in #747868,
>> gstreamer0.10-ffmpeg should be removed and jitsi is fixed in NEW
>> since 8 months).
>> Two (gpac and xbmc) would require changes for the next Libav version
>> as well, because they use private API, which was changed.
>> The hardcoded sonames in taoframework have to be updated, like in every
>> Libav transition. (Maybe someone can fix taoframework to use the sonames
>> present in the build environment?)
>> So only gst-libav1.0 needs a FFmpeg specific change, which is an additional
>> build-dependency on libavresample-dev, because it currently relies on
>> libavcodec-dev to bring this indirectly, but FFmpeg's libavcodec uses
>> libswresample instead of libavresample.
> 
> Good. Did you check whether current FFmpeg is ABI compatible to the
> current libav packages,

No, because I don't know how that could be done.

> or do we need to recompile all packages in any case?

This is the safer choice, unless it can be proven that
using a program compiled against the Libav library wouldn't
(subtly) break, when running with the FFmpeg library.

>> So while Michael is clearly the most active FFmpeg developer, it is not
>> at all a "one man show".
>> Claiming that is a bit like claiming that Linux development was a
>> "one man show", because Linus is the most important developer.
> 
> I think the statistics indicate that the situation is much worse than
> I expected. Please keep in mind that because of the daily merges, all
> commits that are made in Libav also appear in FFmpeg, but not the
> other way round.

That's exactly the problem.

> With a finger-in-the-wind estimate of subtracting the
> numbers in the Libav table from the FFmpeg table, the distance between
> Michael and the other FFmpeg developers because even more extreme.

To summarize: Commits to the FFmpeg repository can be sorted in
three roughly equally large categories:
 * from Michael Niedermayer (1869 since n2.4 [2])
 * from Libav developers (1410 since v11 [3])
 * from other FFmpeg contributors (1750 since n2.4 [4])

Thus Libav lacks more than two thirds of the development done in FFmpeg.

> I'm sorry if you felt that my characterization was disrespectful - it
> was certainly not my intention to undermine anyone's contribution.

That's good to hear, thanks for clarifying.

> I was rather trying to make a point with risk assessment of two
> competing products: the FFmpeg codebase and the Libav codebase. As
> outline above, the product itself is not the only consideration in
> this debate - the health of the upstream project is also very
> important to consider.

As you can see from above numbers, even in the uttermost worst case,
that is if both Michael and the Libav developers stopped working on
the codebase, FFmpeg would still have more commits than Libav
currently does.

Best regards,
Andreas


1: https://ffmpeg.org/pipermail/ffmpeg-devel/2015-June/173814.html
2: FFmpeg: $ git shortlog -ns --no-merges n2.4..HEAD | head -n1
3: Libav:  $ git log --oneline --no-merges v11..HEAD | wc -
4: FFmpeg: $ git log --oneline --no-merges n2.4..HEAD | wc -l
           - [2] - [3]