Update to Speex

[Ron]

On Tue, Mar 18, 2008 at 08:45:58AM +1100, Jean-Marc Valin wrote:
> Ron a écrit :
> >> Ron a écrit :
> >>> So if I've got this straight, the problem here is that we never should
> >>> have distributed 1.1 at all in the first place, as it is not perfectly
> >>> compatible with either 1.0 or 1.2 ?  (and you never claimed it would be)
> >> Distributing 1.1.x was the right thing to do. Depending on the API/ABI
> >> for the *new* stuff it included (noise suppression and AEC) was the
> >> wrong thing to do. None of the apps that worked with 1.0.x broke with
> >> 1.1.x or with 1.2.
> > 
> > hrm.  I think that changes the balance of things a little in my head ...
> 
> I don't understand what this changes. What did you originally think?

If the last 'official' release had been 1.0, which is what I understood
you were 'guaranteeing' compatibility with, then indeed 1.0 -> 1.2.0
(or whatever beta release we decide before then) would not require an
soname change, as things built with libspeex.so.1 (1.0) will always work
perfectly if the system suddenly gives them libspeex.so.1 (1.2).

In that situation, the problem would have been a social one, you had not
promised compatibility for 1.1 users, so any promises we made when
distributing it are our obligation to honour.  We'd have dug a hole it
was entirely our problem to get out of.

As I understand it now though, the promise was a bit more complex.
You were promising libspeex 1.1+ as better and compatible, but there
were extra bits included in it, where all bets were still off as to
what might have to change in them.

I think that is where this first became a technical problem.  And why
I'm not sure a purely social answer is the best one to choose.


Assuming conservatively, that everyone who used the extra bits
understood perfectly what they were getting into by doing that:
still the only way to use them for some time was to link with
libspeex.so.1.

Assuming also that some of them linked with the system libspeex
because the distro builds had included them ...  then:

We now have some number of applications in the wild, with naive users
happily using them.  When libspeex was updated in the past, all users
of the stable api had no problems at all.  Some of the dsp users had
a few problems that went away with a rebuild or a patch from upstream.
But even the latter was a dice roll, many of them probably had no
problems either, because the bits you were still changing were the
most bleeding edge and they weren't using those yet anyway ...

At least that's how I surmise we've heard little about this to date.


What will happen when we split speexdsp out though is a bit different.
All of those applications are now guaranteed to break.  Worst of all,
with lazy linking, they might not break until some point well into the
running when the missing feature is actually used -- at which point
the whole application will ungracefully die.

Worse than that, we're not really sure which (of many) applications
is going to fall into which camp yet.  And without quite a bit of
analysis, we may not know until people's apps crash which ones we
had missed.  Knowing which aren't affected becomes a bit like the
halting problem, though it does have a rough bound.

The only reliable fix for this again becomes a social problem ...
Every maintainer of a dependent package needs to rebuild their code
with the new libspeex and -z defs, fix and re-upload any that break.

The downside of that means that from the time we upload the new
libspeex to the time that all the affected packages are fixed,
they are all immediately broken for all users.


I think using the technical solution is superior in this case,
because we can effect the same transition without any interruption
to the existing users at all.

If the soname were bumped, in line with the strict interpretation
that public symbols have been removed from the library (the library
does not know about the social contract you placed on some of them),
then rather than forcing an all or nothing transition, we get a nice
buffer, where old things work for as long as they need to, and
everything gets updated to the newer version in its own good time.

Uploading libspeex.so.2 won't break any existing apps for any distro,
it won't replace libspeex1, it will coexist with it.  We won't have
to force other app authors into a premature release of their code
because it no longer works with the new libspeex1 package, which I'd
like to avoid as much as I'd like to avoid rushing you (beyond the
hard deadline of "too late for lenny" now ;).

libspeex1 will remain in all the distros so long as some app still
remains to be recompiled with libspeex2 (and if that's all it takes,
autobuilders can fix that problem overnight).

libspeex2 will become the default, so as people upload new versions
of their apps, they will have been fixed however they need to be.
All the end user sees is one day their app gets updated, at no point
it is ever broken because "nobody has updated it yet for the speex
transition" ...

Do you have a scenario where things do break unnecessarily if we
bump the soname?  The only real downside I see is that apps which
could automatically get the new lib won't get it until they are
next rebuilt?  Obviously we don't want to force that on people
any more than ever necessary, but compared to the 'big bang' type
transition, the cost does not seem high.

The big unknown here though is still how big will the bang actually
be in this case ...  a whimper we could suffer, but if we light the
fuse blind, it may be bigger than we are hoping for.

> >   +#define SPEEX_STEREO_STATE_INIT {1,.5,1,1,0,0}
> 
> The extra zeros just prevent something from being left uninitialised.

Cool.  If the size of the struct didn't change, that's fine.  It's just
another thing they won't get the fix for until they recompile ;)

> > The bigger issue though is that while adding all the extra code
> > was forward compatible, removing it again is not ...
> 
> The added stuff was never *ever* marked as stable or anything like that.
> The few who used it were often reminded of that fact.

The catch is that they linked to libspeex to get it, and that is going
to abruptly stop working.  Even if they don't mind fixing it because
they knew it was coming, the users will be left in limbo unless libspeex
and all the updated apps have all been updated together first.

Managing a large group of people to do that in a timely fashion is Hard.
And almost never happens quickly.

The soname change smooths that out in an orderly fashion.  For the cost
of a little temporary space, you buy a few orders of magnitude of cheap
time to spend on resolving the problem.

> Keep in mind that this solution breaks more stuff than it fixes. It
> breaks all the code that was "well-behaved" by only depending on the
> codec.

It won't break them.  They will keep working with the libspeex1 package
they already use, until the next rebuild when they will use libspeex2.
Both will remain available for as long as required.

> On the other hand, for the stuff that depended on the unstable
> features, the only difference is that they'd get an "undefined symbol"
> at link instead of getting a "wrong library version". No big deal. It's
> broken anyway.

It's the already existing binaries that are the problem, they may link
to the new lib and start up fine, only to crash 'mysteriously' later
when the lazy linker goes looking for a speexdsp symbol.

We can't protect them from that, because we are replacing the
libspeex.so.1 they were linked to rather than providing a newer
alternative to it.

> > The messiest, but we can do it that way if we have to option,
> > is to use the package dependency relationships to manually specify
> > a conflict with every app that may have used the removed symbols.
> 
> What do you mean here?

For any package that we could identify in advance, such as ekiga,
we would have to make the libspeex1 (1.2beta3) package (and all
future libspeex packages for at least two debian releases ... )
conflict with ekiga <= $whatever_version_its_at_now.

That way the build of ekiga that cannot work with the replacement
libspeex.so.1 will be forced off the system when speex 1.2beta3 +
are installed.

If the user wants to keep ekiga they will either need to keep the
old libspeex instead, or wait until a new version of ekiga is
uploaded that does work with the new libspeex.

lather, rinse, repeat for any other apps that are reported to break.

It's usually an iterative process, that feeds off of users getting
burned and reporting it, so we like to avoid this one where we can.


> > If the upgrade path was 1.0 -> 1.2beta3+, I think I'd agree not
> > bumping the soname was best, but 1.1 -> 1.2beta2 have been in
> > circulation for quite some time...  I'd wager a lot of people
> > you've never heard of have code using the extra functions ...
> 
> Note that you've already broke the API/ABI several times, with each of
> the 1.1.x releases you made (1.1.6 to 1.1.10 to 1.1.11, ...). I don't
> understand why suddenly things are different.

I think what is different is that the earlier changes rolled the dice
and won, either nobody else noticed them at all, or they noticed them,
remembered what you'd told them, fixed whatever broke and moved on.

This time, everyone is sure to notice.  They still may not grumble,
but some otherwise innocent users are almost surely going to get
caught up in the flak.  We've changed the equation from slim chance
it will break something, to certainty it will break everything that
currently touches speexdsp.

With a change in the soname, we can guarantee everyone to have a
smooth transition, but without one, the best we can do is put out
the fires as they flare up.

> I'd vote for stick with the same soname for libspeex,

If you still think that with all the above taken into account,
then we'll call that final.  Whichever way it goes, we'll all
learn something from how it unfolds  ;)

> then do it right for libspeexdsp.

definitely ...

> The resampler in beta2 was definitely broken, badly. It would produce
> bad quality output in many cases and would crash in some corner cases.

Ahh, bad output I could have missed, up to this point I've largely been
feeding pure tones through it and I don't expect them to perform well,
it's not really what you're optimising for.  They were just good for
spotting the sort of things I was looking for at the time.

The fidelity degraded heavily on some of those in 1.2beta2 relative
to whatever I was using prior to that (probably 1.1.12?) -- if that
shouldn't be, let me know and I'll dig up a test case.

> > I guess with 20/20 hindsight libspeexdsp.so.0 might have been a better
> > starting point, but the numbers themselves aren't particularly important
> > so the law of least surprise would favour libspeexdsp1 for beta3.  The
> > vast majority of users will expect libspeexdsp.so.1 to be provided by
> > that, and a good number of them will get confused if its not.
> 
> The idea is that libspeexdsp.so.0 would be beta3 (or whatever). Then,
> when 1.2 is released, we can have both libspeexdsp1 and libspeex1 to
> make things less confusing.

If there really will be only one more API change (is there definitely
at least one you have planned), that probably would make some sense,
but it will diverge the first time either of them ever does change
in future, and that might make it even more confusing again.

We could aim for libspeexdsp1 and libspeex2 now, and then have 2/2 for
the stable release ;-)

But seriously, that bit doesn't really matter, or at least doesn't
block getting the work done, I'll punt this decision to someone else
to build a consensus around and race them to a result ;)

> The current code is not very different from beta3 actually. You could
> pretty much ship it as a bugfix version of beta3... or wait for the next
> release.

Cool, I'll get beta3 ready to go, then we'll have another look at
how the big picture is shaping up from there.  If you have a known
api change coming in the pipeline as a certainty, we'll probably
want to factor the anticipated timing of that into what we upload
where and when.

> > Do you have a list of things in particular that need help right now?
> 
> Basically, the TODO list goes:
> 1) Lots of testing, especially for the new stuff

I'll run through the odder things I saw when I update our code to beta3
and if they are still odd, I'll see if you think so too ...
It's been on my TODO for a while, but I'll bump it up now release time
is really closing in.

> 2) Documentation

Someone more concise than me please ? ;)

> 3) Need to convert the AGC to fixed-point

hrm...  its just one tiny little function...  but with some
nasty, nasty, operations for fixed-point.  Do you really want
to convert this, or is there some better algorithm we should be
looking at for fixed-point AGC ?

... after hrm'ing some more, I'll get back to you on this one,
I need to rummage around in some old code first ...

> 4) Need to put back a decent VAD.

We might have to define 'decent' before I can decide if I'll
be of any help there ;-)

 Ron