[Aptitude-devel] Acquire considered non-threadsafe
Julian Andres Klode
jak at debian.org
Thu Nov 19 17:09:42 UTC 2009
On Thu, Nov 19, 2009 at 04:31:13AM -0800, Daniel Burrows wrote:
> On Wed, Nov 18, 2009 at 04:03:05PM +0100, Julian Andres Klode <jak at debian.org> was heard to say:
> > On Sun, Nov 15, 2009 at 12:42:38PM -0800, Daniel Burrows wrote:
> > > There are some other very nice options involving rewriting or
> > > reimplementing the "download queue" logic (i.e., just pkgAcquire), but
> > > getting threadsafety does require changing the behavior of
> > > pkgAcquire::Item in any event. I'm assuming that we don't want to have
> > > two parallel implementations of this functionality, and that
> > > non-backwards-compatible changes are considered unacceptable, in which
> > > case the opt-in approach is the only solution I can think of.
> > We should really start talking about APT 0.8 sometime soon, and break
> > some APIs. The problem I have with Acquire is its low speed when queuing
> > items which is probably caused by the use of a linked list (20000 items
> > take multiple minutes to be queued, whereas APT2 queues them in less than
> > a second).
> Acquire is also really painful to develop against, although that's
> more just the general pain of dealing with apt.
> Also, apt2? When did that happen?
APT2 will be a package management library + applications written in Vala,
but at the moment it's just a few parsers and a simple acquire system. And
it's intented to be cross-distro, just like PackageKit. I just had no better
name when I started it. It's currently MIT-licensed, but may switch to the
LGPL-2.1+ soon (as this is what all its dependencies like glib use).
Code is at http://git.debian.org/?p=users/jak/apt2.git;a=summary, but it
is slightly outdated compared to my local branch (which is LGPL-2.1+
> > (b) a new cache format which can be resized in reality
> I wonder if apt really needs an mmap'd cache here. In fact, I wonder
> about using sqlite -- I did a little experimentation with it and got
> decent performance. The big problem (from my point of view) would be
> teaching client code to treat the database transactionally rather than
> as a linked set of in-memory objects, but it way more cleaner and more
> flexible than what we have now.
Working with databases is a bit complicated. It makes development not
> > (c) new dependency resolver [porting aptitude's one?]
> > (d) external dependency resolvers
> > (e) new buildsystem (e.g. raw autotools) and reworked
> > packaging
> Those sound very good.
I already have a CMake buildsystem in the works which can already build
apt-pkg and apt-inst. I expect this to be able to replace the current
one and build exactly the same packages afterwards; so backwards
compatibility is preserved. I will publish a new branch soon.
I chose CMake because it does not require hundreds of files in the tree
or shipping generated files. And it also appears to be faster.
> > (f) storing errno inside the error handling objects,
> > for bindings (nice to have errno).
> Please, please, kill the global error queue. This is a disaster from
> a threadsafety point of view, but also just from a general design
> perspective. There's way too much code that does
> when the error could have been caused by something entirely unrelated
> to what this code is interested in.
> Put error-handling oun either a return-code basis or an exception
> basis, but not this wird intermediate thing where random errors cause
> failure. If you want to know about inner errors, use a logging
> framework (I prefer log4cxx).
As far as I can tell from the documentation, the APT error handling should
only be used for boolean functions. If the function has an error, it pushes
a message to the error stack and returns false. If we used it this way only,
we would have no multi-threading problems.
Exceptions are a bit tricky in C++ as they can happen unexpectedly. Google's
C++ style guide says no; so we should probably also say no.
Log4cxx would effectively cause APT binaries to be GPL-3+ licensed, as
GPL-2 is incompatible with the Apache License. This could be a problem
for GPL-2 only programs using them. But it may still be worth it.
> I would add one more item:
> (g) use modern C++ coding conventions (at least for new code)
> apt was designed in 1996 or so, and it shows. Modern C++ is a lot
> cleaner, easier to maintain and easier to write than the weird mishmash
> of C and "object-oriented programming" that apt is written in. Also,
> a lot of modern C++ programs link against Boost, which provides a ton
> of well-designed utility libraries for stuff that you would otherwise
> have to write yourself, and that you'd end up writing badly.
> (apt-pkg/contrib/ is full of that sort of thing)
We should not forget that APT has priority important, and boost libraries
are optional. But we could switch APT to the subset of C++0x which is
supported by GCC 4.3, the version in stable.
> > But I don't think anyone wants to do this.
> Personally, I would totally love to clean up the apt code and fix
> some of my chronic annoyances with it. I just don't have the spare
> time, and apparently neither does anyone else (except some guy who
> thinks it should be written in Perl (!?!?)). :-(
Cupt is great as a prototype and can give us a direction in terms of
features. As long as you are not running an embedded system and as long
as Perl works things are okay.
Some kind of a roadmap:
* identify stuff we don't use anymore and remove it, eg.
apt-pkg/vendor*, apt-inst/database*, apt-inst/deb/dpkgdb*.
* drop compatibility for GCC < 4.3, and other stuff older
then what is included in Debian Lenny.
* rework the buildsystem using CMake
Julian Andres Klode - Debian Developer, Ubuntu Member
See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.
More information about the Aptitude-devel