[Pkg-utopia-maintainers] Bug#773525: Randomly excludes available connections [when there are too many?]

[Dan Williams]

On Thu, 2015-07-16 at 23:56 +0100, Simon McVittie wrote:
> On 16/07/15 19:56, Dan Williams wrote:
> > There really isn't any solution that I can think of, except serializing
> > the requests in the client libraries.  Unfortunately, that's not a great
> > way to go about it and it simply complicates the code on the client
> > side.
> 
> In the medium to long term I think the only way is to have some sort of
> queue of requests, or improve NM's D-Bus API so that things can be
> batched (for instance getting properties of more than one AP per
> round-trip, which would also make it faster).
> 
> > It's quite easy to run over 128 pending replies.
> 
> How many do you need? Is the answer in fact "arbitrarily many"?
> 
> With some appropriate benchmarks we might be able to increase the limit
> by an order of magnitude or two, but I'm concerned that going back to
> the 1K that NetworkManager historically used might be too many.
> 
> The problem is that if you have an unbounded number of requests
> in-flight, the system dbus-daemon uses an unbounded amount of memory to
> track them; and the system dbus-daemon is a shared resource acting on
> behalf of various trusted and untrusted processes, so that would be bad.
> The reasons it tracks them at all are so that unsolicited "replies" can
> be rejected; so that if a client can call a method on a service, the
> service allowed to reply; and so that if a service falls off the bus
> without replying to all method calls it should have handled, dbus-daemon
> can synthesize error replies immediately, rather than leaving the
> clients to time out.
> 
> At the moment dbus-daemon also doesn't use clever enough data
> structures, resulting in a CPU-consumption denial of service attack:
> with a lot of pending replies and a lot of connections allowed, an
> attacker can make dbus-daemon use a lot of CPU time. We dropped the
> pending reply limit from 8K to 128 because there was a practical
> denial-of-service attack with 8K (CVE-2014-3638,
> https://bugs.freedesktop.org/show_bug.cgi?id=81053): simple method calls
> could be made to take more than 25 seconds (the default timeout).
> 
> https://bugs.freedesktop.org/show_bug.cgi?id=83938 has some attempts at
> better data structures, but it's significant code churn, so we're
> unlikely to land anything like that in a dbus stable-branch. I'd be
> happy to review a cleaned up implementation, but that isn't going to
> help for distributions that aren't tracking bleeding-edge dbus.
> 
> kdbus' hard-coded limit was also 128 pending replies per sender, last
> time I looked (although I think it was just coincidence that Alban
> suggested the same number after experimenting with denial of service
> attacks). It needs to track requested replies for basically the same
> reasons as dbus-daemon, and also has to be fairly conservative with its
> limits because it's allocating kernel memory. So moving to kdbus is
> probably not going to save us from this.
> 
> > We do have the 'libnm' library with NM 1.0+ that uses GDBus all the way
> > through, so if GDBus somehow manages to avoid this problem then great.
> > Otherwise, we'll have the same problem in libnm too...
> 
> It sounds as though this is really an issue with the high-level design
> of NM's D-Bus API, rather than the specifics of how the client library
> is implemented, so I don't think GDBus is going to help.

I'd take issue with that assertion actually.  The object model of NM is
such that the number of objects can be determined by either (a)
configuration, eg number of saved network connections or (b) external
network properties like number of access points, etc.  I don't think
that's really unreasonable, especially given how D-Bus wants objects +
properties + interfaces.  I'd rather not go all libpurple-style and
stuff all properties into one interface so we can save 2 GetAll() calls
per object.

eg, think of it as each contact in Telepathy being an object, and I can
easily see somebody having 100+ contacts/friends etc.  Each of those
contacts has properties to be retrieved too.  So would a client just try
to get 50 of those at a time, wait for the reply, then get the next 50?
And then fire off GetAll() on each of those in 50-item batches?  And
then for multiple interfaces?

Also, how is a client supposed to know how big the limit is?  Or is a
client just supposed to use a sufficiently low number that it is known
to work in all cases?

The solution here is likely to transition the libnm implementation over
to the ObjectManager interface's GetManagedObjects() method for the
initial setup, to get everything in one call.  A lot of data, but likely
faster than doing it piece-by-piece.  That's only going to happen with
NM 1.2 and later though when the daemon is converted to GDBus.  Which
might well leave a lot of people out in the cold if they start using
kdbus with Debian Jessie or something where NM is still at 0.9.10.  Any
chance we could get dbus-glib to implement GetManagedObjects()?

Dan