[Pkg-utopia-maintainers] Bug#879898: dbus: memory leak in dbus-daemon consumes over 5GB memory

[Simon McVittie]

On Thu, 26 Oct 2017 at 23:58:59 -0400, Nicholas D Steeves wrote:
> I hope to hear back from you soon, because Monday I will need to use
> my laptop and I will have to kill the misbehaving dbus-daemon to
> reclaim memory at that time.

This will end your KDE login session rather unceremoniously. If
possible, log out from KDE rather than letting the dbus-daemon die
from the OOM killer or a manual kill command. This should make the
dbus-daemon exit automatically, but if it doesn't, log in to a
text console (as your ordinary user or as root) and kill the
`dbus-daemon --session` process (process 958) from there.

Please make sure you don't kill `dbus-daemon --system` by mistake
(only root can do this); if you do, you'll have to reboot to fix it.

> Here is the only section that jumps out to me:
>   0x00007f87feb730d3 in __epoll_wait_nocancel () at ../sysdeps/unix/syscall-template.S:84
>   84	../sysdeps/unix/syscall-template.S: No such file or directory.
>   #0  0x00007f87feb730d3 in __epoll_wait_nocancel () at ../sysdeps/unix/syscall-template.S:84

Unfortunately this backtrace only indicates "dbus-daemon is idle and
waiting for something interesting to happen". It should normally be in
this state 99% of the time. If it was in a busy-loop that was consuming
all the CPU, a backtrace might be more interesting, but for memory
consumption it's unlikely to help.

The process using all the memory is the standard session bus (dbus-daemon
--session), which has essentially no resource limits (everything on it
is considered equally trusted), so the root cause is probably that some
misbehaving applications or services in your user login session have
used a calling pattern that makes the dbus-daemon use a lot of memory
on their behalf. Sending messages very rapidly to a service that is slow
to process them is one thing that is known to have this effect.

As far as I know, all ways to trigger rapid memory consumption involve
rapid message sending and receiving, so if the dbus-daemon's memory
use is still growing, you can probably see what's happening by running
dbus-monitor, or by looking at top and seeing which other D-Bus-related
processes are actively running. While debugging, you can use "kill -STOP"
and "kill -CONT" to pause and resume the dbus-daemon (which will pause
message processing, so many applications and services will not work
while it is paused, in particular many graphical applications and
dbus-send).

You can correlate bus names (the things that look like :1.23 or
org.gnome.Software) with process IDs by using:

dbus-send --session --dest=org.freedesktop.DBus --type=method_call \
    --print-reply /org/freedesktop/DBus \
    org.freedesktop.DBus.GetConnectionCredentials "string:$name"

where $name is the bus name.

The dbus-daemon also records some potentially useful statistics for
memory consumption and related things, and this feature is compiled in
to Debian's dbus-daemon. Please try running this:

dbus-send --session --dest=org.freedesktop.DBus --type=method_call \
    --print-reply /org/freedesktop/DBus \
    org.freedesktop.DBus.Debug.Stats.GetStats

and send the results to this bug report. You could also try retrieving
details of individual connections:

dir="$(mktemp -d)"
echo "Look in $dir for connection details"
dbus-send --session --dest=org.freedesktop.DBus --type=method_call \
    --print-reply /org/freedesktop/DBus org.freedesktop.DBus.ListNames | \
perl -ne 'print("$1\n") if /^\s*string "(.*)"$/' | \
while read name; do \
    dbus-send --session --dest=org.freedesktop.DBus --type=method_call \
        --print-reply /org/freedesktop/DBus \
        org.freedesktop.DBus.Debug.Stats.GetConnectionStats "string:$name" \
        > "$dir/$name.stats"
done

(It is expected that you get an error message "Error
org.freedesktop.DBus.Error.InvalidArgs: GetConnectionStats is not
meaningful for the message bus "org.freedesktop.DBus" itself",
and it is also OK to get a small number of messages like "Error
org.freedesktop.DBus.Error.NameHasNoOwner: Could not get statistics of
name ':1.92': no such name".)

This will leave a lot of text files in $dir/*.stats (in my GNOME session
there are 127 of them). Then look in $dir/*.stats for a connection or
connections with much larger numbers than the rest. Again, you can use
GetConnectionCredentials to correlate them with process IDs.

Regards,
    smcv