[Pkg-utopia-maintainers] Bug#570165: Hung avahi-daemon causes gethostbyaddr() to never return

[Richard Darst]

Package: avahi-daemon
Version: 0.6.27-2
Severity: normal

Hi,

I have more info.

I was seeing a very annoying problem where openssh sshd would not
allow logins:

  $ ssh-nomux pyke
  <120 seconds pass, the sshd connection timeout>
  Connection closed by 192.168.100.200

After a while searching for this, I found that sshd would hang here:
  Dec 20 23:53:00 pyke sshd[30251]: debug3: Trying to reverse map address 192.168.100.188.
and that using "-o UseDNS=no" to sshd would work around the no-login
problem.

Now to the avahi part:

I also found that gethostbyaddr() would hang and never return
(after leaving it going overnight):
  $ python -c 'import socket ; print socket.gethostbyaddr("192.168.100.188")'

It is correlated with this dbus error:
  $ avahi-resolve -a 192.168.100.188
  Failed to create client object: An unexpected D-Bus error occured

If avahi isn't running at all, we get this error instead and there is
*not* a problem with gethostbyaddr():
  $ avahi-resolve -a 192.168.100.188
  Failed to create client object: Daemon not running

Attempting to invoke-rc.d avahi restart/stop does not work - I have to
kill it.  It also gets into some sort of a bad state where
"invoke-rc.d avahi-daemon start" won't cleanly start it again.

Here are straces on the avahi-daemon processes when they are hung
(when not hung, the first process waits at poll() and the second
waits at read()):
  <process: avahi-daemon: running [pyke.local]>
  select(13, NULL, [12], NULL, NULL

  <process: avahi-daemon: chroot helper>
  read(4, 

A C-c at the console results in more strace output but it hangs again
at a select() call again.

I see this in the output of avahi-daemon --debug, but there are many
lines like this and they they go on for a while before it hangs:
  Invalid response packet from host 128.59.x.y.
  Invalid response packet from host 128.59.z.w.

Here is a gdb backtrace from when it is hung (retrived from gdb --args
avahi-daemon --debug, there are two avahi processes so I'm not sure if
this is the most useful output):

#0  0x00007ffff686b8b3 in select () from /lib/libc.so.6
#1  0x00007ffff79c3093 in avahi_wait_for_write (fd=<value optimized out>)
    at fdutil.c:66
#2  0x00007ffff79b5a2c in sendmsg_loop (fd=12, msg=0x7fffffffe3f0, 
    flags=<value optimized out>) at socket.c:463
#3  0x00007ffff79b5ebe in avahi_send_dns_packet_ipv6 (fd=12, interface=3, 
    p=0x7fc7f0, src_address=0x63ae7c, dst_address=0x0, 
    dst_port=<value optimized out>) at socket.c:623
#4  0x00007ffff79a93dc in avahi_interface_send_packet_unicast (i=0x63ae40, 
    p=0x7fc7f0, a=0x0, port=0) at iface.c:601
#5  0x00007ffff79b7d04 in append_known_answers_and_send (
    e=<value optimized out>, data=<value optimized out>) at query-sched.c:269
#6  elapse_callback (e=<value optimized out>, data=<value optimized out>)
    at query-sched.c:310
#7  0x00007ffff79a8890 in expiration_event (timeout=<value optimized out>, 
    userdata=0x623ad0) at timeeventq.c:94
#8  0x00007ffff7bda8f0 in avahi_simple_poll_dispatch (s=0x622810)
    at simple-watch.c:570
#9  0x0000000000408f33 in run_server (argc=<value optimized out>, 
    argv=<value optimized out>) at main.c:1219
#10 main (argc=<value optimized out>, argv=<value optimized out>)
    at main.c:1637


This happens intermittently, so it may be avahi-daemon getting into a
bad state caused by network traffic.  However, it *is* reproducible if
I leave the daemon running for a few hours, so give me tasks and I can
get more debugging information.

This is effectively a possible DOS (including DOS of ssh) from IPs
which don't have rDNS set up.  Workarounds are UseDNS=no for sshd or
sshing from an IP with a valid rDNS.

Let me know if there is anything more I can do.

Thanks,

- richard



-- System Information:
Debian Release: squeeze/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.32-5-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages avahi-daemon depends on:
ii  adduser                3.112+nmu2        add and remove users and groups
ii  bind9-host [host]      1:9.7.2.dfsg.P3-1 Version of 'host' bundled with BIN
ii  dbus                   1.2.24-3          simple interprocess messaging syst
ii  libavahi-common3       0.6.27-2          Avahi common library
ii  libavahi-core7         0.6.27-2          Avahi's embeddable mDNS/DNS-SD lib
ii  libc6                  2.11.2-7          Embedded GNU C Library: Shared lib
ii  libcap2                1:2.19-3          support for getting/setting POSIX.
ii  libdaemon0             0.14-2            lightweight C library for daemons 
ii  libdbus-1-3            1.2.24-3          simple interprocess messaging syst
ii  libexpat1              2.0.1-7           XML parsing C library - runtime li
ii  lsb-base               3.2-23.1          Linux Standard Base 3.2 init scrip

Versions of packages avahi-daemon recommends:
ii  libnss-mdns                   0.10-3.1   NSS module for Multicast DNS name 

Versions of packages avahi-daemon suggests:
pn  avahi-autoipd                 <none>     (no description available)

-- no debconf information