[Pkg-tigervnc-devel] Bug#873151: tigervnc-standalone-server: Forced disconnections clog up the VNC server

[Mikko Rasa]

Package: tigervnc-standalone-server
Version: 1.7.0+dfsg-7
Severity: normal

I am running a graphical IRC client in a VNC server.  Previously I used
realvnc, but since it got removed from Debian I switched to tigervnc around
the start of the year.  Since then I've noticed that after the VNC server has
been running for a month or two, I can no longer get in on the first try.  A
second attempt is successful, until eventually it will require three attempts
and so on.  Eventually enough attempts are required that my client is
blacklisted before I can successfully connect.

Some methodical testing revealed a way to reproduce the issue.  Start a vnc
server process:

tigervncserver :10 -SecurityTypes None

Then run a loop to start client processes:

while :; do xvncviewer localhost:10 & sleep 1; done

You can do a tail -f on the server log in another terminal if you wish.  Soon
enough you will start getting popups for "Too many security failures".  The
log will look something like this:

Fri Aug 25 01:51:26 2017
 Connections: accepted: 127.0.0.1::34667

Fri Aug 25 01:51:27 2017
 Connections: accepted: 127.0.0.1::34668

Fri Aug 25 01:51:28 2017
 Connections: accepted: 127.0.0.1::34669
 SConnection: Client needs protocol version 3.8
 SConnection: Client requests security type None(1)
 VNCSConnST:  Server default pixel format depth 24 (32bpp) little-endian rgb888
 Connections: closed: 127.0.0.1::34668 (Non-shared connection requested)
 EncodeManager: Framebuffer updates: 0
 EncodeManager:   Total: 0 rects, 0 pixels
 EncodeManager:          0 B (1:-nan ratio)
 Connections: closed: 127.0.0.1::34667 (Non-shared connection requested)
 EncodeManager: Framebuffer updates: 0
 EncodeManager:   Total: 0 rects, 0 pixels
 EncodeManager:          0 B (1:-nan ratio)
 Connections: closed: 127.0.0.1::34666 (Non-shared connection requested)
 EncodeManager: Framebuffer updates: 1
 EncodeManager:   Tight:
 EncodeManager:     Solid: 1 rects, 2.28 Mpixels
 EncodeManager:            16 B (1:570001 ratio)
 EncodeManager:   Total: 1 rects, 2.28 Mpixels
 EncodeManager:          16 B (1:570001 ratio)
 VNCSConnST:  Client pixel format depth 24 (32bpp) little-endian rgb888

Note that no client information was printed for the two stub attempts.  After
a successful connection is made, the two stub connections are terminated, as
well as the previous live connection.

Using shared connections can be used to work around the problem.  The failure
mechanism seems to rely on connection "slots"; after a connection is forcibly
terminated, its slot will become permanently unusable and the next connection
to land in that slot will remain in a limbo indefinitely.  However since a
shared connection does not terminate other connections, the defunct slots
remain filled with stub connections and each new connection gets a viable
slot.

I have not yet delved into the code, but based on this observation my theory
is that there's an array of connections that gets reused, and new connections
are incompletely initialized.  A forced disconnection and subsequent new
connection in the same array index would then land in a bad state where the
server won't process it further.

-- System Information:
Debian Release: buster/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.19.1-core2 (SMP w/8 CPU cores)
Locale: LANG=en_US.utf8, LC_CTYPE=en_US.utf8 (charmap=UTF-8), LANGUAGE=en_US.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: sysvinit (via /sbin/init)

Versions of packages tigervnc-standalone-server depends on:
ii  libaudit1                 1:2.7.7-1+b2
ii  libbsd0                   0.8.6-1
ii  libc6                     2.24-14
ii  libgcc1                   1:7.2.0-1
ii  libgcrypt20               1.7.8-2
ii  libgl1-mesa-glx [libgl1]  13.0.6-1+b2
ii  libgnutls30               3.5.13-2
ii  libjpeg62-turbo           1:1.5.1-2
ii  libpam0g                  1.1.8-3.6
ii  libpixman-1-0             0.34.0-1
ii  libselinux1               2.6-3+b2
ii  libstdc++6                7.2.0-1
ii  libsystemd0               232-25
ii  libx11-6                  2:1.6.4-3
ii  libxau6                   1:1.0.8-1+b2
ii  libxdmcp6                 1:1.1.2-3
ii  libxfont2                 1:2.0.1-3
ii  libxshmfence1             1.2-1+b2
ii  perl                      5.26.0-5
ii  x11-xkb-utils             7.7+3+b1
ii  xauth                     1:1.0.9-1+b2
ii  xkb-data                  2.19-1.1
ii  zlib1g                    1:1.2.8.dfsg-5

Versions of packages tigervnc-standalone-server recommends:
pn  libgl1-mesa-dri    <none>
ii  tigervnc-common    1.7.0+dfsg-7
ii  x11-xserver-utils  7.7+7+b1
ii  xfonts-base        1:1.0.4+nmu1

Versions of packages tigervnc-standalone-server suggests:
ii  xfonts-75dpi     1:1.0.4+nmu1
pn  xfonts-scalable  <none>

-- no debconf information