Bug#856436: xrdp: client is not connecting when security_layer=tls

[Jacco Kwaaitaal]

Package: xrdp
Version: 0.9.1-7
Severity: normal

Dear Maintainer,

If in xrdp.ini the option security_layer=tls is configured, a client is not
able to connect. 

E.g. on the client-side using rdesktop the following error is displayed:
140464326739656:error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number:s3_pkt.c:348:
Failed to connect, SSL required by server.

The xrdp.log shows:
[20170301-01:31:33] [INFO ] A connection received from: -X- port 53758
[20170301-01:31:33] [DEBUG] Closed socket 12 (AF_INET6 -X- port 3389)
[20170301-01:31:33] [DEBUG] Closed socket 11 (AF_INET6 -X- port 3389)
[20170301-01:31:33] [INFO ] Using default X.509 certificate: /etc/xrdp/cert.pem
[20170301-01:31:33] [INFO ] Using default X.509 key file: /etc/xrdp/key.pem
[20170301-01:31:33] [DEBUG] Security layer: requested 1, selected 1
[20170301-01:31:33] [DEBUG] Closed socket 12 (AF_INET6 -X- port 3389)
[20170301-01:31:33] [ERROR] Listening socket is in wrong state, terminating listener

I have tried the option disableSSLv3=true, but that doesn't make any difference.
Other clients (remmina, xfreerdp, windows remote desktop client) won't work either.
The cert/key-files have umask 600 owned by root.
I have tried to explicitly choose non-SSLv3 ciphers with the option
tls_ciphers=HIGH:-SSLv3, but that didn't work.

It should be possible to reproduce this with a standard Stretch installation.

Best regards,
Jacco

-- System Information:
Debian Release: 9.0
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'testing'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.9.0-1-amd64 (SMP w/6 CPU cores)
Locale: LANG=nl_NL.UTF-8, LC_CTYPE=nl_NL.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages xrdp depends on:
ii  adduser              3.115
ii  init-system-helpers  1.47
ii  libc6                2.24-9
ii  libfuse2             2.9.7-1
ii  libjpeg62-turbo      1:1.5.1-2
ii  libopus0             1.2~alpha2-1
ii  libpam0g             1.1.8-3.5
ii  libssl1.1            1.1.0e-1
ii  libx11-6             2:1.6.4-3
ii  libxfixes3           1:5.0.3-1
ii  libxrandr2           2:1.5.1-1
ii  lsb-base             9.20161125
ii  ssl-cert             1.0.38

Versions of packages xrdp recommends:
ii  fuse      2.9.7-1
ii  xorgxrdp  0.9.1-7

Versions of packages xrdp suggests:
pn  guacamole  <none>

Versions of packages xorgxrdp depends on:
ii  libc6                                  2.24-9
pn  xorg-input-abi-24                      <none>
ii  xserver-xorg-core [xorg-video-abi-23]  2:1.19.1-4

Versions of packages xorgxrdp recommends:
ii  xorg  1:7.7+18

Versions of packages xrdp is related to:
pn  vnc-server           <none>
pn  xserver-xorg-legacy  <none>

-- no debconf information