[Pkg-openssl-devel] Bug#514694: openssl: "sslv3 alert unexpected message" error while connecting via TLS to Java ssl sockets

[Max Kirillov]

Package: openssl
Version: 0.9.8g-14
Severity: normal


$openssl s_client -tls1 -connect localhost:12345
CONNECTED(00000003)
26552:error:140943F2:SSL routines:SSL3_READ_BYTES:sslv3 alert unexpected message:s3_pkt.c:1053:SSL alert number 10
26552:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:530:

It happen only with openssl from Debian Lenny (i.e, openssl from
Debian Etch and gnutls-cli do not fail). So I suppose the
error is somewhere in ssl.

probably it's related to #471681, #471896

If turn on debug on server (option -Djavax.net.debug=ssl) it
reports this error:

main, READ: TLSv1 Handshake, length = 88
main, handling exception: javax.net.ssl.SSLException: Unexpected end of handshake data
main, SEND TLSv1 ALERT:  fatal, description = unexpected_message
main, WRITE: TLSv1 Alert, length = 2
main, called closeSocket()
javax.net.ssl.SSLException: Unexpected end of handshake data
...

The server setup (mostly copypasted from tutorial):

$sudo aptitude install openjdk-6-jdk

-------- Server.java ------------
import java.io.OutputStream;
import java.io.InputStream;

import javax.net.ssl.*;

public class Server {
    public static void main(String[] args) {
        int port = 12345;

        SSLServerSocket s;

        try {
            SSLServerSocketFactory sslSrvFact =
                (SSLServerSocketFactory)
                SSLServerSocketFactory.getDefault();
            s =(SSLServerSocket)sslSrvFact.createServerSocket(port);

            SSLSocket c = (SSLSocket)s.accept();

            OutputStream out = c.getOutputStream();
            InputStream in = c.getInputStream();
            byte[] buf = new byte[20];
            int read_size;
            while ((read_size = in.read(buf)) > 0) {
                out.write(buf, 0, read_size);
            }
        } catch (Exception e) {
            e.printStackTrace(System.err);
        }
    }
}
--------

$javac Server.java
$keytool -genkeypair -keyalg RSA  -validity 7 -keystore keystore
(here answer questions)
$java -Djavax.net.ssl.keyStore=keystore -Djavax.net.ssl.keyStorePassword=_PASSWORD_ Server

The last command will handle 1 connection and exit.

-- System Information:
Debian Release: lenny/sid
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.26-1-amd64 (SMP w/2 CPU cores)
Locale: LANG=C, LC_CTYPE=ru_RU.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages openssl depends on:
ii  libc6                  2.7-16            GNU C Library: Shared libraries
ii  libssl0.9.8            0.9.8g-15         SSL shared libraries
ii  zlib1g                 1:1.2.3.3.dfsg-12 compression library - runtime

openssl recommends no packages.

Versions of packages openssl suggests:
ii  ca-certificates               20080809   Common CA certificates

-- no debconf information