[Pkg-openssl-devel] Bug#335703: libssl0.9.8: OpenSSL fails to handle fragmented handshake messages using DTLS

[Mikael Magnusson]

Package: libssl0.9.8
Version: 0.9.8a-2
Severity: important

I'm trying to use OpenSSL DTLS in a program and have some problems with
the handshake which seems to be caused by OpenSSL not handle fragmented
handshake messages (certificate) correctly. As seen in the following example
s_client fails to connect to s_server using the DTLS protocol if the MTU
it set to 1500 (default for Ethernet). The same commands succeeds when using a
large MTU, for example 65000.

$ openssl s_server -accept 5069 -dtls1 -cert /etc/apache/ssl.crt/snakeoil-dsa.crt -key /etc/apache/ssl.key/snakeoil-dsa.key -CAfile /etc/apache/ssl.crt/snakeoil-ca-dsa.crt -mtu 1500
Using default temp DH parameters
Using default temp ECDH parameters
ACCEPT
ERROR
3407:error:143F8412:SSL routines:DTLS1_READ_BYTES:sslv3 alert bad certificate:d1_pkt.c:943:SSL alert number 42
shutting down SSL
CONNECTION CLOSED
ACCEPT


$ openssl s_client -host localhost -port 5069 -dtls1
CONNECTED(00000003)
3409:error:0D07209B:asn1 encoding routines:ASN1_get_object:too long:asn1_lib.c:142:
3409:error:0D068066:asn1 encoding routines:ASN1_CHECK_TLEN:bad object header:tasn_dec.c:1269:
3409:error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:653:
3409:error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:704:
3409:error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:743:Field=subject, Type=X509_CINF
3409:error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:743:Field=cert_info, Type=X509
3409:error:1409000D:SSL routines:SSL3_GET_SERVER_CERTIFICATE:ASN1 lib:s3_clnt.c:866:

/Mikael

-- System Information:
Debian Release: testing/unstable
  APT prefers stable
  APT policy: (871, 'stable'), (50, 'testing'), (30, 'unstable'), (10, 'unstable'), (1, 'experimental')
Architecture: i386 (i686)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.11-vserver-k7
Locale: LANG=sv_SE.UTF-8, LC_CTYPE=sv_SE.UTF-8 (charmap=UTF-8)

Versions of packages libssl0.9.8 depends on:
ii  debconf [debconf-2.0]         1.4.57     Debian configuration management sy
ii  libc6                         2.3.5-6    GNU C Library: Shared libraries an

libssl0.9.8 recommends no packages.

-- debconf information:
  libssl0.9.8/restart-services: