[Pkg-openssl-devel] Bug#747453: Arbitrary key size limitations causing hard-to-diagnose problems when establishing a connection

[Benny Baumann]

Source: openssl
Severity: critical
Tags: security patch

OpenSSL contains a set of arbitrary limitations on the size of accepted key
parameters that make unrelated software fail to establish secure connections.
The problem was found while debugging a XMPP s2s connection issue where two
servers with long certificate keys (8192 Bit RSA) failed to establish a secure
connection because OpenSSL rejected the handshake.

The attached two patches fix the following issues:
1.  Remove the restriction on DSA/DHE parameters to allow for arbitrary size
2.  Increase the maximum allowed size for transmitted (client/server) keys 
    from 516 byte (e.g. 4096 bit RSA) to 8200 byte (e.g. 65536 bit RSA)

The first issue was found with a server using GnuTLS that used DH parameters
with 13337 bits for negotiating the session key. While a website test succeeded
in determining the cipher configuration it failed to negotiate a session key
and did not provide any reasonable error message back to the user. As the issue
depended on the ciphers offered by the client a real client like a webbrowser
would not be able to gracefully fall back to some other algorithm. Thus the
only workaround would be to use no encryption which would be the worst of all
alternatives.

The second issue was found while debugging issues with two ejabberd instances
that both used certificates with 8192 bit RSA. While both servers could
correctly determine the opposite's server's connection parameters (using
provided SRV records) and properly established a cleartext connection they
unexpectedly and without proper diagnosis terminated the SSL connection
after negotiating to upgrade to STARTTLS. After both parties sent their
certificates the connection was suddently terminated without even providing
a SSL fatal error alert - thus no useful information could be provided
by the application layer. Only after increasing the maximum size for key
parameters were both servers able to connect to each other.

This once again demonstrates that you MUST NOT introduce statically compiled-in
magic numbers to place arbitrary limits on the size of used parameters.
Furthermore it should be noted that the parameters used are neither very large,
nor do they require excessive processing power (about 1-2 seconds for one
handshake on average). This might not be an option for everybody but is well
within parameters that are to be expected in casually-paranoid setups.

Please apply both patches ASAP and forward them to be included upstream.

Kind regards,
Benny Baumann

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (900, 'testing'), (800, 'stable'), (750, 'experimental'), (700, 'unstable'), (500, 'oldstable')
Architecture: amd64 (x86_64)

Kernel: Linux 3.13-1-amd64 (SMP w/8 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

-- no debconf informationn
-------------- next part --------------
A non-text attachment was scrubbed...
Name: increase-max-certsize.patch
Type: text/x-diff
Size: 1043 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-openssl-devel/attachments/20140508/292b287b/attachment.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: remove-dsa-dh-keysize-restriction.patch
Type: text/x-diff
Size: 1422 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-openssl-devel/attachments/20140508/292b287b/attachment-0001.patch>