Bug#897410: libpj2 initializes EECDH incorrectly when linked with OpenSSL 1.1.0

[Ondřej Holas]

Package: libpj2
Version: 2.7.2~dfsg-1

Initialization of ephemeral ECDH (EECDH) when accepting TLS session works
incorrectly when linked with OpenSSL 1.1.0. In OpenSSL 1.1.0 the ephemeral
ECDH is already initialized in automatic mode, so there is really no need to
do anything explicit about it.

=== begin citation ===
*) SSL_{CTX_}set_ecdh_auto() has been removed and ECDH is support is
     always enabled now.  If you want to disable the support you should
     exclude it using the list of supported ciphers. This also means that
the
     "-no_ecdhe" option has been removed from s_server.
https://www.openssl.org/news/changelog.html#x10
=== end citation ===

The code in ssl_sock_ossl.c falls to branch initializing only prime256v1
(aka secp256r1) elliptic curve in the context, after the call
SSL_CTX_ctrl(ctx,94,1,NULL) is unsuccessful with OpenSSL 1.1.0. When using
server certificate with EC key based on any other curve, the listener fails
TLS negotiation with misleading alert "no shared cipher", because the
context's curve set applies to both EECDH and ECDSA. (Certificates with RSA
keys work well.) Also, the EECDH itself is limited to use the only (from
today's perspective the weakest acceptable) curve for key negotiation.

When used EC certificate with secp384r1 curve for its key:

=== begin cert key params ===
# openssl x509 -pubkey -noout < /etc/asterisk/boston-ecc-p384-selfcert.pem |
grep -Ev "^-----" | base64 -d | dumpasn1 -
  0 118: SEQUENCE {
  2  16:   SEQUENCE {
  4   7:     OBJECT IDENTIFIER ecPublicKey (1 2 840 10045 2 1)
 13   5:     OBJECT IDENTIFIER secp384r1 (1 3 132 0 34)
       :     }
 20  98:   BIT STRING
       :     ...
       :   }
=== end cert key params ===

=== begin excerpt from /var/log/asterisk/full ===
[2018-05-01 12:44:16.067] DEBUG[1782] pjproject:         ssl0xb500b490 SSL
ECDH initialized (secp256r1), faster PFS cipher-suites enabled
[2018-05-01 12:44:16.067] WARNING[1782] pjproject:                 SSL
SSL_ERROR_SSL (Handshake): Level: 0 err: <337092801> <SSL
routines-tls_post_process_client_hello-no shared cipher> len: 0
[2018-05-01 12:44:16.068] DEBUG[1782] pjproject:         ssl0xb500b490
Handshake failed in accepting [::1]:43352: no shared cipher
=== end excerpt from /var/log/asterisk/full ===

Proposed patch is to remove explicit EECDH setup when used with OpenSSL
1.1.0+:

=== begin patch ===
--- pjproject-2.7.2~dfsg.orig/pjlib/src/pj/ssl_sock_ossl.c
+++ pjproject-2.7.2~dfsg/pjlib/src/pj/ssl_sock_ossl.c
@@ -987,6 +987,7 @@ static pj_status_t create_ssl(pj_ssl_soc
            pj_memcpy(p, "rsa", CERT_TYPE_LEN);
        }

+    #if OPENSSL_VERSION_NUMBER < 0x10100000L
     #ifndef SSL_CTRL_SET_ECDH_AUTO
        #define SSL_CTRL_SET_ECDH_AUTO 94
     #endif
@@ -1008,6 +1009,10 @@ static pj_status_t create_ssl(pj_ssl_soc
            }
     #endif
        }
+    #else // OPENSSL_VERSION_NUMBER < 0x10100000L
+       PJ_LOG(4,(ssock->pool->obj_name, "SSL ECDH already initialized "
+           "(OpenSSL 1.1.0+), faster PFS cipher-suites enabled"));
+    #endif // OPENSSL_VERSION_NUMBER < 0x10100000L
     } else {
        X509_STORE *pkix_validation_store = SSL_CTX_get_cert_store(ctx);
        if (NULL != pkix_validation_store) {
=== end patch ===

Best regards,

Ondrej Holas