[Pkg-openssl-devel] OpenSSL 3.0 performance and Debian 12

[Geoffrey Thomas]

Hi OpenSSL packagers!

OpenSSL 3.0 seems to have orders of magnitude slower performance in common operations for multithreaded applications (e.g., creating a new SSL_CTX, which is how most high-level clients create/accept connections) compared to OpenSSL 1.1. I noticed this when rolling out (self-packaged) OpenSSL 3.0 at my day job [1], and this seems to be a relatively well-known issue for other OpenSSL 3.0 users, including for users of Ubuntu 22.04+ which has the openssl 3 package branched from experimental (e.g., [2]). There's a tracking bug in the OpenSSL bug tracker [3] but there are many more reports not linked there. Wikipedia also has an entire paragraph about it [4] claiming "slowdowns from 80 to 400 times." In practice, many users at our day job are seeing connection timeouts as a result of the slowdowns, so it feels to them like OpenSSL is just not working.

As I understand it, the high-level issue is that the new APIs for loading providers and ENGINEs is very locking-heavy and ends up taking write locks (!) in common paths. The provider architecture is new in 3.x and there isn't a simple obvious thing to revert to get back to 1.1 performance. There are conversations on the bug tracker about changing how locking works (taking read locks, using some sort of RCU instead of rwlocks, allowing apps to freeze the loaded providers/ENGINEs for the remainder of the process, etc.) but there don't seem to be immediate simple fixes.

I suspect this is going to be a pretty big issue for Debian 12 users. At this point in the release cycle, though, I'm not sure whether anything can be done. OpenSSL 3.1 is out and is API/ABI-compatible with 3.0, but it feels like it's way too much for this close to release. Also, performance isn't quite up to 1.1 levels - for one of my test cases, 3.0 is about 25x slower than 1.1 and 3.1 is "only" about 4x slower than 3.1.. I assume reverting the transition is also just not a realistic option, but even if it were, OpenSSL 1.1 will only be supported until September 11 of this year.

Are there any good options for what to do about this? Would 3.1 / 3.x be appropriate for a future point release of bookworm? Is there willingness to backport a large number of performance patches to bookworm's 3.0? Maybe one option is to package up 3.1 in bookworm-backports as soon as it's open and mention this issue in the release notes?

I'd be happy to help if I can, e.g., by forwarding links to relevant patches or opening MRs. (I am a DM but not a DD so I'd need sponsorship for things like maintaining openssl in backports.)

[1] https://github.com/openssl/openssl/issues/20698
[2] https://discourse.haproxy.org/t/openssl-3-x-speed-issues/8494
[3] https://github.com/openssl/openssl/issues/17627
[4] https://en.wikipedia.org/wiki/OpenSSL#Significant_performance_regressions

-- 
Geoffrey Thomas
geofft at ldpreload.com