[Pkg-swan-devel] Bug#1141732: libstrongswan-aesni.so SIGILL crash in both charon and swanctl

[Tobias Brunner]

Hi,

>> So maybe it's the feature detection that runs when the plugin is loaded.
>>  But that's some simple push/pop/mov and cpuid assembler code [1].  Does
>> your CPU lack support for cpuid?
> 
> It has cpuid. Wikipedia agrees cpuid exists since 486, cpuid should be safe.
> Debian 12 cpu_feature.c uses only cpuid leaf 0 and leaf 1, that should be safe.
> 
> Result from gdb: Illegal instruction is punpcklqdq in aesni_plugin_create.
>   <+52>: movd %ecs,%xmm0
>   <+56>: movd %edx,%xmm0
>   <+60>: punpckldq %xmm2,%xmm1
>   <+64>: punpckldq %xmm3,%xmm0
> =><+68>: puncpklqdq %xmm1,%xmm0
>   <+72>: movups %#mm0,(%eax)
>   <+75>: add $0x18,%esp
>   <+78>: pop %ebx
>   <+79>: ret

Oh, interesting.  The complete `aesni` plugin is compiled with `-maes
-mpclmul -mssse3`.  That apparently causes the compiler to "optimize"
the code there.  I guess we could split the plugin into a helper library
that uses those flags and compile the plugin constructor/detection code
without it.  I've pushed such a change to the devel branch [1].

Anyway, another question is if it actually makes sense to ship the
plugin in Debian.  The default crypto library is OpenSSL nowadays and
since that will use AES-NI etc. anyway (likely more optimized than this
plugin), there is probably not much use having `aesni` enabled as well.

Regards,
Tobias

[1]
https://github.com/strongswan/strongswan/commit/553ac5678b50729da896fa07ff8bc502a2a8f3da