[debian-mysql] Bug#977178: mariadb-plugin-rocksdb: Server crashes in rocksdb on startup on pmull on some ARM CPUs

[Michal Povinsky]

Package: mariadb-plugin-rocksdb
Version: 1:10.3.27-0+deb10u1
Severity: important

Dear Maintainer,

After updating from 10.3.23 to either 10.3.25 (from security) or 10.3.27, the package crashes on startup with SIGILL on a Raspberry Pi 4. Running the package under gdb shows the following backtrace:
#0  0x0000007ff467afe4 in crc32c_arm64(unsigned int, unsigned char const*, unsigned int) () from /usr/lib/aarch64-linux-gnu/mariadb19/plugin/ha_rocksdb.so
#1  0x0000007ff44fd15c in rocksdb::log::Reader::ReadPhysicalRecord(rocksdb::Slice*, unsigned long*) () from /usr/lib/aarch64-linux-gnu/mariadb19/plugin/ha_rocksdb.so
#2  0x0000007ff44fd2f4 in rocksdb::log::Reader::ReadRecord(rocksdb::Slice*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, rocksdb::WALRecoveryMode) () from /usr/lib/aarch64-linux-gnu/mariadb19/plugin/ha_rocksdb.so
#3  0x0000007ff4545cdc in rocksdb::VersionSet::ListColumnFamilies(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::FileSystem*) () from /usr/lib/aarch64-linux-gnu/mariadb19/plugin/ha_rocksdb.so
#4  0x0000007ff4460844 in rocksdb::DB::ListColumnFamilies(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*) () from /usr/lib/aarch64-linux-gnu/mariadb19/plugin/ha_rocksdb.so
#5  0x0000007ff44067bc in ?? () from /usr/lib/aarch64-linux-gnu/mariadb19/plugin/ha_rocksdb.so
#6  0x0000005555c8ddb4 in ha_initialize_handlerton(st_plugin_int*) ()
#7  0x0000005555af89b0 in ?? ()
#8  0x0000005555af976c in plugin_init(int*, char**, int) ()
#9  0x0000005555a492cc in ?? ()
#10 0x0000005555a4fd78 in mysqld_main(int, char**) ()
#11 0x0000007ff79eed24 in __libc_start_main (main=0x5555a30198 <main>, argc=1, argv=0x7ffffff658, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:308
#12 0x0000005555a43510 in _start ()

And disassembly shows that the package crashes on this instruction:
=> 0x0000007ff467afe4 <+1116>:  pmull   v2.1q, v2.1d, v1.1d
This seems to be a similar issue to https://jira.mariadb.org/browse/MDEV-23030, but it affects 10.3, and the crash is inside the Rocksdb plugin.
Downgrading to 10.3.23 fixes the problem. I haven't been able to reproduce the problem on a clean install inside a VM - maybe it depends on having some data in the database?

-- System Information:
Debian Release: 10.7
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: arm64 (aarch64)
Foreign Architectures: armhf

Kernel: Linux 5.4.72-v8+ (SMP w/4 CPU cores; PREEMPT)
Kernel taint flags: TAINT_CRAP
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to en_US.UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to en_US.UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages mariadb-plugin-rocksdb depends on:
ii  libc6                2.28-10
ii  liblz4-1             1.8.3-1
ii  libsnappy1v5         1.1.7-1
ii  libstdc++6           8.3.0-6
ii  libzstd1             1.3.8+dfsg-3
ii  mariadb-server-10.3  1:10.3.27-0+deb10u1
ii  rocksdb-tools        5.17.2-3
ii  zlib1g               1:1.2.11.dfsg-1

Versions of packages mariadb-plugin-rocksdb recommends:
ii  python-mysqldb  1.3.10-2

mariadb-plugin-rocksdb suggests no packages.

-- Configuration Files:
/etc/mysql/mariadb.conf.d/rocksdb.cnf changed:
[mariadb]
plugin-load-add=ha_rocksdb.so
rocksdb_unsafe_for_binlog=1
rocksdb_block_cache_size=128M
rocksdb_block_size=128k
rocksdb_large_prefix=1
rocksdb_db_write_buffer_size=128M
rocksdb-override-cf-options='default={bottommost_compression=kZSTD;bottommost_compression_opts=-14:19:0:0:0:true;target_file_size_base=16m;max_bytes_for_level_base=64m;target_file_size_multiplier=2;}'
rocksdb_max_open_files=-1
rocksdb_max_log_file_size=16M
rocksdb_keep_log_file_num=4
rocksdb_wal_recovery_mode=2
gtid_pos_auto_engines = 'InnoDB,RocksDB'


-- no debconf information