[debian-mysql] Bug#1058706: mariadb-server-core: mariadbd fails to start on fresh install on armel
k-i-r at web.de
k-i-r at web.de
Wed Dec 20 09:27:23 GMT 2023
Starting point:
- Installed mariadb-server package which doesn't run
- /var/lib/mysql removed manually
- root at gast1:/var/lib# /usr/bin/mariadb-install-db --rpm --cross-bootstrap --user=mysql -- > error, see below
root at gast1:/var/lib# /usr/bin/mariadb-install-db --rpm --cross-bootstrap --user=mysql
231220 9:21:04 [ERROR] mysqld got signal 4 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.11.4-MariaDB-1~deb12u1 source revision:
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468022 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x5599a37984ae]
/usr/sbin/mariadbd(handle_fatal_signal+0x409)[0x5599a3305f49]
/lib/x86_64-linux-gnu/libc.so.6(+0x3bfd0)[0x7fb58645afd0]
/usr/sbin/mariadbd(+0x6f5d79)[0x5599a2fd2d79]
/usr/sbin/mariadbd(+0xe2590a)[0x5599a370290a]
/usr/sbin/mariadbd(+0x6862e2)[0x5599a2f632e2]
/usr/sbin/mariadbd(+0xcd9671)[0x5599a35b6671]
/usr/sbin/mariadbd(_Z24ha_initialize_handlertonP13st_plugin_int+0x7e)[0x5599a330927e]
Printing to addr2line failed
/usr/sbin/mariadbd(+0x80f997)[0x5599a30ec997]
/usr/sbin/mariadbd(_Z11plugin_initPiPPci+0x85a)[0x5599a30ed95a]
/usr/sbin/mariadbd(+0x701037)[0x5599a2fde037]
/usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0x41d)[0x5599a2fe3a5d]
/lib/x86_64-linux-gnu/libc.so.6(+0x271ca)[0x7fb5864461ca]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7fb586446285]
/usr/sbin/mariadbd(_start+0x21)[0x5599a2fd8c81]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 31372 31372 processes
Max open files 32183 32183 files
Max locked memory 1045245952 1045245952 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 31372 31372 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: core
Kernel version: Linux version 6.1.0-15-amd64 (debian-kernel at lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.66-1 (2023-12-09)
Illegal instruction
Installation of system tables failed! Examine the logs in
/var/lib/mysql for more information.
The problem could be conflicting information in an external
my.cnf files. You can ignore these by doing:
shell> /usr/bin/mariadb-install-db --defaults-file=~/.my.cnf
You can also try to start the mariadbd daemon with:
shell> /usr/sbin/mariadbd --skip-grant-tables --general-log &
and use the command line tool /usr/bin/mariadb
to connect to the mysql database and look at the grant tables:
shell> /usr/bin/mariadb -u root mysql
MariaDB> show tables;
Try '/usr/sbin/mariadbd --help' if you have problems with paths. Using
--general-log gives you a log in /var/lib/mysql that may be helpful.
The latest information about mysql_install_db is available at
https://mariadb.com/kb/en/installing-system-tables-mysql_install_db
You can find the latest source at https://downloads.mariadb.org and
the maria-discuss email list at https://launchpad.net/~maria-discuss
Please check all of the above before submitting a bug report
at https://mariadb.org/jira
root at gast1:/var/lib#
root at gast1:/var/lib# journalctl -u mariadb
Dec 20 09:20:36 gast1 systemd[1]: Starting mariadb.service - MariaDB 10.11.4 database server...
Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] Starting MariaDB 10.11.4-MariaDB-1~deb12u1 source revision as process 1310
Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] InnoDB: Using transactional memory
Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] InnoDB: Number of transaction pools: 1
Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] InnoDB: Using liburing
Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] InnoDB: Completed initialization of buffer pool
Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] InnoDB: Header page consists of zero bytes in datafile: ./ibdata1, Space ID:0, Flags: 0
Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [ERROR] InnoDB: Corrupted page [page id: space=0, page number=0] of datafile './ibdata1' could not be found in the doublewrit>
Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] InnoDB: Starting shutdown...
Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [ERROR] Plugin 'InnoDB' init function returned error.
Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] Plugin 'FEEDBACK' is disabled.
Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [ERROR] Unknown/unsupported storage engine: InnoDB
Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [ERROR] Aborting
Dec 20 09:20:37 gast1 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Dec 20 09:20:37 gast1 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Dec 20 09:20:37 gast1 systemd[1]: Failed to start mariadb.service - MariaDB 10.11.4 database server.
Dec 20 09:20:39 gast1 systemd[1]: Starting mariadb.service - MariaDB 10.11.4 database server...
Dec 20 09:20:39 gast1 mariadbd[1426]: 2023-12-20 9:20:39 0 [Note] Starting MariaDB 10.11.4-MariaDB-1~deb12u1 source revision as process 1426
Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [Note] InnoDB: Using transactional memory
Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [Note] InnoDB: Number of transaction pools: 1
Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [Note] InnoDB: Using liburing
Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [Note] InnoDB: Completed initialization of buffer pool
Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [Note] InnoDB: Header page consists of zero bytes in datafile: ./ibdata1, Space ID:0, Flags: 0
Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [ERROR] InnoDB: Corrupted page [page id: space=0, page number=0] of datafile './ibdata1' could not be found in the doublewrit>
Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [Note] InnoDB: Starting shutdown...
Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [ERROR] Plugin 'InnoDB' init function returned error.
Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [Note] Plugin 'FEEDBACK' is disabled.
Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [ERROR] Unknown/unsupported storage engine: InnoDB
Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [ERROR] Aborting
Dec 20 09:20:40 gast1 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Dec 20 09:20:40 gast1 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Dec 20 09:20:40 gast1 systemd[1]: Failed to start mariadb.service - MariaDB 10.11.4 database server.
root at gast1:/var/lib#
------------------------------
Gesendet: Mittwoch, 20. Dezember 2023 um 09:43 Uhr
Von: "Tuukka Pasanen" <tuukka.pasanen at ilmi.fi>
An: k-i-r at web.de
Cc: "Scott Barker" <scott at mostlylinux.ca>, 1058706 at bugs.debian.org, "Otto Kekäläinen" <otto at debian.org>, "Faustin Lammler" <faustin at mariadb.org>
Betreff: Re: Aw: Re: Re: Bug#1058706: mariadb-server-core: mariadbd fails to start on fresh install on armel
Hello,
Thank you for testing. I assume these are different issues. Other happens in low end ARMv5 machine with 512 MB (memory which is kind of low now a days) and other is in normal server/laptop inside XEN machine (although i686? 32-bit still as armel). As I understand your problem is in initializing mysql-dir (which is mandatory) but MYISAM/Aria and InnoDB works ok?
But you can debug this (I ate my dogfood and tested not just write from the memory sorry for that)
shutdown mariadbd (which is not probably not running)
remove /var/lib/mysql which again will destroy your database (Please note)
use to install your database:
/usr/bin/mariadb-install-db --rpm --cross-bootstrap --user=mysql
Does it print some errors? Is there any errors in syslog? Does MariaDB start after this?
Sincerely,
Tuukka
k-i-r at web.de[mailto:k-i-r at web.de] kirjoitti 19.12.2023 klo 16.33:
Removing mariadb-server via apt purge also removes the user mysql.
--> chown mysql:mylsql doesn't work.
Reinstalling mariadb-server doesn't solve the issue.
Gesendet: Dienstag, 19. Dezember 2023 um 13:01 Uhr
Von: "Tuukka Pasanen" <tuukka.pasanen at ilmi.fi>[mailto:tuukka.pasanen at ilmi.fi]
An: k-i-r at web.de[mailto:k-i-r at web.de]
Cc: "Scott Barker" <scott at mostlylinux.ca>[mailto:scott at mostlylinux.ca], 1058706 at bugs.debian.org[mailto:1058706 at bugs.debian.org], "Otto Kekäläinen" <otto at debian.org>[mailto:otto at debian.org], "Faustin Lammler" <faustin at mariadb.org>[mailto:faustin at mariadb.org]
Betreff: Re: Aw: Re: Bug#1058706: mariadb-server-core: mariadbd fails to start on fresh install on armel
Hello,
I've investigate this yesterday and I could reproduce this one:
Dec 17 11:17:55 gast1 mariadbd[1291]: 2023-12-17 11:17:55 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
Dec 17 11:17:55 gast1 mariadbd[1291]: 2023-12-17 11:17:55 0 [ERROR] Unknown/unsupported storage engine: InnoDB
Dec 17 11:17:55 gast1 mariadbd[1291]: 2023-12-17 11:17:55 0 [ERROR] Aborting
Can you do (Please on testing machine not production will destroy your database!!!)
apt purge mariadb-server
# check is your database is not available /var/lib/mysql if it is remote it
mkdir -p /var/lib/mysql
chown mysql:mysql /var/lib/mysql
and then re-install mariad-server again. Does it work then? Just trying to see if my findings are correct. What comes to this InnoDB corruption I can't reproduce that.
Sincerely,
Tuukka
k-i-r at web.de[mailto:k-i-r at web.de] kirjoitti 19.12.2023 klo 12.18:
Hi Tuukka,
I don't think the issue is related to the armel hardware.
I see the same error on my XEN-VM (1 VCPU) based on a Intel(R) Celeron(R) CPU G3900 @ 2.80GHz processor with 2 cores.
No lack of memory could be seen during installation of mariadb-server package. I tested it with 2 or 8 GB of assigned RAM.
Regards,
Konstantin
Gesendet: Montag, 18. Dezember 2023 um 08:23 Uhr
Von: "Tuukka Pasanen" <tuukka.pasanen at ilmi.fi>[mailto:tuukka.pasanen at ilmi.fi]
An: "Scott Barker" <scott at mostlylinux.ca>[mailto:scott at mostlylinux.ca], 1058706 at bugs.debian.org[mailto:1058706 at bugs.debian.org], "Otto Kekäläinen" <otto at debian.org>[mailto:otto at debian.org], k-i-r at web.de[mailto:k-i-r at web.de]
Cc: "Faustin Lammler" <faustin at mariadb.org>[mailto:faustin at mariadb.org]
Betreff: Re: Bug#1058706: mariadb-server-core: mariadbd fails to start on fresh install on armel
Hello,
Have you monitored that is memory exhausted at the installation? This
could lead error like that.
I assume your XEN/KVM/Docker base machine is some older system with not
so fancy CPU or is there any particular reason to use armel build?
Sincerly,
Tuukka
Scott Barker kirjoitti 17.12.2023 klo 19.15:
Same result with the latest kernel:
Linux nas-1 6.1.0-16-marvell #1 Debian 6.1.67-1 (2023-12-12) armv5tel
GNU/Linux
On Sun, Dec 17, 2023 at 09:59:23AM -0700, Scott Barker wrote:
Prior to the installation, /var/lib/mysql did not exist. After
installation:
61559 4 drwxr-xr-x 2 mysql mysql 4096 Dec 17 09:56
/var/lib/mysql
61975 4 -rw-rw---- 1 mysql mysql 52 Dec 17 09:56
/var/lib/mysql/aria_log_control
61983 12304 -rw-rw---- 1 mysql mysql 12582912 Dec 17 09:56
/var/lib/mysql/ibdata1
61963 0 -rw-r--r-- 1 root root 0 Dec 17 09:56
/var/lib/mysql/debian-10.11.flag
61990 98404 -rw-rw---- 1 mysql mysql 100663296 Dec 17 09:56
/var/lib/mysql/ib_logfile101
61982 16 -rw-rw---- 1 mysql mysql 16384 Dec 17 09:56
/var/lib/mysql/aria_log.00000001
uname -a reports:
Linux nas-1 5.10.0-26-marvell #1 Debian 5.10.197-1 (2023-09-29)
armv5tel GNU/Linux
On Sun, Dec 17, 2023 at 11:48:16PM +0800, Otto Kekäläinen wrote:
Hi Scott and Kr!
Did you note this line?
2023-12-14 14:51:33 0 [Note] InnoDB: Header page consists of zero
bytes in
datafile: ./ibdata1, Space ID:0, Flags: 0
Can you include the output of `find /var/lib/mysql -ls` so we can
see what
files your system has?
What is the hardware you have? The report shows 'Linux
5.10.0-26-marvell '.
What does `uname -a` yield?
The test suite passes on armhf at
e.g. https://launchpad.net/~mysql-ubuntu/[https://launchpad.net/~mysql-ubuntu/]
+archive/ubuntu/mariadb-10.11/+builds?build_text=&build_state=all.
At https://
ci.debian.net/packages/m/mariadb/unstable/armel/ it fails due to
regression in
subselect which is unrelated to what you reported.
--
Scott Barker
--
To unsubscribe, send mail to 1058706-unsubscribe at bugs.debian.org[mailto:1058706-unsubscribe at bugs.debian.org].
More information about the pkg-mysql-maint
mailing list