[debian-mysql] Bug#1058706: mariadb-server-core: mariadbd fails to start on fresh install on armel
Tuukka Pasanen
tuukka.pasanen at ilmi.fi
Thu Dec 21 07:36:02 GMT 2023
Hello,
Please see upstream as this seems to be very similar to your problem:
https://jira.mariadb.org/browse/MDEV-27900
Sincerly,
Tuukka
k-i-r at web.de kirjoitti 20.12.2023 klo 11.27:
> Starting point:
> - Installed mariadb-server package which doesn't run
> - /var/lib/mysql removed manually
> - root at gast1:/var/lib# /usr/bin/mariadb-install-db --rpm --cross-bootstrap --user=mysql -- > error, see below
>
> root at gast1:/var/lib# /usr/bin/mariadb-install-db --rpm --cross-bootstrap --user=mysql
> 231220 9:21:04 [ERROR] mysqld got signal 4 ;
> This could be because you hit a bug. It is also possible that this binary
> or one of the libraries it was linked against is corrupt, improperly built,
> or misconfigured. This error can also be caused by malfunctioning hardware.
>
> To report this bug, see https://mariadb.com/kb/en/reporting-bugs
>
> We will try our best to scrape up some info that will hopefully help
> diagnose the problem, but since we have already crashed,
> something is definitely wrong and this may fail.
>
> Server version: 10.11.4-MariaDB-1~deb12u1 source revision:
> key_buffer_size=134217728
> read_buffer_size=131072
> max_used_connections=0
> max_threads=153
> thread_count=0
> It is possible that mysqld could use up to
> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468022 K bytes of memory
> Hope that's ok; if not, decrease some variables in the equation.
>
> Thread pointer: 0x0
> Attempting backtrace. You can use the following information to find out
> where mysqld died. If you see no messages after this, something went
> terribly wrong...
> stack_bottom = 0x0 thread_stack 0x49000
> /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x5599a37984ae]
> /usr/sbin/mariadbd(handle_fatal_signal+0x409)[0x5599a3305f49]
> /lib/x86_64-linux-gnu/libc.so.6(+0x3bfd0)[0x7fb58645afd0]
> /usr/sbin/mariadbd(+0x6f5d79)[0x5599a2fd2d79]
> /usr/sbin/mariadbd(+0xe2590a)[0x5599a370290a]
> /usr/sbin/mariadbd(+0x6862e2)[0x5599a2f632e2]
> /usr/sbin/mariadbd(+0xcd9671)[0x5599a35b6671]
> /usr/sbin/mariadbd(_Z24ha_initialize_handlertonP13st_plugin_int+0x7e)[0x5599a330927e]
> Printing to addr2line failed
> /usr/sbin/mariadbd(+0x80f997)[0x5599a30ec997]
> /usr/sbin/mariadbd(_Z11plugin_initPiPPci+0x85a)[0x5599a30ed95a]
> /usr/sbin/mariadbd(+0x701037)[0x5599a2fde037]
> /usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0x41d)[0x5599a2fe3a5d]
> /lib/x86_64-linux-gnu/libc.so.6(+0x271ca)[0x7fb5864461ca]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7fb586446285]
> /usr/sbin/mariadbd(_start+0x21)[0x5599a2fd8c81]
> The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
> information that should help you find out what is causing the crash.
> Writing a core file...
> Working directory at /var/lib/mysql
> Resource Limits:
> Limit Soft Limit Hard Limit Units
> Max cpu time unlimited unlimited seconds
> Max file size unlimited unlimited bytes
> Max data size unlimited unlimited bytes
> Max stack size 8388608 unlimited bytes
> Max core file size 0 unlimited bytes
> Max resident set unlimited unlimited bytes
> Max processes 31372 31372 processes
> Max open files 32183 32183 files
> Max locked memory 1045245952 1045245952 bytes
> Max address space unlimited unlimited bytes
> Max file locks unlimited unlimited locks
> Max pending signals 31372 31372 signals
> Max msgqueue size 819200 819200 bytes
> Max nice priority 0 0
> Max realtime priority 0 0
> Max realtime timeout unlimited unlimited us
> Core pattern: core
>
> Kernel version: Linux version 6.1.0-15-amd64 (debian-kernel at lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.66-1 (2023-12-09)
>
> Illegal instruction
>
> Installation of system tables failed! Examine the logs in
> /var/lib/mysql for more information.
>
> The problem could be conflicting information in an external
> my.cnf files. You can ignore these by doing:
>
> shell> /usr/bin/mariadb-install-db --defaults-file=~/.my.cnf
>
> You can also try to start the mariadbd daemon with:
>
> shell> /usr/sbin/mariadbd --skip-grant-tables --general-log &
>
> and use the command line tool /usr/bin/mariadb
> to connect to the mysql database and look at the grant tables:
>
> shell> /usr/bin/mariadb -u root mysql
> MariaDB> show tables;
>
> Try '/usr/sbin/mariadbd --help' if you have problems with paths. Using
> --general-log gives you a log in /var/lib/mysql that may be helpful.
>
> The latest information about mysql_install_db is available at
> https://mariadb.com/kb/en/installing-system-tables-mysql_install_db
> You can find the latest source at https://downloads.mariadb.org and
> the maria-discuss email list at https://launchpad.net/~maria-discuss
>
> Please check all of the above before submitting a bug report
> at https://mariadb.org/jira
>
> root at gast1:/var/lib#
>
> root at gast1:/var/lib# journalctl -u mariadb
> Dec 20 09:20:36 gast1 systemd[1]: Starting mariadb.service - MariaDB 10.11.4 database server...
> Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] Starting MariaDB 10.11.4-MariaDB-1~deb12u1 source revision as process 1310
> Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
> Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] InnoDB: Using transactional memory
> Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] InnoDB: Number of transaction pools: 1
> Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
> Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] InnoDB: Using liburing
> Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
> Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] InnoDB: Completed initialization of buffer pool
> Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] InnoDB: Header page consists of zero bytes in datafile: ./ibdata1, Space ID:0, Flags: 0
> Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [ERROR] InnoDB: Corrupted page [page id: space=0, page number=0] of datafile './ibdata1' could not be found in the doublewrit>
> Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
> Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] InnoDB: Starting shutdown...
> Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [ERROR] Plugin 'InnoDB' init function returned error.
> Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
> Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [Note] Plugin 'FEEDBACK' is disabled.
> Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
> Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [ERROR] Unknown/unsupported storage engine: InnoDB
> Dec 20 09:20:37 gast1 mariadbd[1310]: 2023-12-20 9:20:37 0 [ERROR] Aborting
> Dec 20 09:20:37 gast1 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
> Dec 20 09:20:37 gast1 systemd[1]: mariadb.service: Failed with result 'exit-code'.
> Dec 20 09:20:37 gast1 systemd[1]: Failed to start mariadb.service - MariaDB 10.11.4 database server.
> Dec 20 09:20:39 gast1 systemd[1]: Starting mariadb.service - MariaDB 10.11.4 database server...
> Dec 20 09:20:39 gast1 mariadbd[1426]: 2023-12-20 9:20:39 0 [Note] Starting MariaDB 10.11.4-MariaDB-1~deb12u1 source revision as process 1426
> Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
> Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [Note] InnoDB: Using transactional memory
> Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [Note] InnoDB: Number of transaction pools: 1
> Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
> Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [Note] InnoDB: Using liburing
> Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
> Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [Note] InnoDB: Completed initialization of buffer pool
> Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [Note] InnoDB: Header page consists of zero bytes in datafile: ./ibdata1, Space ID:0, Flags: 0
> Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [ERROR] InnoDB: Corrupted page [page id: space=0, page number=0] of datafile './ibdata1' could not be found in the doublewrit>
> Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
> Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [Note] InnoDB: Starting shutdown...
> Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [ERROR] Plugin 'InnoDB' init function returned error.
> Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
> Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [Note] Plugin 'FEEDBACK' is disabled.
> Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
> Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [ERROR] Unknown/unsupported storage engine: InnoDB
> Dec 20 09:20:40 gast1 mariadbd[1426]: 2023-12-20 9:20:40 0 [ERROR] Aborting
> Dec 20 09:20:40 gast1 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
> Dec 20 09:20:40 gast1 systemd[1]: mariadb.service: Failed with result 'exit-code'.
> Dec 20 09:20:40 gast1 systemd[1]: Failed to start mariadb.service - MariaDB 10.11.4 database server.
> root at gast1:/var/lib#
>
> ------------------------------
>
>
>
>
>
>
>
>
>
> Gesendet: Mittwoch, 20. Dezember 2023 um 09:43 Uhr
> Von: "Tuukka Pasanen" <tuukka.pasanen at ilmi.fi>
> An: k-i-r at web.de
> Cc: "Scott Barker" <scott at mostlylinux.ca>, 1058706 at bugs.debian.org, "Otto Kekäläinen" <otto at debian.org>, "Faustin Lammler" <faustin at mariadb.org>
> Betreff: Re: Aw: Re: Re: Bug#1058706: mariadb-server-core: mariadbd fails to start on fresh install on armel
> Hello,
> Thank you for testing. I assume these are different issues. Other happens in low end ARMv5 machine with 512 MB (memory which is kind of low now a days) and other is in normal server/laptop inside XEN machine (although i686? 32-bit still as armel). As I understand your problem is in initializing mysql-dir (which is mandatory) but MYISAM/Aria and InnoDB works ok?
>
> But you can debug this (I ate my dogfood and tested not just write from the memory sorry for that)
> shutdown mariadbd (which is not probably not running)
> remove /var/lib/mysql which again will destroy your database (Please note)
> use to install your database:
>
> /usr/bin/mariadb-install-db --rpm --cross-bootstrap --user=mysql
>
> Does it print some errors? Is there any errors in syslog? Does MariaDB start after this?
> Sincerely,
> Tuukka
>
>
> k-i-r at web.de[mailto:k-i-r at web.de] kirjoitti 19.12.2023 klo 16.33:
>
> Removing mariadb-server via apt purge also removes the user mysql.
> --> chown mysql:mylsql doesn't work.
> Reinstalling mariadb-server doesn't solve the issue.
>
>
>
> Gesendet: Dienstag, 19. Dezember 2023 um 13:01 Uhr
>
> Von: "Tuukka Pasanen" <tuukka.pasanen at ilmi.fi>[mailto:tuukka.pasanen at ilmi.fi]
> An: k-i-r at web.de[mailto:k-i-r at web.de]
> Cc: "Scott Barker" <scott at mostlylinux.ca>[mailto:scott at mostlylinux.ca], 1058706 at bugs.debian.org[mailto:1058706 at bugs.debian.org], "Otto Kekäläinen" <otto at debian.org>[mailto:otto at debian.org], "Faustin Lammler" <faustin at mariadb.org>[mailto:faustin at mariadb.org]
> Betreff: Re: Aw: Re: Bug#1058706: mariadb-server-core: mariadbd fails to start on fresh install on armel
> Hello,
> I've investigate this yesterday and I could reproduce this one:
> Dec 17 11:17:55 gast1 mariadbd[1291]: 2023-12-17 11:17:55 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
> Dec 17 11:17:55 gast1 mariadbd[1291]: 2023-12-17 11:17:55 0 [ERROR] Unknown/unsupported storage engine: InnoDB
> Dec 17 11:17:55 gast1 mariadbd[1291]: 2023-12-17 11:17:55 0 [ERROR] Aborting
>
> Can you do (Please on testing machine not production will destroy your database!!!)
> apt purge mariadb-server
> # check is your database is not available /var/lib/mysql if it is remote it
> mkdir -p /var/lib/mysql
> chown mysql:mysql /var/lib/mysql
> and then re-install mariad-server again. Does it work then? Just trying to see if my findings are correct. What comes to this InnoDB corruption I can't reproduce that.
>
> Sincerely,
> Tuukka
>
>
> k-i-r at web.de[mailto:k-i-r at web.de] kirjoitti 19.12.2023 klo 12.18:
> Hi Tuukka,
> I don't think the issue is related to the armel hardware.
> I see the same error on my XEN-VM (1 VCPU) based on a Intel(R) Celeron(R) CPU G3900 @ 2.80GHz processor with 2 cores.
> No lack of memory could be seen during installation of mariadb-server package. I tested it with 2 or 8 GB of assigned RAM.
>
> Regards,
> Konstantin
>
>
> Gesendet: Montag, 18. Dezember 2023 um 08:23 Uhr
> Von: "Tuukka Pasanen" <tuukka.pasanen at ilmi.fi>[mailto:tuukka.pasanen at ilmi.fi]
> An: "Scott Barker" <scott at mostlylinux.ca>[mailto:scott at mostlylinux.ca], 1058706 at bugs.debian.org[mailto:1058706 at bugs.debian.org], "Otto Kekäläinen" <otto at debian.org>[mailto:otto at debian.org], k-i-r at web.de[mailto:k-i-r at web.de]
> Cc: "Faustin Lammler" <faustin at mariadb.org>[mailto:faustin at mariadb.org]
> Betreff: Re: Bug#1058706: mariadb-server-core: mariadbd fails to start on fresh install on armel
> Hello,
>
> Have you monitored that is memory exhausted at the installation? This
> could lead error like that.
>
> I assume your XEN/KVM/Docker base machine is some older system with not
> so fancy CPU or is there any particular reason to use armel build?
>
> Sincerly,
>
> Tuukka
>
> Scott Barker kirjoitti 17.12.2023 klo 19.15:
> Same result with the latest kernel:
>
> Linux nas-1 6.1.0-16-marvell #1 Debian 6.1.67-1 (2023-12-12) armv5tel
> GNU/Linux
>
> On Sun, Dec 17, 2023 at 09:59:23AM -0700, Scott Barker wrote:
> Prior to the installation, /var/lib/mysql did not exist. After
> installation:
>
> 61559 4 drwxr-xr-x 2 mysql mysql 4096 Dec 17 09:56
> /var/lib/mysql
> 61975 4 -rw-rw---- 1 mysql mysql 52 Dec 17 09:56
> /var/lib/mysql/aria_log_control
> 61983 12304 -rw-rw---- 1 mysql mysql 12582912 Dec 17 09:56
> /var/lib/mysql/ibdata1
> 61963 0 -rw-r--r-- 1 root root 0 Dec 17 09:56
> /var/lib/mysql/debian-10.11.flag
> 61990 98404 -rw-rw---- 1 mysql mysql 100663296 Dec 17 09:56
> /var/lib/mysql/ib_logfile101
> 61982 16 -rw-rw---- 1 mysql mysql 16384 Dec 17 09:56
> /var/lib/mysql/aria_log.00000001
>
> uname -a reports:
>
> Linux nas-1 5.10.0-26-marvell #1 Debian 5.10.197-1 (2023-09-29)
> armv5tel GNU/Linux
>
> On Sun, Dec 17, 2023 at 11:48:16PM +0800, Otto Kekäläinen wrote:
> Hi Scott and Kr!
>
> Did you note this line?
>
> 2023-12-14 14:51:33 0 [Note] InnoDB: Header page consists of zero
> bytes in
> datafile: ./ibdata1, Space ID:0, Flags: 0
>
> Can you include the output of `find /var/lib/mysql -ls` so we can
> see what
> files your system has?
>
> What is the hardware you have? The report shows 'Linux
> 5.10.0-26-marvell '.
> What does `uname -a` yield?
>
> The test suite passes on armhf at
> e.g. https://launchpad.net/~mysql-ubuntu/[https://launchpad.net/~mysql-ubuntu/]
> +archive/ubuntu/mariadb-10.11/+builds?build_text=&build_state=all.
> At https://
> ci.debian.net/packages/m/mariadb/unstable/armel/ it fails due to
> regression in
> subselect which is unrelated to what you reported.
>
> --
> Scott Barker
>
> --
> To unsubscribe, send mail to 1058706-unsubscribe at bugs.debian.org[mailto:1058706-unsubscribe at bugs.debian.org].
>
>
>
>
>
>
More information about the pkg-mysql-maint
mailing list