[debian-mysql] Bug#798080: mysql-server-5.6: service stop hangs forever on systemd

[Etienne Dechamps]

Package: mysql-server-5.6
Version: 5.6.25-3
Severity: important

"systemctl stop mysql" hangs forever after I upgraded from MySQL 5.5 to
5.6. The systemd journal shows the following:

    Sep 05 10:53:45 zyklos.edechamps.fr systemd[1]: Stopping MySQL Community Server...
    Sep 05 10:53:45 zyklos.edechamps.fr mysqld_safe[14465]: Could not open required defaults file: /etc/mysql/debian.cnf
    Sep 05 10:53:45 zyklos.edechamps.fr mysqld_safe[14465]: Fatal error in defaults handling. Program aborted
    Sep 05 10:53:45 zyklos.edechamps.fr mysqld_safe[14465]: 150905 10:53:45 mysqld_safe Number of processes running now: 1
    Sep 05 10:53:45 zyklos.edechamps.fr mysqld_safe[14465]: 150905 10:53:45 mysqld_safe mysqld process hanging, pid 14781 - killed
    Sep 05 10:53:45 zyklos.edechamps.fr mysqld_safe[14465]: 150905 10:53:45 mysqld_safe mysqld restarted

Digging further, I took a look at the offending file:

    $ ls -l /etc/mysql/debian.cnf
    -rw------- 1 root root 333 Oct 26  2011 /etc/mysql/debian.cnf

Then I tried to fix its permissions so that mysqld_safe (which is
running as the mysql user) can access it:

    # chown root:mysql /etc/mysql/debian.cnf
    # chmod 640 /etc/mysql/debian.cnf

Unfortunately, that seems to only have moved the problem elsewhere:

    Sep 05 10:54:50 zyklos.edechamps.fr systemd[1]: Stopping MySQL Community Server...
    Sep 05 10:54:50 zyklos.edechamps.fr mysqld_safe[15017]: ^G/usr/bin/mysqladmin: connect to server at 'localhost' failed
    Sep 05 10:54:50 zyklos.edechamps.fr mysqld_safe[15017]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
    Sep 05 10:54:50 zyklos.edechamps.fr mysqld_safe[15017]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
    Sep 05 10:54:50 zyklos.edechamps.fr mysqld_safe[15017]: 150905 10:54:50 mysqld_safe Number of processes running now: 1
    Sep 05 10:54:50 zyklos.edechamps.fr mysqld_safe[15017]: 150905 10:54:50 mysqld_safe mysqld process hanging, pid 15334 - killed
    Sep 05 10:54:50 zyklos.edechamps.fr mysqld_safe[15017]: 150905 10:54:50 mysqld_safe mysqld restarted

Error 2 is ENOENT (No such file or directory). I find this puzzling,
because the file *does* exist (even after I started the stop operation):

    $ ls -l /var/run/mysqld/mysqld.sock
    srwxrwxrwx 1 mysql mysql 0 Sep  5 10:54 /var/run/mysqld/mysqld.sock

Even more puzzling, the following works, even if I run it as the mysql
user:

    # /usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf shutdown

For now, I have come up with the following workaround:

    # chown root:mysql /etc/mysql/debian.cnf
    # chmod 640 /etc/mysql/debian.cnf
    # cat > /etc/systemd/system/mysql.service.d/workaround-shutdown-bug.conf <<EOF
    [Service]
    ExecStop=/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf shutdown
    EOF
    # systemctl daemon-reload

My /etc/mysql/debian.cnf contains:

    # Automatically generated for Debian scripts. DO NOT TOUCH!
    [client]
    host     = localhost
    user     = debian-sys-maint
    password = <snip>
    socket   = /var/run/mysqld/mysqld.sock
    [mysql_upgrade]
    host     = localhost
    user     = debian-sys-maint
    password = <snip>
    socket   = /var/run/mysqld/mysqld.sock
    basedir  = /usr

-- System Information:
Debian Release: stretch/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.1.0-2-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages mysql-server-5.6 depends on:
ii  adduser                3.113+nmu3
ii  debconf [debconf-2.0]  1.5.57
ii  init-system-helpers    1.23
ii  initscripts            2.88dsf-59.2
ii  libaio1                0.3.110-1
ii  libc6                  2.19-19
ii  libdbi-perl            1.633-1
ii  libgcc1                1:5.2.1-16
ii  libstdc++6             5.2.1-16
ii  libwrap0               7.6.q-25
ii  lsb-base               4.1+Debian14
ii  mysql-client-5.6       5.6.25-3
ii  mysql-common           5.6.25-3
ii  mysql-server-core-5.6  5.6.25-3
ii  passwd                 1:4.2-3
ii  perl                   5.20.2-6
ii  psmisc                 22.21-2.1
ii  zlib1g                 1:1.2.8.dfsg-2+b1

Versions of packages mysql-server-5.6 recommends:
ii  libhtml-template-perl  2.95-2

Versions of packages mysql-server-5.6 suggests:
ii  mailutils [mailx]  1:2.99.98-2+b1
pn  tinyca             <none>

-- debconf information:
  mysql-server/no_upgrade_when_using_ndb:
  mysql-server-5.6/really_downgrade: false
  mysql-server/password_mismatch:
  mysql-server-5.6/nis_warning:
  mysql-server-5.6/postrm_remove_databases: false
  mysql-server-5.6/start_on_boot: true
  mysql-server/error_setting_password: