Bug#918839: systemd: LXC Container with debian buster can no longer start services after updating to systemd_240

[Fabian Grünbichler]

the original reporter did not mention it explicitly (although the kernel
version already indicates it ;)) - the original occurrence of this issue
was on a Debian derivative (Proxmox Virtual Environment), which does not
use Debian's kernel (and thus neither the same AppArmor LSM code nor
AppArmor feature set) as well as a different LXC version.

it is however easily reproduced using Debian Sid as well:

root at host:/# lxc-create -n test -t debian -- -r buster
[...]

(underlying storage is irrelevant)

then setup network (none in this case to just use host network, config
otherwise unedited)

root at host:/# lxc-attach -n test
root at test:/# apt install apache2
Reading package lists... Done
Building dependency tree... Done
The following additional packages will be installed:
  apache2-bin apache2-data apache2-utils bzip2 file libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap
  libbrotli1 libcurl4 libexpat1 libgdbm-compat4 libgdbm6 libicu63 libjansson4 libldap-2.4-2 libldap-common
  liblua5.2-0 libmagic-mgc libmagic1 libnghttp2-14 libperl5.28 libpsl5 librtmp1 libsasl2-2 libsasl2-modules
  libsasl2-modules-db libsqlite3-0 libssh2-1 libxml2 mime-support perl perl-modules-5.28 publicsuffix ssl-cert
  xz-utils
Suggested packages:
  apache2-doc apache2-suexec-pristine | apache2-suexec-custom www-browser bzip2-doc gdbm-l10n
  libsasl2-modules-gssapi-mit | libsasl2-modules-gssapi-heimdal libsasl2-modules-ldap libsasl2-modules-otp
  libsasl2-modules-sql perl-doc libterm-readline-gnu-perl | libterm-readline-perl-perl make libb-debug-perl
  liblocale-codes-perl openssl-blacklist
The following NEW packages will be installed:
  apache2 apache2-bin apache2-data apache2-utils bzip2 file libapr1 libaprutil1 libaprutil1-dbd-sqlite3
  libaprutil1-ldap libbrotli1 libcurl4 libexpat1 libgdbm-compat4 libgdbm6 libicu63 libjansson4 libldap-2.4-2
  libldap-common liblua5.2-0 libmagic-mgc libmagic1 libnghttp2-14 libperl5.28 libpsl5 librtmp1 libsasl2-2
  libsasl2-modules libsasl2-modules-db libsqlite3-0 libssh2-1 libxml2 mime-support perl perl-modules-5.28
  publicsuffix ssl-cert xz-utils
0 upgraded, 38 newly installed, 0 to remove and 0 not upgraded.
Need to get 21.6 MB of archives.
After this operation, 102 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 https://cdn-aws.deb.debian.org/debian buster/main amd64 perl-modules-5.28 all 5.28.1-3 [2,873 kB]
Get:2 https://cdn-aws.deb.debian.org/debian buster/main amd64 libgdbm6 amd64 1.18.1-2 [64.5 kB]
Get:3 https://cdn-aws.deb.debian.org/debian buster/main amd64 libgdbm-compat4 amd64 1.18.1-2 [44.0 kB]

[...]

Setting up apache2-bin (2.4.37-1) ...
Setting up apache2 (2.4.37-1) ...
Enabling module mpm_event.
Enabling module authz_core.
Enabling module authz_host.
Enabling module authn_core.
Enabling module auth_basic.
Enabling module access_compat.
Enabling module authn_file.
Enabling module authz_user.
Enabling module alias.
Enabling module dir.
Enabling module autoindex.
Enabling module env.
Enabling module mime.
Enabling module negotiation.
Enabling module setenvif.
Enabling module filter.
Enabling module deflate.
Enabling module status.
Enabling module reqtimeout.
Enabling conf charset.
Enabling conf localized-error-pages.
Enabling conf other-vhosts-access-log.
Enabling conf security.
Enabling conf serve-cgi-bin.
Enabling site 000-default.
Created symlink /etc/systemd/system/multi-user.target.wants/apache2.service → /lib/systemd/system/apache2.service.
Created symlink /etc/systemd/system/multi-user.target.wants/apache-htcacheclean.service → /lib/systemd/system/apache-htcacheclean.service.
Job for apache2.service failed because the control process exited with error code.
See "systemctl status apache2.service" and "journalctl -xe" for details.
invoke-rc.d: initscript apache2, action "start" failed.
● apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2019-01-11 20:32:56 UTC; 15ms ago
  Process: 902 ExecStart=/usr/sbin/apachectl start (code=exited, status=226/NAMESPACE)

Jan 11 20:32:56 test systemd[1]: Starting The Apache HTTP Server...
Jan 11 20:32:56 test systemd[902]: apache2.service: Failed to set up mount namespacing: Permission denied
Jan 11 20:32:56 test systemd[902]: apache2.service: Failed at step NAMESPACE spawning /usr/sbin/apachectl: Permission denied
Jan 11 20:32:56 test systemd[1]: apache2.service: Control process exited, code=exited, status=226/NAMESPACE
Jan 11 20:32:56 test systemd[1]: apache2.service: Failed with result 'exit-code'.
Jan 11 20:32:56 test systemd[1]: Failed to start The Apache HTTP Server.
Processing triggers for libc-bin (2.28-2) ...
Processing triggers for systemd (240-2) ...

root at host:/# journalctl --since "-1min"
[...]
Jan 11 21:40:15 host audit[23555]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=23555 comm="(pachectl)" flags="rw, rslave"
Jan 11 21:40:15 host kernel: audit: type=1400 audit(1547239215.720:230): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=23555 comm="(pachectl)" flags="rw, rslave"
[...]

adding the config stanzas from the nesting.conf file shipped with LXC
changes the AppArmor message, but Apache2 does not start either:

Jan 11 21:52:30 core audit[4616]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-with-nesting" name="/run/systemd/unit-root/" pid=4616 comm="(pachectl)" srcname="/" flags="rw, rbind"
Jan 11 21:52:30 core kernel: audit: type=1400 audit(1547239950.506:234): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-with-nesting" name="/run/systemd/unit-root/" pid=4616 comm="(pachectl)" srcname="/" flags="rw, rbind"

without apparmor it works of course, but that is hardly how you want to
run LXC instances, yet alone privileged ones ;)