[pkg-lxc-devel] Bug#947863: lxc: apparmor denied mount with unprivileged lxc

Wed Jan 1 01:05:37 GMT 2020

Package: lxc
Version: 1:3.1.0+really3.0.4-2
Severity: normal

Hi,

when booting into a system started with unprivileged lxc, I'm getting
the following errors:

[    5.818300] audit: type=1400 audit(1577840118.455:15): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=388 comm="(md-udevd)" flags="rw, rslave"
[    5.842226] audit: type=1400 audit(1577840118.479:16): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=389 comm="(md-udevd)" flags="rw, rslave"
[    5.875326] audit: type=1400 audit(1577840118.511:17): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=390 comm="(md-udevd)" flags="rw, rslave"
[    5.894556] audit: type=1400 audit(1577840118.531:18): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=391 comm="(md-udevd)" flags="rw, rslave"
[    5.919489] audit: type=1400 audit(1577840118.555:19): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=392 comm="(md-udevd)" flags="rw, rslave"
[    6.368326] audit: type=1400 audit(1577840119.003:20): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=398 comm="(modprobe)" flags="rw, rslave"
[    6.390053] audit: type=1400 audit(1577840119.027:21): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=400 comm="(d-logind)" flags="rw, rslave"
[    6.400681] audit: type=1400 audit(1577840119.035:22): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=405 comm="(modprobe)" flags="rw, rslave"
[    6.406682] audit: type=1400 audit(1577840119.043:23): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=406 comm="(d-logind)" flags="rw, rslave"
[    6.416232] audit: type=1400 audit(1577840119.051:24): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=409 comm="(modprobe)" flags="rw, rslave"

These errors keep repeating. The only way to silence them I found so far
is to disable apparmor. But maybe the default lxc apparmor profile could
be adjusted to prevent these errors?

To reproduce the problem, please see the attached shell script which you
can run without superuser privileges if you have
kernel.unprivileged_userns_clone set to 1. After running the script, the
errors above can be seen in qemu.log.

Thanks!

cheers, josch
-------------- next part --------------
#!/bin/sh

# Copyright 2019 Johannes 'josch' Schauer <josch at debian.org>
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.

set -exu

# Reasons to use qemu:
#   - can also be run with autopkgtest backends that do not support 
#   - can be run without superuser privileges outside of autopkgtest

if [ -z ${AUTOPKGTEST_TMP+x} ]; then
	# if AUTOPKGTEST_TMP is not set, then this script is probably not
	# executed under autopkgtest, choose unshare mode for mmdebstrap so
	# that this script can be run without superuser privileges
	MODE="unshare"
else
	# since AUTOPKGTEST_TMP is set, we assume that this script is executed
	# under autopkgtest --> switch to the temporary directory
	cd "$AUTOPKGTEST_TMP"
	# We have to use root mode on salsa ci because:
	#  - unshare mode fails because /sys is mounted read-only
	#    and kernel.unprivileged_userns_clone is not set to 1
	#  - fakechroot mode fails because of #944929
	#  - proot mode produces wrong permissions
	MODE="root"
fi

# setting up /etc/fstab, /etc/hostname and /etc/hosts is required to raise
# network interfaces
if [ ! -e debian-unstable-host.tar ]; then
mmdebstrap --mode=$MODE --variant=apt \
	--include=openssh-server,systemd-sysv,ifupdown,netbase,isc-dhcp-client,udev,policykit-1,linux-image-amd64,lxc,uidmap,apparmor,bridge-utils,procps,iptables,iptables-persistent,psmisc \
	--customize-hook='echo host > "$1/etc/hostname"' \
	--customize-hook='echo "127.0.0.1 localhost host" > "$1/etc/hosts"' \
	--customize-hook='echo "/dev/vda1 / auto errors=remount-ro 0 1" > "$1/etc/fstab"' \
	--customize-hook='cat /etc/resolv.conf > "$1/etc/resolv.conf"' \
	unstable debian-unstable-host.tar
fi

# we prepare a second tarball now instead of later inside qemu because
# running mmdebstrap without kvm just wastes cpu cycles
if [ ! -e debian-unstable-container.tar ]; then
mmdebstrap --mode=$MODE --variant=apt --include=dbus,udev,systemd-sysv,policykit-1 \
	--customize-hook='echo container > "$1/etc/hostname"' \
	--customize-hook='cat /etc/resolv.conf > "$1/etc/resolv.conf"' \
	unstable - \
	| python3 -c '
import tarfile
import sys

with tarfile.open(fileobj=sys.stdin.buffer, mode="r|*") as in_tar, \
    tarfile.open(fileobj=sys.stdout.buffer, mode="w|") as out_tar:
    for member in in_tar:
        member.uid += 1000000
        member.gid += 1000000
        if member.isfile():
            with in_tar.extractfile(member) as file:
                out_tar.addfile(member, file)
        else:
            out_tar.addfile(member)
' > debian-unstable-container.tar
fi

echo 'net.ipv4.ip_forward=1' > sysctl.conf

cat << END > rules.v4
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -i eth0 -p tcp -m tcp --dport 8002 -j DNAT --to-destination 10.0.0.2:8003
-A POSTROUTING -o eth0 -j MASQUERADE
COMMIT
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
END

# extlinux config to boot from /dev/vda1 with predictable network interface
# naming and a serial console for logging
cat << END > extlinux.conf
default linux
timeout 0

label linux
kernel /vmlinuz
append initrd=/initrd.img root=/dev/vda1 net.ifnames=0 console=ttyS0
END

# network interface config
# we can use eth0 because we boot with net.ifnames=0 for predictable interface
# names
cat << END > interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

auto lxcbridge
iface lxcbridge inet static
  address 10.0.0.1
  netmask 255.255.255.0
  bridge_stp off
  bridge_waitport 0
  bridge_fd 0
  bridge_ports none
END

echo 'root:1000000:65536' >> subuid
echo 'root:1000000:65536' >> subgid

cat << END > config
lxc.include = /usr/share/lxc/config/common.conf
lxc.include = /usr/share/lxc/config/userns.conf
lxc.uts.name = container
lxc.rootfs.path = dir:/srv/container
lxc.idmap = u 0 1000000 65536
lxc.idmap = g 0 1000000 65536
lxc.net.0.ipv4.address.address = 10.0.0.2/24
lxc.net.0.hwaddr = ee:ec:fa:e9:56:7d
lxc.net.0.type = veth
lxc.net.0.flags = up
lxc.net.0.link = lxcbridge
lxc.net.0.name = eth0
lxc.net.0.mtu = 1500
lxc.net.0.ipv4.gateway = 10.0.0.1
lxc.start.auto = 1
lxc.tty.max = 1
END

if [ ! -e id_rsa ]; then
ssh-keygen -q -t rsa -f ./id_rsa -N ""
fi

# use guestfish to prepare the host system
#
#  - create a single 2G partition and unpack the rootfs tarball into it
#  - copy the public key into .ssh/authorized_keys for the root user
#  - copy in extlinux.conf and /etc/network/interfaces
#  - unpack the tarball of the container into /srv/container
#  - put a syslinux MBR into the first 440 bytes of the drive
#  - install extlinux and make partition bootable
#
# useful stuff to debug any errors:
#   LIBGUESTFS_BACKEND_SETTINGS=force_tcg
#   libguestfs-test-tool || true
#   export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
guestfish -N host.img=disk:2G -- \
	part-disk /dev/sda mbr : \
	mkfs ext2 /dev/sda1 : \
	mount /dev/sda1 / : \
	tar-in debian-unstable-host.tar / : \
	mkdir /root/.ssh : \
	upload id_rsa.pub /root/.ssh/authorized_keys : \
	chown 0 0 /root/.ssh/authorized_keys : \
	copy-in extlinux.conf / : \
	copy-in interfaces /etc/network : \
	copy-in sysctl.conf /etc : \
	copy-in rules.v4 /etc/iptables : \
	copy-in subuid /etc : \
	copy-in subgid /etc : \
	mkdir /var/lib/lxc/container : \
	copy-in config /var/lib/lxc/container : \
	mkdir /srv/container : \
	tar-in debian-unstable-container.tar /srv/container : \
	upload /usr/lib/SYSLINUX/mbr.bin /mbr.bin : \
	copy-file-to-device /mbr.bin /dev/sda size:440 : \
	rm /mbr.bin : \
	extlinux / : \
	sync : \
	umount / : \
	part-set-bootable /dev/sda 1 true : \
	shutdown

# start the host system
# redirect tcp connections on port 10011 localhost to the host system port 22
# redirect all output to a file
# run in the background
qemu-system-x86_64 \
	-enable-kvm \
	-no-user-config \
	-object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0 \
	-m 1G \
	-net nic,model=virtio \
	-nographic \
	-serial mon:stdio \
	-net user,hostfwd=tcp:127.0.0.1:10022-:22,hostfwd=tcp:127.0.0.1:8001-:8002 \
	-drive file=host.img,format=raw,if=virtio \
	>qemu.log </dev/null 2>&1 &

# store the pid
QEMUPID=$!

# show the log and kill qemu in case the script exits first
trap "cat --show-nonprinting qemu.log; kill $QEMUPID" EXIT

# the default ssh command does not store known hosts and even ignores host keys
# it identifies itself with the rsa key generated above
# pseudo terminal allocation is disabled or otherwise, programs executed via
# ssh might wait for input on stdin of the ssh process
ssh="ssh -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no -i id_rsa -T"

# we use sleepenh to make sure that we wait the right number of seconds
# independent on how long the command took beforehand
TIMESTAMP=$(sleepenh 0 || [ $? -eq 1 ])
# the timeout in seconds
TIMEOUT=5
# the maximum number of tries
NUM_TRIES=20
i=0
while true; do
	rv=0
	$ssh -p 10022 -o ConnectTimeout=$TIMEOUT root at localhost echo success || rv=1
	# with an exit code of zero, the ssh connection was successful
	# and we break out of the loop
	[ $rv -eq 0 ] && break
	# if the command before took less than $TIMEOUT seconds, wait the remaining time
	TIMESTAMP=$(sleepenh $TIMESTAMP $TIMEOUT || [ $? -eq 1 ]);
	# increment the counter and break out of the loop if we tried
	# too often
	i=$((i+1))
	if [ $i -ge $NUM_TRIES ]; then
		break
	fi
done

# if all tries were exhausted, the process failed
if [ $i -eq $NUM_TRIES ]; then
	echo "timeout reached: unable to connect to qemu via ssh"
	exit 1
fi

# ip of host system outside qemu: 10.0.2.2
# ip of host system inside qemu: 10.0.2.15
# ip of container: 10.0.0.2

# execute a shell script on the host system
cat << END > expected
NAME      STATE   AUTOSTART IPV4     UNPRIVILEGED 
container RUNNING 1         10.0.0.2 true         
END

$ssh -p 10022 root at localhost lxc-ls --fancy --fancy-format=NAME,STATE,AUTOSTART,IPV4,UNPRIVILEGED | diff -u expected -

$ssh -p 10022 root at localhost lxc-attach --name=container -- apt-get update
$ssh -p 10022 root at localhost lxc-attach --name=container -- apt-get install --no-install-recommends --yes netcat-traditional procps

$ssh -p 10022 root at localhost lxc-attach --name=container -- nc -lp 8003 | grep foobar &
NCPID=$!

sleep 5

echo foobar | nc -q0 127.0.0.1 8001

wait $NCPID

LXCPID=$($ssh -p 10022 root at localhost lxc-info --name container | awk '/^PID:/ { print $2; }')
$ssh -p 10022 root at localhost ps -jf -u 1000000 | awk "\$2 == $LXCPID" | grep /sbin/init
$ssh -p 10022 root at localhost lxc-attach --name=container -- ps -jf -u 0 | grep /sbin/init

trap - EXIT

$ssh -p 10022 root at localhost systemctl poweroff || true

wait $QEMUPID

# debian-unstable-container.tar debian-unstable-host.tar id_rsa id_rsa.pub qemu.log
for f in extlinux.conf interfaces host.img; do
	rm "$f"
done