[pkg-lxc-devel] Bug#978065: lxc: After upgrade lxc to 4.0.5-1, cannot start with lxc.cap.drop sys_admin
Andras Korn
korn-debbugs at elan.rulez.org
Mon Jan 25 23:59:52 GMT 2021
Hi,
I hit the same issue.
I upgraded from 1:4.0.4-6 to 1:4.0.5-2, and from kernel 5.9.0-4-amd64 to 5.10.0-2-amd64, and some of my containers that used to work before don't work anyomre. The ones that still work don't drop sys_admin.
stracing lxc-start I see this:
openat2(33</usr/lib/x86_64-linux-gnu/lxc/rootfs>, "/sys/fs/cgroup", {flags=O_RDONLY|O_CLOEXEC|O_PATH, resolve=RESOLVE_NO_XDEV|RESOLVE_NO_MAGICLINKS|RESOLVE_NO_SYMLINKS|RESOLVE_BENEATH}, 24) = -1 EXDEV (Invalid cross-device link)
The corresponding message from lxc-start with loglevel debug is:
lxc-start unifiadmin 20210125231743.129 ERROR conf - conf.c:lxc_mount_auto_mounts:727 - Invalid cross-device link - Failed to mount "/sys/fs/cgroup"
Some context from lxc-start log output:
lxc-start unifiadmin 20210125231742.854 INFO start - start.c:lxc_init:837 - Container "unifiadmin" is initialized
lxc-start unifiadmin 20210125231742.876 WARN cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1152 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.monitor.unifiadmin"
lxc-start unifiadmin 20210125231742.886 INFO cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1368 - The monitor process uses "lxc.monitor.unifiadmin" as cgroup
lxc-start unifiadmin 20210125231742.904 WARN cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1152 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.payload.unifiadmin"
lxc-start unifiadmin 20210125231742.916 INFO cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1471 - The container process uses "lxc.payload.unifiadmin" as cgroup
lxc-start unifiadmin 20210125231742.944 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWNS
lxc-start unifiadmin 20210125231742.944 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWPID
lxc-start unifiadmin 20210125231742.945 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWUTS
lxc-start unifiadmin 20210125231742.945 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWIPC
lxc-start unifiadmin 20210125231742.945 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWNET
lxc-start unifiadmin 20210125231742.945 DEBUG start - start.c:lxc_try_preserve_namespaces:166 - Preserved mnt namespace via fd 31
lxc-start unifiadmin 20210125231742.945 DEBUG start - start.c:lxc_try_preserve_namespaces:166 - Preserved pid namespace via fd 32
lxc-start unifiadmin 20210125231742.946 DEBUG start - start.c:lxc_try_preserve_namespaces:166 - Preserved uts namespace via fd 33
lxc-start unifiadmin 20210125231742.946 DEBUG start - start.c:lxc_try_preserve_namespaces:166 - Preserved ipc namespace via fd 34
lxc-start unifiadmin 20210125231742.946 DEBUG start - start.c:lxc_try_preserve_namespaces:166 - Preserved net namespace via fd 35
lxc-start unifiadmin 20210125231742.949 INFO cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits_legacy:2881 - Limits for the legacy cgroup hierarchies have been setup
lxc-start unifiadmin 20210125231742.955 WARN cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2942 - Invalid argument - Ignoring cgroup2 limits on legacy cgroup system
lxc-start unifiadmin 20210125231743.315 INFO network - network.c:instantiate_veth:285 - Retrieved mtu 1500 from intra
lxc-start unifiadmin 20210125231743.666 INFO network - network.c:instantiate_veth:333 - Attached "veth-unifi" to bridge "intra"
lxc-start unifiadmin 20210125231743.687 DEBUG network - network.c:instantiate_veth:449 - Instantiated veth tunnel "veth-unifi <--> vethv7jzuF"
lxc-start unifiadmin 20210125231743.699 WARN start - start.c:do_start:1166 - Using /dev/null from the host for container init's standard file descriptors. Migration will not work
lxc-start unifiadmin 20210125231743.704 INFO start - start.c:do_start:1198 - Unshared CLONE_NEWCGROUP
lxc-start unifiadmin 20210125231743.731 DEBUG storage - storage/storage.c:get_storage_by_name:211 - Detected rootfs type "dir"
lxc-start unifiadmin 20210125231743.734 DEBUG conf - conf.c:lxc_mount_rootfs:1259 - Mounted rootfs "/var/lib/lxc/unifiadmin/rootfs" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs" with options "(null)"
lxc-start unifiadmin 20210125231743.738 INFO conf - conf.c:setup_utsname:751 - Set hostname to "unifiadmin"
lxc-start unifiadmin 20210125231743.740 DEBUG network - network.c:lxc_network_setup_in_child_namespaces_common:3510 - Network device "" has been setup
lxc-start unifiadmin 20210125231743.977 DEBUG network - network.c:setup_hw_addr:3360 - Mac address "00:16:3e:11:22:33" on "eth0" has been setup
lxc-start unifiadmin 20210125231743.103 DEBUG network - network.c:lxc_network_setup_in_child_namespaces_common:3510 - Network device "eth0" has been setup
lxc-start unifiadmin 20210125231743.103 INFO network - network.c:lxc_setup_network_in_child_namespaces:3532 - Network has been setup
lxc-start unifiadmin 20210125231743.116 DEBUG conf - conf.c:mount_entry:1943 - Remounting "/shared/cache/apt/lists" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/var/lib/apt/lists" to respect bind or remount options
lxc-start unifiadmin 20210125231743.116 DEBUG conf - conf.c:mount_entry:1962 - Flags for "/shared/cache/apt/lists" were 1038, required extra flags are 14
lxc-start unifiadmin 20210125231743.117 DEBUG conf - conf.c:mount_entry:1971 - Mountflags already were 5134, skipping remount
lxc-start unifiadmin 20210125231743.117 DEBUG conf - conf.c:mount_entry:2006 - Mounted "/shared/cache/apt/lists" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/var/lib/apt/lists" with filesystem type "none"
lxc-start unifiadmin 20210125231743.118 DEBUG conf - conf.c:mount_entry:1943 - Remounting "/shared/cache/apt/archives" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/var/cache/apt/archives" to respect bind or remount options
lxc-start unifiadmin 20210125231743.118 DEBUG conf - conf.c:mount_entry:1962 - Flags for "/shared/cache/apt/archives" were 1038, required extra flags are 14
lxc-start unifiadmin 20210125231743.118 DEBUG conf - conf.c:mount_entry:1971 - Mountflags already were 5134, skipping remount
lxc-start unifiadmin 20210125231743.118 DEBUG conf - conf.c:mount_entry:2006 - Mounted "/shared/cache/apt/archives" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/var/cache/apt/archives" with filesystem type "none"
lxc-start unifiadmin 20210125231743.119 DEBUG conf - conf.c:mount_entry:1943 - Remounting "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/null" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/kcore" to respect bind or remount options
lxc-start unifiadmin 20210125231743.119 DEBUG conf - conf.c:mount_entry:1962 - Flags for "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/null" were 1024, required extra flags are 0
lxc-start unifiadmin 20210125231743.120 DEBUG conf - conf.c:mount_entry:1971 - Mountflags already were 4096, skipping remount
lxc-start unifiadmin 20210125231743.120 DEBUG conf - conf.c:mount_entry:2006 - Mounted "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/null" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/kcore" with filesystem type "none"
lxc-start unifiadmin 20210125231743.123 DEBUG conf - conf.c:mount_entry:1943 - Remounting "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" to respect bind or remount options
lxc-start unifiadmin 20210125231743.123 DEBUG conf - conf.c:mount_entry:1962 - Flags for "/sys/fs/fuse/connections" were 4110, required extra flags are 14
lxc-start unifiadmin 20210125231743.123 DEBUG conf - conf.c:mount_entry:2006 - Mounted "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" with filesystem type "none"
lxc-start unifiadmin 20210125231743.125 DEBUG conf - conf.c:mount_entry:1943 - Remounting "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" to respect bind or remount options
lxc-start unifiadmin 20210125231743.125 DEBUG conf - conf.c:mount_entry:1962 - Flags for "/sys/fs/fuse/connections" were 4110, required extra flags are 14
lxc-start unifiadmin 20210125231743.125 DEBUG conf - conf.c:mount_entry:2006 - Mounted "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" with filesystem type "none"
lxc-start unifiadmin 20210125231743.127 DEBUG conf - conf.c:mount_entry:2006 - Mounted "run" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/run" with filesystem type "tmpfs"
lxc-start unifiadmin 20210125231743.128 DEBUG conf - conf.c:mount_entry:2006 - Mounted "none" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/shm" with filesystem type "tmpfs"
lxc-start unifiadmin 20210125231743.129 ERROR conf - conf.c:lxc_mount_auto_mounts:727 - Invalid cross-device link - Failed to mount "/sys/fs/cgroup"
lxc-start unifiadmin 20210125231743.130 ERROR conf - conf.c:lxc_setup:3365 - Failed to setup remaining automatic mounts
lxc-start unifiadmin 20210125231743.130 ERROR start - start.c:do_start:1218 - Failed to setup container "unifiadmin"
lxc-start unifiadmin 20210125231743.131 ERROR sync - sync.c:__sync_wait:36 - An error occurred in another process (expected sequence number 5)
lxc-start unifiadmin 20210125231743.132 DEBUG network - network.c:lxc_delete_network:3665 - Deleted network devices
lxc-start unifiadmin 20210125231743.133 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:859 - Received container state "ABORTING" instead of "RUNNING"
lxc-start unifiadmin 20210125231743.134 ERROR lxc_start - tools/lxc_start.c:main:308 - The container failed to start
lxc-start unifiadmin 20210125231743.135 ERROR lxc_start - tools/lxc_start.c:main:311 - To get more details, run the container in foreground mode
lxc-start unifiadmin 20210125231743.135 ERROR start - start.c:__lxc_start:1999 - Failed to spawn container "unifiadmin"
lxc-start unifiadmin 20210125231743.136 ERROR lxc_start - tools/lxc_start.c:main:313 - Additional information can be obtained by setting the --logfile and --logpriority options
lxc-start unifiadmin 20210125231743.136 WARN start - start.c:lxc_abort:1012 - No such process - Failed to send SIGKILL via pidfd 30 for process 15227
lxc-start unifiadmin 20210125231743.748 INFO conf - conf.c:run_script_argv:342 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "unifiadmin"
lxc-start unifiadmin 20210125231744.288 INFO conf - conf.c:run_script_argv:342 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "unifiadmin"
If I don't drop the sys_admin capability, it works again.
Before the upgrade, it also worked if I dropped sys_admin.
The configfile for this guest is:
----- 8< -----
lxc.include = /usr/share/lxc/config/common.conf
lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 0
lxc.hook.version = 1
lxc.mount.entry = run run tmpfs rw,nodev,relatime,mode=755,size=20m,create=dir 0 0
lxc.mount.entry = none dev/shm tmpfs rw,nosuid,nodev,mode=1777,size=100m,create=dir 0 0
lxc.cap.drop = sys_resource audit_write block_suspend linux_immutable mac_admin mac_override sys_admin sys_module sys_pacct sys_rawio sys_resource sys_time sys_tty_config syslog
lxc.start.auto = 1
lxc.cgroup.devices.deny = a
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
lxc.cgroup.devices.allow = c 1:7 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:2 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 10:229 rwm
lxc.cgroup.devices.allow = c 254:0 rm
lxc.cgroup.devices.allow = c 10:200 rwm
lxc.cgroup.devices.allow = c 10:228 rwm
lxc.cgroup.devices.allow = c 10:232 rwm
lxc.autodev = 0
lxc.tty.dir =
lxc.tty.max = 0
# Container specific configuration
lxc.rootfs.path = dir:/var/lib/lxc/unifiadmin/rootfs
lxc.uts.name = unifiadmin
lxc.arch = amd64
# Network configuration
lxc.net.0.type = empty
lxc.net.1.type = veth
lxc.net.1.link = intra
lxc.net.1.flags = up
lxc.net.1.name = eth0
lxc.net.1.veth.pair = veth-unifi
lxc.net.1.hwaddr = 00:16:3e:11:22:33
lxc.mount.fstab = /var/lib/lxc/unifiadmin/fstab
----- >8 -----
The fstab doesn't reference cgroup or /sys.
I googled around and found this post from 2012: https://lists.linuxfoundation.org/pipermail/containers/2012-November/030827.html -- based on this, maybe the problem is that cap_sys_admin is dropped too early now?
Also, https://github.com/lxc/lxc/issues/1737 looks related. https://blog.iwakd.de/lxc-cap_sys_admin-jessie also suggests that running containers without cap_sys_admin used to be possible.
Or maybe I should be using cgroup2?
FWIW, both host and guest use runit, so systemd is not involved; runit doesn't interfere with cgroups or capabilities on its own in any way.
András
--
Nothing screws up a good story like an eyewitness.
More information about the Pkg-lxc-devel
mailing list