[pkg-lxc-devel] Bug#887787: lxc: CentOS 7 amd64 container can't be stopped

Toni Mueller toni at debian.org
Fri Jan 19 21:51:13 UTC 2018


Package: lxc
Version: 1:2.0.7-2+deb9u1
Severity: normal


Hi,


I installed an unprivileged CentOS 7 container with

$ lxc-create -n centos -t download

after setting my system up according to the instructions given here:

https://wiki.debian.org/LXC


The resulting container starts as expected, but any attempt to shut it
down again, fails. The container remains responsive, but just does not
stop. Inside the container, the process list looks as follows:

$ lxc-attach -n centos
bash-4.2# ps auwwx
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  42544  2332 ?        Ds   Jan20   0:00 /sbin/init
root       132  0.0  0.1 113380 13416 ?        Ss   Jan20   0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid -H centos eth0
root       191  0.0  0.0 115396  3160 ?        Ss   Jan20   0:00 /bin/bash
root       192  0.0  0.0 107912   616 ?        D+   Jan20   0:00 sync
root       193  0.0  0.0 115396  3064 ?        Ss   Jan20   0:00 /bin/bash
root       194  0.0  0.0 151072  3564 ?        R+   Jan20   0:00 ps auwwx
bash-4.2# 


In a different shell, things look like this:

$ lxc-attach -n centos
bash-4.2# halt -n -f
c^C^Z^C


(ie, no reaction)


In the kernel log of the host, I get a lot of these:


Jan 19 22:20:37 debian kernel: [39269.678133] INFO: task systemd:6381 blocked for more than 120 seconds.
Jan 19 22:20:37 debian kernel: [39269.678144]       Tainted: G           O    4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
Jan 19 22:20:37 debian kernel: [39269.678149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 19 22:20:37 debian kernel: [39269.678154] systemd         D    0  6381   6371 0x00000104
Jan 19 22:20:37 debian kernel: [39269.678163]  ffff976453d38000 ffff976553415800 ffff976504ffafc0 ffff97655fc98940
Jan 19 22:20:37 debian kernel: [39269.678170]  ffff976500564480 ffffa7a7040b3bc0 ffffffffa6802923 ffff976504ffafc0
Jan 19 22:20:37 debian kernel: [39269.678176]  00ff976504ffafc0 ffff97655fc98940 ffff976504161dc0 ffff976504ffafc0
Jan 19 22:20:37 debian kernel: [39269.678181] Call Trace:
Jan 19 22:20:37 debian kernel: [39269.678194]  [<ffffffffa6802923>] ? __schedule+0x233/0x6d0
Jan 19 22:20:37 debian kernel: [39269.678200]  [<ffffffffa6802df2>] ? schedule+0x32/0x80
Jan 19 22:20:37 debian kernel: [39269.678207]  [<ffffffffa6805999>] ? rwsem_down_write_failed+0x1f9/0x360
Jan 19 22:20:37 debian kernel: [39269.678214]  [<ffffffffa64801d0>] ? kernfs_sop_show_options+0x30/0x30
Jan 19 22:20:37 debian kernel: [39269.678220]  [<ffffffffa6538213>] ? call_rwsem_down_write_failed+0x13/0x20
Jan 19 22:20:37 debian kernel: [39269.678225]  [<ffffffffa6805039>] ? down_write+0x29/0x40
Jan 19 22:20:37 debian kernel: [39269.678231]  [<ffffffffa64055ab>] ? grab_super+0x2b/0x90
Jan 19 22:20:37 debian kernel: [39269.678237]  [<ffffffffa6405b43>] ? sget_userns+0x163/0x490
Jan 19 22:20:37 debian kernel: [39269.678242]  [<ffffffffa6480240>] ? kernfs_sop_show_path+0x40/0x40
Jan 19 22:20:37 debian kernel: [39269.678246]  [<ffffffffa648044a>] ? kernfs_mount_ns+0x7a/0x220
Jan 19 22:20:37 debian kernel: [39269.678252]  [<ffffffffa630dba4>] ? cgroup_mount+0x334/0x810
Jan 19 22:20:37 debian kernel: [39269.678259]  [<ffffffffa6406ae6>] ? mount_fs+0x36/0x150
Jan 19 22:20:37 debian kernel: [39269.678264]  [<ffffffffa6423f32>] ? vfs_kern_mount+0x62/0x100
Jan 19 22:20:37 debian kernel: [39269.678268]  [<ffffffffa64263ff>] ? do_mount+0x1cf/0xc80
Jan 19 22:20:37 debian kernel: [39269.678273]  [<ffffffffa64271de>] ? SyS_mount+0x7e/0xd0
Jan 19 22:20:37 debian kernel: [39269.678279]  [<ffffffffa6203b1c>] ? do_syscall_64+0x7c/0xf0
Jan 19 22:20:37 debian kernel: [39269.678285]  [<ffffffffa68076ee>] ? entry_SYSCALL64_slow_path+0x25/0x25
Jan 19 22:20:37 debian kernel: [39269.678294] INFO: task sync:6958 blocked for more than 120 seconds.
Jan 19 22:20:37 debian kernel: [39269.678300]       Tainted: G           O    4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
Jan 19 22:20:37 debian kernel: [39269.678303] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 19 22:20:37 debian kernel: [39269.678307] sync            D    0  6958   6951 0x00000104
Jan 19 22:20:37 debian kernel: [39269.678312]  ffff976553fc4800 0000000000000000 ffff97653103a140 ffff97655fcd8940
Jan 19 22:20:37 debian kernel: [39269.678318]  ffff976556666f00 ffffa7a706043df0 ffffffffa6802923 ffff9765500a5760
Jan 19 22:20:37 debian kernel: [39269.678324]  0000000000000286 ffff97655fcd8940 ffffa7a706043df8 ffff97653103a140
Jan 19 22:20:37 debian kernel: [39269.678329] Call Trace:
Jan 19 22:20:37 debian kernel: [39269.678335]  [<ffffffffa6802923>] ? __schedule+0x233/0x6d0
Jan 19 22:20:37 debian kernel: [39269.678340]  [<ffffffffa6802df2>] ? schedule+0x32/0x80
Jan 19 22:20:37 debian kernel: [39269.678345]  [<ffffffffa6805740>] ? rwsem_down_read_failed+0xf0/0x150
Jan 19 22:20:37 debian kernel: [39269.678350]  [<ffffffffa641f86e>] ? iput+0x7e/0x210
Jan 19 22:20:37 debian kernel: [39269.678356]  [<ffffffffa6436870>] ? SyS_tee+0x390/0x390
Jan 19 22:20:37 debian kernel: [39269.678361]  [<ffffffffa65381e4>] ? call_rwsem_down_read_failed+0x14/0x30
Jan 19 22:20:37 debian kernel: [39269.678366]  [<ffffffffa6804ffc>] ? down_read+0x1c/0x30
Jan 19 22:20:37 debian kernel: [39269.678371]  [<ffffffffa64064ac>] ? iterate_supers+0x9c/0x100
Jan 19 22:20:37 debian kernel: [39269.678376]  [<ffffffffa6436bc2>] ? sys_sync+0x42/0xb0
Jan 19 22:20:37 debian kernel: [39269.678380]  [<ffffffffa6203b1c>] ? do_syscall_64+0x7c/0xf0
Jan 19 22:20:37 debian kernel: [39269.678386]  [<ffffffffa68076ee>] ? entry_SYSCALL64_slow_path+0x25/0x25
Jan 19 22:20:37 debian kernel: [39269.678401] INFO: task halt:7352 blocked for more than 120 seconds.
Jan 19 22:20:37 debian kernel: [39269.678406]       Tainted: G           O    4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
Jan 19 22:20:37 debian kernel: [39269.678409] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 19 22:20:37 debian kernel: [39269.678412] halt            D    0  7352   7211 0x00000100
Jan 19 22:20:37 debian kernel: [39269.678417]  ffff976554b10000 0000000000000000 ffff97650de10200 ffff97655fc18940
Jan 19 22:20:37 debian kernel: [39269.678422]  ffffffffa6e11500 ffffa7a7059bfdf0 ffffffffa6802923 ffff9765500a5760
Jan 19 22:20:37 debian kernel: [39269.678428]  0000000000000286 ffff97655fc18940 ffffa7a7059bfdf8 ffff97650de10200
Jan 19 22:20:37 debian kernel: [39269.678433] Call Trace:
Jan 19 22:20:37 debian kernel: [39269.678438]  [<ffffffffa6802923>] ? __schedule+0x233/0x6d0
Jan 19 22:20:37 debian kernel: [39269.678443]  [<ffffffffa6802df2>] ? schedule+0x32/0x80
Jan 19 22:20:37 debian kernel: [39269.678448]  [<ffffffffa6805740>] ? rwsem_down_read_failed+0xf0/0x150
Jan 19 22:20:37 debian kernel: [39269.678452]  [<ffffffffa641f86e>] ? iput+0x7e/0x210
Jan 19 22:20:37 debian kernel: [39269.678458]  [<ffffffffa6436870>] ? SyS_tee+0x390/0x390
Jan 19 22:20:37 debian kernel: [39269.678463]  [<ffffffffa65381e4>] ? call_rwsem_down_read_failed+0x14/0x30
Jan 19 22:20:37 debian kernel: [39269.678468]  [<ffffffffa6804ffc>] ? down_read+0x1c/0x30
Jan 19 22:20:37 debian kernel: [39269.678473]  [<ffffffffa64064ac>] ? iterate_supers+0x9c/0x100
Jan 19 22:20:37 debian kernel: [39269.678478]  [<ffffffffa6436bc2>] ? sys_sync+0x42/0xb0
Jan 19 22:20:37 debian kernel: [39269.678482]  [<ffffffffa6203b1c>] ? do_syscall_64+0x7c/0xf0
Jan 19 22:20:37 debian kernel: [39269.678487]  [<ffffffffa68076ee>] ? entry_SYSCALL64_slow_path+0x25/0x25
Jan 19 22:22:38 debian kernel: [39390.499861] INFO: task systemd:6381 blocked for more than 120 seconds.
Jan 19 22:22:38 debian kernel: [39390.499872]       Tainted: G           O    4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
Jan 19 22:22:38 debian kernel: [39390.499877] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 19 22:22:38 debian kernel: [39390.499881] systemd         D    0  6381   6371 0x00000104
Jan 19 22:22:38 debian kernel: [39390.499890]  ffff976453d38000 ffff976553415800 ffff976504ffafc0 ffff97655fc98940
Jan 19 22:22:38 debian kernel: [39390.499897]  ffff976500564480 ffffa7a7040b3bc0 ffffffffa6802923 ffff976504ffafc0
Jan 19 22:22:38 debian kernel: [39390.499903]  00ff976504ffafc0 ffff97655fc98940 ffff976504161dc0 ffff976504ffafc0
Jan 19 22:22:38 debian kernel: [39390.499908] Call Trace:
Jan 19 22:22:38 debian kernel: [39390.499922]  [<ffffffffa6802923>] ? __schedule+0x233/0x6d0
Jan 19 22:22:38 debian kernel: [39390.499927]  [<ffffffffa6802df2>] ? schedule+0x32/0x80
Jan 19 22:22:38 debian kernel: [39390.499934]  [<ffffffffa6805999>] ? rwsem_down_write_failed+0x1f9/0x360
Jan 19 22:22:38 debian kernel: [39390.499941]  [<ffffffffa64801d0>] ? kernfs_sop_show_options+0x30/0x30
Jan 19 22:22:38 debian kernel: [39390.499947]  [<ffffffffa6538213>] ? call_rwsem_down_write_failed+0x13/0x20
Jan 19 22:22:38 debian kernel: [39390.499952]  [<ffffffffa6805039>] ? down_write+0x29/0x40
Jan 19 22:22:38 debian kernel: [39390.499958]  [<ffffffffa64055ab>] ? grab_super+0x2b/0x90
Jan 19 22:22:38 debian kernel: [39390.499963]  [<ffffffffa6405b43>] ? sget_userns+0x163/0x490
Jan 19 22:22:38 debian kernel: [39390.499968]  [<ffffffffa6480240>] ? kernfs_sop_show_path+0x40/0x40
Jan 19 22:22:38 debian kernel: [39390.499973]  [<ffffffffa648044a>] ? kernfs_mount_ns+0x7a/0x220
Jan 19 22:22:38 debian kernel: [39390.499978]  [<ffffffffa630dba4>] ? cgroup_mount+0x334/0x810
Jan 19 22:22:38 debian kernel: [39390.499985]  [<ffffffffa6406ae6>] ? mount_fs+0x36/0x150
Jan 19 22:22:38 debian kernel: [39390.499990]  [<ffffffffa6423f32>] ? vfs_kern_mount+0x62/0x100
Jan 19 22:22:38 debian kernel: [39390.499995]  [<ffffffffa64263ff>] ? do_mount+0x1cf/0xc80
Jan 19 22:22:38 debian kernel: [39390.500000]  [<ffffffffa64271de>] ? SyS_mount+0x7e/0xd0
Jan 19 22:22:38 debian kernel: [39390.500006]  [<ffffffffa6203b1c>] ? do_syscall_64+0x7c/0xf0
Jan 19 22:22:38 debian kernel: [39390.500012]  [<ffffffffa68076ee>] ? entry_SYSCALL64_slow_path+0x25/0x25
Jan 19 22:22:38 debian kernel: [39390.500021] INFO: task sync:6958 blocked for more than 120 seconds.
Jan 19 22:22:38 debian kernel: [39390.500026]       Tainted: G           O    4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
Jan 19 22:22:38 debian kernel: [39390.500030] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 19 22:22:38 debian kernel: [39390.500033] sync            D    0  6958   6951 0x00000104
Jan 19 22:22:38 debian kernel: [39390.500038]  ffff976553fc4800 0000000000000000 ffff97653103a140 ffff97655fcd8940
Jan 19 22:22:38 debian kernel: [39390.500045]  ffff976556666f00 ffffa7a706043df0 ffffffffa6802923 ffff9765500a5760
Jan 19 22:22:38 debian kernel: [39390.500050]  0000000000000286 ffff97655fcd8940 ffffa7a706043df8 ffff97653103a140
Jan 19 22:22:38 debian kernel: [39390.500055] Call Trace:
Jan 19 22:22:38 debian kernel: [39390.500061]  [<ffffffffa6802923>] ? __schedule+0x233/0x6d0
Jan 19 22:22:38 debian kernel: [39390.500066]  [<ffffffffa6802df2>] ? schedule+0x32/0x80
Jan 19 22:22:38 debian kernel: [39390.500071]  [<ffffffffa6805740>] ? rwsem_down_read_failed+0xf0/0x150
Jan 19 22:22:38 debian kernel: [39390.500076]  [<ffffffffa641f86e>] ? iput+0x7e/0x210
Jan 19 22:22:38 debian kernel: [39390.500082]  [<ffffffffa6436870>] ? SyS_tee+0x390/0x390
Jan 19 22:22:38 debian kernel: [39390.500088]  [<ffffffffa65381e4>] ? call_rwsem_down_read_failed+0x14/0x30
Jan 19 22:22:38 debian kernel: [39390.500092]  [<ffffffffa6804ffc>] ? down_read+0x1c/0x30
Jan 19 22:22:38 debian kernel: [39390.500098]  [<ffffffffa64064ac>] ? iterate_supers+0x9c/0x100
Jan 19 22:22:38 debian kernel: [39390.500103]  [<ffffffffa6436bc2>] ? sys_sync+0x42/0xb0
Jan 19 22:22:38 debian kernel: [39390.500107]  [<ffffffffa6203b1c>] ? do_syscall_64+0x7c/0xf0
Jan 19 22:22:38 debian kernel: [39390.500112]  [<ffffffffa68076ee>] ? entry_SYSCALL64_slow_path+0x25/0x25
Jan 19 22:22:38 debian kernel: [39390.500127] INFO: task halt:7352 blocked for more than 120 seconds.
Jan 19 22:22:38 debian kernel: [39390.500132]       Tainted: G           O    4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
Jan 19 22:22:38 debian kernel: [39390.500135] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 19 22:22:38 debian kernel: [39390.500138] halt            D    0  7352   7211 0x00000100
Jan 19 22:22:38 debian kernel: [39390.500143]  ffff976554b10000 0000000000000000 ffff97650de10200 ffff97655fc18940
Jan 19 22:22:38 debian kernel: [39390.500149]  ffffffffa6e11500 ffffa7a7059bfdf0 ffffffffa6802923 ffff9765500a5760
Jan 19 22:22:38 debian kernel: [39390.500154]  0000000000000286 ffff97655fc18940 ffffa7a7059bfdf8 ffff97650de10200
Jan 19 22:22:38 debian kernel: [39390.500159] Call Trace:
Jan 19 22:22:38 debian kernel: [39390.500165]  [<ffffffffa6802923>] ? __schedule+0x233/0x6d0
Jan 19 22:22:38 debian kernel: [39390.500169]  [<ffffffffa6802df2>] ? schedule+0x32/0x80
Jan 19 22:22:38 debian kernel: [39390.500174]  [<ffffffffa6805740>] ? rwsem_down_read_failed+0xf0/0x150
Jan 19 22:22:38 debian kernel: [39390.500179]  [<ffffffffa641f86e>] ? iput+0x7e/0x210
Jan 19 22:22:38 debian kernel: [39390.500184]  [<ffffffffa6436870>] ? SyS_tee+0x390/0x390
Jan 19 22:22:38 debian kernel: [39390.500189]  [<ffffffffa65381e4>] ? call_rwsem_down_read_failed+0x14/0x30
Jan 19 22:22:38 debian kernel: [39390.500194]  [<ffffffffa6804ffc>] ? down_read+0x1c/0x30
Jan 19 22:22:38 debian kernel: [39390.500199]  [<ffffffffa64064ac>] ? iterate_supers+0x9c/0x100
Jan 19 22:22:38 debian kernel: [39390.500204]  [<ffffffffa6436bc2>] ? sys_sync+0x42/0xb0
Jan 19 22:22:38 debian kernel: [39390.500208]  [<ffffffffa6203b1c>] ? do_syscall_64+0x7c/0xf0
Jan 19 22:22:38 debian kernel: [39390.500213]  [<ffffffffa68076ee>] ? entry_SYSCALL64_slow_path+0x25/0x25


Trying to 'lxc-stop -n centos' (name of the container) also does not
work. A KVM machine with CentOS 7 on the same host works without
problems. From the host machine, the container's processes look like
this:


$ ps auwwx|grep 231072
231072    6381  0.0  0.0  42544  2100 ?        Ds   22:10   0:00 /sbin/init
231072    6708  0.0  0.1 113380 14640 ?        Ss   22:12   0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid -H centos eth0
231072    6951  0.0  0.0 115396  3160 pts/17   Ss   22:15   0:00 /bin/bash
231072    6958  0.0  0.0 107912   616 pts/17   D+   22:15   0:00 sync
231072    7211  0.0  0.0 115396  3168 pts/18   Ss   22:16   0:00 /bin/bash
231072    7352  0.0  0.0 128432  2576 pts/18   D+   22:18   0:00 halt -f
231072    7525  0.0  0.0 115396  3116 pts/20   Ss   22:19   0:00 /bin/bash
231072    7589  0.0  0.0 128432  2796 pts/20   D+   22:19   0:00 halt -n -f
toni     10162  0.0  0.0  12720   980 pts/5    S+   22:49   0:00 grep --color=auto 231072
231072   27515  0.0  0.0   1508     8 ?        Ss   18:55   0:00 /sbin/init
231072   27827  0.0  0.0   1508    48 ?        Ss   18:55   0:00 /sbin/syslogd -Z
231072   27848  0.0  0.0   1512    48 ?        Ss   18:55   0:00 /usr/sbin/crond -c /etc/crontabs
231072   27917  0.0  0.0   1508    48 ?        Ss   18:56   0:00 udhcpc -b -p /var/run/udhcpc.eth0.pid -i eth0 -x hostname:alpine
231072   27944  0.0  0.0   7320   308 ?        S    18:56   0:00 /usr/sbin/sshd
231072   27948  0.0  0.0   1508     4 pts/0    Ss+  18:56   0:00 /sbin/getty 38400 tty1
231072   27949  0.0  0.0   1508     4 pts/1    Ss+  18:56   0:00 /sbin/getty 38400 tty2
231072   27950  0.0  0.0   1508     4 pts/2    Ss+  18:56   0:00 /sbin/getty 38400 tty3
231072   27951  0.0  0.0   1508     4 pts/3    Ss+  18:56   0:00 /sbin/getty 38400 tty4
231072   27952  0.0  0.0   1508     4 pts/9    Ss+  18:56   0:00 /sbin/getty 38400 console



It would be great if lxc could prevent a container misbehaving like
that. There's a side effect on the host machine: Since the 'init'
process of the container hangs, the host machine cannot shut down
properly anymore.



Cheers,
Toni



-- System Information:
Debian Release: 9.3
  APT prefers stable
  APT policy: (990, 'stable'), (90, 'testing'), (70, 'unstable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.9.0-5-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.utf8 (charmap=UTF-8), LANGUAGE=en_US:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages lxc depends on:
ii  init-system-helpers  1.48
ii  libapparmor1         2.11.0-3
ii  libc6                2.24-11+deb9u1
ii  libcap2              1:2.25-1
ii  libgnutls30          3.5.8-5+deb9u3
ii  liblxc1              1:2.0.7-2+deb9u1
ii  libseccomp2          2.3.1-2.1
ii  libselinux1          2.6-3+b3
ii  lsb-base             9.20161125
ii  python3              3.5.3-1
ii  python3-lxc          1:2.0.7-2+deb9u1

Versions of packages lxc recommends:
ii  bridge-utils  1.5-13+deb9u1
ii  debootstrap   1.0.89
ii  dirmngr       2.1.18-8~deb9u1
ii  dnsmasq-base  2.76-5+deb9u1
ii  gnupg         2.1.18-8~deb9u1
ii  iptables      1.6.0+snapshot20161117-6
ii  libpam-cgfs   2.0.7-1
ii  lxcfs         2.0.7-1
ii  openssl       1.1.0f-3+deb9u1
ii  rsync         3.1.2-1+deb9u1
ii  uidmap        1:4.4-4.1

Versions of packages lxc suggests:
pn  apparmor     <none>
ii  btrfs-tools  4.7.3-1
ii  lvm2         2.02.168-2

-- Configuration Files:
/etc/lxc/default.conf changed:
lxc.network.type = veth
lxc.network.link = virbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:xx:xx:xx


-- no debconf information



More information about the Pkg-lxc-devel mailing list