[Debian-ha-maintainers] Bug#979450: booth: autopkgtest fails on ci-worker-ppc64el-01 (but apparently not on other ppc64el workers)

[Paul Gevers]

reassign 980964 autopkgtest
reassign 980965 autopkgtest
reassign 979450 autopkgtest
force-merge 980964 980965 979450
severity 980964 important
retitle 980964 autopkgtest-build-lxc doesn't copy <container>/config
thanks

Hi,

tl;dr; issue found and worked around: recreation of containers is
flawed, so fresh containers were using extremely old (wrong) configuration.

On 28-01-2021 11:59, wferi at niif.hu wrote:
> These systemd messages are emitted during service setup, before the
> service binary is even started, and are very much characteristic to the
> Apparmor misconfiguration described in the LXC 3 NEWS file.  I can
> readily reproduce them with another systemd-hardened package:
> 
> systemd[697]: coturn.service: Failed to set up mount namespacing: Permission denied
> systemd[697]: coturn.service: Failed at step NAMESPACE spawning /usr/bin/turnserver: Permission denied
> 
> and such messages are neatly paired with these in the host syslog:
> 
> audit: type=1400 audit(1611830306.349:157): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=27587 comm="(rnserver)" flags="rw, rslave"

I see messages like these on the worker where the autopkgtest fails,
*but* also on the workers where they pass. There is a delta though, on
the worker where it fails this message is regularly followed by (so,
more flags):

Jan 24 00:10:18 ci-worker-ppc64el-01 kernel: [1048816.624446] audit:
type=1400 audit(1611447018.564:209866): apparmor="DENIED"
operation="mount" info="failed flags match" error=-13
profile="lxc-container-default-cgns" name="/sys/fs/cgroup/" pid=20099
comm="systemd" fstype="cgroup2" srcname="cgroup2" flags="rw, nosuid,
nodev, noexec"
Jan 24 00:10:18 ci-worker-ppc64el-01 kernel: [1048816.632674] audit:
type=1400 audit(1611447018.572:209867): apparmor="DENIED"
operation="mount" info="failed flags match" error=-13
profile="lxc-container-default-cgns" name="/sys/fs/cgroup/" pid=20099
comm="systemd" fstype="cgroup2" srcname="cgroup2" flags="rw, nosuid,
nodev, noexec"
Jan 24 00:10:18 ci-worker-ppc64el-01 kernel: [1048816.632786] audit:
type=1400 audit(1611447018.572:209868): apparmor="DENIED"
operation="mount" info="failed flags match" error=-13
profile="lxc-container-default-cgns" name="/sys/fs/cgroup/" pid=20099
comm="systemd" fstype="cgroup2" srcname="cgroup2" flags="rw, nosuid,
nodev, noexec"

> Can you see such messages?  Are you sure that the failed runs had
> 
> lxc.apparmor.profile = generated
> lxc.apparmor.allow_nesting = 1
> 
> in their LXC configuration?

Grr. I'm now sure they don't. Although we generate new containers every
day, it seems that the configuration of those containers in
/var/lib/lxc/* *doesn't* get refreshed. I have just destroyed all
containers before creating new ones, and now they contain this. So,
somehow our container recreation is flawed.

I ran a booth, pdns and pdns-recursor autopkgtest manually on this host,
and they now pass.

I've reassigned the bugs to autopkgtest, it needs to be fixed there IMHO.

Thanks everybody for helping along.

Paul

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <http://alioth-lists.debian.net/pipermail/debian-ha-maintainers/attachments/20210128/5f30cb15/attachment.sig>