[pkg-lxc-devel] Bug#911806: Please provide a way to opt out of AppArmor confinement when running tests
intrigeri
intrigeri at debian.org
Fri Oct 26 12:58:21 BST 2018
Hi,
Michael Biebl:
> E.g. if I try to run autopkgtest for systemd on sid in a AA-confined LXC
> container, it explodes left and right, whereas it passes with
> AA-confinement turned off.
Thanks for reporting this!
Disclaimer: I don't use LXC.
I've spent some time reading the various related systemd upstream
issues and commits and my understanding is that we have two problems:
1. Recent systemd sets up namespacing with calls to mount that are not
allowed under the default AppArmor policy shipped in the lxc
package (or rather: it has always done so but previously, failures
to set up said namespacing were non-fatal). If I got it right, this
means that lots of services will be broken when run in LXC once
systemd v240 is released and uploaded to Debian, i.e. the problem
is broader than our CI: most typical LXC use cases will be affected
as well.
It's an instance of a classical problem when nesting security
systems:
- Either the outside one (in this case, AppArmor) allows the
inside one (in this case, systemd) whatever it takes so the
latter can do its job. Ideally this cooperation gives us the
best of both worlds. But in practice, this sometimes requires
the outside security system to give permissions that are so
relaxed that it becomes essentially useless (which might be the
case here, I don't know); when this happens, we basically can't
rely on the outside security system anymore, and instead we rely
mostly on the inside one. When the inside security system is
controlled by untrusted parties, such as an untrusted
container's root user, that's a risky approach, to say the
least. In other words, there's a risk to break isolation between
containers in order to improve isolation between services inside
a given container.
- Or the outside security system is restrictive to the point that
it breaks the inside one, and then it's the opposite: weaker
service isolation inside the container but stronger isolation
between containers.
Ideally, picking the best strategy and implementing it would be
a matter of coordination between LXC and systemd (ideally upstream,
but quite possibly distro maintainers will need to be involved
here). I'll raise this issue to the Ubuntu LXC and AppArmor folks.
Worst case, if this does not get resolved in time for Buster, we
may have to disable AppArmor confinement for LXC containers in
Buster; that would be sad but it may be the best we can do for
Buster, and it would not be a regression vs. Stretch for most users
(the Stretch kernel lacks the AppArmor features that LXC needs for
meaningful confinement so even if one opts in for AppArmor on
Stretch, they get pretty weak isolation). And then whoever
particularly cares about hardening LXC on Debian has a couple more
years to sort this out for Bullseye :)
2. Regardless of #1, some software is broken when run inside
a container, be it due to the way it the container technology uses
Linux namespaces or due to additional LSM-based sandboxing. In this
case the best approach is probably to run these tests inside a KVM
runner instead. During the autopkgtest at last DebConf, with
another similar issue in mind, I've asked about KVM runners for
ci.debian.net and I was told this was WIP and would have it
soonish, which will provide a tool to solve this class of problems:
any affected autopkgtest can declare the isolation-machine restriction.
A special sub-case is when the software would run just fine in
a container but its test suite exercises it in a way that requires
different privileges than normal usage of said software inside
a container, e.g. by setting up stuff in super custom way, which
might break under the default sandboxing rules. In this case,
either I would argue that the test suite should instead exercise
something closer to what actual users will run under that
confinement: that's a guiding design principle of autopkgtests.
Whenever fixing such a test suite is not possible for some reason,
then running the affected tests in KVM could be an
acceptable workaround.
> It is my understanding, that Ubuntu itself turns off AA-confinement in
> their LXC based CI or at least ship a AA policy which is much less
> strict [1].
If I got it right, that's correct for armhf but for every other
architecture, they run tests in standard Ubuntu KVM cloud instances
and not in LXC. Advantage: apart of armhf, no need to deal with the
peculiarities of LXC+AppArmor, and AppArmor-wise they're testing
exactly what users will experience outside of containers. Drawback: to
exercise software that would be broken under the default LXC+AppArmor
sandboxing, they need to relax the AppArmor sandboxing for _all_ tests
on armhf, meaning they don't test software in exactly the same sandbox
that will be used by actual users of Ubuntu LXC containers.
On the Debian side, currently we run tests only in LXC containers,
which is probably not the most common use case of Debian. In most
cases it does not matter because the software under test behaves in
exactly the same way regardless of whether it runs in a container or
not. In some cases it does matter.
> Looking ahead at the buster release, we either need to turn off AA
> confinement in our Debian CI as well, ship a custom AA policy or provide
> a way for individual packages to opt out of AA confinement. In the
> latter case, this will probably need some changes to autopkgtest as
> well.
I hope the problem will be solved in time for Buster with
a combination of a) fixing the cooperation between LXC + AppArmor and
systemd for the general LXC use case and incidentally for most
autopkgtests too; b) having KVM runners available, for cases that
won't run fine in LXC (possibly due to AppArmor).
If that does not happen, then my guiding principles would be "in
doubt, do no harm" and the idea that autopkgtests should be run in an
environment that's as close as possible as the production one, which
you seem to agree with
(https://github.com/systemd/systemd/issues/10166#issuecomment-432743680).
So turning off AppArmor entirely in our CI *only* would feel very
wrong. The remaining options are:
- Shipping a custom AppArmor policy for adt runners. This would feel
wrong too, for the same reason, if that policy is different from
the production one is important ways, e.g. if it lets systemd do
stuff that a production container would block.
- Disabling AppArmor support for LXC by default (not only for our
CI). As I've discussed above, this would be sad but acceptable.
At least we would run our CI in an environment that matches what
production usage of Debian in a container.
- Introducing a new autopkgtest restriction like
isolation-container-without-lsm is probably doable, but I'm wary
that this is a slippery slope towards getting requests for having
all kinds of less restricted containers, which may lead to
introducing more low-level autopkgtest restrictions, which does not
fit into the abstraction level at which they seem to be designed.
(As a side note, when running autopkgtests in a KVM runner,
AppArmor-wise one is testing in the exact environment that people
running Debian on bare metal or in a VM use, so we're all good.
But when running autopkgtests in containers, AppArmor-wise things are
very different; ideally we want policy stacking, i.e. there's the
policy applied to the container as a whole by the host system to
provide container isolation and on top of this, if the container is
sufficiently privileged, it may load its own AppArmor policy that will
be stacked on top of the host system's one; ideally we want
a container that has done the equivalent of "systemctl start
apparmor", to be closer to what actual users will do. AFAIK this
policy stacking setup is used mostly for LXC/LXD-based Ubuntu systems
and I'm not sure what's the status of AppArmor policy stacking support
in the kernel + userspace we will ship in Buster. I gave it a quick
try (using the lxc-ubuntu template) and was not able to load AppArmor
policy inside a LXC container running on sid.)
Cheers,
--
intrigeri
More information about the Pkg-lxc-devel
mailing list