[pkg-lxc-devel] Bug#860974: lxc: systemd and other inotify-based programs fail with "Too many open files" with a large number of LXC containers

[Baptiste Jonglez]

Package: lxc
Version: 1:2.0.7-2
Severity: important
Tags: patch

Dear Maintainer,

I am using LXC on stretch, running a number of stretch containers.

When running around 15 containers, systemd starts to fail, both on the host
and inside containers:

    # journalctl -f
    Failed to get journal fd: Too many open files
    # systemctl stop nginx
    Failed to allocate directory watch: Too many open files

The problem seems to be that systemd uses inotify quite a lot, and the limit
on inotify listeners is very low by default:

    # sysctl fs.inotify.max_user_instances
    fs.inotify.max_user_instances = 128

Increasing this value indeed solves the problem.  The issue was described
in this blog post:

    https://kdecherf.com/blog/2015/09/12/systemd-and-the-fd-exhaustion/

Attached is a patch for lxc that configures this sysctl setting to a higher
value.  I don't know if this is the "Debian way" or if some other solution
is preferred.

-- System Information:
Debian Release: 9.0
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64
 (x86_64)

Kernel: Linux 4.9.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages lxc depends on:
ii  init-system-helpers  1.47
ii  libapparmor1         2.11.0-3
ii  libc6                2.24-9
ii  libcap2              1:2.25-1
ii  libgnutls30          3.5.8-5
ii  liblxc1              1:2.0.7-2
ii  libseccomp2          2.3.1-2.1
ii  libselinux1          2.6-3+b1
ii  lsb-base             9.20161125
ii  python3-lxc          1:2.0.7-2
pn  python3:any          <none>

Versions of packages lxc recommends:
pn  bridge-utils  <none>
ii  debootstrap   1.0.89
ii  dirmngr       2.1.18-6
pn  dnsmasq-base  <none>
ii  gnupg         2.1.18-6
ii  iptables      1.6.0+snapshot20161117-6
pn  libpam-cgfs   <none>
pn  lxcfs         <none>
ii  openssl       1.1.0e-1
ii  rsync         3.1.2-1
pn  uidmap        <none>

Versions of packages lxc suggests:
pn  apparmor     <none>
pn  btrfs-tools  <none>
ii  lvm2         2.02.168-2

-- no debconf information
-------------- next part --------------
>From 023651197841198cd10e797451ae591ec523ff45 Mon Sep 17 00:00:00 2001
From: Baptiste Jonglez <git at bitsofnetworks.org>
Date: Sat, 22 Apr 2017 23:51:22 +0200
Subject: [PATCH] Increase the maximum number of inotify listeners

This avoid issues with systemd containers, where as little as 15 LXC
containers can trigger "Too many open files" errors in systemd.
---
 debian/40-lxc-inotify.conf | 8 ++++++++
 debian/rules               | 4 ++++
 2 files changed, 12 insertions(+)
 create mode 100644 debian/40-lxc-inotify.conf

diff --git a/debian/40-lxc-inotify.conf b/debian/40-lxc-inotify.conf
new file mode 100644
index 0000000..c16dd5f
--- /dev/null
+++ b/debian/40-lxc-inotify.conf
@@ -0,0 +1,8 @@
+# Defines the maximum number of inotify listeners.
+# By default, this value is 128, which is quickly exhausted when using
+# systemd-based LXC containers (15 containers are enough).
+# When the limit is reached, systemd becomes mostly unusable, throwing
+# "Too many open files" all around (both on the host and in containers).
+# See https://kdecherf.com/blog/2015/09/12/systemd-and-the-fd-exhaustion/
+
+fs.inotify.max_user_instances=4096
diff --git a/debian/rules b/debian/rules
index 7d334ad..8443d0f 100755
--- a/debian/rules
+++ b/debian/rules
@@ -48,6 +48,10 @@ override_dh_auto_install:
 	rm -f debian/tmp/usr/share/man/*/lxc-top.*
 	rm -f debian/tmp/usr/share/man/*/*/lxc-top.*
 
+	# increase limit of inotify listeners
+	mkdir -p debian/tmp/etc/sysctl.d
+	cp debian/40-lxc-inotify.conf debian/tmp/etc/sysctl.d/
+
 	dh_apparmor -p lxc --profile-name=usr.bin.lxc-start
 
 override_dh_compress:
-- 
2.12.2