[DRE-maint] Bug#852787: gitlab: systemd configuration for sidekiq tears down gitlab service

[Patrik Hagedorn]

Package: gitlab
Version: 8.13.11+dfsg1-8
Followup-For: Bug #852787

Dear Maintainer,

we had the same issue with sidekiq. Sidekiq is known to suffer from memory
leaks, that's what probably triggers the OOM killer mentioned by Libor Klepáč.
We activated the Sidekiq MemoryKiller as documented by GitLab:
https://docs.gitlab.com/ce/administration/operations/sidekiq_memory_killer.html
As result, the Sidekiq MemoryKiller shuts down the whole gitlab.service.

As mentioned by Libor Klepáč the gitlab-sidekiq.service returns after OOM killer
due to the "Restart=on-abnormal" directive, but every signal send to the sidekiq
process will result in the shutdown of gitlab.service as the
gitlab-sidekiq.service is referenced as a dependency in the "BindsTo" directive.

In our scenario the Sidekiq MemoryKiller sends a SIGTERM after a 15min grace
period or a SIGKILL after additional 30sec. Both signals have the same effect on
the gitlab.service, which shuts down completely.

A patch to fix the configuration of gitlab.service and gitlab-sidekiq.service is
attached. We configured systemd to not tear down the whole gitlab.service when
gitlab-sidekiq.service terminates by removing gitlab-sidekiq.service from the
"BindsTo" directive and using a softer "Wants" directive for
gitlab-sidekiq.service in the gitlab.service instead.

The gitlab-sidekiq.service itself is configured with "Restart" directive to
restart "always" instead of just "on-abnormal", so that it even restarts when
it gets a clean exit code 0 by the SIGTERM of the Sidekiq MemoryKiller.

As recommended by GitLab and used in the Omnibus package, we should use the
Sidekiq MemoryKiller as a measure against the memory leaks (patch attached).

Best
-- 
Patrik Hagedorn
System Developer - Data Center Automation

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin

Email: patrik.hagedorn at profitbricks.com
URL:   https://www.profitbricks.de

Sitz der Gesellschaft: Berlin
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B
Geschäftsführer: Achim Weiss, Matthias Steinberg
-------------- next part --------------
>From 1f177089791f0d6bde137182f9aa857ef1a36d89 Mon Sep 17 00:00:00 2001
From: Patrik Hagedorn <patrik.hagedorn at profitbricks.com>
Date: Thu, 28 Sep 2017 17:50:28 +0200
Subject: [PATCH 1/2] Prevent dying sidekiq to shutdown gitlab.service

Sidekiq has memory leaks and when it gets terminated by OOM killer
or GitLab's Sidekiq MemoryKiller [1] the whole gitlab.service shuts
down with it, due to the "BindsTo" directive.

Configure systemd to not tear down the whole gitlab.service when
gitlab-sidekiq.service terminates by removing gitlab-sidekiq.service
from the "BindsTo" directive and using a softer "Wants" directive for
gitlab-sidekiq.service in the gitlab.service instead.

The gitlab-sidekiq.service is configured with "Restart=always"
directive, so that it even restarts when it gets a clean exit code 0 by
the SIGTERM of the Sidekiq MemoryKiller.

Bug-Debian: https://bugs.debian.org/852787

[1] https://docs.gitlab.com/ce/administration/operations/sidekiq_memory_killer.html
---
 debian/gitlab.gitlab-sidekiq.service | 2 +-
 debian/gitlab.service                | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/debian/gitlab.gitlab-sidekiq.service b/debian/gitlab.gitlab-sidekiq.service
index 2ac84d9..d1e87c0 100644
--- a/debian/gitlab.gitlab-sidekiq.service
+++ b/debian/gitlab.gitlab-sidekiq.service
@@ -15,7 +15,7 @@ SyslogIdentifier=gitlab-sidekiq
 ExecStart=/usr/bin/bundle exec sidekiq \
     -C /etc/gitlab/sidekiq_queues.yml \
     -e $RAILS_ENV
-Restart=on-abnormal
+Restart=always
 
 ## No "Type=notify" support; wait for service to initialise:
 ExecStartPost=-/bin/sh -c "for i in 4 4 4 4 4 4 4 4; do sleep $i; (ps -h -o command -p $MAINPID | grep -q -P \"sidekiq\s\d\.\d\.\d\") && exit 0; done"
diff --git a/debian/gitlab.service b/debian/gitlab.service
index 363826b..97b533b 100644
--- a/debian/gitlab.service
+++ b/debian/gitlab.service
@@ -1,6 +1,7 @@
 [Unit]
 Description=GitLab Services
-BindsTo=gitlab-unicorn.service gitlab-sidekiq.service gitlab-mailroom.service gitlab-workhorse.service
+Wants=gitlab-sidekiq.service
+BindsTo=gitlab-unicorn.service gitlab-mailroom.service gitlab-workhorse.service
 After=gitlab-unicorn.service gitlab-sidekiq.service gitlab-mailroom.service gitlab-workhorse.service
 
 [Service]
-- 
2.11.0

-------------- next part --------------
>From ea18d7aeac0b3760527dd34fd595f50308977442 Mon Sep 17 00:00:00 2001
From: Patrik Hagedorn <patrik.hagedorn at profitbricks.com>
Date: Thu, 28 Sep 2017 18:46:12 +0200
Subject: [PATCH 2/2] Enable Sidekiq MemoryKiller

GitLab recommends to enable Sidekiq MemoryKiller [1]. The GitLab Omnibus
package uses 1 GB as default threshold [2].

Configure Sidekiq MemoryKiller to kill sidekiq when exceeding the
threshold of 1 GB allocated memory.

Bug-Debian: https://bugs.debian.org/852787

[1] https://docs.gitlab.com/ce/administration/operations/sidekiq_memory_killer.html
[2] https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb
---
 debian/conf/gitlab-debian.conf.example | 1 +
 1 file changed, 1 insertion(+)

diff --git a/debian/conf/gitlab-debian.conf.example b/debian/conf/gitlab-debian.conf.example
index 1ef2d43..e68271a 100644
--- a/debian/conf/gitlab-debian.conf.example
+++ b/debian/conf/gitlab-debian.conf.example
@@ -1,6 +1,7 @@
 # Variables with all small letters are debian specific
 # Variables with all caps are passed to gitlab app
 RAILS_ENV=production
+SIDEKIQ_MEMORY_KILLER_MAX_RSS=1000000
 DB=postgres
 gitlab_app_root=/usr/share/gitlab
 gitlab_data_dir=/var/lib/gitlab
-- 
2.11.0