[DRE-maint] Bug#1059561: gitlab: huge amount of sidekiq processes and OOM after 16.2.8->16.4.4 upgrade
Ondrej Zary
zary at gsystem.sk
Thu Dec 28 11:46:50 GMT 2023
Package: gitlab
Version: 16.4.4+ds2-2~fto12+4
Severity: important
X-Debbugs-Cc: zary at gsystem.sk
Dear Maintainer,
upgrading gitlab from 16.2.8+ds1-6~fto12+1 to 16.4.4+ds2-2~fto12+4 failed
during gitlab:check phase. There were about 60 sidekiq processes running,
eating all the memory and causing OOM, crashing other server processes.
Looks like the systemd unit starts a sidekiq process for each file in /usr/share/gitlab
- WTF?
# ps axu | grep side
root 840598 0.0 0.0 18956 5840 pts/0 S+ 11:23 0:00 systemctl restart gitlab-sidekiq
gitlab 840644 0.0 0.0 2576 892 ? Ss 11:24 0:00 /bin/sh -c GEM_PATH=$(gem env gempath); /usr/bin/bundle exec bin/sidekiq-cluster $SIDEKIQ_QUEUES -e $RAILS_ENV
gitlab 840645 0.0 0.0 2576 932 ? Ds 11:24 0:00 /bin/sh -c for i in 4 4 4 4 4 4 4 4; do sleep $i; (ps -h -o command -p $MAINPID | grep -q -P "sidekiq \d\.\d\.\d") && exit 0; done
gitlab 840648 11.8 0.8 1235524 51196 ? Sl 11:24 0:03 bin/sidekiq-cluster CHANGELOG.md CONTRIBUTING.md Dangerfile Dockerfile.assets GITALY_SERVER_VERSION GITLAB_ELASTICSEARCH_INDEXER_VERSION GITLAB_KAS_VERSION GITLAB_PAGES_VERSION GITLAB_SHELL_VERSION GITLAB_WORKHORSE_VERSION Gemfile Gemfile.lock Guardfile INSTALLATION_TYPE MAINTENANCE.md PHILOSOPHY.md PROCESS.md README.md Rakefile VERSION app babel.config.js bin builds changelogs config config.ru crowdin.yml danger data db doc docker-compose.yml fixtures gems generator_templates glfm_specification haml_lint jest.config.base.js jest.config.contract.js jest.config.integration.js jest_resolver.js lib locale log metrics_server node_modules package.json postcss.config.js public rubocop scripts security.txt shared sidekiq_cluster spec storybook tests.yml tmp tooling vendor vite.config.js yarn-error.log yarn.lock -e production
gitlab 840671 3.2 0.3 87288 21380 ? R 11:24 0:00 /usr/bin/ruby /usr/bin/sidekiq -c4 -eproduction -t25 -gqueues:CHANGELOG.md,default,mailers -r/usr/share/gitlab -qCHANGELOG.md,1 -qdefault,1 -qmailers,1
gitlab 840673 3.3 0.3 87296 21148 ? R 11:24 0:00 /usr/bin/ruby /usr/bin/sidekiq -c4 -eproduction -t25 -gqueues:CONTRIBUTING.md,default,mailers -r/usr/share/gitlab -qCONTRIBUTING.md,1 -qdefault,1 -qmailers,1
...
Using the old gitlab-sidekiq.service file from 16.2.8 fixes the problem.
-- System Information:
Debian Release: 12.4
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'oldoldstable-updates'), (500, 'oldoldstable'), (500, 'stable'), (100, 'bookworm-fasttrack'), (100, 'bookworm-backports-staging')
Architecture: amd64 (x86_64)
Kernel: Linux 6.1.0-16-amd64 (SMP w/2 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=sk_SK.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages gitlab depends on:
ii apache2 [httpd] 2.4.57-2
ii asciidoctor 2.0.18-2
ii bc 1.07.1-3+b1
ii bundler 2.3.15-2
ii bzip2 1.0.8-5+b1
ii dbconfig-pgsql 2.0.24
ii debconf [debconf-2.0] 1.5.82
ii fonts-font-awesome [node-font-awesome 5.0.10+really4.7.0~dfsg-4.1
]
ii gitlab-common 16.4.4+ds1-4~bpo12+1
ii gitlab-workhorse 16.4.4+ds2-2~fto12+4
ii katex [node-katex] 0.16.4+~cs6.1.0-1
ii libjs-bootstrap4 [node-bootstrap] 4.6.1+dfsg1-4
ii libjs-pdf [node-pdfjs-dist] 2.14.305+dfsg-2
ii libjs-popper.js [node-popper.js] 1.16.1+ds-6
ii libruby3.1 [ruby-rexml] 3.1.2-7
ii libssl-dev 3.0.11-1~deb12u2
ii lsb-base 11.6
ii node-autosize 4.0.4~dfsg1+~4.0.0-2
ii node-axios 1.2.1+dfsg-1
ii node-babel-loader 9.1.0-3
ii node-babel-plugin-lodash 3.3.4+~cs2.0.1-6
ii node-babel7 7.20.15+ds1+~cs214.269.168-3+deb12u1
ii node-brace-expansion 2.0.1-2
ii node-cache-loader 4.1.0+~cs2.0.0-4
ii node-clipboard 2.0.11+ds+~cs9.6.11-1
ii node-compression-webpack-plugin 10.0.0-2
ii node-copy-webpack-plugin 11.0.0-3
ii node-core-js 3.26.1-3
ii node-core-js-compat 3.26.1-3
ii node-core-js-pure 3.26.1-3
ii node-cron-validator 1.3.1-3
ii node-css-loader 6.7.2+~cs14.0.11-1
ii node-d3 5.16.0-10
ii node-d3-selection 1.4.0-8
ii node-dateformat 5.0.3-5
ii node-dompurify 2.4.1+dfsg+~2.4.0-1
ii node-exports-loader 4.0.0-1
ii node-file-loader 6.2.0-3
ii node-fuzzaldrin-plus 0.6.0+dfsg+~0.6.2-3
ii node-glob 8.0.3+~cs8.4.15-1
ii node-imports-loader 0.8.0-6
ii node-jed 1.1.1-4
ii node-jquery 3.6.1+dfsg+~3.5.14-1
ii node-jquery-ujs 1.2.3-2
ii node-js-cookie 3.0.1+~3.0.0-3
ii node-js-yaml 4.1.0+dfsg+~4.0.5-7
ii node-jszip 3.10.1+dfsg-1
ii node-jszip-utils 0.1.0+dfsg-1
ii node-lodash 4.17.21+dfsg+~cs8.31.198.20210220-9
ii node-marked 4.2.3+ds+~4.0.7-2
ii node-minimatch 5.1.1+~5.1.2-1
ii node-miragejs 0.1.46+~cs5.6.11-1
ii node-mousetrap 1.6.5~ds+~1.6.8-1
ii node-postcss 8.4.20+~cs8.0.23-1
ii node-prismjs 1.29.0+dfsg+~1.26.0-1
ii node-prosemirror-markdown 1.8.0-1
ii node-prosemirror-model 1.16.1+~cs1.1.5-2
ii node-prosemirror-state 1.3.4-2
ii node-prosemirror-view 1.23.13-2
ii node-raven-js 3.22.1+dfsg-7
ii node-raw-loader 4.0.2-3
ii node-style-loader 3.3.1-2
ii node-three-orbit-controls 82.1.0-3
ii node-three-stl-loader 1.0.6-4
ii node-timeago.js 4.0.2-7
ii node-underscore 1.13.4~dfsg+~1.11.4-3
ii node-url-loader 4.1.1-5
ii node-uuid 8.3.2+~8.3.3-3
ii node-vue 2.6.14+dfsg-5
ii node-vue-resource 1.5.3+dfsg+~1.3.6-3
ii node-webpack-stats-plugin 1.1.1-1
ii node-worker-loader 3.0.8-5
ii node-xterm 3.8.1+~cs0.9.0-1
ii node-yaml 2.1.3-2
ii nodejs 18.19.0+dfsg-6~deb12u1
ii ohai 17.9.0-1~bpo12+1
ii openssh-client 1:9.2p1-2+deb12u2
ii postfix [mail-transport-agent] 3.7.6-0+deb12u2
ii postgresql-client-15 [postgresql-clie 15.5-0+deb12u1
nt]
ii postgresql-contrib 15+248
ii puma 6.4.0-2~bpo12+1
ii rake 13.0.6-3
ii redis-server 5:7.0.11-1+deb12u1
ii ruby 1:3.1
ii ruby-ace-rails-ap 4.1.1-1.1
ii ruby-acme-client 2.0.9-1
ii ruby-actioncable [node-rails-actionca 2:6.1.7.3+dfsg-1
ble]
ii ruby-activerecord-explain-analyze 0.1.0-2
ii ruby-acts-as-taggable-on 9.0.1-2
ii ruby-addressable 2.8.1-1
ii ruby-akismet 3.0.0-1
ii ruby-apollo-upload-server 2.1.0-1
ii ruby-app-store-connect 0.37.0-2~bpo12+1
ii ruby-asana 0.10.13-1
ii ruby-asciidoctor-include-ext 0.4.0+gh-1
ii ruby-asciidoctor-kroki 0.8.0-1~bpo12+1
ii ruby-asciidoctor-plantuml 0.0.16-1
ii ruby-autoprefixer-rails 10.4.7.0+dfsg1+~cs24.8.7-5
ii ruby-aws-sdk-cloudformation 1.41.0-2
ii ruby-aws-sdk-core 3.190.0-2~bpo12+1
ii ruby-aws-sdk-s3 1.141.0-2~bpo12+1
ii ruby-azure-storage-blob 2.0.0-3
ii ruby-azure-storage-common 2.0.1-6
ii ruby-babosa 2.0.0-1~bpo12+1
ii ruby-base32 0.3.4-1
ii ruby-batch-loader 2.0.1+dfsg-3
ii ruby-bcrypt 3.1.18-1
ii ruby-bootsnap 1.16.0-1~bpo12+1
ii ruby-bootstrap-form 4.2.0-2
ii ruby-browser 5.3.1-3~bpo12+1
ii ruby-carrierwave 1.3.2-2
ii ruby-cfpropertylist 3.0.6-1~bpo12+1
ii ruby-charlock-holmes 0.7.7-3+b2
ii ruby-chronic [ruby-gitlab-chronic] 0.10.5-2
ii ruby-chronic-duration [ruby-gitlab-ch 0.12.0-1~bpo12+1
ronic-duration]
ii ruby-circuitbox 2.0.0-1~bpo12+1
ii ruby-commonmarker 0.23.10-1~bpo12+1
ii ruby-connection-pool 2.2.5-1
ii ruby-countries 4.2.2-1~bpo12+1
ii ruby-creole 0.5.0-3
ii ruby-cvss-suite 3.1.0-2
ii ruby-declarative-policy 1.1.0-1
ii ruby-dev 1:3.1
ii ruby-device-detector 1.0.1-2
ii ruby-devise 4.8.1-1
ii ruby-devise-two-factor 4.0.2-1
ii ruby-diff-match-patch 0.1.0-2
ii ruby-diffy 3.4.2-1
ii ruby-digest-crc 0.4.0-2~bpo12+1
ii ruby-discordrb-webhooks 3.4.2-2
ii ruby-doorkeeper 5.6.6-2~bpo12+1
ii ruby-doorkeeper-openid-connect 1.8.7-1~bpo12+1
ii ruby-duo-api 1.3.0-2~bpo12+1
ii ruby-ed25519 1.3.0+ds-1+b4
ii ruby-elasticsearch 7.17.1-1
ii ruby-elasticsearch-api 7.17.1-1
ii ruby-elasticsearch-model 7.2.1-1
ii ruby-elasticsearch-rails 7.2.1-1
ii ruby-email-reply-trimmer 0.1.6-2
ii ruby-erubi 1.9.0-2
ii ruby-escape-utils 1.2.1-4+b3
ii ruby-excon 0.88.0-1
ii ruby-faraday 1.1.0-7
ii ruby-faraday-middleware-aws-sigv4 0.3.0-2
ii ruby-fast-blank 1.0.1-1+b4
ii ruby-ffaker 2.20.0-1
ii ruby-flipper [ruby-flipper-active-sup 0.26.2-1~bpo12+1
port-cache-store]
pn ruby-flipper-active-record <none>
ii ruby-flowdock 0.7.1-1
ii ruby-fog-aliyun 0.4.0-1~bpo12+1
ii ruby-fog-aws 3.18.0-1~bpo12+1
ii ruby-fog-core 2.3.0-1
ii ruby-fog-google 1.19.0-2~bpo12+1
ii ruby-fog-local 0.8.0-2
ii ruby-fogbugz 0.3.0-2
ii ruby-fugit 1.8.1-2~bpo12+1
ii ruby-gettext 3.3.3-2
ii ruby-gettext-i18n-rails 1.12.0-1~bpo12+1
ii ruby-gettext-i18n-rails-js 1.3.0+dfsg-3
ii ruby-gitaly 16.4.4+ds1-4~bpo12+1
ii ruby-github-markup 1.7.0+dfsg-6
ii ruby-gitlab-experiment 0.8.0-1~bpo12+1
ii ruby-gitlab-fog-azure-rm 1.8.0-1~bpo12+1
ii ruby-gitlab-labkit 0.34.0-1~bpo12+1
ii ruby-gitlab-markup 1.9.0-1
ii ruby-gon 6.4.0-2
ii ruby-google-apis-androidpublisher-v3 0.34.0-1~bpo12+1
ii ruby-google-apis-cloudbilling-v1 0.21.0-1~bpo12+1
ii ruby-google-apis-cloudresourcemanager 0.31.0-1~bpo12+1
-v1
ii ruby-google-apis-compute-v1 0.57.0-1~bpo12+1
ii ruby-google-apis-container-v1 0.43.0-1~bpo12+1
ii ruby-google-apis-container-v1beta1 0.43.0-1~bpo12+1
ii ruby-google-apis-core 0.11.0-1~bpo12+1
ii ruby-google-apis-iam-v1 0.36.0-1~bpo12+1
ii ruby-google-apis-serviceusage-v1 0.28.0-1~bpo12+1
ii ruby-google-apis-sqladmin-v1beta4 0.41.0-1~bpo12+1
ii ruby-google-protobuf 1:3.21.12-3~bpo12+1
ii ruby-gpgme 2.0.23-1~bpo12+1
ii ruby-grape 1.7.1-1~bpo12+1
ii ruby-grape-entity 0.10.1-3
ii ruby-grape-logging 1.8.4-1
ii ruby-grape-path-helpers 1.7.1-1
ii ruby-graphiql-rails 1.4.10-1
ii ruby-graphlient 0.5.0-3
ii ruby-graphql 1.13.20-1~bpo12+1
ii ruby-grpc 1.51.1-3~bpo12+1
ii ruby-hamlit 2.15.1-3
ii ruby-hangouts-chat 0.0.5-2
ii ruby-hashie 5.0.0-2
ii ruby-health-check 3.0.0-1
ii ruby-html-pipeline 2.14.3-1
ii ruby-html2text 0.2.0-1.1
ii ruby-http 4.4.1-5
ii ruby-httparty 0.21.0-1
ii ruby-icalendar 2.8.0-1
ii ruby-invisible-captcha 2.1.0-1~bpo12+1
ii ruby-ipaddr 1.2.5-1~bpo12+1
ii ruby-ipynbdiff 0.4.7-1
ii ruby-jira 2.1.5-4
ii ruby-js-regex 3.8.0-1
ii ruby-json 2.6.3+dfsg-1
ii ruby-json-schema 2.8.1-4
ii ruby-json-schemer 0.2.18-2
ii ruby-jsonb-accessor 1.3.10-2~bpo12+1
ii ruby-jwt 2.5.0-1
ii ruby-kaminari 1.2.2-1
ii ruby-kas-grpc 16.1.3-1~bpo12+1
ii ruby-kramdown 2.4.0-2
ii ruby-kubeclient 4.11.0-1~bpo12+1
ii ruby-licensee 9.16.0-1~bpo12+1
ii ruby-lockbox 1.1.1-1~bpo12+1
ii ruby-lograge 0.10.0-1.1
ii ruby-loofah 2.21.3-1~bpo12+1
ii ruby-lru-redux 1.1.0-2
ii ruby-mail 2.8.1+dfsg1-1~bpo12+1
ii ruby-mail-room 0.10.0+really0.0.23-1~bpo12+1
ii ruby-marginalia 1.11.1-1
ii ruby-memory-profiler 1.0.1-1~bpo12+1
ii ruby-method-source 1.0.0-2
ii ruby-mini-magick 4.11.0-1
ii ruby-multi-json 1.14.1-2
ii ruby-neighbor 0.2.3-1~bpo12+1
ii ruby-net-dns [ruby-gitlab-net-dns] 0.9.2-2~bpo12+1
ii ruby-net-http-persistent 4.0.1-1~bpo12+1
ii ruby-net-ldap 0.17.1-1~bpo12+1
ii ruby-net-ntp 2.1.3-4
ii ruby-nokogiri 1.15.4+dfsg-1~bpo12+1
ii ruby-ntlm 0.6.3-2
ii ruby-oauth2 2.0.7-2~bpo12+1
ii ruby-octokit 6.1.1-1~bpo12+1
ii ruby-oj 3.14.2-1
ii ruby-oj-introspect 0.7.1-3
ii ruby-omniauth 2.1.1-1
ii ruby-omniauth-alicloud 2.0.1-1~bpo12+1
ii ruby-omniauth-atlassian-oauth2 0.2.0-2
ii ruby-omniauth-auth0 3.1.0-2~bpo12+1
ii ruby-omniauth-authentiq 0.3.3-1
ii ruby-omniauth-azure-activedirectory-v 2.0.0-1
2
ii ruby-omniauth-dingtalk-oauth2 1.0.1-1
ii ruby-omniauth-facebook 4.0.0-3~bpo12+1
ii ruby-omniauth-github 2.0.1-2
ii ruby-omniauth-gitlab 4.1.0-1
ii ruby-omniauth-google-oauth2 1.1.1-2~bpo12+1
ii ruby-omniauth-kerberos 0.3.0-3.1
ii ruby-omniauth-ldap 2.2.0-1
ii ruby-omniauth-oauth 1.2.0-2
ii ruby-omniauth-oauth2 1.8.0-1
ii ruby-omniauth-oauth2-generic 0.2.8-1~bpo12+1
ii ruby-omniauth-openid-connect 0.10.0-2
ii ruby-omniauth-saml 2.1.0-2
ii ruby-omniauth-twitter 1.4.0-1
ii ruby-openid-connect 1.3.0-1~bpo12+1
ii ruby-org 0.9.12-2.1
ii ruby-parslet 1.8.2-4
ii ruby-peek 1.1.0-1
ii ruby-pg 1.5.4-1~bpo12+1
ii ruby-pg-query 4.2.3-1~bpo12+1
ii ruby-posix-spawn 0.3.13-3+b5
ii ruby-premailer-rails 1.10.3-3
ii ruby-prof 1.4.5-2
ii ruby-progressbar 1.11.0-1
ii ruby-prometheus-client-mmap 0.23.1-1~bpo12+1
ii ruby-puma-worker-killer 0.3.1-3
ii ruby-rack 2.2.7-1~bpo12+1
ii ruby-rack-attack 6.7.0-1~bpo12+1
ii ruby-rack-cors 2.0.1-1~bpo12+1
ii ruby-rack-oauth2 1.21.3-1
ii ruby-rack-proxy 0.7.7-1~bpo12+1
ii ruby-rack-timeout 0.6.3-1
ii ruby-rails 2:6.1.7.3+dfsg-1
ii ruby-rails-i18n 7.0.5-1
ii ruby-rainbow 3.1.1-1
ii ruby-rbtrace 0.4.14-1+b2
ii ruby-re2 2.5.0-1~bpo12+1
ii ruby-recaptcha 5.12.3-1~bpo12+1
ii ruby-recursive-open-struct 1.1.3-1
ii ruby-redcloth 4.3.2-4+b3
ii ruby-redis 4.8.0-1
ii ruby-redis-actionpack 5.3.0-1
ii ruby-redis-namespace 1.9.0-1
ii ruby-request-store 1.5.1-1
ii ruby-responders 3.0.1-2
ii ruby-retriable 3.1.2-1
ii ruby-rouge 4.1.3-2~bpo12+1
ii ruby-rqrcode 1.2.0-1
ii ruby-rqrcode-rails3 0.1.7-3
ii ruby-ruby-magic-static [ruby-ruby-mag 0.6.0-1~bpo12+1
ic]
ii ruby-ruby-openai 3.7.0-2~bpo12+1
ii ruby-ruby-parser 3.20.0-2~bpo12+1
ii ruby-rugged 1.7.1+ds-1~bpo12+1
ii ruby-saml 1.15.0-1~bpo12+1
ii ruby-sanitize 6.0.2-1~bpo12+1
ii ruby-sassc 2.4.0+ds-1
ii ruby-sassc-rails 2.1.2-6
ii ruby-sd-notify 0.1.1-2
ii ruby-seed-fu 2.3.7-3
ii ruby-semver-dialects 1.2.1-2~bpo12+1
ii ruby-sentry-rails 5.9.0-1~bpo12+1
ii ruby-sentry-raven 3.1.2+ds-3
ii ruby-sentry-ruby 5.9.0-2~bpo12+1
ii ruby-sentry-sidekiq 5.9.0-1~bpo12+1
ii ruby-sidekiq 6.5.7+dfsg3-1~bpo12+1
ii ruby-sidekiq-cron 1.8.0-1
ii ruby-slack-messenger 2.3.4-2
ii ruby-snowplow-tracker 0.8.0-1~bpo12+1
ii ruby-spamcheck 1.10.1-2~bpo12+1
ii ruby-sprockets 3.7.2-4
ii ruby-ssh-data 1.3.0-2
ii ruby-stackprof 0.2.25-1~bpo12+1
ii ruby-state-machines-activemodel 0.8.0-3
ii ruby-state-machines-activerecord 0.8.0-2
ii ruby-sys-filesystem 1.4.3-1
ii ruby-tanuki-emoji 0.7.0-1~bpo12+1
ii ruby-task-list [node-deckar01-task-li 2.3.2-2
st]
ii ruby-telesign 2.2.4-1
ii ruby-telesignenterprise 2.2.2-2
ii ruby-terser 1.1.12+dfsg-2
ii ruby-thrift 0.16.0-1~bpo12+1
ii ruby-toml-rb 2.2.0-1
ii ruby-truncato 0.7.12-1
ii ruby-typhoeus 1.4.0-3
ii ruby-unf 0.1.4-2.1
ii ruby-unf-ext 0.0.7.7-1+b4
ii ruby-unleash 3.2.5-2
ii ruby-valid-email 0.1.3-3
ii ruby-validates-hostname 1.0.13-2~bpo12+1
ii ruby-version-sorter 2.3.0-1
ii ruby-view-component 3.8.0-1~bpo12+1
ii ruby-virtus 2.0.0-1
ii ruby-vmstat 2.3.0-3+b5
ii ruby-warning 1.3.0-1
ii ruby-webauthn 3.0.0-1~bpo12+1
ii ruby-webpack-rails 0.9.11+git-1
ii ruby-webrick 1.8.1-1
ii ruby-websocket-extensions 0.1.5-2
ii ruby-whitequark-parser 3.1.3.0-1
ii ruby-wikicloth 0.8.1+dfsg-4.1
ii ruby-yajl 1.4.3-1+b2
ii ruby-zeitwerk 2.6.1-1
ii ruby-zip 2.3.2-1
ii rubygems-integration 1.18
ii sysvinit-utils [lsb-base] 3.06-4
ii ucf 3.0043+nmu1
ii webpack 5.75.0+dfsg+~cs17.16.14-1+deb12u1
ii yarnpkg 1.22.19+~cs24.27.18-2+deb12u1
Versions of packages gitlab recommends:
ii certbot 2.1.0-4
ii gitaly 16.4.4+ds1-4~bpo12+1
ii openssh-server 1:9.2p1-2+deb12u2
gitlab suggests no packages.
-- debconf-show failed
More information about the Pkg-ruby-extras-maintainers
mailing list