[DRE-maint] Bug#852787: gitlab: Doesn't restart services properly on OOM kill
Libor Klepáč
libor.klepac at bcom.cz
Fri Jan 27 09:53:08 UTC 2017
Package: gitlab
Severity: important
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hello,
we have low mem instance of gitlab - it used to have 2GB of ram, now it has 3GB.
Sometimes, it runs out of memory. (usually at 22:xx )
OOM killer usually kills one of sidekiq children.
It triggers restart of sidekiq, but also of other gitlab services.
But gitlab-sidekiq is the only one, which is started, other are left dead.
Can this be somehow addressed in systemd service dependecies?
Libor
-----BEGIN PGP SIGNATURE-----
iQJJBAEBCAAzFiEEPGZVVU37tFmB0TQv8O+MbsKfR44FAliLGIQVHGxpYm9yLmts
ZXBhY0BiY29tLmN6AAoJEPDvjG7Cn0eOcGgP/3eJAMqQoBWxG48YXLnfx1D6YZ6h
K4CLe0fpz8rFG87v4qsfyszA5WF3qY3QTffYKXxDJq5Z9QAaV2V+k5Jn0j0Sqhes
IsZqN5g4XvOdyYB6UEy6IGdDhiogiqCeHTz/gRL83WYDb2c+aIykZVYUWSS2MELl
fIpDyIn5FvlRs42VQu0u6x/Wi6b1+j55Y7KWo3j/bET+8MQzg5fOEdIO3qXGTGrf
A5wM0QQtFsOmcvQNWz7hWn7Lm4Fd5Q5U48O0MRiaJNTOUA/OOfoHgRBwHcBAUtvR
2alVQwb9FlH90wdRhUHARJEX10m2+duTvRw4KrP2PHaFcy5l/GJiVp8RV5NrAkHF
Fo9xDrmNjDnCAhjT54gOaudAYW5P9MpqJDHOtcaenvsoK8OHaEFh85kBxYhDj4DA
T8f7qnoHzBsHXzDr3zLMas8TGVfWRc/H7HhhL19WTDwCyXJfm+NBcwYd5RUXysl/
TdZE9K09uhxKtMNbnhAqfPk08XTmTRLObRGx/2U8wWaU8f3X1aZOJLANOQUKuq2Q
57eEIQ3Ed8eY8AUN95jfxq5s9QbgLhwGf8hl6eyzXKrAtHxq/coeN2rNzdaEYIac
A7VRgk4id1qnwqdAFgy6DoNTzqK2vcAjy5Ix+DmoFPHmaxqIuXj6O9pNr5+/9lL4
/0DxTlrb5zieOvBi
=h2TC
-----END PGP SIGNATURE-----
-------------- next part --------------
Jan 26 22:15:18 gitlab kernel: kworker/0:0 invoked oom-killer: gfp_mask=0x24200c2(GFP_HIGHUSER), order=0, oom_score_adj=0
Jan 26 22:15:18 gitlab kernel: kworker/0:0 cpuset=/ mems_allowed=0
Jan 26 22:15:18 gitlab kernel: CPU: 0 PID: 9360 Comm: kworker/0:0 Not tainted 4.8.0-2-amd64 #1 Debian 4.8.15-2
Jan 26 22:15:18 gitlab kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
Jan 26 22:15:18 gitlab kernel: Workqueue: events_freezable vmballoon_work [vmw_balloon]
Jan 26 22:15:18 gitlab kernel: 0000000000000286 000000001c838855 ffffffff8831f925 ffff972ef1ab3d08
Jan 26 22:15:18 gitlab kernel: ffff972f38b02080 ffffffff881f7491 0000000000000206 ffffffff8818f2e9
Jan 26 22:15:18 gitlab kernel: ffff972f3fffccc0 0000000c00000246 ffff972f2d66c080 ffff972f38b02080
Jan 26 22:15:18 gitlab kernel: Call Trace:
Jan 26 22:15:18 gitlab kernel: [<ffffffff8831f925>] ? dump_stack+0x5c/0x77
Jan 26 22:15:18 gitlab kernel: [<ffffffff881f7491>] ? dump_header+0x59/0x1dc
Jan 26 22:15:18 gitlab kernel: [<ffffffff8818f2e9>] ? do_try_to_free_pages+0x269/0x350
Jan 26 22:15:18 gitlab kernel: [<ffffffff8817a392>] ? oom_kill_process+0x222/0x3e0
Jan 26 22:15:18 gitlab kernel: [<ffffffff8817a8fc>] ? out_of_memory+0x35c/0x3f0
Jan 26 22:15:18 gitlab kernel: [<ffffffff8817f987>] ? __alloc_pages_nodemask+0xd67/0xda0
Jan 26 22:15:18 gitlab kernel: [<ffffffff880a7101>] ? update_curr+0xe1/0x160
Jan 26 22:15:18 gitlab kernel: [<ffffffff881d0221>] ? alloc_pages_current+0x91/0x140
Jan 26 22:15:18 gitlab kernel: [<ffffffffc04fcf5f>] ? vmballoon_work+0x47f/0x704 [vmw_balloon]
Jan 26 22:15:18 gitlab kernel: [<ffffffff8808dc90>] ? process_one_work+0x160/0x410
Jan 26 22:15:18 gitlab kernel: [<ffffffff8808df8d>] ? worker_thread+0x4d/0x480
Jan 26 22:15:18 gitlab kernel: [<ffffffff8808df40>] ? process_one_work+0x410/0x410
Jan 26 22:15:18 gitlab kernel: [<ffffffff88093d5d>] ? kthread+0xcd/0xf0
Jan 26 22:15:18 gitlab kernel: [<ffffffff885e8eef>] ? ret_from_fork+0x1f/0x40
Jan 26 22:15:18 gitlab kernel: [<ffffffff88093c90>] ? kthread_create_on_node+0x190/0x190
Jan 26 22:15:18 gitlab kernel: Mem-Info:
Jan 26 22:15:18 gitlab kernel: active_anon:98986 inactive_anon:102318 isolated_anon:0
active_file:23 inactive_file:47 isolated_file:0
unevictable:0 dirty:0 writeback:0 unstable:0
slab_reclaimable:3358 slab_unreclaimable:9520
mapped:1580 shmem:1984 pagetables:3919 bounce:0
free:14226 free_pcp:30 free_cma:0
Jan 26 22:15:18 gitlab kernel: Node 0 active_anon:395944kB inactive_anon:409272kB active_file:92kB inactive_file:188kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:6320kB dirty:0kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 7936kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
Jan 26 22:15:18 gitlab kernel: Node 0 DMA free:12136kB min:232kB low:288kB high:344kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jan 26 22:15:18 gitlab kernel: lowmem_reserve[]: 0 2976 2976 2976 2976
Jan 26 22:15:18 gitlab kernel: Node 0 DMA32 free:44768kB min:44820kB low:56024kB high:67228kB active_anon:395944kB inactive_anon:409272kB active_file:92kB inactive_file:188kB unevictable:0kB writepending:0kB present:3129280kB managed:3068696kB mlocked:0kB slab_reclaimable:13432kB slab_unreclaimable:38080kB kernel_stack:3648kB pagetables:15676kB bounce:0kB free_pcp:120kB local_pcp:120kB free_cma:0kB
Jan 26 22:15:18 gitlab kernel: lowmem_reserve[]: 0 0 0 0 0
Jan 26 22:15:18 gitlab kernel: Node 0 DMA: 0*4kB 1*8kB (U) 0*16kB 1*32kB (U) 1*64kB (U) 0*128kB 1*256kB (U) 1*512kB (U) 1*1024kB (U) 1*2048kB (M) 2*4096kB (ME) = 12136kB
Jan 26 22:15:18 gitlab kernel: Node 0 DMA32: 10060*4kB (UME) 576*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44848kB
Jan 26 22:15:18 gitlab kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jan 26 22:15:18 gitlab kernel: 12168 total pagecache pages
Jan 26 22:15:18 gitlab kernel: 10120 pages in swap cache
Jan 26 22:15:18 gitlab kernel: Swap cache stats: add 987900, delete 977780, find 2147165/2439256
Jan 26 22:15:18 gitlab kernel: Free swap = 0kB
Jan 26 22:15:18 gitlab kernel: Total swap = 999420kB
Jan 26 22:15:18 gitlab kernel: 786318 pages RAM
Jan 26 22:15:18 gitlab kernel: 0 pages HighMem/MovableOnly
Jan 26 22:15:18 gitlab kernel: 15167 pages reserved
Jan 26 22:15:18 gitlab kernel: 0 pages hwpoisoned
Jan 26 22:15:18 gitlab kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Jan 26 22:15:18 gitlab kernel: [ 253] 0 253 14533 80 24 3 82 0 systemd-journal
Jan 26 22:15:18 gitlab kernel: [ 276] 0 276 26800 0 19 4 85 0 lvmetad
Jan 26 22:15:18 gitlab kernel: [ 280] 0 280 11688 12 24 4 503 -1000 systemd-udevd
Jan 26 22:15:18 gitlab kernel: [ 475] 0 475 11639 32 27 3 129 0 systemd-logind
Jan 26 22:15:18 gitlab kernel: [ 476] 0 476 6991 0 19 4 54 0 atd
Jan 26 22:15:18 gitlab kernel: [ 477] 0 477 62528 62 29 4 226 0 rsyslogd
Jan 26 22:15:18 gitlab kernel: [ 482] 105 482 11278 50 26 3 67 -900 dbus-daemon
Jan 26 22:15:18 gitlab kernel: [ 513] 0 513 6997 32 18 3 46 0 cron
Jan 26 22:15:18 gitlab kernel: [ 530] 0 530 51506 102 57 3 258 0 vmtoolsd
Jan 26 22:15:18 gitlab kernel: [ 537] 0 537 1054 0 8 3 30 0 acpid
Jan 26 22:15:18 gitlab kernel: [ 719] 108 719 5949 14 18 3 90 -500 nrpe
Jan 26 22:15:18 gitlab kernel: [ 728] 0 728 17489 11 39 3 183 -1000 sshd
Jan 26 22:15:18 gitlab kernel: [ 737] 0 737 3662 0 11 3 34 0 agetty
Jan 26 22:15:18 gitlab kernel: [ 756] 110 756 25549 70 26 3 108 0 ntpd
Jan 26 22:15:18 gitlab kernel: [ 765] 0 765 12592 105 32 4 2247 0 munin-node
Jan 26 22:15:18 gitlab kernel: [ 818] 114 818 10828 365 19 3 1130 0 redis-server
Jan 26 22:15:18 gitlab kernel: [ 831] 113 831 68188 329 57 3 369 -900 postgres
Jan 26 22:15:18 gitlab kernel: [ 832] 113 832 72342 376 67 3 402 -900 postgres
Jan 26 22:15:18 gitlab kernel: [ 841] 0 841 20287 17 28 3 185 0 master
Jan 26 22:15:18 gitlab kernel: [ 852] 109 852 20845 20 32 3 199 0 qmgr
Jan 26 22:15:18 gitlab kernel: [ 859] 0 859 24571 26 53 3 453 0 apache2
Jan 26 22:15:18 gitlab kernel: [ 919] 33 919 5393 18 15 3 32 0 htcacheclean
Jan 26 22:15:18 gitlab kernel: [ 922] 113 922 72342 77 54 3 420 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 923] 113 923 72342 82 55 3 414 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 924] 113 924 72342 64 54 3 422 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 925] 113 925 72454 228 57 3 445 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 926] 113 926 36092 47 52 3 412 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 939] 113 939 68221 329 62 3 399 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 940] 113 940 68188 117 50 3 378 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 941] 113 941 68188 61 51 3 388 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 942] 113 942 68282 276 53 3 396 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 943] 113 943 32010 106 46 3 365 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1130] 115 1130 404015 64570 410 6 56881 0 sidekiq
Jan 26 22:15:18 gitlab kernel: [ 1133] 115 1133 160074 7295 269 4 55461 0 unicorn_rails
Jan 26 22:15:18 gitlab kernel: [ 1134] 115 1134 28119 23 58 3 5872 0 mail_room
Jan 26 22:15:18 gitlab kernel: [ 1135] 115 1135 31701 423 21 5 74 0 gitlab-workhors
Jan 26 22:15:18 gitlab kernel: [ 1613] 113 1613 70206 481 59 3 747 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1698] 115 1698 175756 53397 295 4 23918 0 unicorn_rails
Jan 26 22:15:18 gitlab kernel: [ 1701] 115 1701 178468 62348 298 4 16520 0 unicorn_rails
Jan 26 22:15:18 gitlab kernel: [ 1784] 113 1784 104096 4142 135 3 28366 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1793] 113 1793 109673 5080 148 4 33780 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1825] 113 1825 71767 854 71 3 1486 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1826] 113 1826 71698 891 71 3 1367 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1870] 113 1870 72056 853 72 3 1724 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1963] 113 1963 72127 730 72 3 1942 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1975] 113 1975 71909 970 71 3 1503 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1976] 113 1976 72064 759 72 3 1813 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1977] 113 1977 71127 721 70 3 1449 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1978] 113 1978 72159 905 72 3 1775 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1979] 113 1979 72078 929 72 3 1667 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1983] 113 1983 71024 850 70 3 1254 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1990] 113 1990 71668 831 71 3 1890 0 postgres
Jan 26 22:15:18 gitlab kernel: [30182] 33 30182 326710 450 97 5 1507 0 apache2
Jan 26 22:15:18 gitlab kernel: [30183] 33 30183 326630 477 97 4 1356 0 apache2
Jan 26 22:15:18 gitlab kernel: [18263] 109 18263 20803 194 31 3 0 0 pickup
Jan 26 22:15:18 gitlab kernel: Out of memory: Kill process 1130 (sidekiq) score 119 or sacrifice child
Jan 26 22:15:18 gitlab kernel: Killed process 1130 (sidekiq) total-vm:1616060kB, anon-rss:258280kB, file-rss:0kB, shmem-rss:0kB
Jan 26 22:15:18 gitlab kernel: oom_reaper: reaped process 1130 (sidekiq), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Jan 26 22:15:18 gitlab systemd[1]: gitlab-sidekiq.service: Main process exited, code=killed, status=9/KILL
Jan 26 22:15:18 gitlab systemd[1]: gitlab-sidekiq.service: Unit entered failed state.
Jan 26 22:15:18 gitlab systemd[1]: gitlab-sidekiq.service: Failed with result 'signal'.
Jan 26 22:15:18 gitlab systemd[1]: Stopped target GitLab Service.
Jan 26 22:15:18 gitlab systemd[1]: Stopped GitLab Services.
Jan 26 22:15:18 gitlab systemd[1]: Stopping Gitlab mailroom Worker...
Jan 26 22:15:18 gitlab systemd[1]: Stopping Gitlab Workhorse handles slow HTTP requests for Gitlab....
Jan 26 22:15:18 gitlab systemd[1]: Stopped Gitlab Workhorse handles slow HTTP requests for Gitlab..
Jan 26 22:15:19 gitlab systemd[1]: Stopped Gitlab mailroom Worker.
Jan 26 22:15:19 gitlab systemd[1]: Stopping GitLab Unicorn Server...
Jan 26 22:15:20 gitlab systemd[1]: Stopped GitLab Unicorn Server.
Jan 26 22:15:28 gitlab systemd[1]: gitlab-sidekiq.service: Service hold-off time over, scheduling restart.
Jan 26 22:15:28 gitlab systemd[1]: Stopped GitLab Sidekiq Worker.
Jan 26 22:15:28 gitlab systemd[1]: Starting GitLab Sidekiq Worker...
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.243Z 18868 TID-gn57l97t8 INFO: Booting Sidekiq 4.2.3 with redis options {:host=>"localhost", :port=>6379, :password=>nil, :namespace=>"resque:gitlab", :url=>nil}
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.295Z 18868 TID-gn57l97t8 INFO: Cron Jobs - add job with name: stuck_ci_builds_worker
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.301Z 18868 TID-gn57l97t8 INFO: Cron Jobs - add job with name: expire_build_artifacts_worker
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.310Z 18868 TID-gn57l97t8 INFO: Cron Jobs - add job with name: repository_check_worker
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.320Z 18868 TID-gn57l97t8 INFO: Cron Jobs - add job with name: admin_email_worker
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.325Z 18868 TID-gn57l97t8 INFO: Cron Jobs - add job with name: repository_archive_cache_worker
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.330Z 18868 TID-gn57l97t8 INFO: Cron Jobs - add job with name: import_export_project_cleanup_worker
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.334Z 18868 TID-gn57l97t8 INFO: Cron Jobs - add job with name: requests_profiles_worker
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.339Z 18868 TID-gn57l97t8 INFO: Cron Jobs - add job with name: remove_expired_members_worker
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.344Z 18868 TID-gn57l97t8 INFO: Cron Jobs - add job with name: remove_expired_group_links_worker
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.347Z 18868 TID-gn57l97t8 INFO: Cron Jobs - add job with name: prune_old_events_worker
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.354Z 18868 TID-gn57l97t8 INFO: Cron Jobs - add job with name: trending_projects_worker
Jan 26 22:16:01 gitlab systemd[1]: Started GitLab Sidekiq Worker.
Jan 26 22:16:04 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:16:04.347Z 18868 TID-gn57l97t8 INFO: Running in ruby 2.3.3p222 (2016-11-21) [x86_64-linux-gnu]
Jan 26 22:16:04 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:16:04.347Z 18868 TID-gn57l97t8 INFO: See LICENSE and the LGPL-3.0 for licensing details.
Jan 26 22:16:04 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:16:04.347Z 18868 TID-gn57l97t8 INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org
Jan 26 22:16:04 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:16:04.354Z 18868 TID-gn57l97t8 INFO: Starting processing, hit Ctrl-C to stop
Jan 26 22:20:05 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:20:05.289Z 18868 TID-gn58yenr0 RepositoryCheck::BatchWorker JID-5d784fb4b3f6f9c8f0381712 INFO: start
Jan 26 22:20:05 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:20:05.292Z 18868 TID-gn58ydpxs INFO: Cron Jobs - add job with name: repository_check_worker
Jan 26 22:20:05 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:20:05.325Z 18868 TID-gn58yenr0 RepositoryCheck::BatchWorker JID-5d784fb4b3f6f9c8f0381712 INFO: done: 0.036 sec
More information about the Pkg-ruby-extras-maintainers
mailing list