[DRE-maint] Bug#814714: gitlab: no http interface and no error in logs

Sun Feb 14 15:33:23 UTC 2016

Package: gitlab
Version: 8.4.3+dfsg-6
Severity: grave
Justification: renders package unusable

Hi,

this problem appeared after my upgrade to 8.4.3+dfsg-6 without me
changing anything in my configuration, so I assume that this must be
gitlab's fault, thus the high severity.

The symptom is, that since after the upgrade (and after manually
creating the /var/run/gitlab directories as I described in #814713), I
only get the following message in my browser when trying to access the
website:

                                   502
            Whoops, GitLab is taking too much time to respond.

   Try refreshing the page, or going back and attempting the action again.

     Please contact your GitLab administrator if this problem persists.

Now I *am* the gitlab administrator and I fail to figure out how to find
the source of the problem. My logs show the following:

	root at 22gitlab:~# tail -n10000 /usr/share/gitlab/log/* /var/log/gitlab/*        
	==> /usr/share/gitlab/log/mail_room.log <==
	fatal: Not a git repository (or any of the parent directories): .git

	==> /usr/share/gitlab/log/production.log <==
	** [Raven] Raven 0.15.3 configured not to send errors.
	** [Raven] Raven 0.15.3 configured not to send errors.
	** [Raven] Raven 0.15.3 configured not to send errors.
	** [Raven] Raven 0.15.3 configured not to send errors.
	Scoped order and limit are ignored, it's forced to be batch order and batch size
	** [Raven] Raven 0.15.3 configured not to send errors.
	Scoped order and limit are ignored, it's forced to be batch order and batch size

	==> /var/log/gitlab/sidekiq.log <==
	No such file or directory @ rb_sysopen - /var/run/gitlab/pids/sidekiq.pid
	/usr/lib/ruby/vendor_ruby/sidekiq/cli.rb:355:in `initialize'
	/usr/lib/ruby/vendor_ruby/sidekiq/cli.rb:355:in `open'
	/usr/lib/ruby/vendor_ruby/sidekiq/cli.rb:355:in `write_pid'
	/usr/lib/ruby/vendor_ruby/sidekiq/cli.rb:42:in `parse'
	/usr/bin/sidekiq:12:in `<main>'
	Pidfile /var/run/gitlab/pids/sidekiq.pid does not exist
	2016-02-14T15:21:41.835Z 293 TID-b4ak0 INFO: Booting Sidekiq 4.0.1 with redis options {:url=>"redis://localhost:6379", :namespace=>"resque:gitlab"}
	2016-02-14T15:21:41.983Z 293 TID-b4ak0 INFO: Cron Jobs - add job with name: stuck_ci_builds_worker
	fatal: Not a git repository (or any of the parent directories): .git
	2016-02-14T15:21:57.436Z 293 TID-b4ak0 INFO: Running in ruby 2.2.4p230 (2015-12-16) [x86_64-linux-gnu]
	2016-02-14T15:21:57.436Z 293 TID-b4ak0 INFO: See LICENSE and the LGPL-3.0 for licensing details.
	2016-02-14T15:21:57.436Z 293 TID-b4ak0 INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org

	==> /var/log/gitlab/unicorn.stderr.log <==
	I, [2016-02-14T16:21:07.730554 #287]  INFO -- : Refreshing Gem list
	fatal: Not a git repository (or any of the parent directories): .git
	I, [2016-02-14T16:21:58.169242 #287]  INFO -- : listening on addr=/var/run/gitlab/pids/gitlab.socket fd=12
	I, [2016-02-14T16:21:58.170435 #287]  INFO -- : listening on addr=127.0.0.1:8080 fd=15
	I, [2016-02-14T16:21:58.195925 #346]  INFO -- : worker=0 ready
	I, [2016-02-14T16:21:58.205334 #349]  INFO -- : worker=1 ready
	I, [2016-02-14T16:21:58.220556 #287]  INFO -- : master process ready
	I, [2016-02-14T16:21:58.223929 #352]  INFO -- : worker=2 ready

	==> /var/log/gitlab/unicorn.stdout.log <==

Does there exist a way to increase the verbosity of the logging so that
I can see what causes the 502 problem? Here is the (shortened, because I
have lots of repositories) output of gitlab:check:

	fatal: Not a git repository (or any of the parent directories): .git
	Checking GitLab Shell ...

	GitLab Shell version >= 2.6.10 ? ... OK (2.6.10)
	Repo base directory exists? ... yes
	Repo base directory is a symlink? ... no
	Repo base owned by gitlab:gitlab? ... yes
	Repo base access is drwxrws---? ... yes
	hooks directories in repos are links: ... 
	[...]
	Running /usr/share/gitlab-shell/bin/check
	Check GitLab API access: FAILED. code: 502
	gitlab-shell self-check failed
	  Try fixing it:
	  Make sure GitLab is running;
	  Check the gitlab-shell configuration file:
	  sudo -u gitlab -H editor /usr/share/gitlab-shell/config.yml
	  Please fix the error above and rerun the checks.

	Checking GitLab Shell ... Finished

	Checking Sidekiq ...

	Running? ... yes
	Number of Sidekiq processes ... 1

	Checking Sidekiq ... Finished

	Checking Reply by email ...

	Reply by email is disabled in config/gitlab.yml

	Checking Reply by email ... Finished

	Checking LDAP ...

	LDAP is disabled in config/gitlab.yml

	Checking LDAP ... Finished

	Checking GitLab ...

	Git configured with autocrlf=input? ... yes
	Database config exists? ... yes
	Database is SQLite ... no
	All migrations up? ... yes
	Database contains orphaned GroupMembers? ... no
	GitLab config exists? ... yes
	GitLab config outdated? ... no
	Log directory writable? ... yes
	Tmp directory writable? ... yes
	Uploads directory setup correctly? ... skipped (no tmp uploads folder yet)
	Init script exists? ... yes
	Init script up-to-date? ... yes
	projects have namespace: ... 
	[...]
	Redis version >= 2.8.0? ... yes
	Ruby version >= 2.1.0 ? ... yes (2.2.4)
	Your git bin path is "/usr/bin/git"
	Git version >= 1.7.10 ? ... yes (2.1.4)
	Active users: 16

	Checking GitLab ... Finished

So this is all green except for the gitlab-shell/bin/check output. But I
guess it's expected that this is red because I cannot access the web
interface through my browser either, right?

I'd appreciate some advise in how to tell you more about the reasons for
this problem.

Thanks!

cheers, josch