[DRE-maint] Bug#1015301: gitlab: Upgrade to 15.0.4 fails because BackgroundMigrations could not finalize

[Maximilian Stein]

Package: gitlab
Version: 15.0.4+ds1-1~fto11+2
Severity: important

Dear Maintainer,

I just tried to upgrade to Gitlab 15.0.4 on my two instances. While I
had no issues in my smaller instance, my bigger one had some issues.

First, postinst failed with the message

    /etc/systemd/system/gitaly.service.d/override.conf already exist

I moved the file away and the upgrade could continue. At the end of
the upgrade, gitaly did not start anymore, so I restored the file (it
actually contains the setting of the user running gitaly).

Then, however, I stumbled upon a much more serious issue: The database
migration could not finish as there were pending background jobs that
failed to finalize:

    gitlab_production database is not empty, skipping gitlab setup
    Attention: used pure ruby version of MurmurHash3
    /usr/share/gitlab/lib/gitlab.rb:47: warning: already initialized constant Gitlab::APP_DIRS_PATTERN
    /usr/share/gitlab/lib/gitlab.rb:47: warning: previous definition of APP_DIRS_PATTERN was here
    /usr/share/gitlab/lib/gitlab.rb:48: warning: already initialized constant Gitlab::VERSION
    /usr/share/gitlab/lib/gitlab.rb:48: warning: previous definition of VERSION was here
    /usr/share/gitlab/lib/gitlab.rb:49: warning: already initialized constant Gitlab::INSTALLATION_TYPE
    /usr/share/gitlab/lib/gitlab.rb:49: warning: previous definition of INSTALLATION_TYPE was here
    /usr/share/gitlab/lib/gitlab.rb:50: warning: already initialized constant Gitlab::HTTP_PROXY_ENV_VARS
    /usr/share/gitlab/lib/gitlab.rb:50: warning: previous definition of HTTP_PROXY_ENV_VARS was here
    == 20220213103859 RemoveIntegrationsType: migrating ===========================
    rake aborted!
    StandardError: An error has occurred, all later migrations canceled:

    Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize
    …

I did some research, and found somebody else having a similar issue
[1]. Unfortunately, manually running the background jobs [2] did not
work either as they continued to fail. In the database I identified
the following stuck background migrations [3]:


    gitlab_production=# select id,status,job_class_name, table_name, column_name, job_arguments from batched_background_migrations where status <> 3;
     id | status |                  job_class_name                  |  table_name  | column_name | job_arguments 
    ----+--------+--------------------------------------------------+--------------+-------------+---------------
     17 |      4 | BackfillNamespaceIdForNamespaceRoute             | routes       | id          | []
     19 |      4 | BackfillMemberNamespaceForGroupMembers           | members      | id          | []
     20 |      4 | MigratePersonalNamespaceProjectMaintainerToOwner | members      | id          | []
     23 |      4 | BackfillGroupFeatures                            | namespaces   | id          | [10000]
     15 |      4 | BackfillIntegrationsTypeNew                      | integrations | id          | []
     16 |      4 | BackfillUserNamespace                            | namespaces   | id          | []
     18 |      4 | BackfillIssueSearchData                          | issues       | id          | []


I then proceeded to simply change the status of the jobs to 3 in the
database as proposed in the issue mentioned above [1]. I could then
finish the upgrade normally with `apt upgrade`.

Afterwards, I simply undid the database change (i.e., reverted the
status of the failed background migrations to 4) and then restarted
the migrations in the web UI.

As far as I can tell, everything seems normal now. The seven
background migration jobs are still pending, but I can continue to use
Gitlab normally.

Do you need any more information on the matter?

Thanks for your investigation into the issue!

Best,
Maximilian


[1]: https://gitlab.com/gitlab-org/gitlab/-/issues/340193
[2]: https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#manually-finishing-a-batched-background-migration
[3]: https://docs.gitlab.com/ee/update/index.html#batched-background-migrations