How to fix failed Gitlab repository checks?

Question

Today I got notified that repository checks on our Gitlab server have failed:

18 projects failed their last repository check.

View details:
http⁠s://git.example.org/admin/projects?last_repository_check_failed=1
You are receiving this message because you are a GitLab administrator for http⁠s://git.example.org.

Looking at /var/log/gitlab/gitlab-rails/repocheck.log I saw errors like the following for various repositories:

E, [2022-08-26T06:25:57.986682 #10920] ERROR -- : mirror/puppetlabs-stdlib: Could not fsck repository: error: Could not read b1ae33a87b6bd7d966b573d8b65c277a0f531cdc
failed to parse commit b1ae33a87b6bd7d966b573d8b65c277a0f531cdc from object database for commit-graph

The help page explains how to manually run the check on the command line, by which I can reproduce the error message, and also how to clear check results from scheduled runs. However, after clearing the errors and manually triggering the checks most of the errors re-appeared (which is probably to be expected), and the help page does not explain how to actually fix the errors.

Answer 1

According to the error message there is an unreachable object in the database, maybe a result of an upstream force-push. The error should be cleaned up automatically after a while, due to Gitlab's automatic housekeeping. If you don't want to wait that long, you can run the housekeeping task manually on the faulty repositories:

First identify the Gitaly relative path for the faulty repository:

• Web UI: In the admin area go to Overview → Projects and select the project in question. The path is displayed in the field "Gitaly relative path" in the "Project info" box.

• Command line: Start a Rails console as root (this may take a long time, so be patient).

  gitlab-rails console
  

  After the prompt appears enter one of the following commands:

  Project.find(42).disk_path                                      # find by project ID
  Project.find_by_full_path('GROUP_NAME/PROJECT_NAME').disk_path  # find by project name
  

  Alternatively you can use the runner subcommand for non-interactive execution. In that case you need to prefix the command with puts or pp ("pretty-print") to actually display the returned value.

  gitlab-rails runner "puts Project.find(42).disk_path"
  

Once you have determined the Gitaly path log into your Gitlab server (if you haven't already) and run the following command as root:

/opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/c0/ff/eed0...0d.git

Replace the example relative path (@hashed/c0/ff/eed0...0d.git) with your actual relative path.