Guidelines

This site is for tech Q&A. Please keep your posts focused on the subject at hand.

Ask one question at a time. Don't conflate multiple problems into a single question.

Make sure to include all relevant information in your posts. Try to avoid linking to external sites.

Links to documentation are fine, but in addition you should also quote the relevant parts in your posts.

0 votes
101 views
101 views

Today I got notified that repository checks on our Gitlab server have failed:

18 projects failed their last repository check.

View details:
http⁠s://git.example.org/admin/projects?last_repository_check_failed=1
You are receiving this message because you are a GitLab administrator for http⁠s://git.example.org.

Looking at /var/log/gitlab/gitlab-rails/repocheck.log I saw errors like the following for various repositories:

E, [2022-08-26T06:25:57.986682 #10920] ERROR -- : mirror/puppetlabs-stdlib: Could not fsck repository: error: Could not read b1ae33a87b6bd7d966b573d8b65c277a0f531cdc
failed to parse commit b1ae33a87b6bd7d966b573d8b65c277a0f531cdc from object database for commit-graph

The help page explains how to manually run the check on the command line, by which I can reproduce the error message, and also how to clear check results from scheduled runs. However, after clearing the errors and manually triggering the checks most of the errors re-appeared (which is probably to be expected), and the help page does not explain how to actually fix the errors.

in Sysadmin
edited by
by (100)
1 6 20
edit history

Please log in or register to answer this question.

1 Answer

0 votes
 

According to the error message there is an unreachable object in the database, maybe a result of an upstream force-push. The error should be cleaned up automatically after a while, due to Gitlab's automatic housekeeping. If you don't want to wait that long, you can run the housekeeping task manually on the faulty repositories:

First identify the Gitaly relative path for the faulty repository:

  • Web UI: In the admin area go to Overview → Projects and select the project in question. The path is displayed in the field "Gitaly relative path" in the "Project info" box.

  • Command line: Start a Rails console as root (this may take a long time, so be patient).

    gitlab-rails console
    

    After the prompt appears enter one of the following commands:

    Project.find(42).disk_path                                      # find by project ID
    Project.find_by_full_path('GROUP_NAME/PROJECT_NAME').disk_path  # find by project name
    

    Alternatively you can use the runner subcommand for non-interactive execution. In that case you need to prefix the command with puts or pp ("pretty-print") to actually display the returned value.

    gitlab-rails runner "puts Project.find(42).disk_path"
    

Once you have determined the Gitaly path log into your Gitlab server (if you haven't already) and run the following command as root:

/opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/c0/ff/eed0...0d.git

Replace the example relative path (@hashed/c0/ff/eed0...0d.git) with your actual relative path.


edited by
by (100)
1 6 20
edit history
...