mirror of https://github.com/go-gitea/gitea.git
e28cc79c92
This addresses https://github.com/go-gitea/gitea/issues/18352 It aims to improve performance (and resource use) of the `SyncReleasesWithTags` operation for pull-mirrors. For large repositories with many tags, `SyncReleasesWithTags` can be a costly operation (taking several minutes to complete). The reason is two-fold: 1. on sync, every upstream repo tag is compared (for changes) against existing local entries in the release table to ensure that they are up-to-date. 2. the procedure for getting _each tag_ involves a series of git operations ```bash git show-ref --tags -- v8.2.4477 git cat-file -t 29ab6ce9f36660cffaad3c8789e71162e5db5d2f git cat-file -p 29ab6ce9f36660cffaad3c8789e71162e5db5d2f git rev-list --count 29ab6ce9f36660cffaad3c8789e71162e5db5d2f ``` of which the `git rev-list --count` can be particularly heavy. This PR optimizes performance for pull-mirrors. We utilize the fact that a pull-mirror is always identical to its upstream and rebuild the entire release table on every sync and use a batch `git for-each-ref .. refs/tags` call to retrieve all tags in one go. For large mirror repos, with hundreds of annotated tags, this brings down the duration of the sync operation from several minutes to a few seconds. A few unscientific examples run on my local machine: - https://github.com/spring-projects/spring-boot (223 tags) - before: `0m28,673s` - after: `0m2,244s` - https://github.com/kubernetes/kubernetes (890 tags) - before: `8m00s` - after: `0m8,520s` - https://github.com/vim/vim (13954 tags) - before: `14m20,383s` - after: `0m35,467s` I added a `foreachref` package which contains a flexible way of specifying which reference fields are of interest (`git-for-each-ref(1)`) and to produce a parser for the expected output. These could be reused in other places where `for-each-ref` is used. I'll add unit tests for those if the overall PR looks promising. |
||
|---|---|---|
| .. | ||
| activitypub | ||
| analyze | ||
| appstate | ||
| auth | ||
| avatar | ||
| base | ||
| cache | ||
| charset | ||
| container | ||
| context | ||
| convert | ||
| csv | ||
| doctor | ||
| emoji | ||
| eventsource | ||
| generate | ||
| git | ||
| gitgraph | ||
| graceful | ||
| hcaptcha | ||
| highlight | ||
| hostmatcher | ||
| httpcache | ||
| httplib | ||
| indexer | ||
| json | ||
| lfs | ||
| log | ||
| markup | ||
| metrics | ||
| migration | ||
| nosql | ||
| notification | ||
| options | ||
| packages | ||
| password | ||
| pprof | ||
| private | ||
| process | ||
| proxy | ||
| public | ||
| queue | ||
| recaptcha | ||
| references | ||
| repository | ||
| secret | ||
| session | ||
| setting | ||
| ssh | ||
| storage | ||
| structs | ||
| svg | ||
| sync | ||
| templates | ||
| test | ||
| timeutil | ||
| translation | ||
| typesniffer | ||
| updatechecker | ||
| upload | ||
| uri | ||
| user | ||
| util | ||
| validation | ||
| web | ||