Mirrors
/
gitea
mirror of https://github.com/go-gitea/gitea.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add push to remote mirror repository (#15157) 

* Added push mirror model.

* Integrated push mirror into queue.

* Moved methods into own file.

* Added basic implementation.

* Mirror wiki too.

* Removed duplicated method.

* Get url for different remotes.

* Added migration.

* Unified remote url access.

* Add/Remove push mirror remotes.

* Prevent hangs with missing credentials.

* Moved code between files.

* Changed sanitizer interface.

* Added push mirror backend methods.

* Only update the mirror remote.

* Limit refs on push.

* Added UI part.

* Added missing table.

* Delete mirror if repository gets removed.

* Changed signature. Handle object errors.

* Added upload method.

* Added "upload" unit tests.

* Added transfer adapter unit tests.

* Send correct headers.

* Added pushing of LFS objects.

* Added more logging.

* Simpler body handling.

* Process files in batches to reduce HTTP calls.

* Added created timestamp.

* Fixed invalid column name.

* Changed name to prevent xorm auto setting.

* Remove table header im empty.

* Strip exit code from error message.

* Added docs page about mirroring.

* Fixed date.

* Fixed merge errors.

* Moved test to integrations.

* Added push mirror test.

* Added test.
This commit is contained in:
KN4CK3R 2021-06-14 19:20:43 +02:00 committed by GitHub
parent 5d113bdd19
commit 440039c0cc
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
39 changed files with 2468 additions and 885 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

88
docs/content/doc/advanced/repo-mirror.en-us.md Normal file
View File

@ -0,0 +1,88 @@
---
date: "2021-05-13T00:00:00-00:00"
title: "Repository Mirror"
slug: "repo-mirror"
weight: 45
toc: false
draft: false
menu:
sidebar:
parent: "advanced"
name: "Repository Mirror"
weight: 45
identifier: "repo-mirror"
---
# Repository Mirror
Repository mirroring allows for the mirroring of repositories to and from external sources. You can use it to mirror branches, tags, and commits between repositories.
**Table of Contents**
{{< toc >}}
## Use cases
The following are some possible use cases for repository mirroring:
- You migrated to Gitea but still need to keep your project in another source. In that case, you can simply set it up to mirror to Gitea (pull) and all the essential history of commits, tags, and branches are available in your Gitea instance.
- You have old projects in another source that you don’t use actively anymore, but don’t want to remove for archiving purposes. In that case, you can create a push mirror so that your active Gitea repository can push its changes to the old location.
## Pulling from a remote repository
For an existing remote repository, you can set up pull mirroring as follows:
1. Select **New Migration** in the **Create...** menu on the top right.
2. Select the remote repository service.
3. Enter a repository URL.
4. If the repository needs authentication fill in your authentication information.
5. Check the box **This repository will be a mirror**.
5. Select **Migrate repository** to save the configuration.
The repository now gets mirrored periodically from the remote repository. You can force a sync by selecting **Synchronize Now** in the repository settings.
## Pushing to a remote repository
For an existing repository, you can set up push mirroring as follows:
1. In your repository, go to **Settings** > **Repository**, and then the **Mirror Settings** section.
2. Enter a repository URL.
3. If the repository needs authentication expand the **Authorization** section and fill in your authentication information.
4. Select **Add Push Mirror** to save the configuration.
The repository now gets mirrored periodically to the remote repository. You can force a sync by selecting **Synchronize Now**. In case of an error a message displayed to help you resolve it.
:exclamation::exclamation: **NOTE:** This will force push to the remote repository. This will overwrite any changes in the remote repository! :exclamation::exclamation:
### Setting up a push mirror from Gitea to GitHub
To set up a mirror from Gitea to GitHub, you need to follow these steps:
1. Create a [GitHub personal access token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) with the *public_repo* box checked.
2. Fill in the **Git Remote Repository URL**: `https://github.com/<your_github_group>/<your_github_project>.git`.
3. Fill in the **Authorization** fields with your GitHub username and the personal access token.
4. Select **Add Push Mirror** to save the configuration.
The repository pushes shortly thereafter. To force a push, select the **Synchronize Now** button.
### Setting up a push mirror from Gitea to GitLab
To set up a mirror from Gitea to GitLab, you need to follow these steps:
1. Create a [GitLab personal access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) with *write_repository* scope.
2. Fill in the **Git Remote Repository URL**: `https://<destination host>/<your_gitlab_group_or_name>/<your_gitlab_project>.git`.
3. Fill in the **Authorization** fields with `oauth2` as **Username** and your GitLab personal access token as **Password**.
4. Select **Add Push Mirror** to save the configuration.
The repository pushes shortly thereafter. To force a push, select the **Synchronize Now** button.
### Setting up a push mirror from Gitea to Bitbucket
To set up a mirror from Gitea to Bitbucket, you need to follow these steps:
1. Create a [Bitbucket app password](https://support.atlassian.com/bitbucket-cloud/docs/app-passwords/) with the *Repository Write* box checked.
2. Fill in the **Git Remote Repository URL**: `https://bitbucket.org/<your_bitbucket_group_or_name>/<your_bitbucket_project>.git`.
3. Fill in the **Authorization** fields with your Bitbucket username and the app password as **Password**.
4. Select **Add Push Mirror** to save the configuration.
The repository pushes shortly thereafter. To force a push, select the **Synchronize Now** button.

16
services/mirror/mirror_test.go → integrations/mirror_pull_test.go
View File

@ -2,28 +2,24 @@
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package mirror
package integrations
import (
"context"
"path/filepath"
"testing"
"code.gitea.io/gitea/models"
"code.gitea.io/gitea/modules/git"
migration "code.gitea.io/gitea/modules/migrations/base"
"code.gitea.io/gitea/modules/repository"
mirror_service "code.gitea.io/gitea/services/mirror"
release_service "code.gitea.io/gitea/services/release"
"github.com/stretchr/testify/assert"
)
func TestMain(m *testing.M) {
models.MainTest(m, filepath.Join("..", ".."))
}
func TestRelease_MirrorDelete(t *testing.T) {
assert.NoError(t, models.PrepareTestDatabase())
func TestMirrorPull(t *testing.T) {
defer prepareTestEnv(t)()
user := models.AssertExistsAndLoadBean(t, &models.User{ID: 2}).(*models.User)
repo := models.AssertExistsAndLoadBean(t, &models.Repository{ID: 1}).(*models.Repository)
@ -76,7 +72,7 @@ func TestRelease_MirrorDelete(t *testing.T) {
err = mirror.GetMirror()
assert.NoError(t, err)
_, ok := runSync(ctx, mirror.Mirror)
ok := mirror_service.SyncPullMirror(ctx, mirror.ID)
assert.True(t, ok)
count, err := models.GetReleaseCountByRepoID(mirror.ID, findOptions)
@ -87,7 +83,7 @@ func TestRelease_MirrorDelete(t *testing.T) {
assert.NoError(t, err)
assert.NoError(t, release_service.DeleteReleaseByID(release.ID, user, true))
_, ok = runSync(ctx, mirror.Mirror)
ok = mirror_service.SyncPullMirror(ctx, mirror.ID)
assert.True(t, ok)
count, err = models.GetReleaseCountByRepoID(mirror.ID, findOptions)

86
integrations/mirror_push_test.go Normal file
View File

@ -0,0 +1,86 @@
// Copyright 2021 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package integrations
import (
"context"
"fmt"
"net/http"
"net/url"
"testing"
"code.gitea.io/gitea/models"
"code.gitea.io/gitea/modules/git"
"code.gitea.io/gitea/modules/repository"
"code.gitea.io/gitea/modules/setting"
mirror_service "code.gitea.io/gitea/services/mirror"
"github.com/stretchr/testify/assert"
)
func TestMirrorPush(t *testing.T) {
onGiteaRun(t, testMirrorPush)
}
func testMirrorPush(t *testing.T, u *url.URL) {
defer prepareTestEnv(t)()
setting.Migrations.AllowLocalNetworks = true
user := models.AssertExistsAndLoadBean(t, &models.User{ID: 2}).(*models.User)
srcRepo := models.AssertExistsAndLoadBean(t, &models.Repository{ID: 1}).(*models.Repository)
mirrorRepo, err := repository.CreateRepository(user, user, models.CreateRepoOptions{
Name: "test-push-mirror",
})
assert.NoError(t, err)
ctx := NewAPITestContext(t, user.LowerName, srcRepo.Name)
doCreatePushMirror(ctx, fmt.Sprintf("%s%s/%s", u.String(), url.PathEscape(ctx.Username), url.PathEscape(mirrorRepo.Name)), user.LowerName, userPassword)(t)
mirrors, err := models.GetPushMirrorsByRepoID(srcRepo.ID)
assert.NoError(t, err)
assert.Len(t, mirrors, 1)
ok := mirror_service.SyncPushMirror(context.Background(), mirrors[0].ID)
assert.True(t, ok)
srcGitRepo, err := git.OpenRepository(srcRepo.RepoPath())
assert.NoError(t, err)
defer srcGitRepo.Close()
srcCommit, err := srcGitRepo.GetBranchCommit("master")
assert.NoError(t, err)
mirrorGitRepo, err := git.OpenRepository(mirrorRepo.RepoPath())
assert.NoError(t, err)
defer mirrorGitRepo.Close()
mirrorCommit, err := mirrorGitRepo.GetBranchCommit("master")
assert.NoError(t, err)
assert.Equal(t, srcCommit.ID, mirrorCommit.ID)
}
func doCreatePushMirror(ctx APITestContext, address, username, password string) func(t *testing.T) {
return func(t *testing.T) {
csrf := GetCSRF(t, ctx.Session, fmt.Sprintf("/%s/%s/settings", url.PathEscape(ctx.Username), url.PathEscape(ctx.Reponame)))
req := NewRequestWithValues(t, "POST", fmt.Sprintf("/%s/%s/settings", url.PathEscape(ctx.Username), url.PathEscape(ctx.Reponame)), map[string]string{
"_csrf": csrf,
"action": "push-mirror-add",
"push_mirror_address": address,
"push_mirror_username": username,
"push_mirror_password": password,
"push_mirror_interval": "0",
})
ctx.Session.MakeRequest(t, req, http.StatusFound)
flashCookie := ctx.Session.GetCookie("macaron_flash")
assert.NotNil(t, flashCookie)
assert.Contains(t, flashCookie.Value, "success")
}
}

2
models/migrations/migrations.go
View File

@ -315,6 +315,8 @@ var migrations = []Migration{
NewMigration("Always save primary email on email address table", addPrimaryEmail2EmailAddress),
// v182 -> v183
NewMigration("Add issue resource index table", addIssueResourceIndexTable),
// v183 -> v184
NewMigration("Create PushMirror table", createPushMirrorTable),
}
// GetCurrentDBVersion returns the current db version

2
models/migrations/v180.go
View File

@ -64,7 +64,7 @@ func removeCredentials(payload string) (string, error) {
opts.AuthPassword = ""
opts.AuthToken = ""
opts.CloneAddr = util.SanitizeURLCredentials(opts.CloneAddr, true)
opts.CloneAddr = util.NewStringURLSanitizer(opts.CloneAddr, true).Replace(opts.CloneAddr)
confBytes, err := json.Marshal(opts)
if err != nil {

39
models/migrations/v183.go Normal file
View File

@ -0,0 +1,39 @@
// Copyright 2021 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package migrations
import (
"fmt"
"time"
"code.gitea.io/gitea/modules/timeutil"
"xorm.io/xorm"
)
func createPushMirrorTable(x *xorm.Engine) error {
type PushMirror struct {
ID int64 `xorm:"pk autoincr"`
RepoID int64 `xorm:"INDEX"`
RemoteName string
Interval time.Duration
CreatedUnix timeutil.TimeStamp `xorm:"created"`
LastUpdateUnix timeutil.TimeStamp `xorm:"INDEX last_update"`
LastError string `xorm:"text"`
}
sess := x.NewSession()
defer sess.Close()
if err := sess.Begin(); err != nil {
return err
}
if err := sess.Sync2(new(PushMirror)); err != nil {
return fmt.Errorf("Sync2: %v", err)
}
return sess.Commit()
}

1
models/models.go
View File

@ -135,6 +135,7 @@ func init() {
new(Session),
new(RepoTransfer),
new(IssueIndex),
new(PushMirror),
)
gonicNames := []string{"SSL", "UID"}

15
models/repo.go
View File

@ -221,6 +221,7 @@ type Repository struct {
IsArchived bool `xorm:"INDEX"`
IsMirror bool `xorm:"INDEX"`
*Mirror `xorm:"-"`
PushMirrors []*PushMirror `xorm:"-"`
Status RepositoryStatus `xorm:"NOT NULL DEFAULT 0"`
RenderingMetas map[string]string `xorm:"-"`
@ -255,7 +256,12 @@ func (repo *Repository) SanitizedOriginalURL() string {
if repo.OriginalURL == "" {
return ""
}
return util.SanitizeURLCredentials(repo.OriginalURL, false)
u, err := url.Parse(repo.OriginalURL)
if err != nil {
return ""
}
u.User = nil
return u.String()
}
// ColorFormat returns a colored string to represent this repo
@ -657,6 +663,12 @@ func (repo *Repository) GetMirror() (err error) {
return err
}
// LoadPushMirrors populates the repository push mirrors.
func (repo *Repository) LoadPushMirrors() (err error) {
repo.PushMirrors, err = GetPushMirrorsByRepoID(repo.ID)
return err
}
// GetBaseRepo populates repo.BaseRepo for a fork repository and
// returns an error on failure (NOTE: no error is returned for
// non-fork repositories, and BaseRepo will be left untouched)
@ -1487,6 +1499,7 @@ func DeleteRepository(doer *User, uid, repoID int64) error {
&Notification{RepoID: repoID},
&ProtectedBranch{RepoID: repoID},
&PullRequest{BaseRepoID: repoID},
&PushMirror{RepoID: repoID},
&Release{RepoID: repoID},
&RepoIndexerStatus{RepoID: repoID},
&RepoRedirect{RedirectRepoID: repoID},

16
models/repo_mirror.go
View File

@ -14,6 +14,12 @@ import (
"xorm.io/xorm"
)
// RemoteMirrorer defines base methods for pull/push mirrors.
type RemoteMirrorer interface {
GetRepository() *Repository
GetRemoteName() string
}
// Mirror represents mirror information of a repository.
type Mirror struct {
ID int64 `xorm:"pk autoincr"`
@ -52,6 +58,16 @@ func (m *Mirror) AfterLoad(session *xorm.Session) {
}
}
// GetRepository returns the repository.
func (m *Mirror) GetRepository() *Repository {
return m.Repo
}
// GetRemoteName returns the name of the remote.
func (m *Mirror) GetRemoteName() string {
return "origin"
}
// ScheduleNextUpdate calculates and sets next update time.
func (m *Mirror) ScheduleNextUpdate() {
if m.Interval != 0 {

106
models/repo_pushmirror.go Normal file
View File

@ -0,0 +1,106 @@
// Copyright 2021 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package models
import (
"errors"
"time"
"code.gitea.io/gitea/modules/log"
"code.gitea.io/gitea/modules/timeutil"
"xorm.io/xorm"
)
var (
// ErrPushMirrorNotExist mirror does not exist error
ErrPushMirrorNotExist = errors.New("PushMirror does not exist")
)
// PushMirror represents mirror information of a repository.
type PushMirror struct {
ID int64 `xorm:"pk autoincr"`
RepoID int64 `xorm:"INDEX"`
Repo *Repository `xorm:"-"`
RemoteName string
Interval time.Duration
CreatedUnix timeutil.TimeStamp `xorm:"created"`
LastUpdateUnix timeutil.TimeStamp `xorm:"INDEX last_update"`
LastError string `xorm:"text"`
}
// AfterLoad is invoked from XORM after setting the values of all fields of this object.
func (m *PushMirror) AfterLoad(session *xorm.Session) {
if m == nil {
return
}
var err error
m.Repo, err = getRepositoryByID(session, m.RepoID)
if err != nil {
log.Error("getRepositoryByID[%d]: %v", m.ID, err)
}
}
// GetRepository returns the path of the repository.
func (m *PushMirror) GetRepository() *Repository {
return m.Repo
}
// GetRemoteName returns the name of the remote.
func (m *PushMirror) GetRemoteName() string {
return m.RemoteName
}
// InsertPushMirror inserts a push-mirror to database
func InsertPushMirror(m *PushMirror) error {
_, err := x.Insert(m)
return err
}
// UpdatePushMirror updates the push-mirror
func UpdatePushMirror(m *PushMirror) error {
_, err := x.ID(m.ID).AllCols().Update(m)
return err
}
// DeletePushMirrorByID deletes a push-mirrors by ID
func DeletePushMirrorByID(ID int64) error {
_, err := x.ID(ID).Delete(&PushMirror{})
return err
}
// DeletePushMirrorsByRepoID deletes all push-mirrors by repoID
func DeletePushMirrorsByRepoID(repoID int64) error {
_, err := x.Delete(&PushMirror{RepoID: repoID})
return err
}
// GetPushMirrorByID returns push-mirror information.
func GetPushMirrorByID(ID int64) (*PushMirror, error) {
m := &PushMirror{}
has, err := x.ID(ID).Get(m)
if err != nil {
return nil, err
} else if !has {
return nil, ErrPushMirrorNotExist
}
return m, nil
}
// GetPushMirrorsByRepoID returns push-mirror informations of a repository.
func GetPushMirrorsByRepoID(repoID int64) ([]*PushMirror, error) {
mirrors := make([]*PushMirror, 0, 10)
return mirrors, x.Where("repo_id=?", repoID).Find(&mirrors)
}
// PushMirrorsIterate iterates all push-mirror repositories.
func PushMirrorsIterate(f func(idx int, bean interface{}) error) error {
return x.
Where("last_update + (`interval` / ?) <= ?", time.Second, time.Now().Unix()).
And("`interval` != 0").
Iterate(new(PushMirror), f)
}

49
models/repo_pushmirror_test.go Normal file
View File

@ -0,0 +1,49 @@
// Copyright 2021 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package models
import (
"testing"
"time"
"code.gitea.io/gitea/modules/timeutil"
"github.com/stretchr/testify/assert"
)
func TestPushMirrorsIterate(t *testing.T) {
assert.NoError(t, PrepareTestDatabase())
now := timeutil.TimeStampNow()
InsertPushMirror(&PushMirror{
RemoteName: "test-1",
LastUpdateUnix: now,
Interval: 1,
})
long, _ := time.ParseDuration("24h")
InsertPushMirror(&PushMirror{
RemoteName: "test-2",
LastUpdateUnix: now,
Interval: long,
})
InsertPushMirror(&PushMirror{
RemoteName: "test-3",
LastUpdateUnix: now,
Interval: 0,
})
time.Sleep(1 * time.Millisecond)
PushMirrorsIterate(func(idx int, bean interface{}) error {
m, ok := bean.(*PushMirror)
assert.True(t, ok)
assert.Equal(t, "test-1", m.RemoteName)
assert.Equal(t, m.RemoteName, m.GetRemoteName())
return nil
})
}

2
models/task.go
View File

@ -234,7 +234,7 @@ func FinishMigrateTask(task *Task) error {
}
conf.AuthPassword = ""
conf.AuthToken = ""
conf.CloneAddr = util.SanitizeURLCredentials(conf.CloneAddr, true)
conf.CloneAddr = util.NewStringURLSanitizer(conf.CloneAddr, true).Replace(conf.CloneAddr)
conf.AuthPasswordEncrypted = ""
conf.AuthTokenEncrypted = ""
conf.CloneAddrEncrypted = ""

6
modules/context/repo.go
View File

@ -360,13 +360,17 @@ func repoAssignment(ctx *Context, repo *models.Repository) {
var err error
ctx.Repo.Mirror, err = models.GetMirrorByRepoID(repo.ID)
if err != nil {
ctx.ServerError("GetMirror", err)
ctx.ServerError("GetMirrorByRepoID", err)
return
}
ctx.Data["MirrorEnablePrune"] = ctx.Repo.Mirror.EnablePrune
ctx.Data["MirrorInterval"] = ctx.Repo.Mirror.Interval
ctx.Data["Mirror"] = ctx.Repo.Mirror
}
if err = repo.LoadPushMirrors(); err != nil {
ctx.ServerError("LoadPushMirrors", err)
return
}
ctx.Repo.Repository = repo
ctx.Data["RepoName"] = ctx.Repo.Repository.Name

31
modules/git/remote.go Normal file
View File

@ -0,0 +1,31 @@
// Copyright 2021 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package git
import "net/url"
// GetRemoteAddress returns the url of a specific remote of the repository.
func GetRemoteAddress(repoPath, remoteName string) (*url.URL, error) {
err := LoadGitVersion()
if err != nil {
return nil, err
}
var cmd *Command
if CheckGitVersionAtLeast("2.7") == nil {
cmd = NewCommand("remote", "get-url", remoteName)
} else {
cmd = NewCommand("config", "--get", "remote."+remoteName+".url")
}
result, err := cmd.RunInDir(repoPath)
if err != nil {
return nil, err
}
if len(result) > 0 {
result = result[:len(result)-1]
}
return url.Parse(result)
}

16
modules/git/repo.go
View File

@ -185,7 +185,9 @@ type PushOptions struct {
Remote string
Branch string
Force bool
Mirror bool
Env []string
Timeout time.Duration
}
// Push pushs local commits to given remote branch.
@ -194,10 +196,20 @@ func Push(repoPath string, opts PushOptions) error {
if opts.Force {
cmd.AddArguments("-f")
}
cmd.AddArguments("--", opts.Remote, opts.Branch)
if opts.Mirror {
cmd.AddArguments("--mirror")
}
cmd.AddArguments("--", opts.Remote)
if len(opts.Branch) > 0 {
cmd.AddArguments(opts.Branch)
}
var outbuf, errbuf strings.Builder
err := cmd.RunInDirTimeoutEnvPipeline(opts.Env, -1, repoPath, &outbuf, &errbuf)
if opts.Timeout == 0 {
opts.Timeout = -1
}
err := cmd.RunInDirTimeoutEnvPipeline(opts.Env, opts.Timeout, repoPath, &outbuf, &errbuf)
if err != nil {
if strings.Contains(errbuf.String(), "non-fast-forward") {
return &ErrPushOutOfDate{

10
modules/lfs/client.go
View File

@ -10,9 +10,17 @@ import (
"net/url"
)
// DownloadCallback gets called for every requested LFS object to process its content
type DownloadCallback func(p Pointer, content io.ReadCloser, objectError error) error
// UploadCallback gets called for every requested LFS object to provide its content
type UploadCallback func(p Pointer, objectError error) (io.ReadCloser, error)
// Client is used to communicate with a LFS source
type Client interface {
Download(ctx context.Context, oid string, size int64) (io.ReadCloser, error)
BatchSize() int
Download(ctx context.Context, objects []Pointer, callback DownloadCallback) error
Upload(ctx context.Context, objects []Pointer, callback UploadCallback) error
}
// NewClient creates a LFS client

1
modules/lfs/client_test.go
View File

@ -6,7 +6,6 @@ package lfs
import (
"net/url"
"testing"
"github.com/stretchr/testify/assert"

61
modules/lfs/filesystem_client.go
View File

@ -19,6 +19,11 @@ type FilesystemClient struct {
lfsdir string
}
// BatchSize returns the preferred size of batchs to process
func (c *FilesystemClient) BatchSize() int {
return 1
}
func newFilesystemClient(endpoint *url.URL) *FilesystemClient {
path, _ := util.FileURLToPath(endpoint)
@ -33,18 +38,56 @@ func (c *FilesystemClient) objectPath(oid string) string {
return filepath.Join(c.lfsdir, oid[0:2], oid[2:4], oid)
}
// Download reads the specific LFS object from the target repository
func (c *FilesystemClient) Download(ctx context.Context, oid string, size int64) (io.ReadCloser, error) {
objectPath := c.objectPath(oid)
// Download reads the specific LFS object from the target path
func (c *FilesystemClient) Download(ctx context.Context, objects []Pointer, callback DownloadCallback) error {
for _, object := range objects {
p := Pointer{object.Oid, object.Size}
if _, err := os.Stat(objectPath); os.IsNotExist(err) {
return nil, err
}
objectPath := c.objectPath(p.Oid)
file, err := os.Open(objectPath)
f, err := os.Open(objectPath)
if err != nil {
return nil, err
return err
}
return file, nil
if err := callback(p, f, nil); err != nil {
return err
}
}
return nil
}
// Upload writes the specific LFS object to the target path
func (c *FilesystemClient) Upload(ctx context.Context, objects []Pointer, callback UploadCallback) error {
for _, object := range objects {
p := Pointer{object.Oid, object.Size}
objectPath := c.objectPath(p.Oid)
if err := os.MkdirAll(filepath.Dir(objectPath), os.ModePerm); err != nil {
return err
}
content, err := callback(p, nil)
if err != nil {
return err
}
err = func() error {
defer content.Close()
f, err := os.Create(objectPath)
if err != nil {
return err
}
_, err = io.Copy(f, content)
return err
}()
if err != nil {
return err
}
}
return nil
}

127
modules/lfs/http_client.go
View File

@ -7,17 +7,19 @@ package lfs
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"strings"
"code.gitea.io/gitea/modules/log"
jsoniter "github.com/json-iterator/go"
)
const batchSize = 20
// HTTPClient is used to communicate with the LFS server
// https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md
type HTTPClient struct {
@ -26,6 +28,11 @@ type HTTPClient struct {
transfers map[string]TransferAdapter
}
// BatchSize returns the preferred size of batchs to process
func (c *HTTPClient) BatchSize() int {
return batchSize
}
func newHTTPClient(endpoint *url.URL) *HTTPClient {
hc := &http.Client{}
@ -55,21 +62,25 @@ func (c *HTTPClient) transferNames() []string {
}
func (c *HTTPClient) batch(ctx context.Context, operation string, objects []Pointer) (*BatchResponse, error) {
log.Trace("BATCH operation with objects: %v", objects)
url := fmt.Sprintf("%s/objects/batch", c.endpoint)
request := &BatchRequest{operation, c.transferNames(), nil, objects}
payload := new(bytes.Buffer)
err := json.NewEncoder(payload).Encode(request)
err := jsoniter.NewEncoder(payload).Encode(request)
if err != nil {
return nil, fmt.Errorf("lfs.HTTPClient.batch json.Encode: %w", err)
log.Error("Error encoding json: %v", err)
return nil, err
}
log.Trace("lfs.HTTPClient.batch NewRequestWithContext: %s", url)
log.Trace("Calling: %s", url)
req, err := http.NewRequestWithContext(ctx, "POST", url, payload)
if err != nil {
return nil, fmt.Errorf("lfs.HTTPClient.batch http.NewRequestWithContext: %w", err)
log.Error("Error creating request: %v", err)
return nil, err
}
req.Header.Set("Content-type", MediaType)
req.Header.Set("Accept", MediaType)
@ -81,18 +92,20 @@ func (c *HTTPClient) batch(ctx context.Context, operation string, objects []Poin
return nil, ctx.Err()
default:
}
return nil, fmt.Errorf("lfs.HTTPClient.batch http.Do: %w", err)
log.Error("Error while processing request: %v", err)
return nil, err
}
defer res.Body.Close()
if res.StatusCode != http.StatusOK {
return nil, fmt.Errorf("lfs.HTTPClient.batch: Unexpected servers response: %s", res.Status)
return nil, fmt.Errorf("Unexpected server response: %s", res.Status)
}
var response BatchResponse
err = json.NewDecoder(res.Body).Decode(&response)
err = jsoniter.NewDecoder(res.Body).Decode(&response)
if err != nil {
return nil, fmt.Errorf("lfs.HTTPClient.batch json.Decode: %w", err)
log.Error("Error decoding json: %v", err)
return nil, err
}
if len(response.Transfer) == 0 {
@ -103,27 +116,99 @@ func (c *HTTPClient) batch(ctx context.Context, operation string, objects []Poin
}
// Download reads the specific LFS object from the LFS server
func (c *HTTPClient) Download(ctx context.Context, oid string, size int64) (io.ReadCloser, error) {
var objects []Pointer
objects = append(objects, Pointer{oid, size})
func (c *HTTPClient) Download(ctx context.Context, objects []Pointer, callback DownloadCallback) error {
return c.performOperation(ctx, objects, callback, nil)
}
result, err := c.batch(ctx, "download", objects)
// Upload sends the specific LFS object to the LFS server
func (c *HTTPClient) Upload(ctx context.Context, objects []Pointer, callback UploadCallback) error {
return c.performOperation(ctx, objects, nil, callback)
}
func (c *HTTPClient) performOperation(ctx context.Context, objects []Pointer, dc DownloadCallback, uc UploadCallback) error {
if len(objects) == 0 {
return nil
}
operation := "download"
if uc != nil {
operation = "upload"
}
result, err := c.batch(ctx, operation, objects)
if err != nil {
return nil, err
return err
}
transferAdapter, ok := c.transfers[result.Transfer]
if !ok {
return nil, fmt.Errorf("lfs.HTTPClient.Download Transferadapter not found: %s", result.Transfer)
return fmt.Errorf("TransferAdapter not found: %s", result.Transfer)
}
if len(result.Objects) == 0 {
return nil, errors.New("lfs.HTTPClient.Download: No objects in result")
for _, object := range result.Objects {
if object.Error != nil {
objectError := errors.New(object.Error.Message)
log.Trace("Error on object %v: %v", object.Pointer, objectError)
if uc != nil {
if _, err := uc(object.Pointer, objectError); err != nil {
return err
}
} else {
if err := dc(object.Pointer, nil, objectError); err != nil {
return err
}
}
continue
}
content, err := transferAdapter.Download(ctx, result.Objects[0])
if uc != nil {
if len(object.Actions) == 0 {
log.Trace("%v already present on server", object.Pointer)
continue
}
link, ok := object.Actions["upload"]
if !ok {
log.Debug("%+v", object)
return errors.New("Missing action 'upload'")
}
content, err := uc(object.Pointer, nil)
if err != nil {
return nil, err
return err
}
return content, nil
err = transferAdapter.Upload(ctx, link, object.Pointer, content)
content.Close()
if err != nil {
return err
}
link, ok = object.Actions["verify"]
if ok {
if err := transferAdapter.Verify(ctx, link, object.Pointer); err != nil {
return err
}
}
} else {
link, ok := object.Actions["download"]
if !ok {
log.Debug("%+v", object)
return errors.New("Missing action 'download'")
}
content, err := transferAdapter.Download(ctx, link)
if err != nil {
return err
}
if err := dc(object.Pointer, content, nil); err != nil {
return err
}
}
}
return nil
}

324
modules/lfs/http_client_test.go
View File

@ -7,13 +7,13 @@ package lfs
import (
"bytes"
"context"
"encoding/json"
"io"
"io/ioutil"
"net/http"
"strings"
"testing"
jsoniter "github.com/json-iterator/go"
"github.com/stretchr/testify/assert"
)
@ -30,69 +30,146 @@ func (a *DummyTransferAdapter) Name() string {
return "dummy"
}
func (a *DummyTransferAdapter) Download(ctx context.Context, r *ObjectResponse) (io.ReadCloser, error) {
func (a *DummyTransferAdapter) Download(ctx context.Context, l *Link) (io.ReadCloser, error) {
return ioutil.NopCloser(bytes.NewBufferString("dummy")), nil
}
func TestHTTPClientDownload(t *testing.T) {
oid := "fb8f7d8435968c4f82a726a92395be4d16f2f63116caf36c8ad35c60831ab041"
size := int64(6)
func (a *DummyTransferAdapter) Upload(ctx context.Context, l *Link, p Pointer, r io.Reader) error {
return nil
}
roundTripHandler := func(req *http.Request) *http.Response {
func (a *DummyTransferAdapter) Verify(ctx context.Context, l *Link, p Pointer) error {
return nil
}
func lfsTestRoundtripHandler(req *http.Request) *http.Response {
var batchResponse *BatchResponse
url := req.URL.String()
if strings.Contains(url, "status-not-ok") {
return &http.Response{StatusCode: http.StatusBadRequest}
}
if strings.Contains(url, "invalid-json-response") {
} else if strings.Contains(url, "invalid-json-response") {
return &http.Response{StatusCode: http.StatusOK, Body: ioutil.NopCloser(bytes.NewBufferString("invalid json"))}
}
if strings.Contains(url, "valid-batch-request-download") {
assert.Equal(t, "POST", req.Method)
assert.Equal(t, MediaType, req.Header.Get("Content-type"), "case %s: error should match", url)
assert.Equal(t, MediaType, req.Header.Get("Accept"), "case %s: error should match", url)
var batchRequest BatchRequest
err := json.NewDecoder(req.Body).Decode(&batchRequest)
assert.NoError(t, err)
assert.Equal(t, "download", batchRequest.Operation)
assert.Len(t, batchRequest.Objects, 1)
assert.Equal(t, oid, batchRequest.Objects[0].Oid)
assert.Equal(t, size, batchRequest.Objects[0].Size)
batchResponse := &BatchResponse{
} else if strings.Contains(url, "valid-batch-request-download") {
batchResponse = &BatchResponse{
Transfer: "dummy",
Objects: make([]*ObjectResponse, 1),
Objects: []*ObjectResponse{
{
Actions: map[string]*Link{
"download": {},
},
},
},
}
payload := new(bytes.Buffer)
json.NewEncoder(payload).Encode(batchResponse)
return &http.Response{StatusCode: http.StatusOK, Body: ioutil.NopCloser(payload)}
} else if strings.Contains(url, "valid-batch-request-upload") {
batchResponse = &BatchResponse{
Transfer: "dummy",
Objects: []*ObjectResponse{
{
Actions: map[string]*Link{
"upload": {},
},
},
},
}
if strings.Contains(url, "invalid-response-no-objects") {
batchResponse := &BatchResponse{Transfer: "dummy"}
payload := new(bytes.Buffer)
json.NewEncoder(payload).Encode(batchResponse)
return &http.Response{StatusCode: http.StatusOK, Body: ioutil.NopCloser(payload)}
} else if strings.Contains(url, "response-no-objects") {
batchResponse = &BatchResponse{Transfer: "dummy"}
} else if strings.Contains(url, "unknown-transfer-adapter") {
batchResponse = &BatchResponse{Transfer: "unknown_adapter"}
} else if strings.Contains(url, "error-in-response-objects") {
batchResponse = &BatchResponse{
Transfer: "dummy",
Objects: []*ObjectResponse{
{
Error: &ObjectError{
Code: 404,
Message: "Object not found",
},
},
},
}
if strings.Contains(url, "unknown-transfer-adapter") {
batchResponse := &BatchResponse{Transfer: "unknown_adapter"}
payload := new(bytes.Buffer)
json.NewEncoder(payload).Encode(batchResponse)
return &http.Response{StatusCode: http.StatusOK, Body: ioutil.NopCloser(payload)}
} else if strings.Contains(url, "empty-actions-map") {
batchResponse = &BatchResponse{
Transfer: "dummy",
Objects: []*ObjectResponse{
{
Actions: map[string]*Link{},
},
},
}
t.Errorf("Unknown test case: %s", url)
} else if strings.Contains(url, "download-actions-map") {
batchResponse = &BatchResponse{
Transfer: "dummy",
Objects: []*ObjectResponse{
{
Actions: map[string]*Link{
"download": {},
},
},
},
}
} else if strings.Contains(url, "upload-actions-map") {
batchResponse = &BatchResponse{
Transfer: "dummy",
Objects: []*ObjectResponse{
{
Actions: map[string]*Link{
"upload": {},
},
},
},
}
} else if strings.Contains(url, "verify-actions-map") {
batchResponse = &BatchResponse{
Transfer: "dummy",
Objects: []*ObjectResponse{
{
Actions: map[string]*Link{
"verify": {},
},
},
},
}
} else if strings.Contains(url, "unknown-actions-map") {
batchResponse = &BatchResponse{
Transfer: "dummy",
Objects: []*ObjectResponse{
{
Actions: map[string]*Link{
"unknown": {},
},
},
},
}
} else {
return nil
}
hc := &http.Client{Transport: RoundTripFunc(roundTripHandler)}
payload := new(bytes.Buffer)
jsoniter.NewEncoder(payload).Encode(batchResponse)
return &http.Response{StatusCode: http.StatusOK, Body: ioutil.NopCloser(payload)}
}
func TestHTTPClientDownload(t *testing.T) {
p := Pointer{Oid: "fb8f7d8435968c4f82a726a92395be4d16f2f63116caf36c8ad35c60831ab041", Size: 6}
hc := &http.Client{Transport: RoundTripFunc(func(req *http.Request) *http.Response {
assert.Equal(t, "POST", req.Method)
assert.Equal(t, MediaType, req.Header.Get("Content-type"))
assert.Equal(t, MediaType, req.Header.Get("Accept"))
var batchRequest BatchRequest
err := jsoniter.NewDecoder(req.Body).Decode(&batchRequest)
assert.NoError(t, err)
assert.Equal(t, "download", batchRequest.Operation)
assert.Equal(t, 1, len(batchRequest.Objects))
assert.Equal(t, p.Oid, batchRequest.Objects[0].Oid)
assert.Equal(t, p.Size, batchRequest.Objects[0].Size)
return lfsTestRoundtripHandler(req)
})}
dummy := &DummyTransferAdapter{}
var cases = []struct {
@ -102,12 +179,12 @@ func TestHTTPClientDownload(t *testing.T) {
// case 0
{
endpoint: "https://status-not-ok.io",
expectederror: "Unexpected servers response: ",
expectederror: "Unexpected server response: ",
},
// case 1
{
endpoint: "https://invalid-json-response.io",
expectederror: "json.Decode: ",
expectederror: "invalid json",
},
// case 2
{
@ -116,13 +193,43 @@ func TestHTTPClientDownload(t *testing.T) {
},
// case 3
{
endpoint: "https://invalid-response-no-objects.io",
expectederror: "No objects in result",
endpoint: "https://response-no-objects.io",
expectederror: "",
},
// case 4
{
endpoint: "https://unknown-transfer-adapter.io",
expectederror: "Transferadapter not found: ",
expectederror: "TransferAdapter not found: ",
},
// case 5
{
endpoint: "https://error-in-response-objects.io",
expectederror: "Object not found",
},
// case 6
{
endpoint: "https://empty-actions-map.io",
expectederror: "Missing action 'download'",
},
// case 7
{
endpoint: "https://download-actions-map.io",
expectederror: "",
},
// case 8
{
endpoint: "https://upload-actions-map.io",
expectederror: "Missing action 'download'",
},
// case 9
{
endpoint: "https://verify-actions-map.io",
expectederror: "Missing action 'download'",
},
// case 10
{
endpoint: "https://unknown-actions-map.io",
expectederror: "Missing action 'download'",
},
}
@ -134,7 +241,116 @@ func TestHTTPClientDownload(t *testing.T) {
}
client.transfers["dummy"] = dummy
_, err := client.Download(context.Background(), oid, size)
err := client.Download(context.Background(), []Pointer{p}, func(p Pointer, content io.ReadCloser, objectError error) error {
if objectError != nil {
return objectError
}
b, err := io.ReadAll(content)
assert.NoError(t, err)
assert.Equal(t, []byte("dummy"), b)
return nil
})
if len(c.expectederror) > 0 {
assert.True(t, strings.Contains(err.Error(), c.expectederror), "case %d: '%s' should contain '%s'", n, err.Error(), c.expectederror)
} else {
assert.NoError(t, err, "case %d", n)
}
}
}
func TestHTTPClientUpload(t *testing.T) {
p := Pointer{Oid: "fb8f7d8435968c4f82a726a92395be4d16f2f63116caf36c8ad35c60831ab041", Size: 6}
hc := &http.Client{Transport: RoundTripFunc(func(req *http.Request) *http.Response {
assert.Equal(t, "POST", req.Method)
assert.Equal(t, MediaType, req.Header.Get("Content-type"))
assert.Equal(t, MediaType, req.Header.Get("Accept"))
var batchRequest BatchRequest
err := jsoniter.NewDecoder(req.Body).Decode(&batchRequest)
assert.NoError(t, err)
assert.Equal(t, "upload", batchRequest.Operation)
assert.Equal(t, 1, len(batchRequest.Objects))
assert.Equal(t, p.Oid, batchRequest.Objects[0].Oid)
assert.Equal(t, p.Size, batchRequest.Objects[0].Size)
return lfsTestRoundtripHandler(req)
})}
dummy := &DummyTransferAdapter{}
var cases = []struct {
endpoint string
expectederror string
}{
// case 0
{
endpoint: "https://status-not-ok.io",
expectederror: "Unexpected server response: ",
},
// case 1
{
endpoint: "https://invalid-json-response.io",
expectederror: "invalid json",
},
// case 2
{
endpoint: "https://valid-batch-request-upload.io",
expectederror: "",
},
// case 3
{
endpoint: "https://response-no-objects.io",
expectederror: "",
},
// case 4
{
endpoint: "https://unknown-transfer-adapter.io",
expectederror: "TransferAdapter not found: ",
},
// case 5
{
endpoint: "https://error-in-response-objects.io",
expectederror: "Object not found",
},
// case 6
{
endpoint: "https://empty-actions-map.io",
expectederror: "",
},
// case 7
{
endpoint: "https://download-actions-map.io",
expectederror: "Missing action 'upload'",
},
// case 8
{
endpoint: "https://upload-actions-map.io",
expectederror: "",
},
// case 9
{
endpoint: "https://verify-actions-map.io",
expectederror: "Missing action 'upload'",
},
// case 10
{
endpoint: "https://unknown-actions-map.io",
expectederror: "Missing action 'upload'",
},
}
for n, c := range cases {
client := &HTTPClient{
client: hc,
endpoint: c.endpoint,
transfers: make(map[string]TransferAdapter),
}
client.transfers["dummy"] = dummy
err := client.Upload(context.Background(), []Pointer{p}, func(p Pointer, objectError error) (io.ReadCloser, error) {
return ioutil.NopCloser(new(bytes.Buffer)), objectError
})
if len(c.expectederror) > 0 {
assert.True(t, strings.Contains(err.Error(), c.expectederror), "case %d: '%s' should contain '%s'", n, err.Error(), c.expectederror)
} else {

4
modules/lfs/shared.go
View File

@ -49,14 +49,14 @@ type ObjectResponse struct {
Error *ObjectError `json:"error,omitempty"`
}
// Link provides a structure used to build a hypermedia representation of an HTTP link.
// Link provides a structure with informations about how to access a object.
type Link struct {
Href string `json:"href"`
Header map[string]string `json:"header,omitempty"`
ExpiresAt *time.Time `json:"expires_at,omitempty"`
}
// ObjectError defines the JSON structure returned to the client in case of an error
// ObjectError defines the JSON structure returned to the client in case of an error.
type ObjectError struct {
Code int `json:"code"`
Message string `json:"message"`

104
modules/lfs/transferadapter.go
View File

@ -5,18 +5,24 @@
package lfs
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"net/http"
"code.gitea.io/gitea/modules/log"
jsoniter "github.com/json-iterator/go"
)
// TransferAdapter represents an adapter for downloading/uploading LFS objects
type TransferAdapter interface {
Name() string
Download(ctx context.Context, r *ObjectResponse) (io.ReadCloser, error)
//Upload(ctx context.Context, reader io.Reader) error
Download(ctx context.Context, l *Link) (io.ReadCloser, error)
Upload(ctx context.Context, l *Link, p Pointer, r io.Reader) error
Verify(ctx context.Context, l *Link, p Pointer) error
}
// BasicTransferAdapter implements the "basic" adapter
@ -30,29 +36,101 @@ func (a *BasicTransferAdapter) Name() string {
}
// Download reads the download location and downloads the data
func (a *BasicTransferAdapter) Download(ctx context.Context, r *ObjectResponse) (io.ReadCloser, error) {
download, ok := r.Actions["download"]
if !ok {
return nil, errors.New("lfs.BasicTransferAdapter.Download: Action 'download' not found")
func (a *BasicTransferAdapter) Download(ctx context.Context, l *Link) (io.ReadCloser, error) {
resp, err := a.performRequest(ctx, "GET", l, nil, nil)
if err != nil {
return nil, err
}
return resp.Body, nil
}
// Upload sends the content to the LFS server
func (a *BasicTransferAdapter) Upload(ctx context.Context, l *Link, p Pointer, r io.Reader) error {
_, err := a.performRequest(ctx, "PUT", l, r, func(req *http.Request) {
if len(req.Header.Get("Content-Type")) == 0 {
req.Header.Set("Content-Type", "application/octet-stream")
}
req, err := http.NewRequestWithContext(ctx, "GET", download.Href, nil)
if err != nil {
return nil, fmt.Errorf("lfs.BasicTransferAdapter.Download http.NewRequestWithContext: %w", err)
if req.Header.Get("Transfer-Encoding") == "chunked" {
req.TransferEncoding = []string{"chunked"}
}
for key, value := range download.Header {
req.ContentLength = p.Size
})
if err != nil {
return err
}
return nil
}
// Verify calls the verify handler on the LFS server
func (a *BasicTransferAdapter) Verify(ctx context.Context, l *Link, p Pointer) error {
b, err := jsoniter.Marshal(p)
if err != nil {
log.Error("Error encoding json: %v", err)
return err
}
_, err = a.performRequest(ctx, "POST", l, bytes.NewReader(b), func(req *http.Request) {
req.Header.Set("Content-Type", MediaType)
})
if err != nil {
return err
}
return nil
}
func (a *BasicTransferAdapter) performRequest(ctx context.Context, method string, l *Link, body io.Reader, callback func(*http.Request)) (*http.Response, error) {
log.Trace("Calling: %s %s", method, l.Href)
req, err := http.NewRequestWithContext(ctx, method, l.Href, body)
if err != nil {
log.Error("Error creating request: %v", err)
return nil, err
}
for key, value := range l.Header {
req.Header.Set(key, value)
}
req.Header.Set("Accept", MediaType)
if callback != nil {
callback(req)
}
res, err := a.client.Do(req)
if err != nil {
select {
case <-ctx.Done():
return nil, ctx.Err()
return res, ctx.Err()
default:
}
return nil, fmt.Errorf("lfs.BasicTransferAdapter.Download http.Do: %w", err)
log.Error("Error while processing request: %v", err)
return res, err
}
return res.Body, nil
if res.StatusCode != http.StatusOK {
return res, handleErrorResponse(res)
}
return res, nil
}
func handleErrorResponse(resp *http.Response) error {
defer resp.Body.Close()
er, err := decodeReponseError(resp.Body)
if err != nil {
return fmt.Errorf("Request failed with status %s", resp.Status)
}
log.Trace("ErrorRespone: %v", er)
return errors.New(er.Message)
}
func decodeReponseError(r io.Reader) (ErrorResponse, error) {
var er ErrorResponse
err := jsoniter.NewDecoder(r).Decode(&er)
if err != nil {
log.Error("Error decoding json: %v", err)
}
return er, err
}

139
modules/lfs/transferadapter_test.go
View File

@ -7,11 +7,13 @@ package lfs
import (
"bytes"
"context"
"io"
"io/ioutil"
"net/http"
"strings"
"testing"
jsoniter "github.com/json-iterator/go"
"github.com/stretchr/testify/assert"
)
@ -21,58 +23,151 @@ func TestBasicTransferAdapterName(t *testing.T) {
assert.Equal(t, "basic", a.Name())
}
func TestBasicTransferAdapterDownload(t *testing.T) {
func TestBasicTransferAdapter(t *testing.T) {
p := Pointer{Oid: "b5a2c96250612366ea272ffac6d9744aaf4b45aacd96aa7cfcb931ee3b558259", Size: 5}
roundTripHandler := func(req *http.Request) *http.Response {
url := req.URL.String()
if strings.Contains(url, "valid-download-request") {
assert.Equal(t, "GET", req.Method)
assert.Equal(t, MediaType, req.Header.Get("Accept"))
assert.Equal(t, "test-value", req.Header.Get("test-header"))
url := req.URL.String()
if strings.Contains(url, "download-request") {
assert.Equal(t, "GET", req.Method)
return &http.Response{StatusCode: http.StatusOK, Body: ioutil.NopCloser(bytes.NewBufferString("dummy"))}
} else if strings.Contains(url, "upload-request") {
assert.Equal(t, "PUT", req.Method)
assert.Equal(t, "application/octet-stream", req.Header.Get("Content-Type"))
b, err := io.ReadAll(req.Body)
assert.NoError(t, err)
assert.Equal(t, "dummy", string(b))
return &http.Response{StatusCode: http.StatusOK}
} else if strings.Contains(url, "verify-request") {
assert.Equal(t, "POST", req.Method)
assert.Equal(t, MediaType, req.Header.Get("Content-Type"))
var vp Pointer
err := jsoniter.NewDecoder(req.Body).Decode(&vp)
assert.NoError(t, err)
assert.Equal(t, p.Oid, vp.Oid)
assert.Equal(t, p.Size, vp.Size)
return &http.Response{StatusCode: http.StatusOK}
} else if strings.Contains(url, "error-response") {
er := &ErrorResponse{
Message: "Object not found",
}
payload := new(bytes.Buffer)
jsoniter.NewEncoder(payload).Encode(er)
return &http.Response{StatusCode: http.StatusNotFound, Body: ioutil.NopCloser(payload)}
} else {
t.Errorf("Unknown test case: %s", url)
return nil
}
}
hc := &http.Client{Transport: RoundTripFunc(roundTripHandler)}
a := &BasicTransferAdapter{hc}
var cases = []struct {
response *ObjectResponse
t.Run("Download", func(t *testing.T) {
cases := []struct {
link *Link
expectederror string
}{
// case 0
{
response: &ObjectResponse{},
expectederror: "Action 'download' not found",
link: &Link{
Href: "https://download-request.io",
Header: map[string]string{"test-header": "test-value"},
},
expectederror: "",
},
// case 1
{
response: &ObjectResponse{
Actions: map[string]*Link{"upload": nil},
},
expectederror: "Action 'download' not found",
},
// case 2
{
response: &ObjectResponse{
Actions: map[string]*Link{"download": {
Href: "https://valid-download-request.io",
link: &Link{
Href: "https://error-response.io",
Header: map[string]string{"test-header": "test-value"},
}},
},
expectederror: "",
expectederror: "Object not found",
},
}
for n, c := range cases {
_, err := a.Download(context.Background(), c.response)
_, err := a.Download(context.Background(), c.link)
if len(c.expectederror) > 0 {
assert.True(t, strings.Contains(err.Error(), c.expectederror), "case %d: '%s' should contain '%s'", n, err.Error(), c.expectederror)
} else {
assert.NoError(t, err, "case %d", n)
}
}
})
t.Run("Upload", func(t *testing.T) {
cases := []struct {
link *Link
expectederror string
}{
// case 0
{
link: &Link{
Href: "https://upload-request.io",
Header: map[string]string{"test-header": "test-value"},
},
expectederror: "",
},
// case 1
{
link: &Link{
Href: "https://error-response.io",
Header: map[string]string{"test-header": "test-value"},
},
expectederror: "Object not found",
},
}
for n, c := range cases {
err := a.Upload(context.Background(), c.link, p, bytes.NewBufferString("dummy"))
if len(c.expectederror) > 0 {
assert.True(t, strings.Contains(err.Error(), c.expectederror), "case %d: '%s' should contain '%s'", n, err.Error(), c.expectederror)
} else {
assert.NoError(t, err, "case %d", n)
}
}
})
t.Run("Verify", func(t *testing.T) {
cases := []struct {
link *Link
expectederror string
}{
// case 0
{
link: &Link{
Href: "https://verify-request.io",
Header: map[string]string{"test-header": "test-value"},
},
expectederror: "",
},
// case 1
{
link: &Link{
Href: "https://error-response.io",
Header: map[string]string{"test-header": "test-value"},
},
expectederror: "Object not found",
},
}
for n, c := range cases {
err := a.Verify(context.Background(), c.link, p)
if len(c.expectederror) > 0 {
assert.True(t, strings.Contains(err.Error(), c.expectederror), "case %d: '%s' should contain '%s'", n, err.Error(), c.expectederror)
} else {
assert.NoError(t, err, "case %d", n)
}
}
})
}

101
modules/repository/repo.go
View File

@ -7,6 +7,7 @@ package repository
import (
"context"
"fmt"
"io"
"net/url"
"path"
"strings"
@ -323,64 +324,90 @@ func StoreMissingLfsObjectsInRepository(ctx context.Context, repo *models.Reposi
errChan := make(chan error, 1)
go lfs.SearchPointerBlobs(ctx, gitRepo, pointerChan, errChan)
err := func() error {
for pointerBlob := range pointerChan {
meta, err := models.NewLFSMetaObject(&models.LFSMetaObject{Pointer: pointerBlob.Pointer, RepositoryID: repo.ID})
downloadObjects := func(pointers []lfs.Pointer) error {
err := client.Download(ctx, pointers, func(p lfs.Pointer, content io.ReadCloser, objectError error) error {
if objectError != nil {
return objectError
}
defer content.Close()
_, err := models.NewLFSMetaObject(&models.LFSMetaObject{Pointer: p, RepositoryID: repo.ID})
if err != nil {
return fmt.Errorf("StoreMissingLfsObjectsInRepository models.NewLFSMetaObject: %w", err)
}
if meta.Existing {
continue
log.Error("Error creating LFS meta object %v: %v", p, err)
return err
}
log.Trace("StoreMissingLfsObjectsInRepository: LFS OID[%s] not present in repository %s", pointerBlob.Oid, repo.FullName())
err = func() error {
exist, err := contentStore.Exists(pointerBlob.Pointer)
if err != nil {
return fmt.Errorf("StoreMissingLfsObjectsInRepository contentStore.Exists: %w", err)
if err := contentStore.Put(p, content); err != nil {
log.Error("Error storing content for LFS meta object %v: %v", p, err)
if _, err2 := repo.RemoveLFSMetaObjectByOid(p.Oid); err2 != nil {
log.Error("Error removing LFS meta object %v: %v", p, err2)
}
if !exist {
if setting.LFS.MaxFileSize > 0 && pointerBlob.Size > setting.LFS.MaxFileSize {
log.Info("LFS OID[%s] download denied because of LFS_MAX_FILE_SIZE=%d < size %d", pointerBlob.Oid, setting.LFS.MaxFileSize, pointerBlob.Size)
return nil
}
stream, err := client.Download(ctx, pointerBlob.Oid, pointerBlob.Size)
if err != nil {
return fmt.Errorf("StoreMissingLfsObjectsInRepository: LFS OID[%s] failed to download: %w", pointerBlob.Oid, err)
}
defer stream.Close()
if err := contentStore.Put(pointerBlob.Pointer, stream); err != nil {
return fmt.Errorf("StoreMissingLfsObjectsInRepository LFS OID[%s] contentStore.Put: %w", pointerBlob.Oid, err)
}
} else {
log.Trace("StoreMissingLfsObjectsInRepository: LFS OID[%s] already present in content store", pointerBlob.Oid)
return err
}
return nil
}()
})
if err != nil {
if _, err2 := repo.RemoveLFSMetaObjectByOid(meta.Oid); err2 != nil {
log.Error("StoreMissingLfsObjectsInRepository RemoveLFSMetaObjectByOid[Oid: %s]: %w", meta.Oid, err2)
}
select {
case <-ctx.Done():
return nil
default:
}
}
return err
}
var batch []lfs.Pointer
for pointerBlob := range pointerChan {
meta, err := repo.GetLFSMetaObjectByOid(pointerBlob.Oid)
if err != nil && err != models.ErrLFSObjectNotExist {
log.Error("Error querying LFS meta object %v: %v", pointerBlob.Pointer, err)
return err
}
return nil
}()
if meta != nil {
log.Trace("Skipping unknown LFS meta object %v", pointerBlob.Pointer)
continue
}
log.Trace("LFS object %v not present in repository %s", pointerBlob.Pointer, repo.FullName())
exist, err := contentStore.Exists(pointerBlob.Pointer)
if err != nil {
log.Error("Error checking if LFS object %v exists: %v", pointerBlob.Pointer, err)
return err
}
if exist {
log.Trace("LFS object %v already present; creating meta object", pointerBlob.Pointer)
_, err := models.NewLFSMetaObject(&models.LFSMetaObject{Pointer: pointerBlob.Pointer, RepositoryID: repo.ID})
if err != nil {
log.Error("Error creating LFS meta object %v: %v", pointerBlob.Pointer, err)
return err
}
} else {
if setting.LFS.MaxFileSize > 0 && pointerBlob.Size > setting.LFS.MaxFileSize {
log.Info("LFS object %v download denied because of LFS_MAX_FILE_SIZE=%d < size %d", pointerBlob.Pointer, setting.LFS.MaxFileSize, pointerBlob.Size)
continue
}
batch = append(batch, pointerBlob.Pointer)
if len(batch) >= client.BatchSize() {
if err := downloadObjects(batch); err != nil {
return err
}
batch = nil
}
}
}
if len(batch) > 0 {
if err := downloadObjects(batch); err != nil {
return err
}
}
err, has := <-errChan
if has {
log.Error("Error enumerating LFS objects for repository: %v", err)
return err
}

2
modules/task/migrate.go
View File

@ -118,7 +118,7 @@ func runMigrateTask(t *models.Task) (err error) {
}
// remoteAddr may contain credentials, so we sanitize it
err = util.URLSanitizedError(err, opts.CloneAddr)
err = util.NewStringURLSanitizedError(err, opts.CloneAddr, true)
if strings.Contains(err.Error(), "Authentication failed") ||
strings.Contains(err.Error(), "could not read Username") {
return fmt.Errorf("Authentication failed: %v", err.Error())

2
modules/task/task.go
View File

@ -74,7 +74,7 @@ func CreateMigrateTask(doer, u *models.User, opts base.MigrateOptions) (*models.
if err != nil {
return nil, err
}
opts.CloneAddr = util.SanitizeURLCredentials(opts.CloneAddr, true)
opts.CloneAddr = util.NewStringURLSanitizer(opts.CloneAddr, true).Replace(opts.CloneAddr)
opts.AuthPasswordEncrypted, err = secret.EncryptSecret(setting.SecretKey, opts.AuthPassword)
if err != nil {
return nil, err

32
modules/templates/helper.go
View File

@ -27,6 +27,7 @@ import (
"code.gitea.io/gitea/models"
"code.gitea.io/gitea/modules/base"
"code.gitea.io/gitea/modules/emoji"
"code.gitea.io/gitea/modules/git"
"code.gitea.io/gitea/modules/log"
"code.gitea.io/gitea/modules/markup"
"code.gitea.io/gitea/modules/repository"
@ -35,7 +36,6 @@ import (
"code.gitea.io/gitea/modules/timeutil"
"code.gitea.io/gitea/modules/util"
"code.gitea.io/gitea/services/gitdiff"
mirror_service "code.gitea.io/gitea/services/mirror"
"github.com/editorconfig/editorconfig-core-go/v2"
jsoniter "github.com/json-iterator/go"
@ -295,10 +295,7 @@ func NewFuncMap() []template.FuncMap {
return float32(n) * 100 / float32(sum)
},
"CommentMustAsDiff": gitdiff.CommentMustAsDiff,
"MirrorAddress": mirror_service.Address,
"MirrorFullAddress": mirror_service.AddressNoCredentials,
"MirrorUserName": mirror_service.Username,
"MirrorPassword": mirror_service.Password,
"MirrorRemoteAddress": mirrorRemoteAddress,
"CommitType": func(commit interface{}) string {
switch commit.(type) {
case models.SignCommitWithStatuses:
@ -963,3 +960,28 @@ func buildSubjectBodyTemplate(stpl *texttmpl.Template, btpl *template.Template,
log.Warn("Failed to parse template [%s/body]: %v", name, err)
}
}
type remoteAddress struct {
Address string
Username string
Password string
}
func mirrorRemoteAddress(m models.RemoteMirrorer) remoteAddress {
a := remoteAddress{}
u, err := git.GetRemoteAddress(m.GetRepository().RepoPath(), m.GetRemoteName())
if err != nil {
log.Error("GetRemoteAddress %v", err)
return a
}
if u.User != nil {
a.Username = u.User.Username()
a.Password, _ = u.User.Password()
}
u.User = nil
a.Address = u.String()
return a
}

61
modules/util/sanitize.go
View File

@ -1,4 +1,4 @@
// Copyright 2017 The Gitea Authors. All rights reserved.
// Copyright 2021 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
@ -9,40 +9,53 @@ import (
"strings"
)
// urlSafeError wraps an error whose message may contain a sensitive URL
type urlSafeError struct {
const userPlaceholder = "sanitized-credential"
const unparsableURL = "(unparsable url)"
type sanitizedError struct {
err error
unsanitizedURL string
replacer *strings.Replacer
}
func (err urlSafeError) Error() string {
return SanitizeMessage(err.err.Error(), err.unsanitizedURL)
func (err sanitizedError) Error() string {
return err.replacer.Replace(err.err.Error())
}
// URLSanitizedError returns the sanitized version an error whose message may
// contain a sensitive URL
func URLSanitizedError(err error, unsanitizedURL string) error {
return urlSafeError{err: err, unsanitizedURL: unsanitizedURL}
// NewSanitizedError wraps an error and replaces all old, new string pairs in the message text.
func NewSanitizedError(err error, oldnew ...string) error {
return sanitizedError{err: err, replacer: strings.NewReplacer(oldnew...)}
}
// SanitizeMessage sanitizes a message which may contains a sensitive URL
func SanitizeMessage(message, unsanitizedURL string) string {
sanitizedURL := SanitizeURLCredentials(unsanitizedURL, true)
return strings.ReplaceAll(message, unsanitizedURL, sanitizedURL)
// NewURLSanitizedError wraps an error and replaces the url credential or removes them.
func NewURLSanitizedError(err error, u *url.URL, usePlaceholder bool) error {
return sanitizedError{err: err, replacer: NewURLSanitizer(u, usePlaceholder)}
}
// SanitizeURLCredentials sanitizes a url, either removing user credentials
// or replacing them with a placeholder.
func SanitizeURLCredentials(unsanitizedURL string, usePlaceholder bool) string {
u, err := url.Parse(unsanitizedURL)
if err != nil {
// don't log the error, since it might contain unsanitized URL.
return "(unparsable url)"
}
// NewStringURLSanitizedError wraps an error and replaces the url credential or removes them.
// If the url can't get parsed it gets replaced with a placeholder string.
func NewStringURLSanitizedError(err error, unsanitizedURL string, usePlaceholder bool) error {
return sanitizedError{err: err, replacer: NewStringURLSanitizer(unsanitizedURL, usePlaceholder)}
}
// NewURLSanitizer creates a replacer for the url with the credential sanitized or removed.
func NewURLSanitizer(u *url.URL, usePlaceholder bool) *strings.Replacer {
old := u.String()
if u.User != nil && usePlaceholder {
u.User = url.User("<credentials>")
u.User = url.User(userPlaceholder)
} else {
u.User = nil
}
return u.String()
return strings.NewReplacer(old, u.String())
}
// NewStringURLSanitizer creates a replacer for the url with the credential sanitized or removed.
// If the url can't get parsed it gets replaced with a placeholder string
func NewStringURLSanitizer(unsanitizedURL string, usePlaceholder bool) *strings.Replacer {
u, err := url.Parse(unsanitizedURL)
if err != nil {
// don't log the error, since it might contain unsanitized URL.
return strings.NewReplacer(unsanitizedURL, unparsableURL)
}
return NewURLSanitizer(u, usePlaceholder)
}

159
modules/util/sanitize_test.go
View File

@ -1,25 +1,164 @@
// Copyright 2020 The Gitea Authors. All rights reserved.
// Copyright 2021 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package util
import (
"errors"
"testing"
"github.com/stretchr/testify/assert"
)
func TestSanitizeURLCredentials(t *testing.T) {
var kases = map[string]string{
"https://github.com/go-gitea/test_repo.git": "https://github.com/go-gitea/test_repo.git",
"https://mytoken@github.com/go-gitea/test_repo.git": "https://github.com/go-gitea/test_repo.git",
"http://github.com/go-gitea/test_repo.git": "http://github.com/go-gitea/test_repo.git",
"/test/repos/repo1": "/test/repos/repo1",
"git@github.com:go-gitea/test_repo.git": "(unparsable url)",
func TestNewSanitizedError(t *testing.T) {
err := errors.New("error while secret on test")
err2 := NewSanitizedError(err)
assert.Equal(t, err.Error(), err2.Error())
var cases = []struct {
input error
oldnew []string
expected string
}{
// case 0
{
errors.New("error while secret on test"),
[]string{"secret", "replaced"},
"error while replaced on test",
},
// case 1
{
errors.New("error while sec-ret on test"),
[]string{"secret", "replaced"},
"error while sec-ret on test",
},
}
for source, value := range kases {
assert.EqualValues(t, value, SanitizeURLCredentials(source, false))
for n, c := range cases {
err := NewSanitizedError(c.input, c.oldnew...)
assert.Equal(t, c.expected, err.Error(), "case %d: error should match", n)
}
}
func TestNewStringURLSanitizer(t *testing.T) {
var cases = []struct {
input string
placeholder bool
expected string
}{
// case 0
{
"https://github.com/go-gitea/test_repo.git",
true,
"https://github.com/go-gitea/test_repo.git",
},
// case 1
{
"https://github.com/go-gitea/test_repo.git",
false,
"https://github.com/go-gitea/test_repo.git",
},
// case 2
{
"https://mytoken@github.com/go-gitea/test_repo.git",
true,
"https://" + userPlaceholder + "@github.com/go-gitea/test_repo.git",
},
// case 3
{
"https://mytoken@github.com/go-gitea/test_repo.git",
false,
"https://github.com/go-gitea/test_repo.git",
},
// case 4
{
"https://user:password@github.com/go-gitea/test_repo.git",
true,
"https://" + userPlaceholder + "@github.com/go-gitea/test_repo.git",
},
// case 5
{
"https://user:password@github.com/go-gitea/test_repo.git",
false,
"https://github.com/go-gitea/test_repo.git",
},
// case 6
{
"https://gi\nthub.com/go-gitea/test_repo.git",
false,
unparsableURL,
},
}
for n, c := range cases {
// uses NewURLSanitizer internally
result := NewStringURLSanitizer(c.input, c.placeholder).Replace(c.input)
assert.Equal(t, c.expected, result, "case %d: error should match", n)
}
}
func TestNewStringURLSanitizedError(t *testing.T) {
var cases = []struct {
input string
placeholder bool
expected string
}{
// case 0
{
"https://github.com/go-gitea/test_repo.git",
true,
"https://github.com/go-gitea/test_repo.git",
},
// case 1
{
"https://github.com/go-gitea/test_repo.git",
false,
"https://github.com/go-gitea/test_repo.git",
},
// case 2
{
"https://mytoken@github.com/go-gitea/test_repo.git",
true,
"https://" + userPlaceholder + "@github.com/go-gitea/test_repo.git",
},
// case 3
{
"https://mytoken@github.com/go-gitea/test_repo.git",
false,
"https://github.com/go-gitea/test_repo.git",
},
// case 4
{
"https://user:password@github.com/go-gitea/test_repo.git",
true,
"https://" + userPlaceholder + "@github.com/go-gitea/test_repo.git",
},
// case 5
{
"https://user:password@github.com/go-gitea/test_repo.git",
false,
"https://github.com/go-gitea/test_repo.git",
},
// case 6
{
"https://gi\nthub.com/go-gitea/test_repo.git",
false,
unparsableURL,
},
}
encloseText := func(input string) string {
return "test " + input + " test"
}
for n, c := range cases {
err := errors.New(encloseText(c.input))
result := NewStringURLSanitizedError(err, c.input, c.placeholder)
assert.Equal(t, encloseText(c.expected), result.Error(), "case %d: error should match", n)
}
}

16
options/locale/locale_en-US.ini
View File

@ -91,8 +91,11 @@ loading = Loading…
step1 = Step 1:
step2 = Step 2:
error = Error
error404 = The page you are trying to reach either <strong>does not exist</strong> or <strong>you are not authorized</strong> to view it.
never = Never
[error]
occurred = An error has occurred
report_message = If you are sure this is a Gitea bug, please search for issue on <a href="https://github.com/go-gitea/gitea/issues">GitHub</a> and open new issue if necessary.
@ -724,7 +727,7 @@ mirror_prune_desc = Remove obsolete remote-tracking references
mirror_interval = Mirror Interval (valid time units are 'h', 'm', 's'). 0 to disable automatic sync.
mirror_interval_invalid = The mirror interval is not valid.
mirror_address = Clone From URL
mirror_address_desc = Put any required credentials in the Clone Authorization section.
mirror_address_desc = Put any required credentials in the Authorization section.
mirror_address_url_invalid = The provided url is invalid. You must escape all components of the url correctly.
mirror_address_protocol_invalid = The provided url is invalid. Only http(s):// or git:// locations can be mirrored from.
mirror_lfs = Large File Storage (LFS)
@ -787,7 +790,7 @@ form.reach_limit_of_creation_n = You have already reached your limit of %d repos
form.name_reserved = The repository name '%s' is reserved.
form.name_pattern_not_allowed = The pattern '%s' is not allowed in a repository name.
need_auth = Clone Authorization
need_auth = Authorization
migrate_options = Migration Options
migrate_service = Migration Service
migrate_options_mirror_helper = This repository will be a <span class="text blue">mirror</span>
@ -1548,6 +1551,15 @@ settings.hooks = Webhooks
settings.githooks = Git Hooks
settings.basic_settings = Basic Settings
settings.mirror_settings = Mirror Settings
settings.mirror_settings.docs = Set up your project to automatically push and/or pull changes to/from another repository. Branches, tags, and commits will be synced automatically. <a target="_blank" rel="noopener noreferrer" href="https://docs.gitea.io/en-us/repo-mirror/">How do I mirror repositories?</a>
settings.mirror_settings.mirrored_repository = Mirrored repository
settings.mirror_settings.direction = Direction
settings.mirror_settings.direction.pull = Pull
settings.mirror_settings.direction.push = Push
settings.mirror_settings.last_update = Last update
settings.mirror_settings.push_mirror.none = No push mirrors configured
settings.mirror_settings.push_mirror.remote_url = Git Remote Repository URL
settings.mirror_settings.push_mirror.add = Add Push Mirror
settings.sync_mirror = Synchronize Now
settings.mirror_sync_in_progress = Mirror synchronization is in progress. Check back in a minute.
settings.email_notifications.enable = Enable Email Notifications

2
routers/api/v1/repo/migrate.go
View File

@ -231,7 +231,7 @@ func handleMigrateError(ctx *context.APIContext, repoOwner *models.User, remoteA
case base.IsErrNotSupported(err):
ctx.Error(http.StatusUnprocessableEntity, "", err)
default:
err = util.URLSanitizedError(err, remoteAddr)
err = util.NewStringURLSanitizedError(err, remoteAddr, true)
if strings.Contains(err.Error(), "Authentication failed") ||
strings.Contains(err.Error(), "Bad credentials") ||
strings.Contains(err.Error(), "could not read Username") {

2
routers/web/repo/migrate.go
View File

@ -101,7 +101,7 @@ func handleMigrateError(ctx *context.Context, owner *models.User, err error, nam
ctx.RenderWithErr(ctx.Tr("repo.form.name_pattern_not_allowed", err.(models.ErrNamePatternNotAllowed).Pattern), tpl, form)
default:
remoteAddr, _ := forms.ParseRemoteAddr(form.CloneAddr, form.AuthUsername, form.AuthPassword)
err = util.URLSanitizedError(err, remoteAddr)
err = util.NewStringURLSanitizedError(err, remoteAddr, true)
if strings.Contains(err.Error(), "Authentication failed") ||
strings.Contains(err.Error(), "Bad credentials") ||
strings.Contains(err.Error(), "could not read Username") {

116
routers/web/repo/setting.go
View File

@ -10,6 +10,7 @@ import (
"fmt"
"io/ioutil"
"net/http"
"strconv"
"strings"
"time"
@ -25,6 +26,7 @@ import (
"code.gitea.io/gitea/modules/structs"
"code.gitea.io/gitea/modules/timeutil"
"code.gitea.io/gitea/modules/typesniffer"
"code.gitea.io/gitea/modules/util"
"code.gitea.io/gitea/modules/validation"
"code.gitea.io/gitea/modules/web"
"code.gitea.io/gitea/routers/utils"
@ -49,6 +51,8 @@ func Settings(ctx *context.Context) {
ctx.Data["Title"] = ctx.Tr("repo.settings")
ctx.Data["PageIsSettingsOptions"] = true
ctx.Data["ForcePrivate"] = setting.Repository.ForcePrivate
ctx.Data["DisabledMirrors"] = setting.Repository.DisableMirrors
ctx.Data["DefaultMirrorInterval"] = setting.Mirror.DefaultInterval
signing, _ := models.SigningKey(ctx.Repo.Repository.RepoPath())
ctx.Data["SigningKeyAvailable"] = len(signing) > 0
@ -167,10 +171,9 @@ func SettingsPost(ctx *context.Context) {
}
}
oldUsername := mirror_service.Username(ctx.Repo.Mirror)
oldPassword := mirror_service.Password(ctx.Repo.Mirror)
if form.MirrorPassword == "" && form.MirrorUsername == oldUsername {
form.MirrorPassword = oldPassword
u, _ := git.GetRemoteAddress(ctx.Repo.Repository.RepoPath(), ctx.Repo.Mirror.GetRemoteName())
if u.User != nil && form.MirrorPassword == "" && form.MirrorUsername == u.User.Username() {
form.MirrorPassword, _ = u.User.Password()
}
address, err := forms.ParseRemoteAddr(form.MirrorAddress, form.MirrorUsername, form.MirrorPassword)
@ -226,6 +229,92 @@ func SettingsPost(ctx *context.Context) {
ctx.Flash.Info(ctx.Tr("repo.settings.mirror_sync_in_progress"))
ctx.Redirect(repo.Link() + "/settings")
case "push-mirror-sync":
m, err := selectPushMirrorByForm(form, repo)
if err != nil {
ctx.NotFound("", nil)
return
}
mirror_service.AddPushMirrorToQueue(m.ID)
ctx.Flash.Info(ctx.Tr("repo.settings.mirror_sync_in_progress"))
ctx.Redirect(repo.Link() + "/settings")
case "push-mirror-remove":
// This section doesn't require repo_name/RepoName to be set in the form, don't show it
// as an error on the UI for this action
ctx.Data["Err_RepoName"] = nil
m, err := selectPushMirrorByForm(form, repo)
if err != nil {
ctx.NotFound("", nil)
return
}
if err = mirror_service.RemovePushMirrorRemote(m); err != nil {
ctx.ServerError("RemovePushMirrorRemote", err)
return
}
if err = models.DeletePushMirrorByID(m.ID); err != nil {
ctx.ServerError("DeletePushMirrorByID", err)
return
}
ctx.Flash.Success(ctx.Tr("repo.settings.update_settings_success"))
ctx.Redirect(repo.Link() + "/settings")
case "push-mirror-add":
// This section doesn't require repo_name/RepoName to be set in the form, don't show it
// as an error on the UI for this action
ctx.Data["Err_RepoName"] = nil
interval, err := time.ParseDuration(form.PushMirrorInterval)
if err != nil || (interval != 0 && interval < setting.Mirror.MinInterval) {
ctx.Data["Err_PushMirrorInterval"] = true
ctx.RenderWithErr(ctx.Tr("repo.mirror_interval_invalid"), tplSettingsOptions, &form)
return
}
address, err := forms.ParseRemoteAddr(form.PushMirrorAddress, form.PushMirrorUsername, form.PushMirrorPassword)
if err == nil {
err = migrations.IsMigrateURLAllowed(address, ctx.User)
}
if err != nil {
ctx.Data["Err_PushMirrorAddress"] = true
handleSettingRemoteAddrError(ctx, err, form)
return
}
remoteSuffix, err := util.RandomString(10)
if err != nil {
ctx.ServerError("RandomString", err)
return
}
m := &models.PushMirror{
RepoID: repo.ID,
Repo: repo,
RemoteName: fmt.Sprintf("remote_mirror_%s", remoteSuffix),
Interval: interval,
}
if err := models.InsertPushMirror(m); err != nil {
ctx.ServerError("InsertPushMirror", err)
return
}
if err := mirror_service.AddPushMirrorRemote(m, address); err != nil {
if err := models.DeletePushMirrorByID(m.ID); err != nil {
log.Error("DeletePushMirrorByID %v", err)
}
ctx.ServerError("AddPushMirrorRemote", err)
return
}
ctx.Flash.Success(ctx.Tr("repo.settings.update_settings_success"))
ctx.Redirect(repo.Link() + "/settings")
case "advanced":
var repoChanged bool
var units []models.RepoUnit
@ -1051,3 +1140,22 @@ func SettingsDeleteAvatar(ctx *context.Context) {
}
ctx.Redirect(ctx.Repo.RepoLink + "/settings")
}
func selectPushMirrorByForm(form *forms.RepoSettingForm, repo *models.Repository) (*models.PushMirror, error) {
id, err := strconv.ParseInt(form.PushMirrorID, 10, 64)
if err != nil {
return nil, err
}
if err = repo.LoadPushMirrors(); err != nil {
return nil, err
}
for _, m := range repo.PushMirrors {
if m.ID == id {
return m, nil
}
}
return nil, fmt.Errorf("PushMirror[%v] not associated to repository %v", id, repo)
}

5
services/forms/repo_form.go
View File

@ -122,6 +122,11 @@ type RepoSettingForm struct {
MirrorPassword string
LFS bool `form:"mirror_lfs"`
LFSEndpoint string `form:"mirror_lfs_endpoint"`
PushMirrorID string
PushMirrorAddress string
PushMirrorUsername string
PushMirrorPassword string
PushMirrorInterval string
Private bool
Template bool
EnablePrune bool

574
services/mirror/mirror.go
View File

@ -7,585 +7,97 @@ package mirror
import (
"context"
"fmt"
"net/url"
"strconv"
"strings"
"time"
"code.gitea.io/gitea/models"
"code.gitea.io/gitea/modules/cache"
"code.gitea.io/gitea/modules/git"
"code.gitea.io/gitea/modules/graceful"
"code.gitea.io/gitea/modules/lfs"
"code.gitea.io/gitea/modules/log"
"code.gitea.io/gitea/modules/notification"
repo_module "code.gitea.io/gitea/modules/repository"
"code.gitea.io/gitea/modules/setting"
"code.gitea.io/gitea/modules/sync"
"code.gitea.io/gitea/modules/timeutil"
"code.gitea.io/gitea/modules/util"
)
// mirrorQueue holds an UniqueQueue object of the mirror
var mirrorQueue = sync.NewUniqueQueue(setting.Repository.MirrorQueueLength)
func readAddress(m *models.Mirror) {
if len(m.Address) > 0 {
return
}
var err error
m.Address, err = remoteAddress(m.Repo.RepoPath())
if err != nil {
log.Error("remoteAddress: %v", err)
}
}
func remoteAddress(repoPath string) (string, error) {
var cmd *git.Command
err := git.LoadGitVersion()
if err != nil {
return "", err
}
if git.CheckGitVersionAtLeast("2.7") == nil {
cmd = git.NewCommand("remote", "get-url", "origin")
} else {
cmd = git.NewCommand("config", "--get", "remote.origin.url")
}
result, err := cmd.RunInDir(repoPath)
if err != nil {
if strings.HasPrefix(err.Error(), "exit status 128 - fatal: No such remote ") {
return "", nil
}
return "", err
}
if len(result) > 0 {
return result[:len(result)-1], nil
}
return "", nil
}
// sanitizeOutput sanitizes output of a command, replacing occurrences of the
// repository's remote address with a sanitized version.
func sanitizeOutput(output, repoPath string) (string, error) {
remoteAddr, err := remoteAddress(repoPath)
if err != nil {
// if we're unable to load the remote address, then we're unable to
// sanitize.
return "", err
}
return util.SanitizeMessage(output, remoteAddr), nil
}
// AddressNoCredentials returns mirror address from Git repository config without credentials.
func AddressNoCredentials(m *models.Mirror) string {
readAddress(m)
u, err := url.Parse(m.Address)
if err != nil {
// this shouldn't happen but just return it unsanitised
return m.Address
}
u.User = nil
return u.String()
}
// UpdateAddress writes new address to Git repository and database
func UpdateAddress(m *models.Mirror, addr string) error {
repoPath := m.Repo.RepoPath()
// Remove old origin
_, err := git.NewCommand("remote", "rm", "origin").RunInDir(repoPath)
if err != nil && !strings.HasPrefix(err.Error(), "exit status 128 - fatal: No such remote ") {
return err
}
_, err = git.NewCommand("remote", "add", "origin", "--mirror=fetch", addr).RunInDir(repoPath)
if err != nil && !strings.HasPrefix(err.Error(), "exit status 128 - fatal: No such remote ") {
return err
}
if m.Repo.HasWiki() {
wikiPath := m.Repo.WikiPath()
wikiRemotePath := repo_module.WikiRemoteURL(addr)
// Remove old origin of wiki
_, err := git.NewCommand("remote", "rm", "origin").RunInDir(wikiPath)
if err != nil && !strings.HasPrefix(err.Error(), "exit status 128 - fatal: No such remote ") {
return err
}
_, err = git.NewCommand("remote", "add", "origin", "--mirror=fetch", wikiRemotePath).RunInDir(wikiPath)
if err != nil && !strings.HasPrefix(err.Error(), "exit status 128 - fatal: No such remote ") {
return err
}
}
m.Repo.OriginalURL = addr
return models.UpdateRepositoryCols(m.Repo, "original_url")
}
// gitShortEmptySha Git short empty SHA
const gitShortEmptySha = "0000000"
// mirrorSyncResult contains information of a updated reference.
// If the oldCommitID is "0000000", it means a new reference, the value of newCommitID is empty.
// If the newCommitID is "0000000", it means the reference is deleted, the value of oldCommitID is empty.
type mirrorSyncResult struct {
refName string
oldCommitID string
newCommitID string
}
// parseRemoteUpdateOutput detects create, update and delete operations of references from upstream.
func parseRemoteUpdateOutput(output string) []*mirrorSyncResult {
results := make([]*mirrorSyncResult, 0, 3)
lines := strings.Split(output, "\n")
for i := range lines {
// Make sure reference name is presented before continue
idx := strings.Index(lines[i], "-> ")
if idx == -1 {
continue
}
refName := lines[i][idx+3:]
switch {
case strings.HasPrefix(lines[i], " * "): // New reference
if strings.HasPrefix(lines[i], " * [new tag]") {
refName = git.TagPrefix + refName
} else if strings.HasPrefix(lines[i], " * [new branch]") {
refName = git.BranchPrefix + refName
}
results = append(results, &mirrorSyncResult{
refName: refName,
oldCommitID: gitShortEmptySha,
})
case strings.HasPrefix(lines[i], " - "): // Delete reference
results = append(results, &mirrorSyncResult{
refName: refName,
newCommitID: gitShortEmptySha,
})
case strings.HasPrefix(lines[i], " + "): // Force update
if idx := strings.Index(refName, " "); idx > -1 {
refName = refName[:idx]
}
delimIdx := strings.Index(lines[i][3:], " ")
if delimIdx == -1 {
log.Error("SHA delimiter not found: %q", lines[i])
continue
}
shas := strings.Split(lines[i][3:delimIdx+3], "...")
if len(shas) != 2 {
log.Error("Expect two SHAs but not what found: %q", lines[i])
continue
}
results = append(results, &mirrorSyncResult{
refName: refName,
oldCommitID: shas[0],
newCommitID: shas[1],
})
case strings.HasPrefix(lines[i], " "): // New commits of a reference
delimIdx := strings.Index(lines[i][3:], " ")
if delimIdx == -1 {
log.Error("SHA delimiter not found: %q", lines[i])
continue
}
shas := strings.Split(lines[i][3:delimIdx+3], "..")
if len(shas) != 2 {
log.Error("Expect two SHAs but not what found: %q", lines[i])
continue
}
results = append(results, &mirrorSyncResult{
refName: refName,
oldCommitID: shas[0],
newCommitID: shas[1],
})
default:
log.Warn("parseRemoteUpdateOutput: unexpected update line %q", lines[i])
}
}
return results
}
// runSync returns true if sync finished without error.
func runSync(ctx context.Context, m *models.Mirror) ([]*mirrorSyncResult, bool) {
repoPath := m.Repo.RepoPath()
wikiPath := m.Repo.WikiPath()
timeout := time.Duration(setting.Git.Timeout.Mirror) * time.Second
log.Trace("SyncMirrors [repo: %-v]: running git remote update...", m.Repo)
gitArgs := []string{"remote", "update"}
if m.EnablePrune {
gitArgs = append(gitArgs, "--prune")
}
stdoutBuilder := strings.Builder{}
stderrBuilder := strings.Builder{}
if err := git.NewCommand(gitArgs...).
SetDescription(fmt.Sprintf("Mirror.runSync: %s", m.Repo.FullName())).
RunInDirTimeoutPipeline(timeout, repoPath, &stdoutBuilder, &stderrBuilder); err != nil {
stdout := stdoutBuilder.String()
stderr := stderrBuilder.String()
// sanitize the output, since it may contain the remote address, which may
// contain a password
stderrMessage, sanitizeErr := sanitizeOutput(stderr, repoPath)
if sanitizeErr != nil {
log.Error("sanitizeOutput failed on stderr: %v", sanitizeErr)
}
stdoutMessage, sanitizeErr := sanitizeOutput(stdout, repoPath)
if sanitizeErr != nil {
log.Error("sanitizeOutput failed: %v", sanitizeErr)
}
log.Error("Failed to update mirror repository %v:\nStdout: %s\nStderr: %s\nErr: %v", m.Repo, stdoutMessage, stderrMessage, err)
desc := fmt.Sprintf("Failed to update mirror repository '%s': %s", m.Repo.FullName(), stderrMessage)
if err = models.CreateRepositoryNotice(desc); err != nil {
log.Error("CreateRepositoryNotice: %v", err)
}
return nil, false
}
output := stderrBuilder.String()
gitRepo, err := git.OpenRepository(repoPath)
if err != nil {
log.Error("OpenRepository: %v", err)
return nil, false
}
defer gitRepo.Close()
log.Trace("SyncMirrors [repo: %-v]: syncing releases with tags...", m.Repo)
if err = repo_module.SyncReleasesWithTags(m.Repo, gitRepo); err != nil {
log.Error("Failed to synchronize tags to releases for repository: %v", err)
}
if m.LFS && setting.LFS.StartServer {
log.Trace("SyncMirrors [repo: %-v]: syncing LFS objects...", m.Repo)
readAddress(m)
ep := lfs.DetermineEndpoint(m.Address, m.LFSEndpoint)
if err = repo_module.StoreMissingLfsObjectsInRepository(ctx, m.Repo, gitRepo, ep); err != nil {
log.Error("Failed to synchronize LFS objects for repository: %v", err)
}
}
log.Trace("SyncMirrors [repo: %-v]: updating size of repository", m.Repo)
if err := m.Repo.UpdateSize(models.DefaultDBContext()); err != nil {
log.Error("Failed to update size for mirror repository: %v", err)
}
if m.Repo.HasWiki() {
log.Trace("SyncMirrors [repo: %-v Wiki]: running git remote update...", m.Repo)
stderrBuilder.Reset()
stdoutBuilder.Reset()
if err := git.NewCommand("remote", "update", "--prune").
SetDescription(fmt.Sprintf("Mirror.runSync Wiki: %s ", m.Repo.FullName())).
RunInDirTimeoutPipeline(timeout, wikiPath, &stdoutBuilder, &stderrBuilder); err != nil {
stdout := stdoutBuilder.String()
stderr := stderrBuilder.String()
// sanitize the output, since it may contain the remote address, which may
// contain a password
stderrMessage, sanitizeErr := sanitizeOutput(stderr, wikiPath)
if sanitizeErr != nil {
log.Error("sanitizeOutput failed on stderr: %v", sanitizeErr)
}
stdoutMessage, sanitizeErr := sanitizeOutput(stdout, wikiPath)
if sanitizeErr != nil {
log.Error("sanitizeOutput failed: %v", sanitizeErr)
}
log.Error("Failed to update mirror repository wiki %v:\nStdout: %s\nStderr: %s\nErr: %v", m.Repo, stdoutMessage, stderrMessage, err)
desc := fmt.Sprintf("Failed to update mirror repository wiki '%s': %s", m.Repo.FullName(), stderrMessage)
if err = models.CreateRepositoryNotice(desc); err != nil {
log.Error("CreateRepositoryNotice: %v", err)
}
return nil, false
}
log.Trace("SyncMirrors [repo: %-v Wiki]: git remote update complete", m.Repo)
}
log.Trace("SyncMirrors [repo: %-v]: invalidating mirror branch caches...", m.Repo)
branches, _, err := repo_module.GetBranches(m.Repo, 0, 0)
if err != nil {
log.Error("GetBranches: %v", err)
return nil, false
}
for _, branch := range branches {
cache.Remove(m.Repo.GetCommitsCountCacheKey(branch.Name, true))
}
m.UpdatedUnix = timeutil.TimeStampNow()
return parseRemoteUpdateOutput(output), true
}
// Address returns mirror address from Git repository config without credentials.
func Address(m *models.Mirror) string {
readAddress(m)
return util.SanitizeURLCredentials(m.Address, false)
}
// Username returns the mirror address username
func Username(m *models.Mirror) string {
readAddress(m)
u, err := url.Parse(m.Address)
if err != nil {
// this shouldn't happen but if it does return ""
return ""
}
return u.User.Username()
}
// Password returns the mirror address password
func Password(m *models.Mirror) string {
readAddress(m)
u, err := url.Parse(m.Address)
if err != nil {
// this shouldn't happen but if it does return ""
return ""
}
password, _ := u.User.Password()
return password
}
// Update checks and updates mirror repositories.
func Update(ctx context.Context) error {
log.Trace("Doing: Update")
if err := models.MirrorsIterate(func(idx int, bean interface{}) error {
m := bean.(*models.Mirror)
handler := func(idx int, bean interface{}) error {
var item string
if m, ok := bean.(*models.Mirror); ok {
if m.Repo == nil {
log.Error("Disconnected mirror repository found: %d", m.ID)
log.Error("Disconnected mirror found: %d", m.ID)
return nil
}
item = fmt.Sprintf("pull %d", m.RepoID)
} else if m, ok := bean.(*models.PushMirror); ok {
if m.Repo == nil {
log.Error("Disconnected push-mirror found: %d", m.ID)
return nil
}
item = fmt.Sprintf("push %d", m.ID)
} else {
log.Error("Unknown bean: %v", bean)
return nil
}
select {
case <-ctx.Done():
return fmt.Errorf("Aborted")
default:
mirrorQueue.Add(m.RepoID)
mirrorQueue.Add(item)
return nil
}
}); err != nil {
log.Trace("Update: %v", err)
}
if err := models.MirrorsIterate(handler); err != nil {
log.Error("MirrorsIterate: %v", err)
return err
}
if err := models.PushMirrorsIterate(handler); err != nil {
log.Error("PushMirrorsIterate: %v", err)
return err
}
log.Trace("Finished: Update")
return nil
}
// SyncMirrors checks and syncs mirrors.
// syncMirrors checks and syncs mirrors.
// FIXME: graceful: this should be a persistable queue
func SyncMirrors(ctx context.Context) {
func syncMirrors(ctx context.Context) {
// Start listening on new sync requests.
for {
select {
case <-ctx.Done():
mirrorQueue.Close()
return
case repoID := <-mirrorQueue.Queue():
syncMirror(ctx, repoID)
}
}
}
func syncMirror(ctx context.Context, repoID string) {
log.Trace("SyncMirrors [repo_id: %v]", repoID)
defer func() {
err := recover()
if err == nil {
return
}
// There was a panic whilst syncMirrors...
log.Error("PANIC whilst syncMirrors[%s] Panic: %v\nStacktrace: %s", repoID, err, log.Stack(2))
}()
mirrorQueue.Remove(repoID)
id, _ := strconv.ParseInt(repoID, 10, 64)
m, err := models.GetMirrorByRepoID(id)
if err != nil {
log.Error("GetMirrorByRepoID [%s]: %v", repoID, err)
return
}
log.Trace("SyncMirrors [repo: %-v]: Running Sync", m.Repo)
results, ok := runSync(ctx, m)
if !ok {
return
}
log.Trace("SyncMirrors [repo: %-v]: Scheduling next update", m.Repo)
m.ScheduleNextUpdate()
if err = models.UpdateMirror(m); err != nil {
log.Error("UpdateMirror [%s]: %v", repoID, err)
return
}
var gitRepo *git.Repository
if len(results) == 0 {
log.Trace("SyncMirrors [repo: %-v]: no branches updated", m.Repo)
case item := <-mirrorQueue.Queue():
id, _ := strconv.ParseInt(item[5:], 10, 64)
if strings.HasPrefix(item, "pull") {
_ = SyncPullMirror(ctx, id)
} else if strings.HasPrefix(item, "push") {
_ = SyncPushMirror(ctx, id)
} else {
log.Trace("SyncMirrors [repo: %-v]: %d branches updated", m.Repo, len(results))
gitRepo, err = git.OpenRepository(m.Repo.RepoPath())
if err != nil {
log.Error("OpenRepository [%d]: %v", m.RepoID, err)
return
log.Error("Unknown item in queue: %v", item)
}
defer gitRepo.Close()
if ok := checkAndUpdateEmptyRepository(m, gitRepo, results); !ok {
return
mirrorQueue.Remove(item)
}
}
for _, result := range results {
// Discard GitHub pull requests, i.e. refs/pull/*
if strings.HasPrefix(result.refName, "refs/pull/") {
continue
}
tp, _ := git.SplitRefName(result.refName)
// Create reference
if result.oldCommitID == gitShortEmptySha {
if tp == git.TagPrefix {
tp = "tag"
} else if tp == git.BranchPrefix {
tp = "branch"
}
commitID, err := gitRepo.GetRefCommitID(result.refName)
if err != nil {
log.Error("gitRepo.GetRefCommitID [repo_id: %s, ref_name: %s]: %v", m.RepoID, result.refName, err)
continue
}
notification.NotifySyncPushCommits(m.Repo.MustOwner(), m.Repo, &repo_module.PushUpdateOptions{
RefFullName: result.refName,
OldCommitID: git.EmptySHA,
NewCommitID: commitID,
}, repo_module.NewPushCommits())
notification.NotifySyncCreateRef(m.Repo.MustOwner(), m.Repo, tp, result.refName)
continue
}
// Delete reference
if result.newCommitID == gitShortEmptySha {
notification.NotifySyncDeleteRef(m.Repo.MustOwner(), m.Repo, tp, result.refName)
continue
}
// Push commits
oldCommitID, err := git.GetFullCommitID(gitRepo.Path, result.oldCommitID)
if err != nil {
log.Error("GetFullCommitID [%d]: %v", m.RepoID, err)
continue
}
newCommitID, err := git.GetFullCommitID(gitRepo.Path, result.newCommitID)
if err != nil {
log.Error("GetFullCommitID [%d]: %v", m.RepoID, err)
continue
}
commits, err := gitRepo.CommitsBetweenIDs(newCommitID, oldCommitID)
if err != nil {
log.Error("CommitsBetweenIDs [repo_id: %d, new_commit_id: %s, old_commit_id: %s]: %v", m.RepoID, newCommitID, oldCommitID, err)
continue
}
theCommits := repo_module.ListToPushCommits(commits)
if len(theCommits.Commits) > setting.UI.FeedMaxCommitNum {
theCommits.Commits = theCommits.Commits[:setting.UI.FeedMaxCommitNum]
}
theCommits.CompareURL = m.Repo.ComposeCompareURL(oldCommitID, newCommitID)
notification.NotifySyncPushCommits(m.Repo.MustOwner(), m.Repo, &repo_module.PushUpdateOptions{
RefFullName: result.refName,
OldCommitID: oldCommitID,
NewCommitID: newCommitID,
}, theCommits)
}
log.Trace("SyncMirrors [repo: %-v]: done notifying updated branches/tags - now updating last commit time", m.Repo)
// Get latest commit date and update to current repository updated time
commitDate, err := git.GetLatestCommitTime(m.Repo.RepoPath())
if err != nil {
log.Error("GetLatestCommitDate [%d]: %v", m.RepoID, err)
return
}
if err = models.UpdateRepositoryUpdatedTime(m.RepoID, commitDate); err != nil {
log.Error("Update repository 'updated_unix' [%d]: %v", m.RepoID, err)
return
}
log.Trace("SyncMirrors [repo: %-v]: Successfully updated", m.Repo)
}
func checkAndUpdateEmptyRepository(m *models.Mirror, gitRepo *git.Repository, results []*mirrorSyncResult) bool {
if !m.Repo.IsEmpty {
return true
}
hasDefault := false
hasMaster := false
hasMain := false
defaultBranchName := m.Repo.DefaultBranch
if len(defaultBranchName) == 0 {
defaultBranchName = setting.Repository.DefaultBranch
}
firstName := ""
for _, result := range results {
if strings.HasPrefix(result.refName, "refs/pull/") {
continue
}
tp, name := git.SplitRefName(result.refName)
if len(tp) > 0 && tp != git.BranchPrefix {
continue
}
if len(firstName) == 0 {
firstName = name
}
hasDefault = hasDefault || name == defaultBranchName
hasMaster = hasMaster || name == "master"
hasMain = hasMain || name == "main"
}
if len(firstName) > 0 {
if hasDefault {
m.Repo.DefaultBranch = defaultBranchName
} else if hasMaster {
m.Repo.DefaultBranch = "master"
} else if hasMain {
m.Repo.DefaultBranch = "main"
} else {
m.Repo.DefaultBranch = firstName
}
// Update the git repository default branch
if err := gitRepo.SetDefaultBranch(m.Repo.DefaultBranch); err != nil {
if !git.IsErrUnsupportedVersion(err) {
log.Error("Failed to update default branch of underlying git repository %-v. Error: %v", m.Repo, err)
desc := fmt.Sprintf("Failed to uupdate default branch of underlying git repository '%s': %v", m.Repo.RepoPath(), err)
if err = models.CreateRepositoryNotice(desc); err != nil {
log.Error("CreateRepositoryNotice: %v", err)
}
return false
}
}
m.Repo.IsEmpty = false
// Update the is empty and default_branch columns
if err := models.UpdateRepositoryCols(m.Repo, "default_branch", "is_empty"); err != nil {
log.Error("Failed to update default branch of repository %-v. Error: %v", m.Repo, err)
desc := fmt.Sprintf("Failed to uupdate default branch of repository '%s': %v", m.Repo.RepoPath(), err)
if err = models.CreateRepositoryNotice(desc); err != nil {
log.Error("CreateRepositoryNotice: %v", err)
}
return false
}
}
return true
}
// InitSyncMirrors initializes a go routine to sync the mirrors
func InitSyncMirrors() {
go graceful.GetManager().RunWithShutdownContext(SyncMirrors)
go graceful.GetManager().RunWithShutdownContext(syncMirrors)
}
// StartToMirror adds repoID to mirror queue
func StartToMirror(repoID int64) {
go mirrorQueue.Add(repoID)
go mirrorQueue.Add(fmt.Sprintf("pull %d", repoID))
}
// AddPushMirrorToQueue adds the push mirror to the queue
func AddPushMirrorToQueue(mirrorID int64) {
go mirrorQueue.Add(fmt.Sprintf("push %d", mirrorID))
}

452
services/mirror/mirror_pull.go Normal file
View File

@ -0,0 +1,452 @@
// Copyright 2021 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package mirror
import (
"context"
"fmt"
"strings"
"time"
"code.gitea.io/gitea/models"
"code.gitea.io/gitea/modules/cache"
"code.gitea.io/gitea/modules/git"
"code.gitea.io/gitea/modules/lfs"
"code.gitea.io/gitea/modules/log"
"code.gitea.io/gitea/modules/notification"
repo_module "code.gitea.io/gitea/modules/repository"
"code.gitea.io/gitea/modules/setting"
"code.gitea.io/gitea/modules/timeutil"
"code.gitea.io/gitea/modules/util"
)
// gitShortEmptySha Git short empty SHA
const gitShortEmptySha = "0000000"
// UpdateAddress writes new address to Git repository and database
func UpdateAddress(m *models.Mirror, addr string) error {
remoteName := m.GetRemoteName()
repoPath := m.Repo.RepoPath()
// Remove old remote
_, err := git.NewCommand("remote", "rm", remoteName).RunInDir(repoPath)
if err != nil && !strings.HasPrefix(err.Error(), "exit status 128 - fatal: No such remote ") {
return err
}
_, err = git.NewCommand("remote", "add", remoteName, "--mirror=fetch", addr).RunInDir(repoPath)
if err != nil && !strings.HasPrefix(err.Error(), "exit status 128 - fatal: No such remote ") {
return err
}
if m.Repo.HasWiki() {
wikiPath := m.Repo.WikiPath()
wikiRemotePath := repo_module.WikiRemoteURL(addr)
// Remove old remote of wiki
_, err := git.NewCommand("remote", "rm", remoteName).RunInDir(wikiPath)
if err != nil && !strings.HasPrefix(err.Error(), "exit status 128 - fatal: No such remote ") {
return err
}
_, err = git.NewCommand("remote", "add", remoteName, "--mirror=fetch", wikiRemotePath).RunInDir(wikiPath)
if err != nil && !strings.HasPrefix(err.Error(), "exit status 128 - fatal: No such remote ") {
return err
}
}
m.Repo.OriginalURL = addr
return models.UpdateRepositoryCols(m.Repo, "original_url")
}
// mirrorSyncResult contains information of a updated reference.
// If the oldCommitID is "0000000", it means a new reference, the value of newCommitID is empty.
// If the newCommitID is "0000000", it means the reference is deleted, the value of oldCommitID is empty.
type mirrorSyncResult struct {
refName string
oldCommitID string
newCommitID string
}
// parseRemoteUpdateOutput detects create, update and delete operations of references from upstream.
func parseRemoteUpdateOutput(output string) []*mirrorSyncResult {
results := make([]*mirrorSyncResult, 0, 3)
lines := strings.Split(output, "\n")
for i := range lines {
// Make sure reference name is presented before continue
idx := strings.Index(lines[i], "-> ")
if idx == -1 {
continue
}
refName := lines[i][idx+3:]
switch {
case strings.HasPrefix(lines[i], " * "): // New reference
if strings.HasPrefix(lines[i], " * [new tag]") {
refName = git.TagPrefix + refName
} else if strings.HasPrefix(lines[i], " * [new branch]") {
refName = git.BranchPrefix + refName
}
results = append(results, &mirrorSyncResult{
refName: refName,
oldCommitID: gitShortEmptySha,
})
case strings.HasPrefix(lines[i], " - "): // Delete reference
results = append(results, &mirrorSyncResult{
refName: refName,
newCommitID: gitShortEmptySha,
})
case strings.HasPrefix(lines[i], " + "): // Force update
if idx := strings.Index(refName, " "); idx > -1 {
refName = refName[:idx]
}
delimIdx := strings.Index(lines[i][3:], " ")
if delimIdx == -1 {
log.Error("SHA delimiter not found: %q", lines[i])
continue
}
shas := strings.Split(lines[i][3:delimIdx+3], "...")
if len(shas) != 2 {
log.Error("Expect two SHAs but not what found: %q", lines[i])
continue
}
results = append(results, &mirrorSyncResult{
refName: refName,
oldCommitID: shas[0],
newCommitID: shas[1],
})
case strings.HasPrefix(lines[i], " "): // New commits of a reference
delimIdx := strings.Index(lines[i][3:], " ")
if delimIdx == -1 {
log.Error("SHA delimiter not found: %q", lines[i])
continue
}
shas := strings.Split(lines[i][3:delimIdx+3], "..")
if len(shas) != 2 {
log.Error("Expect two SHAs but not what found: %q", lines[i])
continue
}
results = append(results, &mirrorSyncResult{
refName: refName,
oldCommitID: shas[0],
newCommitID: shas[1],
})
default:
log.Warn("parseRemoteUpdateOutput: unexpected update line %q", lines[i])
}
}
return results
}
// runSync returns true if sync finished without error.
func runSync(ctx context.Context, m *models.Mirror) ([]*mirrorSyncResult, bool) {
repoPath := m.Repo.RepoPath()
wikiPath := m.Repo.WikiPath()
timeout := time.Duration(setting.Git.Timeout.Mirror) * time.Second
log.Trace("SyncMirrors [repo: %-v]: running git remote update...", m.Repo)
gitArgs := []string{"remote", "update"}
if m.EnablePrune {
gitArgs = append(gitArgs, "--prune")
}
gitArgs = append(gitArgs, m.GetRemoteName())
remoteAddr, remoteErr := git.GetRemoteAddress(repoPath, m.GetRemoteName())
if remoteErr != nil {
log.Error("GetRemoteAddress Error %v", remoteErr)
}
stdoutBuilder := strings.Builder{}
stderrBuilder := strings.Builder{}
if err := git.NewCommand(gitArgs...).
SetDescription(fmt.Sprintf("Mirror.runSync: %s", m.Repo.FullName())).
RunInDirTimeoutPipeline(timeout, repoPath, &stdoutBuilder, &stderrBuilder); err != nil {
stdout := stdoutBuilder.String()
stderr := stderrBuilder.String()
// sanitize the output, since it may contain the remote address, which may
// contain a password
sanitizer := util.NewURLSanitizer(remoteAddr, true)
stderrMessage := sanitizer.Replace(stderr)
stdoutMessage := sanitizer.Replace(stdout)
log.Error("Failed to update mirror repository %v:\nStdout: %s\nStderr: %s\nErr: %v", m.Repo, stdoutMessage, stderrMessage, err)
desc := fmt.Sprintf("Failed to update mirror repository '%s': %s", repoPath, stderrMessage)
if err = models.CreateRepositoryNotice(desc); err != nil {
log.Error("CreateRepositoryNotice: %v", err)
}
return nil, false
}
output := stderrBuilder.String()
gitRepo, err := git.OpenRepository(repoPath)
if err != nil {
log.Error("OpenRepository: %v", err)
return nil, false
}
log.Trace("SyncMirrors [repo: %-v]: syncing releases with tags...", m.Repo)
if err = repo_module.SyncReleasesWithTags(m.Repo, gitRepo); err != nil {
log.Error("Failed to synchronize tags to releases for repository: %v", err)
}
if m.LFS && setting.LFS.StartServer {
log.Trace("SyncMirrors [repo: %-v]: syncing LFS objects...", m.Repo)
ep := lfs.DetermineEndpoint(remoteAddr.String(), m.LFSEndpoint)
if err = repo_module.StoreMissingLfsObjectsInRepository(ctx, m.Repo, gitRepo, ep); err != nil {
log.Error("Failed to synchronize LFS objects for repository: %v", err)
}
}
gitRepo.Close()
log.Trace("SyncMirrors [repo: %-v]: updating size of repository", m.Repo)
if err := m.Repo.UpdateSize(models.DefaultDBContext()); err != nil {
log.Error("Failed to update size for mirror repository: %v", err)
}
if m.Repo.HasWiki() {
log.Trace("SyncMirrors [repo: %-v Wiki]: running git remote update...", m.Repo)
stderrBuilder.Reset()
stdoutBuilder.Reset()
if err := git.NewCommand("remote", "update", "--prune", m.GetRemoteName()).
SetDescription(fmt.Sprintf("Mirror.runSync Wiki: %s ", m.Repo.FullName())).
RunInDirTimeoutPipeline(timeout, wikiPath, &stdoutBuilder, &stderrBuilder); err != nil {
stdout := stdoutBuilder.String()
stderr := stderrBuilder.String()
// sanitize the output, since it may contain the remote address, which may
// contain a password
remoteAddr, remoteErr := git.GetRemoteAddress(wikiPath, m.GetRemoteName())
if remoteErr != nil {
log.Error("GetRemoteAddress Error %v", remoteErr)
}
sanitizer := util.NewURLSanitizer(remoteAddr, true)
stderrMessage := sanitizer.Replace(stderr)
stdoutMessage := sanitizer.Replace(stdout)
log.Error("Failed to update mirror repository wiki %v:\nStdout: %s\nStderr: %s\nErr: %v", m.Repo, stdoutMessage, stderrMessage, err)
desc := fmt.Sprintf("Failed to update mirror repository wiki '%s': %s", wikiPath, stderrMessage)
if err = models.CreateRepositoryNotice(desc); err != nil {
log.Error("CreateRepositoryNotice: %v", err)
}
return nil, false
}
log.Trace("SyncMirrors [repo: %-v Wiki]: git remote update complete", m.Repo)
}
log.Trace("SyncMirrors [repo: %-v]: invalidating mirror branch caches...", m.Repo)
branches, _, err := repo_module.GetBranches(m.Repo, 0, 0)
if err != nil {
log.Error("GetBranches: %v", err)
return nil, false
}
for _, branch := range branches {
cache.Remove(m.Repo.GetCommitsCountCacheKey(branch.Name, true))
}
m.UpdatedUnix = timeutil.TimeStampNow()
return parseRemoteUpdateOutput(output), true
}
// SyncPullMirror starts the sync of the pull mirror and schedules the next run.
func SyncPullMirror(ctx context.Context, repoID int64) bool {
log.Trace("SyncMirrors [repo_id: %v]", repoID)
defer func() {
err := recover()
if err == nil {
return
}
// There was a panic whilst syncMirrors...
log.Error("PANIC whilst syncMirrors[%d] Panic: %v\nStacktrace: %s", repoID, err, log.Stack(2))
}()
m, err := models.GetMirrorByRepoID(repoID)
if err != nil {
log.Error("GetMirrorByRepoID [%d]: %v", repoID, err)
return false
}
log.Trace("SyncMirrors [repo: %-v]: Running Sync", m.Repo)
results, ok := runSync(ctx, m)
if !ok {
return false
}
log.Trace("SyncMirrors [repo: %-v]: Scheduling next update", m.Repo)
m.ScheduleNextUpdate()
if err = models.UpdateMirror(m); err != nil {
log.Error("UpdateMirror [%d]: %v", m.RepoID, err)
return false
}
var gitRepo *git.Repository
if len(results) == 0 {
log.Trace("SyncMirrors [repo: %-v]: no branches updated", m.Repo)
} else {
log.Trace("SyncMirrors [repo: %-v]: %d branches updated", m.Repo, len(results))
gitRepo, err = git.OpenRepository(m.Repo.RepoPath())
if err != nil {
log.Error("OpenRepository [%d]: %v", m.RepoID, err)
return false
}
defer gitRepo.Close()
if ok := checkAndUpdateEmptyRepository(m, gitRepo, results); !ok {
return false
}
}
for _, result := range results {
// Discard GitHub pull requests, i.e. refs/pull/*
if strings.HasPrefix(result.refName, "refs/pull/") {
continue
}
tp, _ := git.SplitRefName(result.refName)
// Create reference
if result.oldCommitID == gitShortEmptySha {
if tp == git.TagPrefix {
tp = "tag"
} else if tp == git.BranchPrefix {
tp = "branch"
}
commitID, err := gitRepo.GetRefCommitID(result.refName)
if err != nil {
log.Error("gitRepo.GetRefCommitID [repo_id: %d, ref_name: %s]: %v", m.RepoID, result.refName, err)
continue
}
notification.NotifySyncPushCommits(m.Repo.MustOwner(), m.Repo, &repo_module.PushUpdateOptions{
RefFullName: result.refName,
OldCommitID: git.EmptySHA,
NewCommitID: commitID,
}, repo_module.NewPushCommits())
notification.NotifySyncCreateRef(m.Repo.MustOwner(), m.Repo, tp, result.refName)
continue
}
// Delete reference
if result.newCommitID == gitShortEmptySha {
notification.NotifySyncDeleteRef(m.Repo.MustOwner(), m.Repo, tp, result.refName)
continue
}
// Push commits
oldCommitID, err := git.GetFullCommitID(gitRepo.Path, result.oldCommitID)
if err != nil {
log.Error("GetFullCommitID [%d]: %v", m.RepoID, err)
continue
}
newCommitID, err := git.GetFullCommitID(gitRepo.Path, result.newCommitID)
if err != nil {
log.Error("GetFullCommitID [%d]: %v", m.RepoID, err)
continue
}
commits, err := gitRepo.CommitsBetweenIDs(newCommitID, oldCommitID)
if err != nil {
log.Error("CommitsBetweenIDs [repo_id: %d, new_commit_id: %s, old_commit_id: %s]: %v", m.RepoID, newCommitID, oldCommitID, err)
continue
}
theCommits := repo_module.ListToPushCommits(commits)
if len(theCommits.Commits) > setting.UI.FeedMaxCommitNum {
theCommits.Commits = theCommits.Commits[:setting.UI.FeedMaxCommitNum]
}
theCommits.CompareURL = m.Repo.ComposeCompareURL(oldCommitID, newCommitID)
notification.NotifySyncPushCommits(m.Repo.MustOwner(), m.Repo, &repo_module.PushUpdateOptions{
RefFullName: result.refName,
OldCommitID: oldCommitID,
NewCommitID: newCommitID,
}, theCommits)
}
log.Trace("SyncMirrors [repo: %-v]: done notifying updated branches/tags - now updating last commit time", m.Repo)
// Get latest commit date and update to current repository updated time
commitDate, err := git.GetLatestCommitTime(m.Repo.RepoPath())
if err != nil {
log.Error("GetLatestCommitDate [%d]: %v", m.RepoID, err)
return false
}
if err = models.UpdateRepositoryUpdatedTime(m.RepoID, commitDate); err != nil {
log.Error("Update repository 'updated_unix' [%d]: %v", m.RepoID, err)
return false
}
log.Trace("SyncMirrors [repo: %-v]: Successfully updated", m.Repo)
return true
}
func checkAndUpdateEmptyRepository(m *models.Mirror, gitRepo *git.Repository, results []*mirrorSyncResult) bool {
if !m.Repo.IsEmpty {
return true
}
hasDefault := false
hasMaster := false
hasMain := false
defaultBranchName := m.Repo.DefaultBranch
if len(defaultBranchName) == 0 {
defaultBranchName = setting.Repository.DefaultBranch
}
firstName := ""
for _, result := range results {
if strings.HasPrefix(result.refName, "refs/pull/") {
continue
}
tp, name := git.SplitRefName(result.refName)
if len(tp) > 0 && tp != git.BranchPrefix {
continue
}
if len(firstName) == 0 {
firstName = name
}
hasDefault = hasDefault || name == defaultBranchName
hasMaster = hasMaster || name == "master"
hasMain = hasMain || name == "main"
}
if len(firstName) > 0 {
if hasDefault {
m.Repo.DefaultBranch = defaultBranchName
} else if hasMaster {
m.Repo.DefaultBranch = "master"
} else if hasMain {
m.Repo.DefaultBranch = "main"
} else {
m.Repo.DefaultBranch = firstName
}
// Update the git repository default branch
if err := gitRepo.SetDefaultBranch(m.Repo.DefaultBranch); err != nil {
if !git.IsErrUnsupportedVersion(err) {
log.Error("Failed to update default branch of underlying git repository %-v. Error: %v", m.Repo, err)
desc := fmt.Sprintf("Failed to uupdate default branch of underlying git repository '%s': %v", m.Repo.RepoPath(), err)
if err = models.CreateRepositoryNotice(desc); err != nil {
log.Error("CreateRepositoryNotice: %v", err)
}
return false
}
}
m.Repo.IsEmpty = false
// Update the is empty and default_branch columns
if err := models.UpdateRepositoryCols(m.Repo, "default_branch", "is_empty"); err != nil {
log.Error("Failed to update default branch of repository %-v. Error: %v", m.Repo, err)
desc := fmt.Sprintf("Failed to uupdate default branch of repository '%s': %v", m.Repo.RepoPath(), err)
if err = models.CreateRepositoryNotice(desc); err != nil {
log.Error("CreateRepositoryNotice: %v", err)
}
return false
}
}
return true
}

242
services/mirror/mirror_push.go Normal file
View File

@ -0,0 +1,242 @@
// Copyright 2021 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package mirror
import (
"context"
"errors"
"io"
"net/url"
"regexp"
"time"
"code.gitea.io/gitea/models"
"code.gitea.io/gitea/modules/git"
"code.gitea.io/gitea/modules/lfs"
"code.gitea.io/gitea/modules/log"
"code.gitea.io/gitea/modules/repository"
"code.gitea.io/gitea/modules/setting"
"code.gitea.io/gitea/modules/timeutil"
"code.gitea.io/gitea/modules/util"
)
var stripExitStatus = regexp.MustCompile(`exit status \d+ - `)
// AddPushMirrorRemote registers the push mirror remote.
func AddPushMirrorRemote(m *models.PushMirror, addr string) error {
addRemoteAndConfig := func(addr, path string) error {
if _, err := git.NewCommand("remote", "add", "--mirror=push", m.RemoteName, addr).RunInDir(path); err != nil {
return err
}
if _, err := git.NewCommand("config", "--add", "remote."+m.RemoteName+".push", "+refs/heads/*:refs/heads/*").RunInDir(path); err != nil {
return err
}
if _, err := git.NewCommand("config", "--add", "remote."+m.RemoteName+".push", "+refs/tags/*:refs/tags/*").RunInDir(path); err != nil {
return err
}
return nil
}
if err := addRemoteAndConfig(addr, m.Repo.RepoPath()); err != nil {
return err
}
if m.Repo.HasWiki() {
wikiRemoteURL := repository.WikiRemoteURL(addr)
if len(wikiRemoteURL) > 0 {
if err := addRemoteAndConfig(wikiRemoteURL, m.Repo.WikiPath()); err != nil {
return err
}
}
}
return nil
}
// RemovePushMirrorRemote removes the push mirror remote.
func RemovePushMirrorRemote(m *models.PushMirror) error {
cmd := git.NewCommand("remote", "rm", m.RemoteName)
if _, err := cmd.RunInDir(m.Repo.RepoPath()); err != nil {
return err
}
if m.Repo.HasWiki() {
if _, err := cmd.RunInDir(m.Repo.WikiPath()); err != nil {
// The wiki remote may not exist
log.Warn("Wiki Remote[%d] could not be removed: %v", m.ID, err)
}
}
return nil
}
// SyncPushMirror starts the sync of the push mirror and schedules the next run.
func SyncPushMirror(ctx context.Context, mirrorID int64) bool {
log.Trace("SyncPushMirror [mirror: %d]", mirrorID)
defer func() {
err := recover()
if err == nil {
return
}
// There was a panic whilst syncPushMirror...
log.Error("PANIC whilst syncPushMirror[%d] Panic: %v\nStacktrace: %s", mirrorID, err, log.Stack(2))
}()
m, err := models.GetPushMirrorByID(mirrorID)
if err != nil {
log.Error("GetPushMirrorByID [%d]: %v", mirrorID, err)
return false
}
m.LastError = ""
log.Trace("SyncPushMirror [mirror: %d][repo: %-v]: Running Sync", m.ID, m.Repo)
err = runPushSync(ctx, m)
if err != nil {
log.Error("SyncPushMirror [mirror: %d][repo: %-v]: %v", m.ID, m.Repo, err)
m.LastError = stripExitStatus.ReplaceAllLiteralString(err.Error(), "")
}
m.LastUpdateUnix = timeutil.TimeStampNow()
if err := models.UpdatePushMirror(m); err != nil {
log.Error("UpdatePushMirror [%d]: %v", m.ID, err)
return false
}
log.Trace("SyncPushMirror [mirror: %d][repo: %-v]: Finished", m.ID, m.Repo)
return err == nil
}
func runPushSync(ctx context.Context, m *models.PushMirror) error {
timeout := time.Duration(setting.Git.Timeout.Mirror) * time.Second
performPush := func(path string) error {
remoteAddr, err := git.GetRemoteAddress(path, m.RemoteName)
if err != nil {
log.Error("GetRemoteAddress(%s) Error %v", path, err)
return errors.New("Unexpected error")
}
if setting.LFS.StartServer {
log.Trace("SyncMirrors [repo: %-v]: syncing LFS objects...", m.Repo)
gitRepo, err := git.OpenRepository(path)
if err != nil {
log.Error("OpenRepository: %v", err)
return errors.New("Unexpected error")
}
defer gitRepo.Close()
ep := lfs.DetermineEndpoint(remoteAddr.String(), "")
if err := pushAllLFSObjects(ctx, gitRepo, ep); err != nil {
return util.NewURLSanitizedError(err, remoteAddr, true)
}
}
log.Trace("Pushing %s mirror[%d] remote %s", path, m.ID, m.RemoteName)
if err := git.Push(path, git.PushOptions{
Remote: m.RemoteName,
Force: true,
Mirror: true,
Timeout: timeout,
}); err != nil {
log.Error("Error pushing %s mirror[%d] remote %s: %v", path, m.ID, m.RemoteName, err)
return util.NewURLSanitizedError(err, remoteAddr, true)
}
return nil
}
err := performPush(m.Repo.RepoPath())
if err != nil {
return err
}
if m.Repo.HasWiki() {
wikiPath := m.Repo.WikiPath()
_, err := git.GetRemoteAddress(wikiPath, m.RemoteName)
if err == nil {
err := performPush(wikiPath)
if err != nil {
return err
}
} else {
log.Trace("Skipping wiki: No remote configured")
}
}
return nil
}
func pushAllLFSObjects(ctx context.Context, gitRepo *git.Repository, endpoint *url.URL) error {
client := lfs.NewClient(endpoint)
contentStore := lfs.NewContentStore()
pointerChan := make(chan lfs.PointerBlob)
errChan := make(chan error, 1)
go lfs.SearchPointerBlobs(ctx, gitRepo, pointerChan, errChan)
uploadObjects := func(pointers []lfs.Pointer) error {
err := client.Upload(ctx, pointers, func(p lfs.Pointer, objectError error) (io.ReadCloser, error) {
if objectError != nil {
return nil, objectError
}
content, err := contentStore.Get(p)
if err != nil {
log.Error("Error reading LFS object %v: %v", p, err)
}
return content, err
})
if err != nil {
select {
case <-ctx.Done():
return nil
default:
}
}
return err
}
var batch []lfs.Pointer
for pointerBlob := range pointerChan {
exists, err := contentStore.Exists(pointerBlob.Pointer)
if err != nil {
log.Error("Error checking if LFS object %v exists: %v", pointerBlob.Pointer, err)
return err
}
if !exists {
log.Trace("Skipping missing LFS object %v", pointerBlob.Pointer)
continue
}
batch = append(batch, pointerBlob.Pointer)
if len(batch) >= client.BatchSize() {
if err := uploadObjects(batch); err != nil {
return err
}
batch = nil
}
}
if len(batch) > 0 {
if err := uploadObjects(batch); err != nil {
return err
}
}
err, has := <-errChan
if has {
log.Error("Error enumerating LFS objects for repository: %v", err)
return err
}
return nil
}

2
templates/repo/header.tmpl
View File

@ -36,7 +36,7 @@
{{end}}
</div>
</div>
{{if .IsMirror}}<div class="fork-flag">{{$.i18n.Tr "repo.mirror_from"}} <a target="_blank" rel="noopener noreferrer" href="{{if .SanitizedOriginalURL}}{{.SanitizedOriginalURL}}{{else}}{{MirrorAddress $.Mirror}}{{end}}">{{if .SanitizedOriginalURL}}{{.SanitizedOriginalURL}}{{else}}{{MirrorAddress $.Mirror}}{{end}}</a></div>{{end}}
{{if .IsMirror}}<div class="fork-flag">{{$.i18n.Tr "repo.mirror_from"}} <a target="_blank" rel="noopener noreferrer" href="{{if .SanitizedOriginalURL}}{{.SanitizedOriginalURL}}{{else}}{{(MirrorRemoteAddress $.Mirror).Address}}{{end}}">{{if .SanitizedOriginalURL}}{{.SanitizedOriginalURL}}{{else}}{{(MirrorRemoteAddress $.Mirror).Address}}{{end}}</a></div>{{end}}
{{if .IsFork}}<div class="fork-flag">{{$.i18n.Tr "repo.forked_from"}} <a href="{{.BaseRepo.Link}}">{{SubStr .BaseRepo.RelLink 1 -1}}</a></div>{{end}}
{{if .IsGenerated}}<div class="fork-flag">{{$.i18n.Tr "repo.generated_from"}} <a href="{{.TemplateRepo.Link}}">{{SubStr .TemplateRepo.RelLink 1 -1}}</a></div>{{end}}
</div>

112
templates/repo/settings/options.tmpl
View File

@ -70,11 +70,39 @@
</div>
{{if .Repository.IsMirror}}
{{if or .Repository.IsMirror (not .DisabledMirrors)}}
<h4 class="ui top attached header">
{{.i18n.Tr "repo.settings.mirror_settings"}}
</h4>
<div class="ui attached segment">
{{$.i18n.Tr "repo.settings.mirror_settings.docs" | Safe}}
<table class="ui table">
{{if or .Repository.IsMirror .Repository.PushMirrors}}
<thead>
<tr>
<th style="width:40%">{{$.i18n.Tr "repo.settings.mirror_settings.mirrored_repository"}}</th>
<th>{{$.i18n.Tr "repo.settings.mirror_settings.direction"}}</th>
<th>{{$.i18n.Tr "repo.settings.mirror_settings.last_update"}}</th>
<th></th>
</tr>
</thead>
{{end}}
{{if .Repository.IsMirror}}
<tbody>
<tr>
<td>{{(MirrorRemoteAddress .Mirror).Address}}</td>
<td>{{$.i18n.Tr "repo.settings.mirror_settings.direction.pull"}}</td>
<td>{{.Mirror.UpdatedUnix.AsTime}}</td>
<td class="right aligned">
<form method="post" style="display: inline-block">
{{.CsrfTokenHtml}}
<input type="hidden" name="action" value="mirror-sync">
<button class="ui blue tiny button inline text-thin">{{$.i18n.Tr "repo.settings.sync_mirror"}}</button>
</form>
</td>
</tr>
<tr>
<td colspan="4">
<form class="ui form" method="post">
{{.CsrfTokenHtml}}
<input type="hidden" name="action" value="mirror">
@ -89,24 +117,25 @@
<label for="interval">{{.i18n.Tr "repo.mirror_interval"}}</label>
<input id="interval" name="interval" value="{{.MirrorInterval}}">
</div>
{{$address := MirrorRemoteAddress .Mirror}}
<div class="field {{if .Err_MirrorAddress}}error{{end}}">
<label for="mirror_address">{{.i18n.Tr "repo.mirror_address"}}</label>
<input id="mirror_address" name="mirror_address" value="{{MirrorFullAddress .Mirror}}" required>
<input id="mirror_address" name="mirror_address" value="{{$address.Address}}" required>
<p class="help">{{.i18n.Tr "repo.mirror_address_desc"}}</p>
</div>
<details class="ui optional field" {{if .Err_Auth}}open{{else if (MirrorUserName .Mirror)}}open{{end}}>
<details class="ui optional field" {{if or .Err_Auth $address.Username}}open{{end}}>
<summary class="p-2">
{{.i18n.Tr "repo.need_auth"}}
</summary>
<div class="p-2">
<div class="inline field {{if .Err_Auth}}error{{end}}">
<label for="mirror_username">{{.i18n.Tr "username"}}</label>
<input id="mirror_username" name="mirror_username" value="{{MirrorUserName .Mirror}}" {{if not .mirror_username}}data-need-clear="true"{{end}}>
<input id="mirror_username" name="mirror_username" value="{{$address.Username}}" {{if not .mirror_username}}data-need-clear="true"{{end}}>
</div>
<input class="fake" type="password">
<div class="inline field {{if .Err_Auth}}error{{end}}">
<label for="mirror_password">{{.i18n.Tr "password"}}</label>
<input id="mirror_password" name="mirror_password" type="password" placeholder="{{if MirrorPassword .Mirror }}{{.i18n.Tr "repo.mirror_password_placeholder"}}{{else}}{{.i18n.Tr "repo.mirror_password_blank_placeholder"}}{{end}}" value="" {{if not .mirror_password}}data-need-clear="true"{{end}} autocomplete="off">
<input id="mirror_password" name="mirror_password" type="password" placeholder="{{if $address.Password}}{{.i18n.Tr "repo.mirror_password_placeholder"}}{{else}}{{.i18n.Tr "repo.mirror_password_blank_placeholder"}}{{end}}" value="" {{if not .mirror_password}}data-need-clear="true"{{end}} autocomplete="off">
</div>
<p class="help">{{.i18n.Tr "repo.mirror_password_help"}}</p>
</div>
@ -126,25 +155,80 @@
<p class="help">{{.i18n.Tr "repo.mirror_lfs_endpoint_desc" "https://github.com/git-lfs/git-lfs/blob/main/docs/api/server-discovery.md#server-discovery" | Str2html}}</p>
</div>
{{end}}
<div class="field">
<button class="ui green button">{{$.i18n.Tr "repo.settings.update_settings"}}</button>
</div>
</form>
<div class="ui divider"></div>
</td>
</tr>
</tbody>
<thead><tr><th colspan="4"></th></tr></thead>
{{end}}
<tbody>
{{range .Repository.PushMirrors}}
<tr>
{{$address := MirrorRemoteAddress .}}
<td>{{$address.Address}}</td>
<td>{{$.i18n.Tr "repo.settings.mirror_settings.direction.push"}}</td>
<td>{{if .LastUpdateUnix}}{{.LastUpdateUnix.AsTime}}{{else}}{{$.i18n.Tr "never"}}{{end}} {{if .LastError}}<div class="ui red label" data-tooltip="{{.LastError}}">{{$.i18n.Tr "error"}}</div>{{end}}</td>
<td class="right aligned">
<form method="post" style="display: inline-block">
{{$.CsrfTokenHtml}}
<input type="hidden" name="action" value="push-mirror-remove">
<input type="hidden" name="push_mirror_id" value="{{.ID}}">
<button class="ui basic red tiny button inline text-thin">{{$.i18n.Tr "remove"}}</button>
</form>
<form method="post" style="display: inline-block">
{{$.CsrfTokenHtml}}
<input type="hidden" name="action" value="push-mirror-sync">
<input type="hidden" name="push_mirror_id" value="{{.ID}}">
<button class="ui blue tiny button inline text-thin">{{$.i18n.Tr "repo.settings.sync_mirror"}}</button>
</form>
</td>
</tr>
{{else}}
<tr>
<td>{{$.i18n.Tr "repo.settings.mirror_settings.push_mirror.none"}}</td>
</tr>
{{end}}
<tr>
<td colspan="4">
<form class="ui form" method="post">
{{.CsrfTokenHtml}}
<input type="hidden" name="action" value="mirror-sync">
<div class="inline field">
<label>{{.i18n.Tr "repo.mirror_last_synced"}}</label>
<span>{{.Mirror.UpdatedUnix.AsTime}}</span>
<input type="hidden" name="action" value="push-mirror-add">
<div class="field {{if .Err_PushMirrorAddress}}error{{end}}">
<label for="push_mirror_address">{{.i18n.Tr "repo.settings.mirror_settings.push_mirror.remote_url"}}</label>
<input id="push_mirror_address" name="push_mirror_address" value="{{.push_mirror_address}}" required>
<p class="help">{{.i18n.Tr "repo.mirror_address_desc"}}</p>
</div>
<details class="ui optional field" {{if or .Err_PushMirrorAuth .push_mirror_username}}open{{end}}>
<summary class="p-2">
{{.i18n.Tr "repo.need_auth"}}
</summary>
<div class="p-2">
<div class="inline field {{if .Err_PushMirrorAuth}}error{{end}}">
<label for="push_mirror_username">{{.i18n.Tr "username"}}</label>
<input id="push_mirror_username" name="push_mirror_username" value="{{.push_mirror_username}}">
</div>
<input class="fake" type="password">
<div class="inline field {{if .Err_PushMirrorAuth}}error{{end}}">
<label for="push_mirror_password">{{.i18n.Tr "password"}}</label>
<input id="push_mirror_password" name="push_mirror_password" type="password" value="{{.push_mirror_password}}" autocomplete="off">
</div>
</div>
</details>
<div class="inline field {{if .Err_PushMirrorInterval}}error{{end}}">
<label for="push_mirror_interval">{{.i18n.Tr "repo.mirror_interval"}}</label>
<input id="push_mirror_interval" name="push_mirror_interval" value="{{if .push_mirror_interval}}{{.push_mirror_interval}}{{else}}{{.DefaultMirrorInterval}}{{end}}">
</div>
<div class="field">
<button class="ui blue button">{{$.i18n.Tr "repo.settings.sync_mirror"}}</button>
<button class="ui green button">{{$.i18n.Tr "repo.settings.mirror_settings.push_mirror.add"}}</button>
</div>
</form>
</td>
</tr>
</tbody>
</table>
</div>
{{end}}