gitea/services/migrations/gitlab.go

660 lines
18 KiB
Go
Raw Normal View History

Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
// Copyright 2019 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package migrations
import (
"context"
"errors"
"fmt"
"io"
"net/http"
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
"net/url"
"path"
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
"strings"
"time"
"code.gitea.io/gitea/modules/log"
base "code.gitea.io/gitea/modules/migration"
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
"code.gitea.io/gitea/modules/structs"
"github.com/xanzy/go-gitlab"
)
var (
_ base.Downloader = &GitlabDownloader{}
_ base.DownloaderFactory = &GitlabDownloaderFactory{}
)
func init() {
RegisterDownloaderFactory(&GitlabDownloaderFactory{})
}
// GitlabDownloaderFactory defines a gitlab downloader factory
type GitlabDownloaderFactory struct{}
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
// New returns a Downloader related to this factory according MigrateOptions
func (f *GitlabDownloaderFactory) New(ctx context.Context, opts base.MigrateOptions) (base.Downloader, error) {
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
u, err := url.Parse(opts.CloneAddr)
if err != nil {
return nil, err
}
baseURL := u.Scheme + "://" + u.Host
repoNameSpace := strings.TrimPrefix(u.Path, "/")
repoNameSpace = strings.TrimSuffix(repoNameSpace, ".git")
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
log.Trace("Create gitlab downloader. BaseURL: %s RepoName: %s", baseURL, repoNameSpace)
return NewGitlabDownloader(ctx, baseURL, repoNameSpace, opts.AuthUsername, opts.AuthPassword, opts.AuthToken)
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
}
// GitServiceType returns the type of git service
func (f *GitlabDownloaderFactory) GitServiceType() structs.GitServiceType {
return structs.GitlabService
}
// GitlabDownloader implements a Downloader interface to get repository information
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
// from gitlab via go-gitlab
// - issueCount is incremented in GetIssues() to ensure PR and Issue numbers do not overlap,
// because Gitlab has individual Issue and Pull Request numbers.
type GitlabDownloader struct {
base.NullDownloader
ctx context.Context
client *gitlab.Client
repoID int
repoName string
issueCount int64
maxPerPage int
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
}
// NewGitlabDownloader creates a gitlab Downloader via gitlab API
// Use either a username/password, personal token entered into the username field, or anonymous/public access
// Note: Public access only allows very basic access
func NewGitlabDownloader(ctx context.Context, baseURL, repoPath, username, password, token string) (*GitlabDownloader, error) {
gitlabClient, err := gitlab.NewClient(token, gitlab.WithBaseURL(baseURL), gitlab.WithHTTPClient(NewMigrationHTTPClient()))
// Only use basic auth if token is blank and password is NOT
// Basic auth will fail with empty strings, but empty token will allow anonymous public API usage
if token == "" && password != "" {
gitlabClient, err = gitlab.NewBasicAuthClient(username, password, gitlab.WithBaseURL(baseURL), gitlab.WithHTTPClient(NewMigrationHTTPClient()))
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
}
if err != nil {
log.Trace("Error logging into gitlab: %v", err)
return nil, err
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
}
// split namespace and subdirectory
pathParts := strings.Split(strings.Trim(repoPath, "/"), "/")
var resp *gitlab.Response
u, _ := url.Parse(baseURL)
for len(pathParts) >= 2 {
_, resp, err = gitlabClient.Version.GetVersion()
if err == nil || resp != nil && resp.StatusCode == http.StatusUnauthorized {
err = nil // if no authentication given, this still should work
break
}
u.Path = path.Join(u.Path, pathParts[0])
baseURL = u.String()
pathParts = pathParts[1:]
_ = gitlab.WithBaseURL(baseURL)(gitlabClient)
repoPath = strings.Join(pathParts, "/")
}
if err != nil {
log.Trace("Error could not get gitlab version: %v", err)
return nil, err
}
log.Trace("gitlab downloader: use BaseURL: '%s' and RepoPath: '%s'", baseURL, repoPath)
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
// Grab and store project/repo ID here, due to issues using the URL escaped path
gr, _, err := gitlabClient.Projects.GetProject(repoPath, nil, nil, gitlab.WithContext(ctx))
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
if err != nil {
log.Trace("Error retrieving project: %v", err)
return nil, err
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
}
if gr == nil {
log.Trace("Error getting project, project is nil")
return nil, errors.New("Error getting project, project is nil")
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
}
return &GitlabDownloader{
ctx: ctx,
client: gitlabClient,
repoID: gr.ID,
repoName: gr.Name,
maxPerPage: 100,
}, nil
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
}
// SetContext set context
func (g *GitlabDownloader) SetContext(ctx context.Context) {
g.ctx = ctx
}
// GetRepoInfo returns a repository information
func (g *GitlabDownloader) GetRepoInfo() (*base.Repository, error) {
gr, _, err := g.client.Projects.GetProject(g.repoID, nil, nil, gitlab.WithContext(g.ctx))
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
if err != nil {
return nil, err
}
var private bool
switch gr.Visibility {
case gitlab.InternalVisibility:
private = true
case gitlab.PrivateVisibility:
private = true
}
var owner string
if gr.Owner == nil {
log.Trace("gr.Owner is nil, trying to get owner from Namespace")
if gr.Namespace != nil && gr.Namespace.Kind == "user" {
owner = gr.Namespace.Path
}
} else {
owner = gr.Owner.Username
}
// convert gitlab repo to stand Repo
return &base.Repository{
Owner: owner,
Name: gr.Name,
IsPrivate: private,
Description: gr.Description,
OriginalURL: gr.WebURL,
CloneURL: gr.HTTPURLToRepo,
DefaultBranch: gr.DefaultBranch,
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
}, nil
}
// GetTopics return gitlab topics
func (g *GitlabDownloader) GetTopics() ([]string, error) {
gr, _, err := g.client.Projects.GetProject(g.repoID, nil, nil, gitlab.WithContext(g.ctx))
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
if err != nil {
return nil, err
}
return gr.TagList, err
}
// GetMilestones returns milestones
func (g *GitlabDownloader) GetMilestones() ([]*base.Milestone, error) {
perPage := g.maxPerPage
state := "all"
milestones := make([]*base.Milestone, 0, perPage)
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
for i := 1; ; i++ {
ms, _, err := g.client.Milestones.ListMilestones(g.repoID, &gitlab.ListMilestonesOptions{
State: &state,
ListOptions: gitlab.ListOptions{
Page: i,
PerPage: perPage,
},
}, nil, gitlab.WithContext(g.ctx))
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
if err != nil {
return nil, err
}
for _, m := range ms {
var desc string
if m.Description != "" {
desc = m.Description
}
state := "open"
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
var closedAt *time.Time
if m.State != "" {
state = m.State
if state == "closed" {
closedAt = m.UpdatedAt
}
}
var deadline *time.Time
if m.DueDate != nil {
deadlineParsed, err := time.Parse("2006-01-02", m.DueDate.String())
if err != nil {
log.Trace("Error parsing Milestone DueDate time")
deadline = nil
} else {
deadline = &deadlineParsed
}
}
milestones = append(milestones, &base.Milestone{
Title: m.Title,
Description: desc,
Deadline: deadline,
State: state,
Created: *m.CreatedAt,
Updated: m.UpdatedAt,
Closed: closedAt,
})
}
if len(ms) < perPage {
break
}
}
return milestones, nil
}
2020-09-10 07:04:30 -06:00
func (g *GitlabDownloader) normalizeColor(val string) string {
val = strings.TrimLeft(val, "#")
val = strings.ToLower(val)
if len(val) == 3 {
c := []rune(val)
val = fmt.Sprintf("%c%c%c%c%c%c", c[0], c[0], c[1], c[1], c[2], c[2])
}
if len(val) != 6 {
return ""
}
return val
}
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
// GetLabels returns labels
func (g *GitlabDownloader) GetLabels() ([]*base.Label, error) {
perPage := g.maxPerPage
labels := make([]*base.Label, 0, perPage)
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
for i := 1; ; i++ {
ls, _, err := g.client.Labels.ListLabels(g.repoID, &gitlab.ListLabelsOptions{ListOptions: gitlab.ListOptions{
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
Page: i,
PerPage: perPage,
}}, nil, gitlab.WithContext(g.ctx))
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
if err != nil {
return nil, err
}
for _, label := range ls {
baseLabel := &base.Label{
Name: label.Name,
2020-09-10 07:04:30 -06:00
Color: g.normalizeColor(label.Color),
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
Description: label.Description,
}
labels = append(labels, baseLabel)
}
if len(ls) < perPage {
break
}
}
return labels, nil
}
func (g *GitlabDownloader) convertGitlabRelease(rel *gitlab.Release) *base.Release {
var zero int
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
r := &base.Release{
TagName: rel.TagName,
TargetCommitish: rel.Commit.ID,
Name: rel.Name,
Body: rel.Description,
Created: *rel.CreatedAt,
PublisherID: int64(rel.Author.ID),
PublisherName: rel.Author.Username,
}
httpClient := NewMigrationHTTPClient()
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
for k, asset := range rel.Assets.Links {
r.Assets = append(r.Assets, &base.ReleaseAsset{
ID: int64(asset.ID),
Name: asset.Name,
ContentType: &rel.Assets.Sources[k].Format,
Size: &zero,
DownloadCount: &zero,
DownloadFunc: func() (io.ReadCloser, error) {
link, _, err := g.client.ReleaseLinks.GetReleaseLink(g.repoID, rel.TagName, asset.ID, gitlab.WithContext(g.ctx))
if err != nil {
return nil, err
}
req, err := http.NewRequest("GET", link.URL, nil)
if err != nil {
return nil, err
}
req = req.WithContext(g.ctx)
resp, err := httpClient.Do(req)
if err != nil {
return nil, err
}
// resp.Body is closed by the uploader
return resp.Body, nil
},
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
})
}
return r
}
// GetReleases returns releases
func (g *GitlabDownloader) GetReleases() ([]*base.Release, error) {
perPage := g.maxPerPage
releases := make([]*base.Release, 0, perPage)
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
for i := 1; ; i++ {
ls, _, err := g.client.Releases.ListReleases(g.repoID, &gitlab.ListReleasesOptions{
Page: i,
PerPage: perPage,
}, nil, gitlab.WithContext(g.ctx))
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
if err != nil {
return nil, err
}
for _, release := range ls {
releases = append(releases, g.convertGitlabRelease(release))
}
if len(ls) < perPage {
break
}
}
return releases, nil
}
type gitlabIssueContext struct {
IsMergeRequest bool
}
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
// GetIssues returns issues according start and limit
// Note: issue label description and colors are not supported by the go-gitlab library at this time
func (g *GitlabDownloader) GetIssues(page, perPage int) ([]*base.Issue, bool, error) {
state := "all"
sort := "asc"
if perPage > g.maxPerPage {
perPage = g.maxPerPage
}
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
opt := &gitlab.ListProjectIssuesOptions{
State: &state,
Sort: &sort,
ListOptions: gitlab.ListOptions{
PerPage: perPage,
Page: page,
},
}
allIssues := make([]*base.Issue, 0, perPage)
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
issues, _, err := g.client.Issues.ListProjectIssues(g.repoID, opt, nil, gitlab.WithContext(g.ctx))
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
if err != nil {
return nil, false, fmt.Errorf("error while listing issues: %v", err)
}
for _, issue := range issues {
labels := make([]*base.Label, 0, len(issue.Labels))
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
for _, l := range issue.Labels {
labels = append(labels, &base.Label{
Name: l,
})
}
var milestone string
if issue.Milestone != nil {
milestone = issue.Milestone.Title
}
var reactions []*base.Reaction
awardPage := 1
for {
awards, _, err := g.client.AwardEmoji.ListIssueAwardEmoji(g.repoID, issue.IID, &gitlab.ListAwardEmojiOptions{Page: awardPage, PerPage: perPage}, gitlab.WithContext(g.ctx))
if err != nil {
return nil, false, fmt.Errorf("error while listing issue awards: %v", err)
}
for i := range awards {
reactions = append(reactions, g.awardToReaction(awards[i]))
}
if len(awards) < perPage {
break
}
awardPage++
}
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
allIssues = append(allIssues, &base.Issue{
Store the foreign ID of issues during migration (#18446) Storing the foreign identifier of an imported issue in the database is a prerequisite to implement idempotent migrations or mirror for issues. It is a baby step towards mirroring that introduces a new table. At the moment when an issue is created by the Gitea uploader, it fails if the issue already exists. The Gitea uploader could be modified so that, instead of failing, it looks up the database to find an existing issue. And if it does it would update the issue instead of creating a new one. However this is not currently possible because an information is missing from the database: the foreign identifier that uniquely represents the issue being migrated is not persisted. With this change, the foreign identifier is stored in the database and the Gitea uploader will then be able to run a query to figure out if a given issue being imported already exists. The implementation of mirroring for issues, pull requests, releases, etc. can be done in three steps: 1. Store an identifier for the element being mirrored (issue, pull request...) in the database (this is the purpose of these changes) 2. Modify the Gitea uploader to be able to update an existing repository with all it contains (issues, pull request...) instead of failing if it exists 3. Optimize the Gitea uploader to speed up the updates, when possible. The second step creates code that does not yet exist to enable idempotent migrations with the Gitea uploader. When a migration is done for the first time, the behavior is not changed. But when a migration is done for a repository that already exists, this new code is used to update it. The third step can use the code created in the second step to optimize and speed up migrations. For instance, when a migration is resumed, an issue that has an update time that is not more recent can be skipped and only newly created issues or updated ones will be updated. Another example of optimization could be that a webhook notifies Gitea when an issue is updated. The code triggered by the webhook would download only this issue and call the code created in the second step to update the issue, as if it was in the process of an idempotent migration. The ForeignReferences table is added to contain local and foreign ID pairs relative to a given repository. It can later be used for pull requests and other artifacts that can be mirrored. Although the foreign id could be added as a single field in issues or pull requests, it would need to be added to all tables that represent something that can be mirrored. Creating a new table makes for a simpler and more generic design. The drawback is that it requires an extra lookup to obtain the information. However, this extra information is only required during migration or mirroring and does not impact the way Gitea currently works. The foreign identifier of an issue or pull request is similar to the identifier of an external user, which is stored in reactions, issues, etc. as OriginalPosterID and so on. The representation of a user is however different and the ability of users to link their account to an external user at a later time is also a logic that is different from what is involved in mirroring or migrations. For these reasons, despite some commonalities, it is unclear at this time how the two tables (foreign reference and external user) could be merged together. The ForeignID field is extracted from the issue migration context so that it can be dumped in files with dump-repo and later restored via restore-repo. The GetAllComments downloader method is introduced to simplify the implementation and not overload the Context for the purpose of pagination. It also clarifies in which context the comments are paginated and in which context they are not. The Context interface is no longer useful for the purpose of retrieving the LocalID and ForeignID since they are now both available from the PullRequest and Issue struct. The Reviewable and Commentable interfaces replace and serve the same purpose. The Context data member of PullRequest and Issue becomes a DownloaderContext to clarify that its purpose is not to support in memory operations while the current downloader is acting but is not otherwise persisted. It is, for instance, used by the GitLab downloader to store the IsMergeRequest boolean and sort out issues. --- [source](https://lab.forgefriends.org/forgefriends/forgefriends/-/merge_requests/36) Signed-off-by: Loïc Dachary <loic@dachary.org> Co-authored-by: Loïc Dachary <loic@dachary.org>
2022-03-17 11:08:35 -06:00
Title: issue.Title,
Number: int64(issue.IID),
PosterID: int64(issue.Author.ID),
PosterName: issue.Author.Username,
Content: issue.Description,
Milestone: milestone,
State: issue.State,
Created: *issue.CreatedAt,
Labels: labels,
Reactions: reactions,
Closed: issue.ClosedAt,
IsLocked: issue.DiscussionLocked,
Updated: *issue.UpdatedAt,
ForeignIndex: int64(issue.IID),
Context: gitlabIssueContext{IsMergeRequest: false},
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
})
// increment issueCount, to be used in GetPullRequests()
g.issueCount++
}
return allIssues, len(issues) < perPage, nil
}
// GetComments returns comments according issueNumber
// TODO: figure out how to transfer comment reactions
Store the foreign ID of issues during migration (#18446) Storing the foreign identifier of an imported issue in the database is a prerequisite to implement idempotent migrations or mirror for issues. It is a baby step towards mirroring that introduces a new table. At the moment when an issue is created by the Gitea uploader, it fails if the issue already exists. The Gitea uploader could be modified so that, instead of failing, it looks up the database to find an existing issue. And if it does it would update the issue instead of creating a new one. However this is not currently possible because an information is missing from the database: the foreign identifier that uniquely represents the issue being migrated is not persisted. With this change, the foreign identifier is stored in the database and the Gitea uploader will then be able to run a query to figure out if a given issue being imported already exists. The implementation of mirroring for issues, pull requests, releases, etc. can be done in three steps: 1. Store an identifier for the element being mirrored (issue, pull request...) in the database (this is the purpose of these changes) 2. Modify the Gitea uploader to be able to update an existing repository with all it contains (issues, pull request...) instead of failing if it exists 3. Optimize the Gitea uploader to speed up the updates, when possible. The second step creates code that does not yet exist to enable idempotent migrations with the Gitea uploader. When a migration is done for the first time, the behavior is not changed. But when a migration is done for a repository that already exists, this new code is used to update it. The third step can use the code created in the second step to optimize and speed up migrations. For instance, when a migration is resumed, an issue that has an update time that is not more recent can be skipped and only newly created issues or updated ones will be updated. Another example of optimization could be that a webhook notifies Gitea when an issue is updated. The code triggered by the webhook would download only this issue and call the code created in the second step to update the issue, as if it was in the process of an idempotent migration. The ForeignReferences table is added to contain local and foreign ID pairs relative to a given repository. It can later be used for pull requests and other artifacts that can be mirrored. Although the foreign id could be added as a single field in issues or pull requests, it would need to be added to all tables that represent something that can be mirrored. Creating a new table makes for a simpler and more generic design. The drawback is that it requires an extra lookup to obtain the information. However, this extra information is only required during migration or mirroring and does not impact the way Gitea currently works. The foreign identifier of an issue or pull request is similar to the identifier of an external user, which is stored in reactions, issues, etc. as OriginalPosterID and so on. The representation of a user is however different and the ability of users to link their account to an external user at a later time is also a logic that is different from what is involved in mirroring or migrations. For these reasons, despite some commonalities, it is unclear at this time how the two tables (foreign reference and external user) could be merged together. The ForeignID field is extracted from the issue migration context so that it can be dumped in files with dump-repo and later restored via restore-repo. The GetAllComments downloader method is introduced to simplify the implementation and not overload the Context for the purpose of pagination. It also clarifies in which context the comments are paginated and in which context they are not. The Context interface is no longer useful for the purpose of retrieving the LocalID and ForeignID since they are now both available from the PullRequest and Issue struct. The Reviewable and Commentable interfaces replace and serve the same purpose. The Context data member of PullRequest and Issue becomes a DownloaderContext to clarify that its purpose is not to support in memory operations while the current downloader is acting but is not otherwise persisted. It is, for instance, used by the GitLab downloader to store the IsMergeRequest boolean and sort out issues. --- [source](https://lab.forgefriends.org/forgefriends/forgefriends/-/merge_requests/36) Signed-off-by: Loïc Dachary <loic@dachary.org> Co-authored-by: Loïc Dachary <loic@dachary.org>
2022-03-17 11:08:35 -06:00
func (g *GitlabDownloader) GetComments(commentable base.Commentable) ([]*base.Comment, bool, error) {
context, ok := commentable.GetContext().(gitlabIssueContext)
if !ok {
Store the foreign ID of issues during migration (#18446) Storing the foreign identifier of an imported issue in the database is a prerequisite to implement idempotent migrations or mirror for issues. It is a baby step towards mirroring that introduces a new table. At the moment when an issue is created by the Gitea uploader, it fails if the issue already exists. The Gitea uploader could be modified so that, instead of failing, it looks up the database to find an existing issue. And if it does it would update the issue instead of creating a new one. However this is not currently possible because an information is missing from the database: the foreign identifier that uniquely represents the issue being migrated is not persisted. With this change, the foreign identifier is stored in the database and the Gitea uploader will then be able to run a query to figure out if a given issue being imported already exists. The implementation of mirroring for issues, pull requests, releases, etc. can be done in three steps: 1. Store an identifier for the element being mirrored (issue, pull request...) in the database (this is the purpose of these changes) 2. Modify the Gitea uploader to be able to update an existing repository with all it contains (issues, pull request...) instead of failing if it exists 3. Optimize the Gitea uploader to speed up the updates, when possible. The second step creates code that does not yet exist to enable idempotent migrations with the Gitea uploader. When a migration is done for the first time, the behavior is not changed. But when a migration is done for a repository that already exists, this new code is used to update it. The third step can use the code created in the second step to optimize and speed up migrations. For instance, when a migration is resumed, an issue that has an update time that is not more recent can be skipped and only newly created issues or updated ones will be updated. Another example of optimization could be that a webhook notifies Gitea when an issue is updated. The code triggered by the webhook would download only this issue and call the code created in the second step to update the issue, as if it was in the process of an idempotent migration. The ForeignReferences table is added to contain local and foreign ID pairs relative to a given repository. It can later be used for pull requests and other artifacts that can be mirrored. Although the foreign id could be added as a single field in issues or pull requests, it would need to be added to all tables that represent something that can be mirrored. Creating a new table makes for a simpler and more generic design. The drawback is that it requires an extra lookup to obtain the information. However, this extra information is only required during migration or mirroring and does not impact the way Gitea currently works. The foreign identifier of an issue or pull request is similar to the identifier of an external user, which is stored in reactions, issues, etc. as OriginalPosterID and so on. The representation of a user is however different and the ability of users to link their account to an external user at a later time is also a logic that is different from what is involved in mirroring or migrations. For these reasons, despite some commonalities, it is unclear at this time how the two tables (foreign reference and external user) could be merged together. The ForeignID field is extracted from the issue migration context so that it can be dumped in files with dump-repo and later restored via restore-repo. The GetAllComments downloader method is introduced to simplify the implementation and not overload the Context for the purpose of pagination. It also clarifies in which context the comments are paginated and in which context they are not. The Context interface is no longer useful for the purpose of retrieving the LocalID and ForeignID since they are now both available from the PullRequest and Issue struct. The Reviewable and Commentable interfaces replace and serve the same purpose. The Context data member of PullRequest and Issue becomes a DownloaderContext to clarify that its purpose is not to support in memory operations while the current downloader is acting but is not otherwise persisted. It is, for instance, used by the GitLab downloader to store the IsMergeRequest boolean and sort out issues. --- [source](https://lab.forgefriends.org/forgefriends/forgefriends/-/merge_requests/36) Signed-off-by: Loïc Dachary <loic@dachary.org> Co-authored-by: Loïc Dachary <loic@dachary.org>
2022-03-17 11:08:35 -06:00
return nil, false, fmt.Errorf("unexpected context: %+v", commentable.GetContext())
}
allComments := make([]*base.Comment, 0, g.maxPerPage)
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
page := 1
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
for {
var comments []*gitlab.Discussion
var resp *gitlab.Response
var err error
if !context.IsMergeRequest {
Store the foreign ID of issues during migration (#18446) Storing the foreign identifier of an imported issue in the database is a prerequisite to implement idempotent migrations or mirror for issues. It is a baby step towards mirroring that introduces a new table. At the moment when an issue is created by the Gitea uploader, it fails if the issue already exists. The Gitea uploader could be modified so that, instead of failing, it looks up the database to find an existing issue. And if it does it would update the issue instead of creating a new one. However this is not currently possible because an information is missing from the database: the foreign identifier that uniquely represents the issue being migrated is not persisted. With this change, the foreign identifier is stored in the database and the Gitea uploader will then be able to run a query to figure out if a given issue being imported already exists. The implementation of mirroring for issues, pull requests, releases, etc. can be done in three steps: 1. Store an identifier for the element being mirrored (issue, pull request...) in the database (this is the purpose of these changes) 2. Modify the Gitea uploader to be able to update an existing repository with all it contains (issues, pull request...) instead of failing if it exists 3. Optimize the Gitea uploader to speed up the updates, when possible. The second step creates code that does not yet exist to enable idempotent migrations with the Gitea uploader. When a migration is done for the first time, the behavior is not changed. But when a migration is done for a repository that already exists, this new code is used to update it. The third step can use the code created in the second step to optimize and speed up migrations. For instance, when a migration is resumed, an issue that has an update time that is not more recent can be skipped and only newly created issues or updated ones will be updated. Another example of optimization could be that a webhook notifies Gitea when an issue is updated. The code triggered by the webhook would download only this issue and call the code created in the second step to update the issue, as if it was in the process of an idempotent migration. The ForeignReferences table is added to contain local and foreign ID pairs relative to a given repository. It can later be used for pull requests and other artifacts that can be mirrored. Although the foreign id could be added as a single field in issues or pull requests, it would need to be added to all tables that represent something that can be mirrored. Creating a new table makes for a simpler and more generic design. The drawback is that it requires an extra lookup to obtain the information. However, this extra information is only required during migration or mirroring and does not impact the way Gitea currently works. The foreign identifier of an issue or pull request is similar to the identifier of an external user, which is stored in reactions, issues, etc. as OriginalPosterID and so on. The representation of a user is however different and the ability of users to link their account to an external user at a later time is also a logic that is different from what is involved in mirroring or migrations. For these reasons, despite some commonalities, it is unclear at this time how the two tables (foreign reference and external user) could be merged together. The ForeignID field is extracted from the issue migration context so that it can be dumped in files with dump-repo and later restored via restore-repo. The GetAllComments downloader method is introduced to simplify the implementation and not overload the Context for the purpose of pagination. It also clarifies in which context the comments are paginated and in which context they are not. The Context interface is no longer useful for the purpose of retrieving the LocalID and ForeignID since they are now both available from the PullRequest and Issue struct. The Reviewable and Commentable interfaces replace and serve the same purpose. The Context data member of PullRequest and Issue becomes a DownloaderContext to clarify that its purpose is not to support in memory operations while the current downloader is acting but is not otherwise persisted. It is, for instance, used by the GitLab downloader to store the IsMergeRequest boolean and sort out issues. --- [source](https://lab.forgefriends.org/forgefriends/forgefriends/-/merge_requests/36) Signed-off-by: Loïc Dachary <loic@dachary.org> Co-authored-by: Loïc Dachary <loic@dachary.org>
2022-03-17 11:08:35 -06:00
comments, resp, err = g.client.Discussions.ListIssueDiscussions(g.repoID, int(commentable.GetForeignIndex()), &gitlab.ListIssueDiscussionsOptions{
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
Page: page,
PerPage: g.maxPerPage,
}, nil, gitlab.WithContext(g.ctx))
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
} else {
Store the foreign ID of issues during migration (#18446) Storing the foreign identifier of an imported issue in the database is a prerequisite to implement idempotent migrations or mirror for issues. It is a baby step towards mirroring that introduces a new table. At the moment when an issue is created by the Gitea uploader, it fails if the issue already exists. The Gitea uploader could be modified so that, instead of failing, it looks up the database to find an existing issue. And if it does it would update the issue instead of creating a new one. However this is not currently possible because an information is missing from the database: the foreign identifier that uniquely represents the issue being migrated is not persisted. With this change, the foreign identifier is stored in the database and the Gitea uploader will then be able to run a query to figure out if a given issue being imported already exists. The implementation of mirroring for issues, pull requests, releases, etc. can be done in three steps: 1. Store an identifier for the element being mirrored (issue, pull request...) in the database (this is the purpose of these changes) 2. Modify the Gitea uploader to be able to update an existing repository with all it contains (issues, pull request...) instead of failing if it exists 3. Optimize the Gitea uploader to speed up the updates, when possible. The second step creates code that does not yet exist to enable idempotent migrations with the Gitea uploader. When a migration is done for the first time, the behavior is not changed. But when a migration is done for a repository that already exists, this new code is used to update it. The third step can use the code created in the second step to optimize and speed up migrations. For instance, when a migration is resumed, an issue that has an update time that is not more recent can be skipped and only newly created issues or updated ones will be updated. Another example of optimization could be that a webhook notifies Gitea when an issue is updated. The code triggered by the webhook would download only this issue and call the code created in the second step to update the issue, as if it was in the process of an idempotent migration. The ForeignReferences table is added to contain local and foreign ID pairs relative to a given repository. It can later be used for pull requests and other artifacts that can be mirrored. Although the foreign id could be added as a single field in issues or pull requests, it would need to be added to all tables that represent something that can be mirrored. Creating a new table makes for a simpler and more generic design. The drawback is that it requires an extra lookup to obtain the information. However, this extra information is only required during migration or mirroring and does not impact the way Gitea currently works. The foreign identifier of an issue or pull request is similar to the identifier of an external user, which is stored in reactions, issues, etc. as OriginalPosterID and so on. The representation of a user is however different and the ability of users to link their account to an external user at a later time is also a logic that is different from what is involved in mirroring or migrations. For these reasons, despite some commonalities, it is unclear at this time how the two tables (foreign reference and external user) could be merged together. The ForeignID field is extracted from the issue migration context so that it can be dumped in files with dump-repo and later restored via restore-repo. The GetAllComments downloader method is introduced to simplify the implementation and not overload the Context for the purpose of pagination. It also clarifies in which context the comments are paginated and in which context they are not. The Context interface is no longer useful for the purpose of retrieving the LocalID and ForeignID since they are now both available from the PullRequest and Issue struct. The Reviewable and Commentable interfaces replace and serve the same purpose. The Context data member of PullRequest and Issue becomes a DownloaderContext to clarify that its purpose is not to support in memory operations while the current downloader is acting but is not otherwise persisted. It is, for instance, used by the GitLab downloader to store the IsMergeRequest boolean and sort out issues. --- [source](https://lab.forgefriends.org/forgefriends/forgefriends/-/merge_requests/36) Signed-off-by: Loïc Dachary <loic@dachary.org> Co-authored-by: Loïc Dachary <loic@dachary.org>
2022-03-17 11:08:35 -06:00
comments, resp, err = g.client.Discussions.ListMergeRequestDiscussions(g.repoID, int(commentable.GetForeignIndex()), &gitlab.ListMergeRequestDiscussionsOptions{
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
Page: page,
PerPage: g.maxPerPage,
}, nil, gitlab.WithContext(g.ctx))
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
}
if err != nil {
return nil, false, fmt.Errorf("error while listing comments: %v %v", g.repoID, err)
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
}
for _, comment := range comments {
// Flatten comment threads
if !comment.IndividualNote {
for _, note := range comment.Notes {
allComments = append(allComments, &base.Comment{
Store the foreign ID of issues during migration (#18446) Storing the foreign identifier of an imported issue in the database is a prerequisite to implement idempotent migrations or mirror for issues. It is a baby step towards mirroring that introduces a new table. At the moment when an issue is created by the Gitea uploader, it fails if the issue already exists. The Gitea uploader could be modified so that, instead of failing, it looks up the database to find an existing issue. And if it does it would update the issue instead of creating a new one. However this is not currently possible because an information is missing from the database: the foreign identifier that uniquely represents the issue being migrated is not persisted. With this change, the foreign identifier is stored in the database and the Gitea uploader will then be able to run a query to figure out if a given issue being imported already exists. The implementation of mirroring for issues, pull requests, releases, etc. can be done in three steps: 1. Store an identifier for the element being mirrored (issue, pull request...) in the database (this is the purpose of these changes) 2. Modify the Gitea uploader to be able to update an existing repository with all it contains (issues, pull request...) instead of failing if it exists 3. Optimize the Gitea uploader to speed up the updates, when possible. The second step creates code that does not yet exist to enable idempotent migrations with the Gitea uploader. When a migration is done for the first time, the behavior is not changed. But when a migration is done for a repository that already exists, this new code is used to update it. The third step can use the code created in the second step to optimize and speed up migrations. For instance, when a migration is resumed, an issue that has an update time that is not more recent can be skipped and only newly created issues or updated ones will be updated. Another example of optimization could be that a webhook notifies Gitea when an issue is updated. The code triggered by the webhook would download only this issue and call the code created in the second step to update the issue, as if it was in the process of an idempotent migration. The ForeignReferences table is added to contain local and foreign ID pairs relative to a given repository. It can later be used for pull requests and other artifacts that can be mirrored. Although the foreign id could be added as a single field in issues or pull requests, it would need to be added to all tables that represent something that can be mirrored. Creating a new table makes for a simpler and more generic design. The drawback is that it requires an extra lookup to obtain the information. However, this extra information is only required during migration or mirroring and does not impact the way Gitea currently works. The foreign identifier of an issue or pull request is similar to the identifier of an external user, which is stored in reactions, issues, etc. as OriginalPosterID and so on. The representation of a user is however different and the ability of users to link their account to an external user at a later time is also a logic that is different from what is involved in mirroring or migrations. For these reasons, despite some commonalities, it is unclear at this time how the two tables (foreign reference and external user) could be merged together. The ForeignID field is extracted from the issue migration context so that it can be dumped in files with dump-repo and later restored via restore-repo. The GetAllComments downloader method is introduced to simplify the implementation and not overload the Context for the purpose of pagination. It also clarifies in which context the comments are paginated and in which context they are not. The Context interface is no longer useful for the purpose of retrieving the LocalID and ForeignID since they are now both available from the PullRequest and Issue struct. The Reviewable and Commentable interfaces replace and serve the same purpose. The Context data member of PullRequest and Issue becomes a DownloaderContext to clarify that its purpose is not to support in memory operations while the current downloader is acting but is not otherwise persisted. It is, for instance, used by the GitLab downloader to store the IsMergeRequest boolean and sort out issues. --- [source](https://lab.forgefriends.org/forgefriends/forgefriends/-/merge_requests/36) Signed-off-by: Loïc Dachary <loic@dachary.org> Co-authored-by: Loïc Dachary <loic@dachary.org>
2022-03-17 11:08:35 -06:00
IssueIndex: commentable.GetLocalIndex(),
Index: int64(note.ID),
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
PosterID: int64(note.Author.ID),
PosterName: note.Author.Username,
PosterEmail: note.Author.Email,
Content: note.Body,
Created: *note.CreatedAt,
})
}
} else {
c := comment.Notes[0]
allComments = append(allComments, &base.Comment{
Store the foreign ID of issues during migration (#18446) Storing the foreign identifier of an imported issue in the database is a prerequisite to implement idempotent migrations or mirror for issues. It is a baby step towards mirroring that introduces a new table. At the moment when an issue is created by the Gitea uploader, it fails if the issue already exists. The Gitea uploader could be modified so that, instead of failing, it looks up the database to find an existing issue. And if it does it would update the issue instead of creating a new one. However this is not currently possible because an information is missing from the database: the foreign identifier that uniquely represents the issue being migrated is not persisted. With this change, the foreign identifier is stored in the database and the Gitea uploader will then be able to run a query to figure out if a given issue being imported already exists. The implementation of mirroring for issues, pull requests, releases, etc. can be done in three steps: 1. Store an identifier for the element being mirrored (issue, pull request...) in the database (this is the purpose of these changes) 2. Modify the Gitea uploader to be able to update an existing repository with all it contains (issues, pull request...) instead of failing if it exists 3. Optimize the Gitea uploader to speed up the updates, when possible. The second step creates code that does not yet exist to enable idempotent migrations with the Gitea uploader. When a migration is done for the first time, the behavior is not changed. But when a migration is done for a repository that already exists, this new code is used to update it. The third step can use the code created in the second step to optimize and speed up migrations. For instance, when a migration is resumed, an issue that has an update time that is not more recent can be skipped and only newly created issues or updated ones will be updated. Another example of optimization could be that a webhook notifies Gitea when an issue is updated. The code triggered by the webhook would download only this issue and call the code created in the second step to update the issue, as if it was in the process of an idempotent migration. The ForeignReferences table is added to contain local and foreign ID pairs relative to a given repository. It can later be used for pull requests and other artifacts that can be mirrored. Although the foreign id could be added as a single field in issues or pull requests, it would need to be added to all tables that represent something that can be mirrored. Creating a new table makes for a simpler and more generic design. The drawback is that it requires an extra lookup to obtain the information. However, this extra information is only required during migration or mirroring and does not impact the way Gitea currently works. The foreign identifier of an issue or pull request is similar to the identifier of an external user, which is stored in reactions, issues, etc. as OriginalPosterID and so on. The representation of a user is however different and the ability of users to link their account to an external user at a later time is also a logic that is different from what is involved in mirroring or migrations. For these reasons, despite some commonalities, it is unclear at this time how the two tables (foreign reference and external user) could be merged together. The ForeignID field is extracted from the issue migration context so that it can be dumped in files with dump-repo and later restored via restore-repo. The GetAllComments downloader method is introduced to simplify the implementation and not overload the Context for the purpose of pagination. It also clarifies in which context the comments are paginated and in which context they are not. The Context interface is no longer useful for the purpose of retrieving the LocalID and ForeignID since they are now both available from the PullRequest and Issue struct. The Reviewable and Commentable interfaces replace and serve the same purpose. The Context data member of PullRequest and Issue becomes a DownloaderContext to clarify that its purpose is not to support in memory operations while the current downloader is acting but is not otherwise persisted. It is, for instance, used by the GitLab downloader to store the IsMergeRequest boolean and sort out issues. --- [source](https://lab.forgefriends.org/forgefriends/forgefriends/-/merge_requests/36) Signed-off-by: Loïc Dachary <loic@dachary.org> Co-authored-by: Loïc Dachary <loic@dachary.org>
2022-03-17 11:08:35 -06:00
IssueIndex: commentable.GetLocalIndex(),
Index: int64(c.ID),
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
PosterID: int64(c.Author.ID),
PosterName: c.Author.Username,
PosterEmail: c.Author.Email,
Content: c.Body,
Created: *c.CreatedAt,
})
}
}
if resp.NextPage == 0 {
break
}
page = resp.NextPage
}
return allComments, true, nil
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
}
// GetPullRequests returns pull requests according page and perPage
func (g *GitlabDownloader) GetPullRequests(page, perPage int) ([]*base.PullRequest, bool, error) {
if perPage > g.maxPerPage {
perPage = g.maxPerPage
}
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
opt := &gitlab.ListProjectMergeRequestsOptions{
ListOptions: gitlab.ListOptions{
PerPage: perPage,
Page: page,
},
}
allPRs := make([]*base.PullRequest, 0, perPage)
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
prs, _, err := g.client.MergeRequests.ListProjectMergeRequests(g.repoID, opt, nil, gitlab.WithContext(g.ctx))
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
if err != nil {
return nil, false, fmt.Errorf("error while listing merge requests: %v", err)
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
}
for _, pr := range prs {
labels := make([]*base.Label, 0, len(pr.Labels))
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
for _, l := range pr.Labels {
labels = append(labels, &base.Label{
Name: l,
})
}
var merged bool
if pr.State == "merged" {
merged = true
pr.State = "closed"
}
mergeTime := pr.MergedAt
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
if merged && pr.MergedAt == nil {
mergeTime = pr.UpdatedAt
}
closeTime := pr.ClosedAt
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
if merged && pr.ClosedAt == nil {
closeTime = pr.UpdatedAt
}
var locked bool
if pr.State == "locked" {
locked = true
}
var milestone string
if pr.Milestone != nil {
milestone = pr.Milestone.Title
}
var reactions []*base.Reaction
awardPage := 1
for {
awards, _, err := g.client.AwardEmoji.ListMergeRequestAwardEmoji(g.repoID, pr.IID, &gitlab.ListAwardEmojiOptions{Page: awardPage, PerPage: perPage}, gitlab.WithContext(g.ctx))
if err != nil {
return nil, false, fmt.Errorf("error while listing merge requests awards: %v", err)
}
for i := range awards {
reactions = append(reactions, g.awardToReaction(awards[i]))
}
if len(awards) < perPage {
break
}
awardPage++
}
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
// Add the PR ID to the Issue Count because PR and Issues share ID space in Gitea
newPRNumber := g.issueCount + int64(pr.IID)
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
allPRs = append(allPRs, &base.PullRequest{
Title: pr.Title,
Number: newPRNumber,
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
PosterName: pr.Author.Username,
PosterID: int64(pr.Author.ID),
Content: pr.Description,
Milestone: milestone,
State: pr.State,
Created: *pr.CreatedAt,
Closed: closeTime,
Labels: labels,
Merged: merged,
MergeCommitSHA: pr.MergeCommitSHA,
MergedTime: mergeTime,
IsLocked: locked,
Reactions: reactions,
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
Head: base.PullRequestBranch{
Ref: pr.SourceBranch,
SHA: pr.SHA,
RepoName: g.repoName,
OwnerName: pr.Author.Username,
CloneURL: pr.WebURL,
},
Base: base.PullRequestBranch{
Ref: pr.TargetBranch,
SHA: pr.DiffRefs.BaseSha,
RepoName: g.repoName,
OwnerName: pr.Author.Username,
},
Store the foreign ID of issues during migration (#18446) Storing the foreign identifier of an imported issue in the database is a prerequisite to implement idempotent migrations or mirror for issues. It is a baby step towards mirroring that introduces a new table. At the moment when an issue is created by the Gitea uploader, it fails if the issue already exists. The Gitea uploader could be modified so that, instead of failing, it looks up the database to find an existing issue. And if it does it would update the issue instead of creating a new one. However this is not currently possible because an information is missing from the database: the foreign identifier that uniquely represents the issue being migrated is not persisted. With this change, the foreign identifier is stored in the database and the Gitea uploader will then be able to run a query to figure out if a given issue being imported already exists. The implementation of mirroring for issues, pull requests, releases, etc. can be done in three steps: 1. Store an identifier for the element being mirrored (issue, pull request...) in the database (this is the purpose of these changes) 2. Modify the Gitea uploader to be able to update an existing repository with all it contains (issues, pull request...) instead of failing if it exists 3. Optimize the Gitea uploader to speed up the updates, when possible. The second step creates code that does not yet exist to enable idempotent migrations with the Gitea uploader. When a migration is done for the first time, the behavior is not changed. But when a migration is done for a repository that already exists, this new code is used to update it. The third step can use the code created in the second step to optimize and speed up migrations. For instance, when a migration is resumed, an issue that has an update time that is not more recent can be skipped and only newly created issues or updated ones will be updated. Another example of optimization could be that a webhook notifies Gitea when an issue is updated. The code triggered by the webhook would download only this issue and call the code created in the second step to update the issue, as if it was in the process of an idempotent migration. The ForeignReferences table is added to contain local and foreign ID pairs relative to a given repository. It can later be used for pull requests and other artifacts that can be mirrored. Although the foreign id could be added as a single field in issues or pull requests, it would need to be added to all tables that represent something that can be mirrored. Creating a new table makes for a simpler and more generic design. The drawback is that it requires an extra lookup to obtain the information. However, this extra information is only required during migration or mirroring and does not impact the way Gitea currently works. The foreign identifier of an issue or pull request is similar to the identifier of an external user, which is stored in reactions, issues, etc. as OriginalPosterID and so on. The representation of a user is however different and the ability of users to link their account to an external user at a later time is also a logic that is different from what is involved in mirroring or migrations. For these reasons, despite some commonalities, it is unclear at this time how the two tables (foreign reference and external user) could be merged together. The ForeignID field is extracted from the issue migration context so that it can be dumped in files with dump-repo and later restored via restore-repo. The GetAllComments downloader method is introduced to simplify the implementation and not overload the Context for the purpose of pagination. It also clarifies in which context the comments are paginated and in which context they are not. The Context interface is no longer useful for the purpose of retrieving the LocalID and ForeignID since they are now both available from the PullRequest and Issue struct. The Reviewable and Commentable interfaces replace and serve the same purpose. The Context data member of PullRequest and Issue becomes a DownloaderContext to clarify that its purpose is not to support in memory operations while the current downloader is acting but is not otherwise persisted. It is, for instance, used by the GitLab downloader to store the IsMergeRequest boolean and sort out issues. --- [source](https://lab.forgefriends.org/forgefriends/forgefriends/-/merge_requests/36) Signed-off-by: Loïc Dachary <loic@dachary.org> Co-authored-by: Loïc Dachary <loic@dachary.org>
2022-03-17 11:08:35 -06:00
PatchURL: pr.WebURL + ".patch",
ForeignIndex: int64(pr.IID),
Context: gitlabIssueContext{IsMergeRequest: true},
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
})
}
return allPRs, len(prs) < perPage, nil
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
}
// GetReviews returns pull requests review
Store the foreign ID of issues during migration (#18446) Storing the foreign identifier of an imported issue in the database is a prerequisite to implement idempotent migrations or mirror for issues. It is a baby step towards mirroring that introduces a new table. At the moment when an issue is created by the Gitea uploader, it fails if the issue already exists. The Gitea uploader could be modified so that, instead of failing, it looks up the database to find an existing issue. And if it does it would update the issue instead of creating a new one. However this is not currently possible because an information is missing from the database: the foreign identifier that uniquely represents the issue being migrated is not persisted. With this change, the foreign identifier is stored in the database and the Gitea uploader will then be able to run a query to figure out if a given issue being imported already exists. The implementation of mirroring for issues, pull requests, releases, etc. can be done in three steps: 1. Store an identifier for the element being mirrored (issue, pull request...) in the database (this is the purpose of these changes) 2. Modify the Gitea uploader to be able to update an existing repository with all it contains (issues, pull request...) instead of failing if it exists 3. Optimize the Gitea uploader to speed up the updates, when possible. The second step creates code that does not yet exist to enable idempotent migrations with the Gitea uploader. When a migration is done for the first time, the behavior is not changed. But when a migration is done for a repository that already exists, this new code is used to update it. The third step can use the code created in the second step to optimize and speed up migrations. For instance, when a migration is resumed, an issue that has an update time that is not more recent can be skipped and only newly created issues or updated ones will be updated. Another example of optimization could be that a webhook notifies Gitea when an issue is updated. The code triggered by the webhook would download only this issue and call the code created in the second step to update the issue, as if it was in the process of an idempotent migration. The ForeignReferences table is added to contain local and foreign ID pairs relative to a given repository. It can later be used for pull requests and other artifacts that can be mirrored. Although the foreign id could be added as a single field in issues or pull requests, it would need to be added to all tables that represent something that can be mirrored. Creating a new table makes for a simpler and more generic design. The drawback is that it requires an extra lookup to obtain the information. However, this extra information is only required during migration or mirroring and does not impact the way Gitea currently works. The foreign identifier of an issue or pull request is similar to the identifier of an external user, which is stored in reactions, issues, etc. as OriginalPosterID and so on. The representation of a user is however different and the ability of users to link their account to an external user at a later time is also a logic that is different from what is involved in mirroring or migrations. For these reasons, despite some commonalities, it is unclear at this time how the two tables (foreign reference and external user) could be merged together. The ForeignID field is extracted from the issue migration context so that it can be dumped in files with dump-repo and later restored via restore-repo. The GetAllComments downloader method is introduced to simplify the implementation and not overload the Context for the purpose of pagination. It also clarifies in which context the comments are paginated and in which context they are not. The Context interface is no longer useful for the purpose of retrieving the LocalID and ForeignID since they are now both available from the PullRequest and Issue struct. The Reviewable and Commentable interfaces replace and serve the same purpose. The Context data member of PullRequest and Issue becomes a DownloaderContext to clarify that its purpose is not to support in memory operations while the current downloader is acting but is not otherwise persisted. It is, for instance, used by the GitLab downloader to store the IsMergeRequest boolean and sort out issues. --- [source](https://lab.forgefriends.org/forgefriends/forgefriends/-/merge_requests/36) Signed-off-by: Loïc Dachary <loic@dachary.org> Co-authored-by: Loïc Dachary <loic@dachary.org>
2022-03-17 11:08:35 -06:00
func (g *GitlabDownloader) GetReviews(reviewable base.Reviewable) ([]*base.Review, error) {
approvals, resp, err := g.client.MergeRequestApprovals.GetConfiguration(g.repoID, int(reviewable.GetForeignIndex()), gitlab.WithContext(g.ctx))
if err != nil {
if resp != nil && resp.StatusCode == http.StatusNotFound {
log.Error(fmt.Sprintf("GitlabDownloader: while migrating a error occurred: '%s'", err.Error()))
return []*base.Review{}, nil
}
return nil, err
}
var createdAt time.Time
if approvals.CreatedAt != nil {
createdAt = *approvals.CreatedAt
} else if approvals.UpdatedAt != nil {
createdAt = *approvals.UpdatedAt
} else {
createdAt = time.Now()
}
reviews := make([]*base.Review, 0, len(approvals.ApprovedBy))
for _, user := range approvals.ApprovedBy {
reviews = append(reviews, &base.Review{
Store the foreign ID of issues during migration (#18446) Storing the foreign identifier of an imported issue in the database is a prerequisite to implement idempotent migrations or mirror for issues. It is a baby step towards mirroring that introduces a new table. At the moment when an issue is created by the Gitea uploader, it fails if the issue already exists. The Gitea uploader could be modified so that, instead of failing, it looks up the database to find an existing issue. And if it does it would update the issue instead of creating a new one. However this is not currently possible because an information is missing from the database: the foreign identifier that uniquely represents the issue being migrated is not persisted. With this change, the foreign identifier is stored in the database and the Gitea uploader will then be able to run a query to figure out if a given issue being imported already exists. The implementation of mirroring for issues, pull requests, releases, etc. can be done in three steps: 1. Store an identifier for the element being mirrored (issue, pull request...) in the database (this is the purpose of these changes) 2. Modify the Gitea uploader to be able to update an existing repository with all it contains (issues, pull request...) instead of failing if it exists 3. Optimize the Gitea uploader to speed up the updates, when possible. The second step creates code that does not yet exist to enable idempotent migrations with the Gitea uploader. When a migration is done for the first time, the behavior is not changed. But when a migration is done for a repository that already exists, this new code is used to update it. The third step can use the code created in the second step to optimize and speed up migrations. For instance, when a migration is resumed, an issue that has an update time that is not more recent can be skipped and only newly created issues or updated ones will be updated. Another example of optimization could be that a webhook notifies Gitea when an issue is updated. The code triggered by the webhook would download only this issue and call the code created in the second step to update the issue, as if it was in the process of an idempotent migration. The ForeignReferences table is added to contain local and foreign ID pairs relative to a given repository. It can later be used for pull requests and other artifacts that can be mirrored. Although the foreign id could be added as a single field in issues or pull requests, it would need to be added to all tables that represent something that can be mirrored. Creating a new table makes for a simpler and more generic design. The drawback is that it requires an extra lookup to obtain the information. However, this extra information is only required during migration or mirroring and does not impact the way Gitea currently works. The foreign identifier of an issue or pull request is similar to the identifier of an external user, which is stored in reactions, issues, etc. as OriginalPosterID and so on. The representation of a user is however different and the ability of users to link their account to an external user at a later time is also a logic that is different from what is involved in mirroring or migrations. For these reasons, despite some commonalities, it is unclear at this time how the two tables (foreign reference and external user) could be merged together. The ForeignID field is extracted from the issue migration context so that it can be dumped in files with dump-repo and later restored via restore-repo. The GetAllComments downloader method is introduced to simplify the implementation and not overload the Context for the purpose of pagination. It also clarifies in which context the comments are paginated and in which context they are not. The Context interface is no longer useful for the purpose of retrieving the LocalID and ForeignID since they are now both available from the PullRequest and Issue struct. The Reviewable and Commentable interfaces replace and serve the same purpose. The Context data member of PullRequest and Issue becomes a DownloaderContext to clarify that its purpose is not to support in memory operations while the current downloader is acting but is not otherwise persisted. It is, for instance, used by the GitLab downloader to store the IsMergeRequest boolean and sort out issues. --- [source](https://lab.forgefriends.org/forgefriends/forgefriends/-/merge_requests/36) Signed-off-by: Loïc Dachary <loic@dachary.org> Co-authored-by: Loïc Dachary <loic@dachary.org>
2022-03-17 11:08:35 -06:00
IssueIndex: reviewable.GetLocalIndex(),
ReviewerID: int64(user.User.ID),
ReviewerName: user.User.Username,
CreatedAt: createdAt,
// All we get are approvals
State: base.ReviewStateApproved,
})
}
return reviews, nil
Add support for migrating from Gitlab (#9084) * First stab at a Gitlab migrations interface. * Modify JS to show migration for Gitlab * Properly strip out #gitlab tag from repo name * Working Gitlab migrations! Still need to figure out how to hide tokens/etc from showing up in opts.CloneAddr * Try #2 at trying to hide credentials. CloneAddr was being used as OriginalURL. Now passing OriginalURL through from the form and saving it. * Add go-gitlab dependency * Vendor go-gitlab * Use gitlab.BasicAuthClient Correct CloneURL. This should be functioning! Previous commits fixed "Migrated from" from including the migration credentials. * Replaced repoPath with repoID globally. RepoID is grabbed in NewGitlabDownloader * Logging touchup * Properly set private repo status. Properly set milestone deadline time. Consistently use Gitlab username for 'Name'. * Add go-gitlab vendor cache * Fix PR migrations: - Count of issues is kept to set a non-conflicting PR.ID - Bool is used to tell whether to fetch Issue or PR comments * Ensure merged PRs are closed and set with the proper time * Remove copyright and some commented code * Rip out '#gitlab' based self-hosted Gitlab support * Hide given credentials for migrated repos. CloneAddr was being saved as OriginalURL. Now passing OriginalURL through from the form and saving it in it's place * Use asset.URL directly, no point in parsing. Opened PRs should fall through to false. * Fix importing Milestones. Allow importing using Personal Tokens or anonymous access. * Fix Gitlab Milestone migration if DueDate isn't set * Empty Milestone due dates properly return nil, not zero time * Add GITLAB_READ_TOKEN to drone unit-test step * Add working gitlab_test.go. A Personal Access Token, given in env variable GITLAB_READ_TOKEN is required to run the test. * Fix linting issues * Add modified JS files * Remove pre-build JS files * Only merged PRs are marged as merged/closed * Test topics * Skip test if gitlab is inaccessible * Grab personal token from username, not password. Matches Github migration implementation * Add SetContext() to GitlabDownloader. * Checking Updated field in Issues. * Actually fetch Issue Updated time from Gitlab * Add Gitlab migration GetReviews() stub * Fix Patch and Clone URLs * check Updated too * fix mod * make vendor with go1.14 Co-authored-by: techknowlogick <techknowlogick@gitea.io> Co-authored-by: 6543 <6543@obermui.de> Co-authored-by: Lauris BH <lauris@nix.lv> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-04-19 09:44:11 -06:00
}
func (g *GitlabDownloader) awardToReaction(award *gitlab.AwardEmoji) *base.Reaction {
return &base.Reaction{
UserID: int64(award.User.ID),
UserName: award.User.Username,
Content: award.Name,
}
}