Commit Graph

16 Commits

Author SHA1 Message Date
zeripath 012e45a4c1
Correctly handle failed migrations (#17575) (#18099)
* Correctly handle failed migrations

There is a bug in handling failed migrations whereby the migration task gets decoupled
from the migration repository. This leads to a failure of the task to get deleted with
the repository and also leads to the migration failed page resulting in a ISE.

This PR removes the zeroing out of the task id from the migration but also makes
the migration handler tolerate missing tasks much nicer.

Fix #17571

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2021-12-25 15:45:51 +00:00
6543 3b13c5d41a
Fix bug of migrated repository not index (#16991) (#16996)
Fix #16986, #16152

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-09-09 07:02:22 +01:00
luzpaz e0296b6a6d
Fix various documentation, user-facing, and source comment typos (#16367)
* Fix various doc, user-facing, and source comment typos

Found via `codespell -q 3 -S ./options/locale,./vendor -L ba,pullrequest,pullrequests,readby`
2021-07-08 13:38:13 +02:00
zeripath 6d69df2804
Add Status Updates whilst Gitea migrations are occurring (#15076)
* Add migrating message

Signed-off-by: Andrew Thornton <art27@cantab.net>

* simplify messenger

Signed-off-by: Andrew Thornton <art27@cantab.net>

* make messenger an interface

Signed-off-by: Andrew Thornton <art27@cantab.net>

* rename

Signed-off-by: Andrew Thornton <art27@cantab.net>

* prepare for merge

Signed-off-by: Andrew Thornton <art27@cantab.net>

* as per tech

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-06-16 18:02:24 -04:00
KN4CK3R 440039c0cc
Add push to remote mirror repository (#15157)
* Added push mirror model.

* Integrated push mirror into queue.

* Moved methods into own file.

* Added basic implementation.

* Mirror wiki too.

* Removed duplicated method.

* Get url for different remotes.

* Added migration.

* Unified remote url access.

* Add/Remove push mirror remotes.

* Prevent hangs with missing credentials.

* Moved code between files.

* Changed sanitizer interface.

* Added push mirror backend methods.

* Only update the mirror remote.

* Limit refs on push.

* Added UI part.

* Added missing table.

* Delete mirror if repository gets removed.

* Changed signature. Handle object errors.

* Added upload method.

* Added "upload" unit tests.

* Added transfer adapter unit tests.

* Send correct headers.

* Added pushing of LFS objects.

* Added more logging.

* Simpler body handling.

* Process files in batches to reduce HTTP calls.

* Added created timestamp.

* Fixed invalid column name.

* Changed name to prevent xorm auto setting.

* Remove table header im empty.

* Strip exit code from error message.

* Added docs page about mirroring.

* Fixed date.

* Fixed merge errors.

* Moved test to integrations.

* Added push mirror test.

* Added test.
2021-06-14 19:20:43 +02:00
Norwin cb940c4312
Encrypt migration credentials at rest (#15895)
* encrypt migration credentials in task persistence

Not sure this is the best approach, we could encrypt the entire
`PayloadContent` instead. Also instead of clearing individual fields in
payload content, we could just delete the task once it has
(successfully) finished..?

* remove credentials of past migrations

* only run DB migration for completed tasks

* fix binding

* add omitempty

* never serialize unencrypted credentials

* fix import order

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-05-31 04:25:47 -04:00
zeripath f0e15250b9
Migrate to use jsoniter instead of encoding/json (#14841)
* Migrate to use jsoniter

* fix tests

* update gitea.com/go-chi/binding

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2021-03-01 22:08:10 +01:00
6543 2b4a08e962
Migrations: Use Process Manager to create own Context (#13792) 2020-12-02 18:36:06 +00:00
zeripath f40a2a4404
Store task errors following migrations and display them (#13246)
* Store task errors following migrations and display them

When migrate tasks fail store the error in the task table
and ensure that they show on the status page.

Fix #13242

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update web_src/js/index.js

* Hide the failed first

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-10-23 19:46:35 -04:00
zeripath 7edf7eb46c
Log the underlying panic in runMigrateTask (#13096)
If there is a panic during runMigrateTask we should capture and log the underlying
panic error.

This PR ensures that the panic is logged and captured as part of the task message.

Fix #13095

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-10-11 14:51:13 -04:00
6543 fd60ebfe14
[API] Migration: Change ServiceType String (#12672)
* use different structs for MigrateRepoOptions on UI and API

* Fix TokenAuth and rename UID to an understandable Name

* fix swagger doc

* simplify & mk redable

* R E F A C T O R:
migration has now internal 3 structs to store its options:
 * the Options for WebUI: modules/auth/repo_form.go
 * the Options for API: modules/structs/repo.go
 * the option struct with after validation for internal prossessing: modules/migrations/base/options.go

* Copyright Header

* Deprecate UID - add RepoOwner

* adopt repo.go -> migrate.go

* add comment about each struct purpose

* lint
2020-09-10 23:29:19 +01:00
Lunny Xiao b465d0d787
Move create/fork repository from models to modules/repository (#9489)
* Move create/fork repository from models to modules/repository

* fix wrong reference

* fix test

* fix test

* fix lint

* Fix DBContext

* remove duplicated TestMain

* fix lint

* fix conflicts
2020-01-12 20:11:17 +08:00
zeripath 62eb1b0f25 Graceful Queues: Issue Indexing and Tasks (#9363)
* Queue: Add generic graceful queues with settings

* Queue & Setting: Add worker pool implementation

* Queue: Add worker settings

* Queue: Make resizing worker pools

* Queue: Add name variable to queues

* Queue: Add monitoring

* Queue: Improve logging

* Issues: Gracefulise the issues indexer

Remove the old now unused specific queues

* Task: Move to generic queue and gracefulise

* Issues: Standardise the issues indexer queue settings

* Fix test

* Queue: Allow Redis to connect to unix

* Prevent deadlock during early shutdown of issue indexer

* Add MaxWorker settings to queues

* Merge branch 'master' into graceful-queues

* Update modules/indexer/issues/indexer.go

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Update modules/indexer/issues/indexer.go

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Update modules/queue/queue_channel.go

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Update modules/queue/queue_disk.go

* Update modules/queue/queue_disk_channel.go

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Rename queue.Description to queue.ManagedQueue as per @guillep2k

* Cancel pool workers when removed

* Remove dependency on queue from setting

* Update modules/queue/queue_redis.go

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* As per @guillep2k add mutex locks on shutdown/terminate

* move unlocking out of setInternal

* Add warning if number of workers < 0

* Small changes as per @guillep2k

* No redis host specified not found

* Clean up documentation for queues

* Update docs/content/doc/advanced/config-cheat-sheet.en-us.md

* Update modules/indexer/issues/indexer_test.go

* Ensure that persistable channel queue is added to manager

* Rename QUEUE_NAME REDIS_QUEUE_NAME

* Revert "Rename QUEUE_NAME REDIS_QUEUE_NAME"

This reverts commit 1f83b4fc9b.

Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: techknowlogick <matti@mdranta.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-01-07 12:23:09 +01:00
Lunny Xiao ffc904b1e0 Sleep longer if request speed is over github limitation (#9335)
* Sleep longer if request speed is over github limitation

* improve code

* remove unused code

* fix lint

* Use github's rate limit remain value to determine how long to sleep

* Save reset time when finished github api request

* fix bug

* fix lint

* Add context.Context for sleep

* fix test

* improve code

* fix bug and lint

* fix import order
2019-12-16 23:16:54 -05:00
Lunny Xiao f02138a148 Fix bug when migrate from API (#8631)
* fix bug when migrate from API

* fix test

* fix test

* improve

* fix error message
2019-11-08 17:21:00 -05:00
Lunny Xiao f2a3abc683
Move migrating repository from frontend to backend (#6200)
* move migrating to backend

* add loading image when migrating and fix tests

* fix format

* fix lint

* add redis task queue support and improve docs

* add redis vendor

* fix vet

* add database migrations and fix app.ini sample

* add comments for task section on app.ini.sample

* Update models/migrations/v84.go

Co-Authored-By: lunny <xiaolunwen@gmail.com>

* Update models/repo.go

Co-Authored-By: lunny <xiaolunwen@gmail.com>

* move migrating to backend

* add loading image when migrating and fix tests

* fix fmt

* add redis task queue support and improve docs

* fix fixtures

* fix fixtures

* fix duplicate function on index.js

* fix tests

* rename repository statuses

* check if repository is being create when SSH request

* fix lint

* fix template

* some improvements

* fix template

* unified migrate options

* fix lint

* fix loading page

* refactor

* When gitea restart, don't restart the running tasks because we may have servel gitea instances, that may break the migration

* fix js

* Update models/repo.go

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Update docs/content/doc/advanced/config-cheat-sheet.en-us.md

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* fix tests

* rename ErrTaskIsNotExist to ErrTaskDoesNotExist

* delete release after add one on tests to make it run happy

* fix tests

* fix tests

* improve codes

* fix lint

* fix lint

* fix migrations
2019-10-13 21:23:14 +08:00