closes#20683
Add an option to gitea dump to skip the bleve indexes, which can become
quite large (in my case the same size as the repo's) and can be
regenerated after restore.
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
Only load SECRET_KEY and INTERNAL_TOKEN if they exist.
Never write the config file if the keys do not exist, which was only a fallback for Gitea upgraded from < 1.5
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
<!--
Please check the following:
1. Make sure you are targeting the `main` branch, pull requests on
release branches are only allowed for bug fixes.
2. Read contributing guidelines:
https://github.com/go-gitea/gitea/blob/main/CONTRIBUTING.md
3. Describe what your pull request does and which issue you're targeting
(if any)
-->
I fixed typo.
A testing cleanup.
This pull request replaces `os.MkdirTemp` with `t.TempDir`. We can use the `T.TempDir` function from the `testing` package to create temporary directory. The directory created by `T.TempDir` is automatically removed when the test and all its subtests complete.
This saves us at least 2 lines (error check, and cleanup) on every instance, or in some cases adds cleanup that we forgot.
Reference: https://pkg.go.dev/testing#T.TempDir
```go
func TestFoo(t *testing.T) {
// before
tmpDir, err := os.MkdirTemp("", "")
require.NoError(t, err)
defer os.RemoveAll(tmpDir)
// now
tmpDir := t.TempDir()
}
```
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* fix hard-coded timeout and error panic in API archive download endpoint
This commit updates the `GET /api/v1/repos/{owner}/{repo}/archive/{archive}`
endpoint which prior to this PR had a couple of issues.
1. The endpoint had a hard-coded 20s timeout for the archiver to complete after
which a 500 (Internal Server Error) was returned to client. For a scripted
API client there was no clear way of telling that the operation timed out and
that it should retry.
2. Whenever the timeout _did occur_, the code used to panic. This was caused by
the API endpoint "delegating" to the same call path as the web, which uses a
slightly different way of reporting errors (HTML rather than JSON for
example).
More specifically, `api/v1/repo/file.go#GetArchive` just called through to
`web/repo/repo.go#Download`, which expects the `Context` to have a `Render`
field set, but which is `nil` for API calls. Hence, a `nil` pointer error.
The code addresses (1) by dropping the hard-coded timeout. Instead, any
timeout/cancelation on the incoming `Context` is used.
The code addresses (2) by updating the API endpoint to use a separate call path
for the API-triggered archive download. This avoids producing HTML-errors on
errors (it now produces JSON errors).
Signed-off-by: Peter Gardfjäll <peter.gardfjall.work@gmail.com>
The recovery, API, Web and package frameworks all create their own HTML
Renderers. This increases the memory requirements of Gitea
unnecessarily with duplicate templates being kept in memory.
Further the reloading framework in dev mode for these involves locking
and recompiling all of the templates on each load. This will potentially
hide concurrency issues and it is inefficient.
This PR stores the templates renderer in the context and stores this
context in the NormalRoutes, it then creates a fsnotify.Watcher
framework to watch files.
The watching framework is then extended to the mailer templates which
were previously not being reloaded in dev.
Then the locales are simplified to a similar structure.
Fix#20210Fix#20211Fix#20217
Signed-off-by: Andrew Thornton <art27@cantab.net>
This PR adds functionality to allow Gitea to sit behind an
HAProxy and HAProxy protocolled connections directly.
Fix#7508
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Disable doctor logging on panic
If permissions are incorrect for writing to the doctor log simply disable the log file
instead of panicing.
Related #20570
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update cmd/doctor.go
* Update cmd/doctor.go
Co-authored-by: delvh <dev.lh@web.de>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* merge `CheckLFSVersion` into `InitFull` (renamed from `InitWithSyncOnce`)
* remove the `Once` during git init, no data-race now
* for doctor sub-commands, `InitFull` should only be called in initialization stage
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* `PROTOCOL`: can be smtp, smtps, smtp+startls, smtp+unix, sendmail, dummy
* `SMTP_ADDR`: domain for SMTP, or path to unix socket
* `SMTP_PORT`: port for SMTP; defaults to 25 for `smtp`, 465 for `smtps`, and 587 for `smtp+startls`
* `ENABLE_HELO`, `HELO_HOSTNAME`: reverse `DISABLE_HELO` to `ENABLE_HELO`; default to false + system hostname
* `FORCE_TRUST_SERVER_CERT`: replace the unclear `SKIP_VERIFY`
* `CLIENT_CERT_FILE`, `CLIENT_KEY_FILE`, `USE_CLIENT_CERT`: clarify client certificates here
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
The current `admin auth` CLI for managing authentication source of type
LDAP via BindDN and Simple LDAP does not allow enabling the respective
source, once disabled via `--not-active`.
The same applies to `--synchronize-users` specifially for LDAP via
BindDN.
These changes add two new flags to LDAP related CLI commands:
- `--active` for both LDAP authentication source types
- `--disable-synchronize-users` for LDAP via BindDN
Signed-off-by: justusbunsi <61625851+justusbunsi@users.noreply.github.com>
This PR adds a new manager command to switch on SQL logging and to turn it off.
```
gitea manager logging log-sql
gitea manager logging log-sql --off
```
Signed-off-by: Andrew Thornton <art27@cantab.net>
fgprof is a sampling Go profiler that allows you to analyze On-CPU as
well as Off-CPU (e.g. I/O) time together.
Go's builtin sampling CPU profiler can only show On-CPU time, but it's
better than fgprof at that. Go also includes tracing profilers that can
analyze I/O, but they can't be combined with the CPU profiler.
fgprof is designed for analyzing applications with mixed I/O and CPU
workloads. This kind of profiling is also known as wall-clock profiling.
Whilst fgprof can cause significant STW latencies in applications with a
lot of goroutines (> 1-10k), these latencies only occur if the profile
is requested - it doesn't cause a delay by simply being available.
The fgprof profile is mounted on
`http://localhost:6060/debug/fgprof?seconds=3`
Signed-off-by: Andrew Thornton <art27@cantab.net>
* clean git support for ver < 2.0
* fine tune tests for markup (which requires git module)
* remove unnecessary comments
* try to fix tests
* try test again
* use const for GitVersionRequired instead of var
* try to fix integration test
* Refactor CheckAttributeReader to make a *git.Repository version
* update document for commit signing with Gitea's internal gitconfig
* update document for commit signing with Gitea's internal gitconfig
Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
- We don't need to buffer the logger with a thousand capacity. It's not
a high-throughput logger, this also caused issue whereby the logger
can't keep up with repeated messages being send(somehow they are lost in
the queue?).
- Resolves#19969
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Fix cli command restore-repo: "units" should be parsed as StringSlice because after #15790 it's read by c.StringSlice("units"). Before, the "units" were processed by strings.Split
* Add checking for invalid unit names
Co-authored-by: 6543 <6543@obermui.de>
* Move access and repo permission to models/perm/access
* fix test
* Move some git related files into sub package models/git
* Fix build
* fix git test
* move lfs to sub package
* move more git related functions to models/git
* Move functions sequence
* Some improvements per @KN4CK3R and @delvh
* Move some repository related code into sub package
* Move more repository functions out of models
* Fix lint
* Some performance optimization for webhooks and others
* some refactors
* Fix lint
* Fix
* Update modules/repository/delete.go
Co-authored-by: delvh <dev.lh@web.de>
* Fix test
* Merge
* Fix test
* Fix test
* Fix test
* Fix test
Co-authored-by: delvh <dev.lh@web.de>
When Gitea is running as PID 1 git will occassionally orphan child processes leading
to (defunct) processes. This PR simply sets Setpgid to true on these child processes
meaning that these defunct processes will also be correctly reaped.
Fix#19077
Signed-off-by: Andrew Thornton <art27@cantab.net>
* docs: update the ROOT documentation and error messages
* The documentation now reflects what happens in the
setting/repository.go::newRepository function:
filepath.Join(AppWorkPath, RepoRootPath) was missing.
* The error message displayed when RepoRootPath is not found now
displays the value of RepoRootPath. Given the complexity of the
construction of this value, only referring to it in the abstract
is likely to be misleading to the Gitea admin trying to interpret
the message.
Co-authored-by: delvh <dev.lh@web.de>
* Apply DefaultUserIsRestricted in CreateUser
* Enforce system defaults in CreateUser
Allow for overwrites with CreateUserOverwriteOptions
* Fix compilation errors
* Add "restricted" option to create user command
* Add "restricted" option to create user admin api
* Respect default setting.Service.RegisterEmailConfirm and setting.Service.RegisterManualConfirm where needed
* Revert "Respect default setting.Service.RegisterEmailConfirm and setting.Service.RegisterManualConfirm where needed"
This reverts commit ee95d3e8dc.
* When dumping trim the standard suffices instead of a random suffix
Instead of using the `path.Ext()` to trim the last "extension" suffix, just iterate
through the supported suffices and trim those.
Fix#19424
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix enum with to have correct supported types only
Co-authored-by: 6543 <6543@obermui.de>
* Warn on SSH connection for incorrect configuration
- When `setting.RepoRootPath` cannot be found(most likely due to
incorrect configuration) show "Gitea: Incorrect configuration" on the
client-side to help easier with debugging the problem.
* Update cmd/serv.go
Co-authored-by: delvh <dev.lh@web.de>
* Don't leak configuration
* Update cmd/serv.go
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Follows: #19284
* The `CopyDir` is only used inside test code
* Rewrite `ToSnakeCase` with more test cases
* The `RedisCacher` only put strings into cache, here we use internal `toStr` to replace the legacy `ToStr`
* The `UniqueQueue` can use string as ID directly, no need to call `ToStr`
Follows #19266, #8553, Close#18553, now there are only three `Run..(&RunOpts{})` functions.
* before: `stdout, err := RunInDir(path)`
* now: `stdout, _, err := RunStdString(&git.RunOpts{Dir:path})`
Continues on from #19202.
Following the addition of pprof labels we can now more easily understand the relationship between a goroutine and the requests that spawn them.
This PR takes advantage of the labels and adds a few others, then provides a mechanism for the monitoring page to query the pprof goroutine profile.
The binary profile that results from this profile is immediately piped in to the google library for parsing this and then stack traces are formed for the goroutines.
If the goroutine is within a context or has been created from a goroutine within a process context it will acquire the process description labels for that process.
The goroutines are mapped with there associate pids and any that do not have an associated pid are placed in a group at the bottom as unbound.
In this way we should be able to more easily examine goroutines that have been stuck.
A manager command `gitea manager processes` is also provided that can export the processes (with or without stacktraces) to the command line.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Remove `db.DefaultContext` usage in routers, use `ctx` directly
* Use `ctx` directly if there is one, remove some `db.DefaultContext` in `services`
* Use ctx instead of db.DefaultContext for `cmd` and some `modules` packages
* fix incorrect context usage
* Add setting for a JSON that maps LDAP groups to Org Teams.
* Add log when removing or adding team members.
* Sync is being run on login and periodically.
* Existing group filter settings are reused.
* Adding and removing team members.
* Sync not existing LDAP group.
* Login with broken group map JSON.
* Refactor jwt.StandardClaims to RegisteredClaims
go-jwt/jwt has deprecated the StandardClaims interface to use RegisteredClaims
instead. This PR migrates to use this new format.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Apply suggestions from code review
Co-authored-by: Gusted <williamzijl7@hotmail.com>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Gusted <williamzijl7@hotmail.com>
Make router logger more friendly, show the related function name/file/line.
[BREAKING]
This PR substantially changes the logging format of the router logger. If you use this logging for monitoring e.g. fail2ban you will need to update this to match the new format.
This PR continues the work in #17125 by progressively ensuring that git
commands run within the request context.
This now means that the if there is a git repo already open in the context it will be used instead of reopening it.
Signed-off-by: Andrew Thornton <art27@cantab.net>
if return a error message to cli, it will print it
to stderr which is duplicate with our code (line 82
in same file). so user will see two line same
error message in git output. I think it's not mecessary,
so suggerst not return error message to cli. Thanks.
Signed-off-by: a1012112796 <1012112796@qq.com>
Migrate from U2F to Webauthn
Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
It appears that there are several places that password length, complexity and ispwned
are not currently been checked when changing passwords. This PR adds these.
Fix#17977
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Add setting to OAuth handlers to override local 2FA settings
This PR adds a setting to OAuth and OpenID login sources to allow the source to
override local 2FA requirements.
Fix#13939
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Fix regression from #16544
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add scopes settings
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix trace logging in auth_openid
Signed-off-by: Andrew Thornton <art27@cantab.net>
* add required claim options
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Move UpdateExternalUser to externalaccount
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Allow OAuth2/OIDC to set Admin/Restricted status
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Allow use of the same group claim name for the prohibit login value
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fixup! Move UpdateExternalUser to externalaccount
* as per wxiaoguang
Signed-off-by: Andrew Thornton <art27@cantab.net>
* add label back in
Signed-off-by: Andrew Thornton <art27@cantab.net>
* adjust localisation
Signed-off-by: Andrew Thornton <art27@cantab.net>
* placate lint
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Move keys to models/keys
* Rename models/keys -> models/asymkey
* change the missed package name
* Fix package alias
* Fix test
* Fix docs
* Fix test
* Fix test
* merge
* Some refactors related repository model
* Move more methods out of repository
* Move repository into models/repo
* Fix test
* Fix test
* some improvements
* Remove unnecessary function
* Improve install code to avoid low-level mistakes.
If a user tries to do a re-install in a Gitea database, they gets a warning and double check.
When Gitea runs, it never create empty app.ini automatically.
Also some small (related) refactoring:
* Refactor db.InitEngine related logic make it more clean (especially for the install code)
* Move some i18n strings out from setting.go to make the setting.go can be easily maintained.
* Show errors in CLI code if an incorrect app.ini is used.
* APP_DATA_PATH is created when installing, and checked when starting (no empty directory is created any more).
This PR registers requests with the process manager and manages hierarchy within the processes.
Git repos are then associated with a context, (usually the request's context) - with sub commands using this context as their base context.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Remove unnecessary functions of User struct
* Move more database methods out of user struct
* Move more database methods out of user struct
* Fix template failure
* Fix bug
* Remove finished FIXME
* remove unnecessary code
* Simplify Gothic to use our session store instead of creating a different store
We have been using xormstore to provide a separate session store for our OAuth2 logins
however, this relies on using gorilla context and some doubling of our session storing.
We can however, simplify and simply use our own chi-based session store. Thus removing
a cookie and some of the weirdness with missing contexts.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* as per review
Signed-off-by: Andrew Thornton <art27@cantab.net>
* as per review
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Handle MaxTokenLength
Signed-off-by: Andrew Thornton <art27@cantab.net>
* oops
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
* Allow LDAP Sources to provide Avatars
Add setting to LDAP source to allow it to provide an Avatar.
Currently this is required to point to the image bytes.
Fix#4144
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Rename as Avatar Attribute (drop JPEG)
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Always synchronize avatar if there is change
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Actually get the avatar from the ldap
Signed-off-by: Andrew Thornton <art27@cantab.net>
* clean-up
Signed-off-by: Andrew Thornton <art27@cantab.net>
* use len()>0 rather than != ""
Signed-off-by: Andrew Thornton <art27@cantab.net>
* slight shortcut in IsUploadAvatarChanged
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
This PR extends #16594 to allow LDAP to be able to be set to skip local 2FA too. The technique used here would be extensible to PAM and SMTP sources.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Johan Van de Wauw