Support compression for Actions logs to save storage space and
bandwidth. Inspired by
https://github.com/go-gitea/gitea/issues/24256#issuecomment-1521153015
The biggest challenge is that the compression format should support
[seekable](https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md).
So when users are viewing a part of the log lines, Gitea doesn't need to
download the whole compressed file and decompress it.
That means gzip cannot help here. And I did research, there aren't too
many choices, like bgzip and xz, but I think zstd is the most popular
one. It has an implementation in Golang with
[zstd](https://github.com/klauspost/compress/tree/master/zstd) and
[zstd-seekable-format-go](https://github.com/SaveTheRbtz/zstd-seekable-format-go),
and what is better is that it has good compatibility: a seekable format
zstd file can be read by a regular zstd reader.
This PR introduces a new package `zstd` to combine and wrap the two
packages, to provide a unified and easy-to-use API.
And a new setting `LOG_COMPRESSION` is added to the config, although I
don't see any reason why not to use compression, I think's it's a good
idea to keep the default with `none` to be consistent with old versions.
`LOG_COMPRESSION` takes effect for only new log files, it adds `.zst` as
an extension to the file name, so Gitea can determine if it needs
decompression according to the file name when reading. Old files will
keep the format since it's not worth converting them, as they will be
cleared after #31735.
<img width="541" alt="image"
src="https://github.com/user-attachments/assets/e9598764-a4e0-4b68-8c2b-f769265183c9">
If the assign the pull request review to a team, it did not show the
members of the team in the "requested_reviewers" field, so the field was
null. As a solution, I added the team members to the array.
fix#31764
Fix#31657.
According to the
[doc](https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#onschedule)
of GitHub Actions, The timezone for cron should be UTC, not the local
timezone. And Gitea Actions doesn't have any reasons to change this, so
I think it's a bug.
However, Gitea Actions has extended the syntax, as it supports
descriptors like `@weekly` and `@every 5m`, and supports specifying the
timezone like `TZ=UTC 0 10 * * *`. So we can make it use UTC only when
the timezone is not specified, to be compatible with GitHub Actions, and
also respect the user's specified.
It does break the feature because the times to run tasks would be
changed, and it may confuse users. So I don't think we should backport
this.
## ⚠️ BREAKING ⚠️
If the server's local time zone is not UTC, a scheduled task would run
at a different time after upgrading Gitea to this version.
Fix#31707.
Also related to #31715.
Some Actions resources could has different types of ownership. It could
be:
- global: all repos and orgs/users can use it.
- org/user level: only the org/user can use it.
- repo level: only the repo can use it.
There are two ways to distinguish org/user level from repo level:
1. `{owner_id: 1, repo_id: 2}` for repo level, and `{owner_id: 1,
repo_id: 0}` for org level.
2. `{owner_id: 0, repo_id: 2}` for repo level, and `{owner_id: 1,
repo_id: 0}` for org level.
The first way seems more reasonable, but it may not be true. The point
is that although a resource, like a runner, belongs to a repo (it can be
used by the repo), the runner doesn't belong to the repo's org (other
repos in the same org cannot use the runner). So, the second method
makes more sense.
And the first way is not user-friendly to query, we must set the repo id
to zero to avoid wrong results.
So, #31715 should be right. And the most simple way to fix#31707 is
just:
```diff
- shared.GetRegistrationToken(ctx, ctx.Repo.Repository.OwnerID, ctx.Repo.Repository.ID)
+ shared.GetRegistrationToken(ctx, 0, ctx.Repo.Repository.ID)
```
However, it is quite intuitive to set both owner id and repo id since
the repo belongs to the owner. So I prefer to be compatible with it. If
we get both owner id and repo id not zero when creating or finding, it's
very clear that the caller want one with repo level, but set owner id
accidentally. So it's OK to accept it but fix the owner id to zero.
Fix#26685
If a commit status comes from Gitea Actions and the user cannot access
the repo's actions unit (the user does not have the permission or the
actions unit is disabled), a 404 page will occur after clicking the
"Details" link. We should hide the "Details" link in this case.
<img
src="https://github.com/go-gitea/gitea/assets/15528715/68361714-b784-4bb5-baab-efde4221f466"
width="400px" />
This leverages the existing `sync_external_users` cron job to
synchronize the `IsActive` flag on users who use an OAuth2 provider set
to synchronize. This synchronization is done by checking for expired
access tokens, and using the stored refresh token to request a new
access token. If the response back from the OAuth2 provider is the
`invalid_grant` error code, the user is marked as inactive. However, the
user is able to reactivate their account by logging in the web browser
through their OAuth2 flow.
Also changed to support this is that a linked `ExternalLoginUser` is
always created upon a login or signup via OAuth2.
### Notes on updating permissions
Ideally, we would also refresh permissions from the configured OAuth
provider (e.g., admin, restricted and group mappings) to match the
implementation of LDAP. However, the OAuth library used for this `goth`,
doesn't seem to support issuing a session via refresh tokens. The
interface provides a [`RefreshToken`
method](https://github.com/markbates/goth/blob/master/provider.go#L20),
but the returned `oauth.Token` doesn't implement the `goth.Session` we
would need to call `FetchUser`. Due to specific implementations, we
would need to build a compatibility function for every provider, since
they cast to concrete types (e.g.
[Azure](https://github.com/markbates/goth/blob/master/providers/azureadv2/azureadv2.go#L132))
---------
Co-authored-by: Kyle D <kdumontnu@gmail.com>
#26996
Added default sorting for milestones by name.
Additional, name for sorting closestduedate and furthestduedate was
broken, so I fixed it.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
We have some instances that only allow using an external authentication
source for authentication. In this case, users changing their email,
password, or linked OpenID connections will not have any effect, and
we'd like to prevent showing that to them to prevent confusion.
Included in this are several changes to support this:
* A new setting to disable user managed authentication credentials
(email, password & OpenID connections)
* A new setting to disable user managed MFA (2FA codes & WebAuthn)
* Fix an issue where some templates had separate logic for determining
if a feature was disabled since it didn't check the globally disabled
features
* Hide more user setting pages in the navbar when their settings aren't
enabled
---------
Co-authored-by: Kyle D <kdumontnu@gmail.com>
Before we had just the plain mail address as recipient. But now we
provide additional Information for the Mail clients.
---
*Sponsored by Kithara Software GmbH*
Fixes#22722
### Problem
Currently, it is not possible to force push to a branch with branch
protection rules in place. There are often times where this is necessary
(CI workflows/administrative tasks etc).
The current workaround is to rename/remove the branch protection,
perform the force push, and then reinstate the protections.
### Solution
Provide an additional section in the branch protection rules to allow
users to specify which users with push access can also force push to the
branch. The default value of the rule will be set to `Disabled`, and the
UI is intuitive and very similar to the `Push` section.
It is worth noting in this implementation that allowing force push does
not override regular push access, and both will need to be enabled for a
user to force push.
This applies to manual force push to a remote, and also in Gitea UI
updating a PR by rebase (which requires force push)
This modifies the `BranchProtection` API structs to add:
- `enable_force_push bool`
- `enable_force_push_whitelist bool`
- `force_push_whitelist_usernames string[]`
- `force_push_whitelist_teams string[]`
- `force_push_whitelist_deploy_keys bool`
### Updated Branch Protection UI:
<img width="943" alt="image"
src="https://github.com/go-gitea/gitea/assets/79623665/7491899c-d816-45d5-be84-8512abd156bf">
### Pull Request `Update branch by Rebase` option enabled with source
branch `test` being a protected branch:
<img width="1038" alt="image"
src="https://github.com/go-gitea/gitea/assets/79623665/57ead13e-9006-459f-b83c-7079e6f4c654">
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
closes#22015
After adding a passkey, you can now simply login with it directly by
clicking `Sign in with a passkey`.
Note for testing. You need to run gitea using `https` to get the full
passkeys experience.
---------
Co-authored-by: silverwind <me@silverwind.io>
This PR enhances the labels handling in issue_search by optimizing the
SQL query and de-duplicate the IDs when generating the query string.
---------
Co-authored-by: techknowlogick <techknowlogick@gitea.com>
1. There are already global "unit consts", no need to use context data, which is fragile
2. Remove the "String()" method from "unit", it would only cause rendering problems in templates
---------
Co-authored-by: silverwind <me@silverwind.io>
Fix#31361, and add tests
And this PR introduces an undocumented & debug-purpose-only config
option: `USE_SUB_URL_PATH`. It does nothing for end users, it only helps
the development of sub-path related problems.
And also fix#31366
Co-authored-by: @ExplodingDragon
fix a bug from #30490
`prs.GetIssueIDs()` will also be used in other places, e.g.
`InvalidateCodeComments`
so we should not add `if pr.Issue == nil` in it, or if `pr.Issue` is
already loaded, you will not get the issueID in the results list and
this is not an expected result.
So this will caused a bug:
before calling `InvalidateCodeComments`, all `pr.Issues` in `prs` are
loaded, so `issueIDs` in this function will always be `[]`.
Enable [unparam](https://github.com/mvdan/unparam) linter.
Often I could not tell the intention why param is unused, so I put
`//nolint` for those cases like webhook request creation functions never
using `ctx`.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: delvh <dev.lh@web.de>
Resolves#31167.
https://github.com/go-gitea/gitea/pull/30885 changed the behavior of
`repo.AvatarLink()` where it can now take the empty string and append it
to the app data URL. This does not point to a valid avatar image URL,
and, as the issue mentions, previous Gitea versions returned the empty
string.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
This PR implemented object storages(LFS/Packages/Attachments and etc.)
for Azure Blob Storage. It depends on azure official golang SDK and can
support both the azure blob storage cloud service and azurite mock
server.
Replace #25458Fix#22527
- [x] CI Tests
- [x] integration test, MSSQL integration tests will now based on
azureblob
- [x] unit test
- [x] CLI Migrate Storage
- [x] Documentation for configuration added
------
TODO (other PRs):
- [ ] Improve performance of `blob download`.
---------
Co-authored-by: yp05327 <576951401@qq.com>
This PR split the `Board` into two parts. One is the struct has been
renamed to `Column` and the second we have a `Template Type`.
But to make it easier to review, this PR will not change the database
schemas, they are just renames. The database schema changes could be in
future PRs.
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: yp05327 <576951401@qq.com>
This PR adds some fields to the gitea webhook payload that
[openproject](https://www.openproject.org/) expects to exists in order
to process the webhooks.
These fields do exists in Github's webhook payload so adding them makes
Gitea's native webhook more compatible towards Github's.
Replace #25741Close#24445Close#30658Close#20646
~Depends on #30805~
Since #25741 has been rewritten totally, to make the contribution
easier, I will continue the work in this PR. Thanks @6543
---------
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Remove "EncodeSha1", it shouldn't be used as a general purpose hasher
(just like we have removed "EncodeMD5" in #28622)
Rewrite the "time-limited code" related code and write better tests, the
old code doesn't seem quite right.
Fix#30872
We will assume the database is consistent before executing the
migration. So the indexes should exist. Removing `IF EXIST` then is safe
enough.
---------
Co-authored-by: silverwind <me@silverwind.io>
# The problem
The previous implementation will start multiple POST requests from the
frontend when moving a column and another bug is moving the default
column will never be remembered in fact.
# What's changed
- [x] This PR will allow the default column to move to a non-first
position
- [x] And it also uses one request instead of multiple requests when
moving the columns
- [x] Use a star instead of a pin as the icon for setting the default
column action
- [x] Inserted new column will be append to the end
- [x] Fix#30701 the newly added issue will be append to the end of the
default column
- [x] Fix when deleting a column, all issues in it will be displayed
from UI but database records exist.
- [x] Add a limitation for columns in a project to 20. So the sorting
will not be overflow because it's int8.
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
If an user is deactivated, it should not be in the list of users who are
suggested to be assigned or review-requested.
old assignees or reviewers are not affected.
---
*Sponsored by Kithara Software GmbH*
Suggested by logs in #30729
- Remove `math/rand.Seed`
`rand.Seed is deprecated: As of Go 1.20 there is no reason to call Seed
with a random value.`
- Replace `math/rand.Read`
`rand.Read is deprecated: For almost all use cases, [crypto/rand.Read]
is more appropriate.`
- Replace `math/rand` with `math/rand/v2`, which is available since Go
1.22
Follow #30472:
When a user is created by command line `./gitea admin user create`:
Old behavior before #30472: the first user (admin or non-admin) doesn't
need to change password.
Revert to the old behavior before #30472
Misspell 0.5.0 supports passing a csv file to extend the list of
misspellings, so I added some common ones from the codebase. There is at
least one typo in a API response so we need to decided whether to revert
that and then likely remove the dict entry.
Follow #30454
And fix#24957
When using "preferred_username", if no such field,
`extractUserNameFromOAuth2` (old `getUserName`) shouldn't return an
error. All other USERNAME options do not return such error.
And fine tune some logic and error messages, make code more stable and
more friendly to end users.
Follow #29468
1. Interpolate runs-on with variables when scheduling tasks.
2. The `GetVariablesOfRun` function will check if the `Repo` of the run
is nil.
---------
Co-authored-by: Giteabot <teabot@gitea.io>
Noteable additions:
- `redefines-builtin-id` forbid variable names that shadow go builtins
- `empty-lines` remove unnecessary empty lines that `gofumpt` does not
remove for some reason
- `superfluous-else` eliminate more superfluous `else` branches
Rules are also sorted alphabetically and I cleaned up various parts of
`.golangci.yml`.
Fixes#28255
The new query uses the id field to sort by "newer". This most not be
correct (usually it is) but it's faster (see #28255).
If someone has a better idea, please propose changes.
Co-authored-by: Giteabot <teabot@gitea.io>
Follow #30495
"HasAccess" behavior wasn't clear, to make it clear:
* Use a new name `HasAnyUnitAccess`, it will be easier to review related
code and permission problems.
* Separate everyone access mode to a separate field, then all calls to
HasAccess are reverted to old behavior before #30495.
* Add new tests.
---------
Co-authored-by: Giteabot <teabot@gitea.io>
The target_url is necessary for the UI, but missed in
commit_status_summary table. This PR fix it.
---------
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Giteabot <teabot@gitea.io>
This PR adds a new table named commit status summary to reduce queries
from the commit status table. After this change, commit status summary
table will be used for the final result, commit status table will be for
details.
---------
Co-authored-by: Jason Song <i@wolfogre.com>
When repo is a mirror, and commit author is an external user, then
`GetUserByEmail` will return error.
reproduce/test:
- mirror Gitea to your instance
- disable action and enable it again, this will trigger
`DetectAndHandleSchedules`
ps: also follow #24706, it only fixed normal runs, not scheduled runs.
As the latest tag of `github.com/denisenkom/go-mssqldb` is in 2022, but
as a fork of it, `github.com/microsoft/go-mssqldb` has more activities
than the original repository. We can convert the driver to the fork.
Since the interface of Go database driver are the same, it should have
no any affect for the end users.
Fix#30243
We only checking unit disabled when detecting workflows, but not in
runner `FetchTask`.
So if a workflow was detected when action unit is enabled, but disabled
later, `FetchTask` will still return these detected actions.
Global setting: repo.ENABLED and repository.`DISABLED_REPO_UNITS` will
not effect this.
Fix#29074 (allow to disable all builtin apps) and don't make the doctor
command remove the builtin apps.
By the way, rename refobject and joincond to camel case.
Fixes#30235
If the key id "front" byte has a single digit, `%X` is missing the 0
prefix.
` 38D1A3EADDBEA9C` instead of
`038D1A3EADDBEA9C`
When using the `IssuerFingerprint` slice `%X` is enough but I changed it
to `%016X` too to be consistent.
1. check `IsActive` before calling `IsLastAdminUser`.
2. Fix some comments and error messages.
3. Don't `return err` if "removing file" fails in `DeleteUser`.
4. Remove incorrect `DeleteInactiveEmailAddresses`. Active users could
also have inactive emails, and inactive emails do not support
"olderThan"
5. Add tests
## Changes
- Adds setting `EXTERNAL_USER_DISABLE_FEATURES` to disable any supported
user features when login type is not plain
- In general, this is necessary for SSO implementations to avoid
inconsistencies between the external account management and the linked
account
- Adds helper functions to encourage correct use
Jason Song