Commit Graph

7722 Commits

Author SHA1 Message Date
Shay 84ae2e3f6f
Fix deletion for Dehydrated Devices (#16046) 2023-08-04 10:49:54 -07:00
Patrick Cloke d98a43d922
Stabilize support for MSC3970: updated transaction semantics (scope to `device_id`) (#15629)
For now this maintains compatible with old Synapses by falling back
to using transaction semantics on a per-access token. A future version
of Synapse will drop support for this.
2023-08-04 07:47:18 -04:00
Shay 0a5f4f7665
Move support for application service query parameter authorization behind a configuration option (#16017) 2023-08-03 11:43:51 -07:00
Mathieu Velten f0a860908b
Allow config of the backoff algorithm for the federation client. (#15754)
Adds three new configuration variables:

* destination_min_retry_interval is identical to before (10mn).
* destination_retry_multiplier is now 2 instead of 5, the maximum value will
  be reached slower.
* destination_max_retry_interval is one day instead of (essentially) infinity.

Capping this will cause destinations to continue to be retried sometimes instead
of being lost forever. The previous value was 2 ^ 62 milliseconds.
2023-08-03 14:36:55 -04:00
reivilibre 9c462f18a4
Allow modules to check whether the current worker is configured to run background tasks. (#15991) 2023-08-03 08:42:19 -04:00
Patrick Cloke 4f5bccbbba
Add forward-compatibility for the redacts property (MSC2174). (#16013)
The location of the redacts field changes in room version 11. Ensure
it is copied to the *new* location for *old* room versions for
forwards-compatibility with clients.

Note that copying it to the *old* location for the *new* room version
was previously handled.
2023-08-02 15:35:54 +00:00
Patrick Cloke 01a45869f0
Update MSC3958 support to interact with intentional mentions. (#15992)
* Updates the rule ID.
* Use `event_property_is` instead of `event_match`.

This updates the implementation of MSC3958 to match the latest
text from the MSC.
2023-08-02 08:41:32 -04:00
Andrew Morgan a51b0862a1
Update `flake.lock` to fix running the nix developer environment on MacOS (#16019) 2023-08-02 07:47:16 +01:00
Patrick Cloke 8fe1fd906a
Update certifi to 2023.7.22 and pygments to 2.15.1. (#16044) 2023-08-01 15:55:58 +00:00
Patrick Cloke 90ad836ed8
Properly setup the additional sequences in the portdb script. (#16043)
The un_partial_stated_event_stream_sequence and
application_services_txn_id_seq were never properly configured
in the portdb script, resulting in an error on start-up.
2023-08-01 10:36:33 -04:00
Mohit Rathee 5eb3fd785b
Trim whitespace when setting display names (#16031) 2023-08-01 09:14:02 -04:00
Jason Little 7cbb2a00d1
Add metrics tracking for eviction to ResponseCache (#16028)
Track whether the ResponseCache is evicting due to invalidation
or due to time.
2023-08-01 08:10:49 -04:00
Patrick Cloke b7695ac388
Combine duplicated code for calculating an event ID from a txn ID (#16023)
Refactoring related to stabilization of MSC3970, refactor to combine
code which has the same logic.
2023-07-31 08:44:45 -04:00
Nils e02f4b7de2
Do not expose Admin API in caddy reverse proxy example (#16027)
Signed-off-by: Nils ANDRÉ-CHANG <nils@nilsand.re>
2023-07-31 13:25:06 +02:00
Erik Johnston ae55cc1e6b
Add ability to wait for locks and add locks to purge history / room deletion (#15791)
c.f. #13476
2023-07-31 10:58:03 +01:00
Shay 68b2611783
Clarify comment on key uploads over replication (#16016) 2023-07-27 15:08:46 -07:00
Mathieu Velten a719b703d9
Fix 404 on /profile when the display name is empty but not the avatar (#16012) 2023-07-27 15:45:05 +02:00
Mathieu Velten a461f1f846
Update PyYAML to 6.0.1 (#16011) 2023-07-27 14:51:26 +02:00
David Robertson f9f3e89354
Attempt to fix labelling in docker workflow (#16009) 2023-07-27 13:47:48 +01:00
Shay f98f4f2e16
Remove support for legacy application service paths (#15964) 2023-07-26 12:59:47 -07:00
Anshul Madnawat 58f8305114
Inline SQL queries using boolean parameters (#15525)
SQLite now supports TRUE and FALSE constants, simplify some
queries by inlining those instead of passing them as arguments.
2023-07-26 18:45:47 +00:00
Mo Balaa 96529c4236
Add synapse version as Docker container label (#15972)
Co-authored-by: Mo Balaa <balaa@fractalnetworks.co>
2023-07-26 16:16:12 +00:00
Mathieu Velten d4ea465496 Remove changelog file 2023-07-26 14:54:37 +02:00
Mathieu Velten 8ebfd577e2
Bump DB version to 79 since synapse v1.88 was already there (#15998) 2023-07-26 14:51:44 +02:00
Mathieu Velten dbee081d14 1.89.0rc1 2023-07-25 14:32:47 +02:00
Shay 641ff9ef7e
Support MSC3814: Dehydrated Devices (#15929)
Signed-off-by: Nicolas Werner <n.werner@famedly.com>
Co-authored-by: Nicolas Werner <n.werner@famedly.com>
Co-authored-by: Nicolas Werner <89468146+nico-famedly@users.noreply.github.com>
Co-authored-by: Hubert Chathi <hubert@uhoreg.ca>
2023-07-24 08:23:19 -07:00
SnipeX_ 05f8dada8b
Fix broken Arch Linux package link (#15981) 2023-07-24 09:06:10 -04:00
Erik Johnston 654902a758
Resync stale devices in background (#15975)
This is so we don't block responding to federation transaction while we
try and fetch the device lists.
2023-07-24 13:43:43 +01:00
Shay 5c7364fea5
Properly handle redactions of creation events (#15973) 2023-07-23 16:32:01 -07:00
Shay f08d05dd2c
Actually stop reading from column `user_id` of tables `profiles` (#15955) 2023-07-23 16:30:54 -07:00
Shay e1fa42249c
Build packages for Debian Trixie (#15961) 2023-07-23 16:30:05 -07:00
Erik Johnston fc1e534e41
Speed up updating state in large rooms (#15971)
This should speed up updating state in rooms with lots of state.
2023-07-20 15:51:28 +01:00
Will Lewis 835174180b
Fixed grafana deploy annotations in the dashboard config, so it shows for those not managing matrix.org (#15957)
Removed the 'matrix.org' hardcorded instance setting

Originally introduced in #15674

Co-authored-by: wrjlewis <will.lewis@askattest.com>
2023-07-20 12:33:06 +00:00
Erik Johnston fd44053b84
Don't log exceptions for every non-200 response (#15969)
Introduced in #15913
2023-07-20 11:07:58 +01:00
Erik Johnston ad52db3b5c
Reduce the amount of state we pull out (#15968) 2023-07-20 10:46:37 +01:00
Erik Johnston 67f9e5293e
Ensure a long state res does not starve CPU (#15960)
We do this by yielding the reactor in hot loops.
2023-07-19 17:00:33 +00:00
Erik Johnston 19796e20aa
Fix bad merge of #15933 (#15958)
This was because we reverted the bump of the schema version, so we were not applying the new deltas.
2023-07-19 12:17:08 +00:00
Erik Johnston 40a3583ba1
Fix race in triggers for read/write locks. (#15933) 2023-07-19 12:06:38 +01:00
Shay cb6e2c6cc7
Fix background schema updates failing over a large upgrade gap (#15887) 2023-07-18 16:59:27 -07:00
Patrick Cloke 6d81aec09f
Support room version 11 (#15912)
And fix a bug in the implementation of the updated redaction
format (MSC2174) where the top-level redacts field was not
properly added for backwards-compatibility.
2023-07-18 08:44:59 -04:00
Jason Little 199c270947
Add a locality to a few presence metrics (#15952) 2023-07-18 10:36:40 +01:00
Eric Eastwood 1c802de626
Re-introduce the outbound federation proxy (#15913)
Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`).

This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.
2023-07-18 09:49:21 +01:00
Andrew Morgan 85e0541db1
Pin the rust version in `flake.nix`, and bump to 1.70.0 to fix installing `ruff` (#15940) 2023-07-17 09:36:12 +01:00
Will Hunt 8d3656b994
Document that you cannot login as yourself on /_synapse/admin/v1/users/<user_id>/login (#15938) 2023-07-14 08:32:13 -04:00
Patrick Cloke 20ae617d14
Stop accepting 'user' parameter for application service registration. (#15928)
This is unspecced, but has existed for a very long time.
2023-07-13 07:23:56 -04:00
Patrick Cloke 204b66c203
Remove unneeded __init__. (#15926)
Remove an __init__ which only calls super() without changing the
input arguments.
2023-07-12 14:30:05 +00:00
Patrick Cloke 5bdf01fccd
Fix running with an empty experimental features section. (#15925) 2023-07-12 12:39:25 +00:00
Erik Johnston 36c6b92bfc
Fix push for invites received over federation (#15820) 2023-07-12 11:02:11 +00:00
Mathieu Velten 8eb7bb975e
Mark get_user_in_directory private since only used in tests (#15884) 2023-07-12 11:09:13 +02:00
Eric Eastwood 3bdb9b07fd
Make it more obvious which Python version runs on a given Linux distribution (#15909)
Make it more obvious which Python version runs on a given Linux distribution so when we end up dropping support for a given Python version, we can more easily find the reference to the Python version and remove any references for the distribution. We don't want to be running tests or building packages on a distribution that no longer has a supported Python version.

This way, we can avoid another situation like when we dropped support for Python 3.7 but forgot to drop the Debian Buster references everywhere (https://github.com/matrix-org/synapse/pull/15893)
2023-07-11 17:15:06 -05:00
Eric Eastwood 0371a354cf
Better clarify how to run a worker instance (pass both configs) (#15921)
Previously, if you just followed the instructions per the docs, you just ran into an error:

```sh
$ poetry run synapse_worker --config-path homeserver_generic_worker1.yaml

Missing mandatory `server_name` config option.
```
2023-07-11 17:13:54 -05:00
Eric Eastwood ae391db777
Better warning in logs when we fail to fetch an alias (#15922)
**Before:**
```
Error retrieving alias
```

**After:**
```
Error retrieving alias #foo:bar -> 401 Unauthorized
```

*Spawning from creating the [manual testing strategy for the outbound federation proxy](https://github.com/matrix-org/synapse/pull/15773).*
2023-07-11 17:12:41 -05:00
Eric Eastwood d7fc87d973
Bump Unix sockets intro version (#15924)
https://github.com/matrix-org/synapse/pull/15708 didn't quite make the cut for `1.88.0` this morning.
2023-07-11 15:32:50 -05:00
Jason Little 224ef0b669
Unix Sockets for HTTP Replication (#15708)
Unix socket support for `federation` and `client` Listeners has existed now for a little while(since [1.81.0](https://github.com/matrix-org/synapse/pull/15353)), but there was one last hold out before it could be complete: HTTP Replication communication. This should finish it up. The Listeners would have always worked, but would have had no way to be talked to/at.

---------

Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
Co-authored-by: Eric Eastwood <erice@element.io>
2023-07-11 13:08:06 -05:00
Patrick Cloke a4243183f0
Add + as an allowed character for Matrix IDs (MSC4009) (#15911) 2023-07-11 12:21:00 -04:00
David Robertson 7c7bd9898b
1.88.0rc1 2023-07-11 10:28:11 +01:00
Michael Telatynski b516d91999
Add `Server` to Access-Control-Expose-Headers header (#15908) 2023-07-11 09:18:50 +01:00
Eric Eastwood 2328e90fbb
Make the media `/upload` tracing less ambiguous (#15888)
A lot of the functions have the same name in this space like `store_file`,
and we also do it multiple times for different reasons (main media repo,
other storage providers, thumbnails, etc) so it's good to differentiate
them so your head doesn't explode.

Follow-up to https://github.com/matrix-org/synapse/pull/15850

Tracing instrumentation to media `/upload` code paths to investigate https://github.com/matrix-org/synapse/issues/15841
2023-07-10 17:23:11 -05:00
Shay 5e82b07d2c
Drop debian buster (#15893) 2023-07-10 10:39:36 -07:00
Eric Eastwood c9bf644fa0
Revert "Federation outbound proxy" (#15910)
Revert "Federation outbound proxy (#15773)"

This reverts commit b07b14b494.
2023-07-10 11:10:20 -05:00
Eric Eastwood a704a35dd7 Revert "Placeholder changelog"
This reverts commit 6e731e86bf.
2023-07-10 10:26:04 -05:00
Erik Johnston e55a9b3e41
Fix downgrading to previous version of Synapse (#15907)
We do this by marking the constraint as deferrable.
2023-07-10 16:24:42 +01:00
Erik Johnston 6774f265b4
Fix building rust with nightly (#15906)
Also fix up a warning.
2023-07-10 16:24:04 +01:00
Eric Eastwood 6e731e86bf Placeholder changelog 2023-07-10 10:23:30 -05:00
Shay 8a529e4fb6
Stop running sytest on buster/python3.7 (#15892) 2023-07-07 12:04:55 -07:00
Shay f25b0f8808
Stop writing to column `user_id` of tables `profiles` and `user_filters` (#15787) 2023-07-07 09:23:27 -07:00
Dirk Klimpel 677272caed
Remove `worker_replication_*` settings from worker doc (#15872)
Co-authored-by: Mathieu Velten <mathieuv@matrix.org>
2023-07-07 08:09:41 +00:00
Jason Little 2481b7dfa4
Remove `worker_replication_*` deprecated settings, with helpful errors on startup (#15860)
Co-authored-by: reivilibre <oliverw@matrix.org>
2023-07-07 07:45:25 +00:00
sarthak shah f19dd39dfc
Update link to the clients webpage, fix #15825 (#15874) 2023-07-06 17:28:09 +02:00
Eric Eastwood b07b14b494
Federation outbound proxy (#15773)
Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`).

This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.

The original code is from @erikjohnston's branches which I've gotten in-shape to merge.
2023-07-05 18:53:55 -05:00
Eric Eastwood 561d06b481
Remove support for Python 3.7 (#15851)
Fix https://github.com/matrix-org/synapse/issues/15836
2023-07-05 18:45:42 -05:00
Erik Johnston 39d131b016
Add basic read/write lock (#15782) 2023-07-05 17:25:00 +01:00
Eric Eastwood ce857c05d5
Add tracing to media `/upload` endpoint (#15850)
Add tracing instrumentation to media `/upload` code paths to investigate https://github.com/matrix-org/synapse/issues/15841
2023-07-05 10:22:21 -05:00
Sumner Evans cc780b3f77
docs/admin_api: fix header level on 'Users' page (#15852)
Signed-off-by: Sumner Evans <sumner@beeper.com>
2023-07-05 16:15:56 +02:00
Jason Little 4cf9f92f39
Fix could not serialize access due to concurrent `DELETE` from presence_stream (#15826)
* Change update_presence to have a isolation level of READ_COMMITTED

* changelog
2023-07-05 11:44:02 +01:00
Erik Johnston 95a96b21eb
Add foreign key constraint to `event_forward_extremities`. (#15751) 2023-07-05 09:43:19 +00:00
an0nfunc c303eca8cc
use Image.LANCZOS instead of Image.ANTIALIAS for thumbnail resize (#15876)
Image.ANTIALIAS is not defined in current pillow releases. Since ANTIALIAS was just using LANCZOS anyways, this is just a cosmetic change, but makes synapse work with most recent pillow releases.

Signed-off-by: Giovanni Harting <539@idlegandalf.com>
2023-07-05 10:52:12 +02:00
Michael Weimann c8e81898b6
Add not_user_type param to the list accounts admin API (#15844)
Signed-off-by: Michael Weimann <michaelw@element.io>
2023-07-04 15:03:20 -07:00
Olivier Wilkinson (reivilibre) 861752b3aa Merge branch 'master' into develop 2023-07-04 17:40:37 +01:00
Olivier Wilkinson (reivilibre) 664ba14080 1.87.0 2023-07-04 16:25:33 +01:00
Paarth Shah 649848627c Pin `pydantic` to <2.0.0 (#15862)
Signed-off-by: Paarth Shah <mail@shahpaarth.com>
2023-07-04 16:22:33 +01:00
Paarth Shah 670d590f8a
Pin `pydantic` to <2.0.0 (#15862)
Signed-off-by: Paarth Shah <mail@shahpaarth.com>
2023-07-04 09:33:24 +02:00
pacien 07d7cbfe69
devices: use combined ANY clause for faster cleanup (#15861)
Old device entries for the same user were being removed in individual
SQL commands, making the batch take way longer than necessary.

This combines the commands into a single one with a IN/ANY clause.

Example of log entry before the change, regularly observed with
"log_min_duration_statement = 10000" in PostgreSQL's config:

    LOG:  duration: 42538.282 ms  statement:
    DELETE FROM device_lists_stream
    WHERE user_id = '@someone' AND device_id = 'someid1'
    AND stream_id < 123456789
    ;
    DELETE FROM device_lists_stream
    WHERE user_id = '@someone' AND device_id = 'someid2'
    AND stream_id < 123456789
    ;
    [repeated for each device ID of that user, potentially a lot...]

With the patch applied on my instance for the past couple of days, I
no longer notice overly long statements of that particular kind.

Signed-off-by: pacien <pacien.trangirard@pacien.net>
2023-07-03 16:39:38 +02:00
reivilibre cd8b73aa97
Fix the `devenv up` configuration which was ignoring the config overrides. (#15854)
* Fix use of config override directory in `devenv up`

`--config-directory` is for the generate config script; `-c` is for usage

* Add homeserver config override directory to gitignore

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-07-03 11:39:52 +01:00
reivilibre 53aa26eddc
Add a timeout that aborts any Postgres statement taking more than 1 hour. (#15853)
* Add a timeout to Postgres statements

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-07-03 11:38:57 +01:00
Eric Eastwood 13fc89148c
Split out 2022 changes from the changelog (#15846)
Split out 2022 changes from the changelog so the rendered version in GitHub doesn't timeout as much.
2023-06-28 15:10:33 -05:00
Eric Eastwood 472c2c72f6 Prepare changelog for v1.87.0rc1 2023-06-27 10:29:20 -05:00
Shay 78cfa55dad
Fix sqlite `user_filters` upgrade (#15817) 2023-06-27 09:41:42 +01:00
Erik Johnston 25c55a9d22
Add login spam checker API (#15838) 2023-06-26 14:12:20 +00:00
Nicolas Werner e0c39d6bb5
Fix forgotten rooms missing in initial sync (#15815)
If you leave a room and forget it, then rejoin it, the room would be
missing from the next initial sync.

fixes #13262

Signed-off-by: Nicolas Werner <n.werner@famedly.com>
2023-06-21 14:56:31 +01:00
Erik Johnston 289ce3b8d9
Fix harmless exception in port DB script (#15814)
The port DB script would try and run database background tasks, which
could fail if the data they acted on was in the process of being ported.
These exceptions were non fatal.

Fixes #15789
2023-06-21 13:20:46 +00:00
Erik Johnston 6c749c5124
Fix typo in faster join docs (#15812)
Fixes #15756
2023-06-21 11:34:32 +01:00
Mathieu Velten 496f73103d
Allow for the configuration of max request retries and min/max retry delays in the matrix federation client (#15783) 2023-06-21 10:41:11 +02:00
Shay 6a5cf1a759
Fix Sytest environmental variable evaluation in CI (#15804) 2023-06-20 07:55:46 -07:00
ew-at-vier 2301a09d7a
Fix admin api documentation typo (#15805)
* Fix admin api documentation typo

Signed-off-by: Eric Wolf <eric.wolf@vier.ai>
2023-06-20 10:45:26 +00:00
Eric Eastwood 887fa4b66b
Switch from `matrix://` to `matrix-federation://` scheme for internal Synapse routing of outbound federation traffic (#15806)
`matrix://` is a registered specced scheme nowadays and doesn't make sense for
our internal to Synapse use case anymore. ([discussion]
(https://github.com/matrix-org/synapse/pull/15773#discussion_r1227598679))
2023-06-20 10:05:31 +01:00
Eric Eastwood 0f02f0b4da
Remove experimental MSC2716 implementation to incrementally import history into existing rooms (#15748)
Context for why we're removing the implementation:

 - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1487441010
 - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1504262734

Anyone wanting to continue MSC2716, should also address these leftover tasks: https://github.com/matrix-org/synapse/issues/10737

Closes https://github.com/matrix-org/synapse/issues/10737 in the fact that it is not longer necessary to track those things.
2023-06-16 14:12:24 -05:00
Andrew Morgan 2ac6c3bbb5
Don't always lock "user_ips" table when performing non-native upsert (#15788) 2023-06-16 15:25:44 +01:00
Mathieu Velten 0618bf94cd
push rules: fix internal conversion from _type to value (#15781)
Also fix wrong rule names for `is_user_mention` and `is_room_mention`.
2023-06-16 14:17:02 +02:00
Mathieu Velten f63d4a3a65
Regularly try to wake up dests instead of waiting for next PDU/EDU (#15743) 2023-06-16 10:15:12 +00:00