Commit Graph

15503 Commits

Author SHA1 Message Date
Eric Eastwood f9561b9e37
Some house keeping on `maybe_backfill()` functions (#15709) 2023-06-05 23:38:52 -05:00
Shay d0c4257f14
`N + 3`: Read from column `full_user_id` rather than `user_id` of tables `profiles` and `user_filters` (#15649) 2023-06-02 17:24:13 -07:00
Mathieu Velten e0f2429d13
Add a catch-all * to the supported relation types when redacting (#15705)
This is an update to MSC3912 implementation
2023-06-02 13:13:50 +00:00
Eric Eastwood 30a5076da8
Log when events are (unexpectedly) filtered out of responses in tests (#14213)
See https://github.com/matrix-org/synapse/pull/14095#discussion_r990335492

This is useful because when see that a relevant event is an `outlier` or `soft-failed`, then that's a good unexpected indicator explaining why it's not showing up. `filter_events_for_client` is used in `/sync`, `/messages`, `/context` which are all common end-to-end assertion touch points (also notifications, relations).
2023-06-01 21:27:18 -05:00
H. Shay 8af29155ec Merge branch 'release-v1.85' into develop 2023-06-01 10:26:37 -07:00
Erik Johnston 5ed0e8c61f
Cache requests for user's devices from federation (#15675)
This should mitigate the issue where lots of different servers requests
the same user's devices all at once.
2023-06-01 13:25:20 +00:00
Hugh Nimmo-Smith d1693f0362
Implement stable support for MSC3882 to allow an existing device/session to generate a login token for use on a new device/session (#15388)
Implements stable support for MSC3882; this involves updating Synapse's support to
match the MSC / the spec says.

Continue to support the unstable version to allow clients to transition.
2023-06-01 08:52:51 -04:00
Shay 6d9e2fd878
Speed up background jobs populate_full_user_id_user_filters and populate_full_user_id_profiles (#15700) 2023-05-31 15:13:48 -07:00
Patrick Cloke 6f18812bb0
Add stubs package for lxml. (#15697)
The stubs have some issues so this has some generous cast
and ignores in it, but it is better than not having stubs.

Note that confusing that Element is a function which creates
_Element instances (and similarly for Comment).
2023-05-31 17:06:57 +00:00
reivilibre 11e15d79b8
Fix a performance issue introduced in Synapse v1.83.0 which meant that purging rooms was very slow and database-intensive. (#15693)
* Add indices required to efficiently validate new foreign key constraints on stream_ordering

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-05-31 14:59:56 +01:00
Gabriel Féron daf3a67908
Add get_canonical_room_alias to module API (#15450)
Co-authored-by: Boxdot <d@zerovolt.org>
2023-05-31 09:18:37 -04:00
Patrick Cloke c01343de43
Add stricter mypy options (#15694)
Enable warn_unused_configs, strict_concatenate, disallow_subclassing_any,
and disallow_incomplete_defs.
2023-05-31 07:18:29 -04:00
Quentin Gliech ceb3dd77db Enforce that an admin token also has the basic Matrix API scope 2023-05-30 09:43:06 -04:00
Quentin Gliech 32a2f05004 Make the config tests spawn the homeserver only when needed 2023-05-30 09:43:06 -04:00
Quentin Gliech f739bde962 Reject tokens with multiple device scopes 2023-05-30 09:43:06 -04:00
Quentin Gliech 98afc57d59 Make OIDC scope constants 2023-05-30 09:43:06 -04:00
Quentin Gliech 14a5be9c4d Handle errors when introspecting tokens
This returns a proper 503 when the introspection endpoint is not working
for some reason, which should avoid logging out clients in those cases.
2023-05-30 09:43:06 -04:00
Quentin Gliech e343125b38 Disable incompatible Admin API endpoints 2023-05-30 09:43:06 -04:00
Quentin Gliech 4d0231b364 Make AS tokens work & allow ASes to /register 2023-05-30 09:43:06 -04:00
Quentin Gliech c008b44b4f Add an admin token for MAS -> Synapse calls 2023-05-30 09:43:06 -04:00
Hugh Nimmo-Smith 249f4a338d Refactor config to be an experimental feature
Also enforce you can't combine it with incompatible config options
2023-05-30 09:43:06 -04:00
Quentin Gliech 31691d6151 Disable account related endpoints when using OAuth delegation 2023-05-30 09:43:06 -04:00
Hugh Nimmo-Smith 5fe96082d0 Actually enforce guest + return www-authenticate header 2023-05-30 09:43:06 -04:00
Hugh Nimmo-Smith a1374b5c70 MSC2967: Check access token scope for use as user and add guest support 2023-05-30 09:43:06 -04:00
Hugh Nimmo-Smith d20669971a Use `name` claim as display name when registering users on the fly.
This makes is so that the `name` claim got when introspecting the token
is used as the display name when registering a user on the fly.
2023-05-30 09:43:06 -04:00
Quentin Gliech f9cd549f64 Record the `sub` claims as an external_id 2023-05-30 09:43:06 -04:00
Quentin Gliech 7628dbf4e9 Handle the Synapse admin scope 2023-05-30 09:43:06 -04:00
Quentin Gliech c5cf1b421d Save the scopes in the requester 2023-05-30 09:43:06 -04:00
Quentin Gliech e82ec6d008 MSC2965: OIDC Provider discovery via well-known document 2023-05-30 09:43:06 -04:00
Quentin Gliech 8f576aa462 Expose the public keys used for client authentication on an endpoint 2023-05-30 09:43:06 -04:00
Quentin Gliech 765244faee Initial MSC3964 support: delegation of auth to OIDC server 2023-05-30 09:43:06 -04:00
Quentin Gliech e2c8458bba Make the api.auth.Auth a Protocol 2023-05-30 09:43:06 -04:00
Sean Quah 5d8c659373
Remove unused `FederationServer.__str__` override (#15690)
Signed-off-by: Sean Quah <seanq@matrix.org>
2023-05-30 14:37:39 +01:00
Jason Little c835befd10
Add Unix socket support for Redis connections (#15644)
Adds a new configuration setting to connect to Redis via a Unix
socket instead of over TCP. Disabled by default.
2023-05-26 15:28:39 -04:00
Travis Ralston 50918c4940
Add `MSC3820opt2` as a known room version (#15678) 2023-05-26 18:05:24 +00:00
Patrick Cloke 2ad91ec628
Set thread_id column to non-null for event_push_{actions,actions_staging,summary} (#15597)
Updates the database schema to require a thread_id (by adding a
constraint that the column is non-null) for event_push_actions,
event_push_actions_staging, and event_push_actions_summary.

For PostgreSQL we add the constraint as NOT VALID, then
VALIDATE the constraint a background job to avoid locking
the table during an upgrade.

Each table is updated as a separate schema delta to avoid
deadlocks between them.

For SQLite we simply rebuild the table & copy the data.
2023-05-26 13:16:08 -04:00
Olivier Wilkinson (reivilibre) a1154dfc20 Merge branch 'master' into develop 2023-05-26 17:16:15 +01:00
reivilibre c775d80b73
Fix a bug introduced in Synapse v1.84.0 where workers do not start up when no `instance_map` was provided. (#15672)
* Fix #15669: always populate instance map even if it was empty

* Fix some tests

* Fix more tests

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* CI fix: don't forget to update apt repository sources before installing olddeps deps

* Add test testing the backwards compatibility

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-05-26 14:28:55 +00:00
Travis Ralston 4e013093a8
Add MSC3820 (room version 11) option 2 unstable room version. (#15666) 2023-05-26 07:46:13 -04:00
Eric Eastwood 77156a4bc1
Process previously failed backfill events in the background (#15585)
Process previously failed backfill events in the background because they are bound to fail again and we don't need to waste time holding up the request for something that is bound to fail again.

Fix https://github.com/matrix-org/synapse/issues/13623

Follow-up to https://github.com/matrix-org/synapse/issues/13621 and https://github.com/matrix-org/synapse/issues/13622

Part of making `/messages` faster: https://github.com/matrix-org/synapse/issues/13356
2023-05-24 23:22:24 -05:00
Shay 8839b6c2f8
Add requesting user id parameter to key claim methods in `TransportLayerClient` (#15663) 2023-05-24 13:23:26 -07:00
Erik Johnston c7e9c1d5ae
Speed up user directory rebuild for users some more... (#15665) 2023-05-24 14:13:28 +00:00
Patrick Cloke 1f55c04cbc
Improve type hints for cached decorator. (#15658)
The cached decorators always return a Deferred, which was not
properly propagated. It was close enough when wrapping coroutines,
but failed if a bare function was wrapped.
2023-05-24 12:59:31 +00:00
Eric Eastwood 379eb2d7ab
Fix `@trace` not wrapping some state methods that return coroutines correctly (#15647)
```
2023-05-21 09:30:09,288 - synapse.logging.opentracing - 940 - ERROR - POST-1 - @trace may not have wrapped StateStorageController.get_state_for_groups correctly! The function is not async but returned a coroutine
```

Tracing instrumentation for these functions originally introduced in https://github.com/matrix-org/synapse/pull/15610
2023-05-23 12:26:25 -05:00
Patrick Cloke 7c9b91790c
Consolidate logic to check for deactivated users. (#15634)
This moves the deactivated user check to the method which
all login types call.

Additionally updates the application service tests to be more
realistic by removing invalid tests and fixing server names.
2023-05-23 10:35:43 -04:00
Jason Little 1df0221bda
Use a custom scheme & the worker name for replication requests. (#15578)
All the information needed is already in the `instance_map`, so
use that instead of passing the hostname / IP & port manually
for each replication request.

This consolidates logic for future improvements of using e.g.
UNIX sockets for workers.
2023-05-23 09:05:30 -04:00
Andrew Morgan 737f7ddf58
Remove outdated comment in log config (#15648) 2023-05-22 17:58:58 +00:00
Patrick Cloke c5d1e6d414
Properly parse event_fields in filters (#15607)
The event_fields property in filters should use the proper
escape rules, namely backslashes can be escaped with
an additional backslash.

This adds tests (adapted from matrix-js-sdk) and implements
the logic to properly split the event_fields strings.
2023-05-22 11:31:22 -04:00
Eric Eastwood 703a8f9c67
Instrument `state` and `state_group` storage related things (tracing) (#15610)
Instrument `state` and `state_group` storage related things (tracing) so it's a little more clear where these database transactions are coming from as there is a lot of wires crossing in these functions.

Part of `/messages` performance investigation: https://github.com/matrix-org/synapse/issues/13356
2023-05-19 12:26:58 -05:00
Eric Eastwood ca3c07e833
Trace how many new events from the backfill response we need to process (#15633)
You can kinda derive this information from how many `_process_pulled_event` spans there are but it would be nice to quickly glance.
2023-05-19 11:18:45 -05:00