Commit Graph

162 Commits

Author SHA1 Message Date
Quentin Gliech 7d52ce7d4b
Format files with Ruff (#17643)
I thought ruff check would also format, but it doesn't.

This runs ruff format in CI and dev scripts. The first commit is just a
run of `ruff format .` in the root directory.
2024-09-02 12:39:04 +01:00
Shay cf69f8d59b
Support MSC3916 by adding a federation /thumbnail endpoint and authenticated `_matrix/client/v1/media/thumbnail` endpoint (#17388)
[MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916)
added the endpoints `_matrix/federation/v1/media/thumbnail` and the
authenticated `_matrix/client/v1/media/thumbnail`.

This PR implements those endpoints, along with stabilizing
`_matrix/client/v1/media/config` and
`_matrix/client/v1/media/preview_url`.

Complement tests are at
https://github.com/matrix-org/complement/pull/728
2024-07-08 10:11:20 +01:00
Shay 8f890447b0
Support MSC3916 by adding `_matrix/client/v1/media/download` endpoint (#17365) 2024-07-02 14:07:04 +01:00
Erik Johnston 2f5a77efae
Limit size of presence EDUs (#17371)
Otherwise they are unbounded.

---------

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-07-01 12:48:36 +01:00
Shay a023538822
Re-introduce federation /download endpoint (#17350) 2024-06-25 14:35:37 +00:00
Andrew Morgan 199223062a
Revert "Support MSC3916 by adding a federation `/download` endpoint" (#17325) 2024-06-18 16:54:19 +01:00
Shay ab94bce02c
Support MSC3916 by adding a federation `/download` endpoint (#17172) 2024-06-07 13:54:28 +01:00
Timshel 34a8652366
Optional whitespace support in Authorization (#1350) (#17145)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-05-08 13:56:16 +00:00
Richard van der Hoff c897ac63e9
Ensure that incoming to-device messages are not dropped (#17127)
... when workers are unreachable, etc.

Fixes https://github.com/element-hq/synapse/issues/17117.

The general principle is just to make sure that we propagate any
exceptions to the JsonResource, so that we return an error code to the
sending server. That means that the sending server no longer considers
the message safely sent, so it will retry later.

In the issue, Erik mentions that an alternative solution would be to
persist the to-device messages into a table so that they can be retried.
This might be an improvement for performance, but even if we did that,
we still need this mechanism, since we might be unable to reach the
database. So, if we want to do that, it can be a later follow-up.

---------

Co-authored-by: Erik Johnston <erik@matrix.org>
2024-04-29 14:11:00 +01:00
Erik Johnston 23740eaa3d
Correctly mention previous copyright (#16820)
During the migration the automated script to update the copyright
headers accidentally got rid of some of the existing copyright lines.
Reinstate them.
2024-01-23 11:26:48 +00:00
Patrick Cloke 8e1e62c9e0 Update license headers 2023-11-21 15:29:58 -05:00
Patrick Cloke f2f2c7c1f0
Use full GitHub links instead of bare issue numbers. (#16637) 2023-11-15 08:02:11 -05:00
Patrick Cloke 679c691f6f
Remove more usages of cursor_to_dict. (#16551)
Mostly to improve type safety.
2023-10-26 15:12:28 -04:00
Patrick Cloke fc31b495b3
Stop sending incorrect knock_state_events. (#16403)
Synapse was incorrectly implemented with a knock_state_events
property on some APIs (instead of knock_room_state). This was
correct in Synapse 1.70.0, but *both* fields were sent to also be
compatible with Synapse versions expecting the wrong field.

Enough time has passed that only the correct field needs to be
included/handled.
2023-10-06 07:27:35 -04:00
Patrick Cloke f84da3c32e
Add a cache around server ACL checking (#16360)
* Pre-compiles the server ACLs onto an object per room and
  invalidates them when new events come in.
* Converts the server ACL checking into Rust.
2023-09-26 11:57:50 -04:00
Erik Johnston d35bed8369
Don't wake up destination transaction queue if they're not due for retry. (#16223) 2023-09-04 17:14:09 +01:00
Patrick Cloke 9ec3da06da
Bump mypy-zope & mypy. (#16188) 2023-08-29 10:38:56 -04:00
Patrick Cloke daf11e26ef
Replace make_awaitable with AsyncMock (#16179)
Python 3.8 provides a native AsyncMock, we can replace the
homegrown version we have.
2023-08-24 19:38:46 -04:00
Eric Eastwood 887fa4b66b
Switch from `matrix://` to `matrix-federation://` scheme for internal Synapse routing of outbound federation traffic (#15806)
`matrix://` is a registered specced scheme nowadays and doesn't make sense for
our internal to Synapse use case anymore. ([discussion]
(https://github.com/matrix-org/synapse/pull/15773#discussion_r1227598679))
2023-06-20 10:05:31 +01:00
Mathieu Velten f63d4a3a65
Regularly try to wake up dests instead of waiting for next PDU/EDU (#15743) 2023-06-16 10:15:12 +00:00
Patrick Cloke 1e89976b26
Rename blacklist/whitelist internally. (#15620)
Avoid renaming configuration settings for now and rename internal code
to use blocklist and allowlist instead.
2023-05-19 12:25:25 +00:00
Patrick Cloke 07771fa487
Remove experimental configuration flags & unstable values for faster joins (#15625)
Synapse will no longer send (or respond to) the unstable flags
for faster joins. These were only available behind a configuration
flag and handled in parallel with the stable flags.
2023-05-19 07:23:09 -04:00
Patrick Cloke ea5c3ede4f
Finish type hints for federation client HTTP code. (#15465) 2023-04-24 13:12:06 -04:00
David Robertson c071cd5a0e
Ensure fed-sender catchup does not block for full state (#15248)
* Reproduce bad scenario in test
* Avoid catchup optimisation for partial state rooms
2023-03-13 12:31:19 +00:00
dependabot[bot] 9bb2eac719
Bump black from 22.12.0 to 23.1.0 (#15103) 2023-02-22 15:29:09 -05:00
Patrick Cloke 42aea0d8af
Add final type hint to tests.unittest. (#15072)
Adds a return type to HomeServerTestCase.make_homeserver and deal
with any variables which are no longer Any.
2023-02-14 14:03:35 -05:00
David Robertson 0f34abed7c
Type hints for tests.federation (#14991)
* Make tests.federation pass mypy

* Untyped defs in tests.federation.transport

* test methods return None

* Remaining type hints in tests.federation

* Changelog

* Avoid an uncessary type-ignore
2023-02-06 16:05:06 +00:00
Andrew Morgan 836c592f15
Fix type hints in knocking tests. (#14887) 2023-01-25 14:38:20 -05:00
David Robertson 5b3af1c7d0
Stabilise serving partial join responses (#14839)
Serving partial join responses is no longer experimental. They will only be served under the stable identifier if the the undocumented config flag experimental.msc3706_enabled is set to true.

Synapse continues to request a partial join only if the undocumented config flag experimental.faster_joins is set to true; this setting remains present and unaffected.
2023-01-17 12:44:15 +00:00
David Robertson 85a7a201fa
Also use stable name in SendJoinResponse struct (#14841)
* Also use stable name in SendJoinResponse struct

follow-up to #14832

* Changelog

* Fix a rename I missed

* Run black

* Update synapse/federation/federation_client.py

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2023-01-16 12:40:25 +00:00
David Robertson 52ae80dd1a
Use stable identifiers for faster joins (#14832)
* Use new query param when requesting a partial join

* Read new query param when serving partial join

* Provide new field names when serving partial joins

* Read new field names from partial join response

* Changelog
2023-01-13 17:58:53 +00:00
realtyem 854a6884d8
Modernize unit tests configuration settings for workers. (#14568)
Use the newer foo_instances configuration instead of the
deprecated flags to enable specific features (e.g. start_pushers).
2022-12-01 07:38:27 -05:00
Mathieu Velten 4569eda944
Use servers list approx to send read receipts when in partial state (#14549)
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
2022-11-30 13:39:47 +01:00
Patrick Cloke d748bbc8f8
Include thread information when sending receipts over federation. (#14466)
Include the thread_id field when sending read receipts over
federation. This might result in the same user having multiple
read receipts per-room, meaning multiple EDUs must be sent
to encapsulate those receipts.

This restructures the PerDestinationQueue APIs to support
multiple receipt EDUs, queue_read_receipt now becomes linear
time in the number of queued threaded receipts in the room for
the given user, it is expected this is a small number since receipt
EDUs are sent as filler in transactions.
2022-11-28 14:40:17 +00:00
David Robertson 1eed795fc5
Include heroes in partial join responses' state (#14442)
* Pull out hero selection logic

* Include heroes in partial join response's state

* Changelog

* Fixup trial test

* Remove TODO
2022-11-15 17:35:19 +00:00
David Robertson 5905ba12d0
Run trial tests against Python 3.11 (#13812) 2022-11-01 13:07:54 +00:00
Eric Eastwood 40fa8294e3
Refactor MSC3030 `/timestamp_to_event` to move away from our snowflake pull from `destination` pattern (#14096)
1. `federation_client.timestamp_to_event(...)` now handles all `destination` looping and uses our generic `_try_destination_list(...)` helper.
 2. Consistently handling `NotRetryingDestination` and `FederationDeniedError` across `get_pdu` , backfill, and the generic `_try_destination_list` which is used for many places we use this pattern.
 3. `get_pdu(...)` now returns `PulledPduInfo` so we know which `destination` we ended up pulling the PDU from
2022-10-26 16:10:55 -05:00
Quentin Gliech 9192d74b0b
Refactor OIDC tests to better mimic an actual OIDC provider. (#13910)
This implements a fake OIDC server, which intercepts calls to the HTTP client.
Improves accuracy of tests by covering more internal methods.

One particular example was the ID token validation, which previously mocked.

This uncovered an incorrect dependency: Synapse actually requires at least
authlib 0.15.1, not 0.14.0.
2022-10-25 14:25:02 +00:00
dependabot[bot] 0b7830e457
Bump flake8-bugbear from 21.3.2 to 22.9.23 (#14042)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Erik Johnston <erik@matrix.org>
Co-authored-by: David Robertson <davidr@element.io>
2022-10-19 19:38:24 +00:00
David Robertson cb20b885cb
Always close _all_ `ijson` coroutines, even if doing so raises Exceptions (#14065) 2022-10-06 18:17:50 +00:00
Eric Eastwood 70a4317692
Track when the pulled event signature fails (#13815)
Because we're doing the recording in `_check_sigs_and_hash_for_pulled_events_and_fetch` (previously named `_check_sigs_and_hash_and_fetch`), this means we will track signature failures for `backfill`, `get_room_state`, `get_event_auth`, and `get_missing_events` (all pulled event scenarios). And we also record signature failures from `get_pdu`.

Part of https://github.com/matrix-org/synapse/issues/13700

Part of https://github.com/matrix-org/synapse/issues/13676 and https://github.com/matrix-org/synapse/issues/13356

This PR will be especially important for https://github.com/matrix-org/synapse/pull/13816 so we can avoid the costly `_get_state_ids_after_missing_prev_event` down the line when `/messages` calls backfill.
2022-10-03 14:53:29 -05:00
Patrick Cloke efd108b45d
Accept & store thread IDs for receipts (implement MSC3771). (#13782)
Updates the `/receipts` endpoint and receipt EDU handler to parse a
`thread_id` from the body and insert it in the database.
2022-09-23 14:33:28 +00:00
reivilibre 7bc110a19e
Generalise the `@cancellable` annotation so it can be used on functions other than just servlet methods. (#13662) 2022-08-31 11:16:05 +00:00
Eric Eastwood 1a209efdb2
Update `get_users_in_room` mis-use to get hosts with dedicated `get_current_hosts_in_room` (#13605)
See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755
2022-08-24 14:15:37 -05:00
Dirk Klimpel e2ed1b7155
Use literals in place of `HTTPStatus` constants in tests (#13463) 2022-08-05 16:59:09 +02:00
Eric Eastwood 0f971ca68e
Update `get_pdu` to return the original, pristine `EventBase` (#13320)
Update `get_pdu` to return the untouched, pristine `EventBase` as it was originally seen over federation (no metadata added). Previously, we returned the same `event` reference that we stored in the cache which downstream code modified in place and added metadata like setting it as an `outlier`  and essentially poisoned our cache. Now we always return a copy of the `event` so the original can stay pristine in our cache and re-used for the next cache call.

Split out from https://github.com/matrix-org/synapse/pull/13205

As discussed at:

 - https://github.com/matrix-org/synapse/pull/13205#discussion_r918365746
 - https://github.com/matrix-org/synapse/pull/13205#discussion_r918366125

Related to https://github.com/matrix-org/synapse/issues/12584. This PR doesn't fix that issue because it hits [`get_event` which exists from the local database before it tries to `get_pdu`](7864f33e28/synapse/federation/federation_client.py (L581-L594)).
2022-07-20 15:58:51 -05:00
Shay a1b62af2af
Validate federation destinations and log an error if server name is invalid. (#13318) 2022-07-20 11:17:26 -07:00
David Robertson b977867358
Rate limit joins per-room (#13276) 2022-07-19 11:45:17 +00:00
Dirk Klimpel 96cf81e312
Use HTTPStatus constants in place of literals in tests. (#13297) 2022-07-15 19:31:27 +00:00
David Robertson 52a0c8f2f7
Rename test case method to `add_hashes_and_signatures_from_other_server` (#13255) 2022-07-12 18:46:32 +00:00