Commit Graph

731 Commits

Author SHA1 Message Date
Patrick Cloke 9ec3da06da
Bump mypy-zope & mypy. (#16188) 2023-08-29 10:38:56 -04:00
V02460 84f441f88f
Prepare unit tests for Python 3.12 (#16099) 2023-08-25 15:05:10 -04:00
Patrick Cloke a8a46b1336
Replace simple_async_mock with AsyncMock (#16180)
Python 3.8 has a native AsyncMock, use it instead of a custom
implementation.
2023-08-25 09:27:21 -04:00
Patrick Cloke daf11e26ef
Replace make_awaitable with AsyncMock (#16179)
Python 3.8 provides a native AsyncMock, we can replace the
homegrown version we have.
2023-08-24 19:38:46 -04:00
Will Hunt 0538e3e2db
Add `Retry-After` to M_LIMIT_EXCEEDED error responses (#16136)
Implements MSC4041 behind an experimental configuration flag.
2023-08-24 10:40:26 -04:00
Patrick Cloke ad3f43be9a
Run pyupgrade for python 3.7 & 3.8. (#16110) 2023-08-15 08:11:20 -04:00
Shay 0328b56468
Support MSC3814: Dehydrated Devices Part 2 (#16010) 2023-08-08 12:04:46 -07:00
Shay 84ae2e3f6f
Fix deletion for Dehydrated Devices (#16046) 2023-08-04 10:49:54 -07:00
Patrick Cloke 4f5bccbbba
Add forward-compatibility for the redacts property (MSC2174). (#16013)
The location of the redacts field changes in room version 11. Ensure
it is copied to the *new* location for *old* room versions for
forwards-compatibility with clients.

Note that copying it to the *old* location for the *new* room version
was previously handled.
2023-08-02 15:35:54 +00:00
Mohit Rathee 5eb3fd785b
Trim whitespace when setting display names (#16031) 2023-08-01 09:14:02 -04:00
Erik Johnston ae55cc1e6b
Add ability to wait for locks and add locks to purge history / room deletion (#15791)
c.f. #13476
2023-07-31 10:58:03 +01:00
Shay 641ff9ef7e
Support MSC3814: Dehydrated Devices (#15929)
Signed-off-by: Nicolas Werner <n.werner@famedly.com>
Co-authored-by: Nicolas Werner <n.werner@famedly.com>
Co-authored-by: Nicolas Werner <89468146+nico-famedly@users.noreply.github.com>
Co-authored-by: Hubert Chathi <hubert@uhoreg.ca>
2023-07-24 08:23:19 -07:00
Patrick Cloke 6d81aec09f
Support room version 11 (#15912)
And fix a bug in the implementation of the updated redaction
format (MSC2174) where the top-level redacts field was not
properly added for backwards-compatibility.
2023-07-18 08:44:59 -04:00
Eric Eastwood 1c802de626
Re-introduce the outbound federation proxy (#15913)
Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`).

This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.
2023-07-18 09:49:21 +01:00
Eric Eastwood c9bf644fa0
Revert "Federation outbound proxy" (#15910)
Revert "Federation outbound proxy (#15773)"

This reverts commit b07b14b494.
2023-07-10 11:10:20 -05:00
Eric Eastwood b07b14b494
Federation outbound proxy (#15773)
Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`).

This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.

The original code is from @erikjohnston's branches which I've gotten in-shape to merge.
2023-07-05 18:53:55 -05:00
Erik Johnston 25c55a9d22
Add login spam checker API (#15838) 2023-06-26 14:12:20 +00:00
Eric Eastwood 0f02f0b4da
Remove experimental MSC2716 implementation to incrementally import history into existing rooms (#15748)
Context for why we're removing the implementation:

 - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1487441010
 - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1504262734

Anyone wanting to continue MSC2716, should also address these leftover tasks: https://github.com/matrix-org/synapse/issues/10737

Closes https://github.com/matrix-org/synapse/issues/10737 in the fact that it is not longer necessary to track those things.
2023-06-16 14:12:24 -05:00
Mathieu Velten 0618bf94cd
push rules: fix internal conversion from _type to value (#15781)
Also fix wrong rule names for `is_user_mention` and `is_room_mention`.
2023-06-16 14:17:02 +02:00
Erik Johnston c485ed1c5a
Clear event caches when we purge history (#15609)
This should help a little with #13476

---------

Co-authored-by: Patrick Cloke <patrickc@matrix.org>
2023-06-08 13:14:40 +01:00
Grant McLean 5c24d7b9eb
Check required power levels earlier in createRoom handler. (#15695)
* Check required power levels earlier in createRoom handler.

- If a server was configured to reject the creation of rooms with E2EE
  enabled (by specifying an unattainably high power level for
  "m.room.encryption" in default_power_level_content_override), the 403
  error was not being triggered until after the room was created and
  before the "m.room.power_levels" was sent.  This allowed a user to
  access the partially-configured room and complete the setup of E2EE
  and power levels manually.

- This change causes the power level overrides to be checked earlier and
  the request to be rejected before the user gains access to the room.

- A new `_validate_room_config` method is added to contain checks that
  should be run before a room is created.

- The new test case confirms that a user request is rejected by the new
  validation method.

Signed-off-by: Grant McLean <grant@catalyst.net.nz>

* Add a changelog file.

* Formatting fix for black.

* Remove unneeded line from test.

---------

Signed-off-by: Grant McLean <grant@catalyst.net.nz>
2023-06-07 16:21:25 +01:00
Shay d0c4257f14
`N + 3`: Read from column `full_user_id` rather than `user_id` of tables `profiles` and `user_filters` (#15649) 2023-06-02 17:24:13 -07:00
Mathieu Velten e0f2429d13
Add a catch-all * to the supported relation types when redacting (#15705)
This is an update to MSC3912 implementation
2023-06-02 13:13:50 +00:00
Hugh Nimmo-Smith d1693f0362
Implement stable support for MSC3882 to allow an existing device/session to generate a login token for use on a new device/session (#15388)
Implements stable support for MSC3882; this involves updating Synapse's support to
match the MSC / the spec says.

Continue to support the unstable version to allow clients to transition.
2023-06-01 08:52:51 -04:00
Patrick Cloke 89a23c9406
Do not allow deactivated users to login with JWT. (#15624)
To improve the organization of this code it moves the JWT login
checks to a separate handler and then fixes the bug (and a
deprecation warning).
2023-05-19 08:06:54 -04:00
Nick Mills-Barrett ad50510a06
Handle missing previous read marker event. (#15464)
If the previous read marker is pointing to an event that no longer exists
(e.g. due to retention) then assume that the newly given read marker
is newer.
2023-05-18 14:37:31 -04:00
Jonathan de Jong e5b4d93770
Update Mutual Rooms (MSC2666) implementation (#15621)
To track changes in MSC2666:

- The change from `/mutual_rooms/{user_id}` to `/mutual_rooms?user_id={user_id}`.
- The addition of `next_batch_token` (and logic).
- Unstable flag now being `uk.half-shot.msc2666.query_mutual_rooms`.
- The error code when your own user is requested.
2023-05-18 12:49:12 -04:00
Andrew Morgan 2e59e97ebd
Move ThirdPartyEventRules into module_api/callbacks (#15535) 2023-05-04 14:18:22 +00:00
Patrick Cloke 07b1c70d6b
Initial implementation of MSC3981: recursive relations API (#15315)
Adds an optional keyword argument to the /relations API which
will recurse a limited number of event relationships.

This will cause the API to return not just the events related to the
parent event, but also events related to those related to the parent
event, etc.

This is disabled by default behind an experimental configuration
flag and is currently implemented using prefixed parameters.
2023-05-02 07:59:55 -04:00
Shay 301b4156d5
Add column `full_user_id` to tables `profiles` and `user_filters`. (#15458) 2023-04-26 16:03:26 -07:00
Andrew Morgan aec639e3e3
Move Spam Checker callbacks to a dedicated file (#15453) 2023-04-18 00:57:40 +00:00
Patrick Cloke 2503126d52
Implement MSC2174: move redacts to a content property. (#15395)
This moves `redacts` from being a top-level property to
a `content` property in a new room version.

MSC2176 (which was previously implemented) states to not
`redact` this property.
2023-04-13 13:47:07 +00:00
Shay 6b23d74ad1
Delete server-side backup keys when deactivating an account. (#15181) 2023-04-04 20:16:08 +00:00
Erik Johnston 6204c3663e
Revert pruning of old devices (#15360)
* Revert "Fix registering a device on an account with lots of devices (#15348)"

This reverts commit f0d8f66eaa.

* Revert "Delete stale non-e2e devices for users, take 3 (#15183)"

This reverts commit 78cdb72cd6.
2023-03-31 13:51:51 +01:00
Erik Johnston f0d8f66eaa
Fix registering a device on an account with lots of devices (#15348)
Fixes up #15183
2023-03-29 13:37:06 +00:00
Andrew Morgan ec9224bf9a
Make `POST /_matrix/client/v3/rooms/{roomId}/report/{eventId}` endpoint return 404 if event exists, but the user lacks access (#15300) 2023-03-21 13:24:03 +00:00
Andrew Morgan b6aef59334
Make `EventHandler.get_event` return `None` when the requested event is not found (#15298) 2023-03-21 13:23:47 +00:00
Andrew Morgan 4953cd71df
Move Account Validity callbacks to a dedicated file (#15237) 2023-03-16 10:35:31 +00:00
David Robertson a1c9869394
Merge branch 'release-v1.79' into develop 2023-03-13 18:35:21 +00:00
Andrew Morgan e157c63f68
Fix missing conditional for registering `on_remove_user_third_party_identifier` module api callbacks (#15227 2023-03-10 10:35:18 +00:00
Quentin Gliech 47bc84dd53
Pass the Requester down to the HttpTransactionCache. (#15200) 2023-03-07 16:05:22 +00:00
Patrick Cloke 05e0a4089a
Stop applying edits to event contents (MSC3925). (#15193)
Enables MSC3925 support by default, which:

* Includes the full edit event in the bundled aggregations of an
  edited event.
* Stops modifying the original event's content to return the new
  content from the edit event.

This is a backwards-incompatible change that is considered to be
"correct" by the spec.
2023-03-06 09:43:01 -05:00
Hugh Nimmo-Smith 916b8061d2
Implementation of MSC3967: Don't require UIA for initial upload of cross signing keys (#15077) 2023-03-02 10:34:59 +00:00
Richard van der Hoff 2b78981736
Remove support for aggregating reactions (#15172)
It turns out that no clients rely on server-side aggregation of `m.annotation`
relationships: it's just not very useful as currently implemented.

It's also non-trivial to calculate.

I want to remove it from MSC2677, so to keep the implementation in line, let's
remove it here.
2023-02-28 18:49:28 +00:00
Andrew Morgan b40657314e
Add module API callbacks for adding and deleting local 3PID associations (#15044 2023-02-27 14:19:19 +00:00
Shay 1c95ddd09b
Batch up storing state groups when creating new room (#14918) 2023-02-24 13:15:29 -08:00
dependabot[bot] 9bb2eac719
Bump black from 22.12.0 to 23.1.0 (#15103) 2023-02-22 15:29:09 -05:00
Patrick Cloke c9b9143655
Fix-up type hints in tests/server.py. (#15084)
This file was being ignored by mypy, we remove that
and add the missing type hints & deal with any fallout.
2023-02-17 18:19:38 +00:00
David Robertson ffc2ee521d
Use mypy 1.0 (#15052)
* Update mypy and mypy-zope
* Remove unused ignores

These used to suppress

```
synapse/storage/engines/__init__.py:28: error: "__new__" must return a
class instance (got "NoReturn")  [misc]
```

and

```
synapse/http/matrixfederationclient.py:1270: error: "BaseException" has no attribute "reasons"  [attr-defined]
```

(note that we check `hasattr(e, "reasons")` above)

* Avoid empty body warnings, sometimes by marking methods as abstract

E.g.

```
tests/handlers/test_register.py:58: error: Missing return statement  [empty-body]
tests/handlers/test_register.py:108: error: Missing return statement  [empty-body]
```

* Suppress false positive about `JaegerConfig`

Complaint was

```
synapse/logging/opentracing.py:450: error: Function "Type[Config]" could always be true in boolean context  [truthy-function]
```

* Fix not calling `is_state()`

Oops!

```
tests/rest/client/test_third_party_rules.py:428: error: Function "Callable[[], bool]" could always be true in boolean context  [truthy-function]
```

* Suppress false positives from ParamSpecs

````
synapse/logging/opentracing.py:971: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]"  [arg-type]
synapse/logging/opentracing.py:1017: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]"  [arg-type]
````

* Drive-by improvement to `wrapping_logic` annotation

* Workaround false "unreachable" positives

See https://github.com/Shoobx/mypy-zope/issues/91

```
tests/http/test_proxyagent.py:626: error: Statement is unreachable  [unreachable]
tests/http/test_proxyagent.py:762: error: Statement is unreachable  [unreachable]
tests/http/test_proxyagent.py:826: error: Statement is unreachable  [unreachable]
tests/http/test_proxyagent.py:838: error: Statement is unreachable  [unreachable]
tests/http/test_proxyagent.py:845: error: Statement is unreachable  [unreachable]
tests/http/federation/test_matrix_federation_agent.py:151: error: Statement is unreachable  [unreachable]
tests/http/federation/test_matrix_federation_agent.py:452: error: Statement is unreachable  [unreachable]
tests/logging/test_remote_handler.py:60: error: Statement is unreachable  [unreachable]
tests/logging/test_remote_handler.py:93: error: Statement is unreachable  [unreachable]
tests/logging/test_remote_handler.py:127: error: Statement is unreachable  [unreachable]
tests/logging/test_remote_handler.py:152: error: Statement is unreachable  [unreachable]
```

* Changelog

* Tweak DBAPI2 Protocol to be accepted by mypy 1.0

Some extra context in:
- https://github.com/matrix-org/python-canonicaljson/pull/57
- https://github.com/python/mypy/issues/6002
- https://mypy.readthedocs.io/en/latest/common_issues.html#covariant-subtyping-of-mutable-protocol-members-is-rejected

* Pull in updated canonicaljson lib

so the protocol check just works

* Improve comments in opentracing

I tried to workaround the ignores but found it too much trouble.

I think the corresponding issue is
https://github.com/python/mypy/issues/12909. The mypy repo has a PR
claiming to fix this (https://github.com/python/mypy/pull/14677) which
might mean this gets resolved soon?

* Better annotation for INTERACTIVE_AUTH_CHECKERS

* Drive-by AUTH_TYPE annotation, to remove an ignore
2023-02-16 16:09:11 +00:00
Patrick Cloke 42aea0d8af
Add final type hint to tests.unittest. (#15072)
Adds a return type to HomeServerTestCase.make_homeserver and deal
with any variables which are no longer Any.
2023-02-14 14:03:35 -05:00