Commit Graph

14380 Commits

Author SHA1 Message Date
David Robertson 6463244375
Remove unused `# type: ignore`s (#12531)
Over time we've begun to use newer versions of mypy, typeshed, stub
packages---and of course we've improved our own annotations. This makes
some type ignore comments no longer necessary. I have removed them.

There was one exception: a module that imports `select.epoll`. The
ignore is redundant on Linux, but I've kept it ignored for those of us
who work on the source tree using not-Linux. (#11771)

I'm more interested in the config line which enforces this. I want
unused ignores to be reported, because I think it's useful feedback when
annotating to know when you've fixed a problem you had to previously
ignore.

* Installing extras before typechecking

Lacking an easy way to install all extras generically, let's bite the bullet and
make install the hand-maintained `all` extra before typechecking.

Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to
the release/v1 branch.
2022-04-27 14:03:44 +01:00
Patrick Cloke 8a23bde823
Consistently use collections.abc.Mapping to check frozendict. (#12564) 2022-04-27 09:00:07 -04:00
Will Hunt e8d1ec0e92
Add option to enable token registration without requiring 3pids (#12526) 2022-04-27 12:57:53 +00:00
Dirk Klimpel b76f1a4d5f
Add some type hints to datastore (#12485) 2022-04-27 13:05:00 +01:00
Nick Mills-Barrett 63ba9ba38b
Bound ephemeral events by key (#12544)
Co-authored-by: Brad Murray <bradtgmurray@gmail.com>
Co-authored-by: Andrew Morgan <andrewm@element.io>
2022-04-26 20:14:21 +01:00
Erik Johnston f59e3f4c90
Mark remote device list updates as already handled (#12557) 2022-04-26 17:07:21 +01:00
David Robertson 6d89f1239c
Comment out dodgy log-kv (#12554) 2022-04-26 15:53:06 +01:00
Erik Johnston c48ab3734e
Fix sending opentracing contexts to remote servers (#12555) 2022-04-26 14:48:16 +00:00
Richard van der Hoff 17d99f758a
Optimise backfill calculation (#12522)
Try to avoid an OOM by checking fewer extremities.

Generally this is a big rewrite of _maybe_backfill, to try and fix some of the TODOs and other problems in it. It's best reviewed commit-by-commit.
2022-04-26 10:27:11 +01:00
Sean Quah 8a87b4435a
Handle cancellation in `EventsWorkerStore._get_events_from_cache_or_db` (#12529)
Multiple calls to `EventsWorkerStore._get_events_from_cache_or_db` can
reuse the same database fetch, which is initiated by the first call.
Ensure that cancelling the first call doesn't cancel the other calls
sharing the same database fetch.

Signed-off-by: Sean Quah <seanq@element.io>
2022-04-25 19:39:17 +01:00
Patrick Cloke 185da8f0f2
Misc. clean-ups to the relations code (#12519)
* Corrects some typos / copy & paste errors in tests.
* Clarifies docstrings.
* Removes an unnecessary method.
2022-04-25 08:25:56 -04:00
Sean Quah a50fb411b3
Update `delay_cancellation` to accept any awaitable (#12468)
This will mainly be useful when dealing with module callbacks, which are
all typed as returning `Awaitable`s instead of coroutines or
`Deferred`s.

Signed-off-by: Sean Quah <seanq@element.io>
2022-04-22 18:20:06 +01:00
Will Hunt b82fff66df
MSC3202: Fix device_unused_fallback_keys -> device_unused_fallback_key_types (#12520)
* Fix device_unused_fallback_keys -> device_unused_fallback_key_types

* changelog
2022-04-22 16:03:46 +01:00
Richard van der Hoff f5668f0b4a
Await un-partial-stating after a partial-state join (#12399)
When we join a room via the faster-joins mechanism, we end up with "partial
state" at some points on the event DAG. Many parts of the codebase need to
wait for the full state to load. So, we implement a mechanism to keep track of
which events have partial state, and wait for them to be fully-populated.
2022-04-21 07:42:03 +01:00
David Robertson e5a76ec00b
Dump setuptools; correct pyproject version number (#12478) 2022-04-20 17:33:20 +01:00
Patrick Cloke 103f51d867
Fix Jinja templating error when generating thumbnail URLs. (#12510)
scale is meant to be a constant string, not refer to a variable.
2022-04-20 12:03:03 -04:00
Tulir Asokan 4bc8cb4669
Implement MSC2815: allow room moderators to view redacted event content (#12427)
Implements matrix-org/matrix-spec-proposals#2815

Signed-off-by: Tulir Asokan <tulir@maunium.net>
2022-04-20 12:57:39 +01:00
Erik Johnston c1482a352a
Fix returned count of delete extremities admin API (#12496) 2022-04-19 16:49:45 +01:00
Richard van der Hoff b80bb7e452
Fix `/room/.../event/...` to return the *original* event after any edits (#12476)
This is what the MSC (now) requires. Fixes https://github.com/matrix-org/synapse/issues/10310.
2022-04-19 16:42:19 +01:00
Jan Christian Grünhage a1f87f57ff
Implement MSC3383: include destination in X-Matrix auth header (#11398)
Co-authored-by: Jan Christian Grünhage <jan.christian@gruenhage.xyz>
Co-authored-by: Marcus Hoffmann <bubu@bubu1.eu>
2022-04-19 16:23:53 +01:00
Olivier Wilkinson (reivilibre) 0921d93dcd Merge branch 'master' into develop 2022-04-19 13:28:28 +01:00
Richard van der Hoff b121a3ad2b
Back out implementation of MSC2314 (#12474)
MSC2314 has now been closed, so we're backing out its implementation, which
originally happened in #6176.

Unfortunately it's not a direct revert, as that PR mixed in a bunch of
unrelated changes to tests etc.
2022-04-19 11:17:29 +00:00
Olivier Wilkinson (reivilibre) 88fe72cc1e 1.57.0 2022-04-19 11:00:37 +01:00
Travis Ralston f8d3ee9570
Fix grammatical error in error message (#12483)
* Fix grammatical error in error message

* changelog
2022-04-18 12:41:55 -06:00
Erik Johnston 0b014eb25e
Only send out device list updates for our own users (#12465)
Broke in #12365
2022-04-14 13:05:31 +01:00
Shay 8e2759f2d8
Limit `device_id` size to 512B (#12454)
*
2022-04-13 10:04:01 -07:00
David Baker 73d8ded0b0
Prevent a sync request from removing a user's busy presence status (#12213)
In trying to use the MSC3026 busy presence status, the user's status
would be set back to 'online' next time they synced. This change makes
it so that syncing does not affect a user's presence status if it
is currently set to 'busy': it must be removed through the presence
API.

The MSC defers to implementations on the behaviour of busy presence,
so this ought to remain compatible with the MSC.
2022-04-13 16:21:07 +01:00
Nick Mills-Barrett e3a49f4784
Fix missing sync events during historical batch imports (#12319)
Discovered after much in-depth investigation in #12281.

Closes: #12281
Closes: #3305

Signed off by: Nick Mills-Barrett nick@beeper.com
2022-04-13 11:38:35 +01:00
Jonathan de Jong 36d8b83888
Rename Mutual Rooms `unstable_features` flag to match MSC (#12445)
Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2022-04-13 10:32:44 +01:00
Erik Johnston aa28110264
Process device list updates asynchronously (#12365) 2022-04-12 16:50:40 +01:00
Patrick Cloke 4bdbebccb9
Remove the unstable event field for `/send_join` per MSC3083. (#12395)
This was missed when initially stabilising room version 8 and was
left in as a compatibility shim. Most homeservers have upgraded
to a version which expects the proper field name, and the failure
mode is reasonable (a user on an older server may have to attempt
joining the room twice with an obscure error message the first time).
2022-04-12 11:27:45 -04:00
Erik Johnston a468768104 Merge branch 'release-v1.57' into develop 2022-04-12 15:28:41 +01:00
Patrick Cloke 9535fd0f9c
Disable groups/communities by default. (#12344)
This disables the endpoints (and sync response fields) for
groups/communities by default.
2022-04-12 10:20:46 -04:00
Richard van der Hoff 320186319a
Resync state after partial-state join (#12394)
We work through all the events with partial state, updating the state at each
of them. Once it's done, we recalculate the state for the whole room, and then
mark the room as having complete state.
2022-04-12 13:23:43 +00:00
Patrick Cloke 86cf6a3a17
Remove references to unstable identifiers from MSC3440. (#12382)
Removes references to unstable thread relation, unstable
identifiers for filtering parameters, and the experimental
config flag.
2022-04-12 08:42:03 -04:00
Erik Johnston 641f43ba81 1.57.0rc1 2022-04-12 13:39:35 +01:00
Dirk Klimpel 1783156dbc
Add some type hints to datastore (#12423)
* Add some type hints to datastore

* newsfile

* change `Collection` to `List`

* refactor return type of `select_users_txn`

* correct type hint in `stream.py`

* Remove `Optional` in `select_users_txn`

* remove not needed return type in `__init__`

* Revert change in `get_stream_id_for_event_txn`

* Remove import from `Literal`
2022-04-12 11:54:00 +01:00
Erik Johnston 3ad74b63e5
Specify tls extra for Twisted dependency. (#12444)
* Specify `tls` extra for Twisted dependency.

It was already pulled in for us by `treq`, but we should be explicit
that we do use the `tls` functionality of Twisted directly.

* Mark `idna` as dev-dependency

This doesn't actually change anything, as `Twisted[tls]` will put it in
as a main dependency anyway.
2022-04-12 10:49:02 +00:00
Patrick Cloke ab3165efb7
Remove experimental configuration flag for MSC3666. (#12436) 2022-04-11 15:37:46 -04:00
Patrick Cloke 4586119f0b
Add missing type hints to config classes. (#12402) 2022-04-11 12:07:23 -04:00
Patrick Cloke 772bad2562
Do not consider events by ignored users for bundled aggregations (#12235)
Consider the requester's ignored users when calculating the
bundled aggregations.

See #12285 / 4df10d3214
for corresponding changes for the `/relations` endpoint.
2022-04-11 10:09:57 -04:00
David Robertson 961ee75a9b
Disallow untyped defs in synapse._scripts (#12422)
Of note: 

* No untyped defs in `register_new_matrix_user`

This one might be contraversial. `request_registration` has three
dependency-injection arguments used for testing. I'm removing the
injection of the `requests` module and using `unitest.mock.patch` in the
test cases instead.

Doing `reveal_type(requests)` and `reveal_type(requests.get)` before the
change:

```
synapse/_scripts/register_new_matrix_user.py:45: note: Revealed type is "Any"
synapse/_scripts/register_new_matrix_user.py:46: note: Revealed type is "Any"
```

And after:

```
synapse/_scripts/register_new_matrix_user.py:44: note: Revealed type is "types.ModuleType"
synapse/_scripts/register_new_matrix_user.py:45: note: Revealed type is "def (url: Union[builtins.str, builtins.bytes], params: Union[Union[_typeshed.SupportsItems[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]], Tuple[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]], typing.Iterable[Tuple[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]]], builtins.str, builtins.bytes], None] =, data: Union[Any, None] =, headers: Union[Any, None] =, cookies: Union[Any, None] =, files: Union[Any, None] =, auth: Union[Any, None] =, timeout: Union[Any, None] =, allow_redirects: builtins.bool =, proxies: Union[Any, None] =, hooks: Union[Any, None] =, stream: Union[Any, None] =, verify: Union[Any, None] =, cert: Union[Any, None] =, json: Union[Any, None] =) -> requests.models.Response"
```

* Drive-by comment in `synapse.storage.types`

* No untyped defs in `synapse_port_db`

This was by far the most painful. I'm happy to break this up into
smaller pieces for review if it's not managable as-is.
2022-04-11 12:41:55 +01:00
reivilibre 85ca963c1a
Add Module API for reading and writing global account data. (#12391) 2022-04-11 10:05:43 +01:00
reivilibre e630722f11
Optimise `_update_client_ips_batch_txn` to batch together database operations. (#12252)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-04-08 15:29:13 +01:00
David Robertson 0cd182f296
Make `synapse._scripts` pass typechecks (#12421) 2022-04-08 14:00:12 +00:00
David Robertson 95a038c106
Unify HTTP query parameter type hints (#12415)
* Pull out query param types to `synapse.http.types`
* Use QueryParams everywhere
* Simplify `encode_query_args`
* Add annotation which would have caught #12410
2022-04-08 13:06:51 +01:00
Jorge Florian 2e2d8cc2f9
Update the server notices user profile in room if changed. (#12115) 2022-04-08 07:51:27 -04:00
Erik Johnston 7732c4902c
Fix rejecting invites over federation (#12409)
Currently causes future incremental syncs to fail.

Broke by #12191
2022-04-07 15:54:16 +01:00
Erik Johnston 36af768c13
Fix fetching public rooms over federation (#12410)
Broke by #12364
2022-04-07 14:18:02 +00:00
Patrick Cloke 1a90c1e3af
Do not add groups to sync results if disabled. (#12408) 2022-04-07 09:29:56 -04:00