Commit Graph

20545 Commits

Author SHA1 Message Date
Patrick Cloke 9535fd0f9c
Disable groups/communities by default. (#12344)
This disables the endpoints (and sync response fields) for
groups/communities by default.
2022-04-12 10:20:46 -04:00
Erik Johnston 9b1f360091 Update changelog 2022-04-12 15:07:08 +01:00
Richard van der Hoff 643c0c50c1
Update UPGRADE notes to include more details about v1.57 upgrade failure mode (#12448) 2022-04-12 14:56:05 +01:00
Richard van der Hoff 320186319a
Resync state after partial-state join (#12394)
We work through all the events with partial state, updating the state at each
of them. Once it's done, we recalculate the state for the whole room, and then
mark the room as having complete state.
2022-04-12 13:23:43 +00:00
Erik Johnston e31d06f6f0 Split changelogs by year 2022-04-12 13:54:36 +01:00
Patrick Cloke 86cf6a3a17
Remove references to unstable identifiers from MSC3440. (#12382)
Removes references to unstable thread relation, unstable
identifiers for filtering parameters, and the experimental
config flag.
2022-04-12 08:42:03 -04:00
Erik Johnston 3810730ba5 Don't line wrap 2022-04-12 13:41:58 +01:00
Erik Johnston 641f43ba81 1.57.0rc1 2022-04-12 13:39:35 +01:00
Dirk Klimpel 1783156dbc
Add some type hints to datastore (#12423)
* Add some type hints to datastore

* newsfile

* change `Collection` to `List`

* refactor return type of `select_users_txn`

* correct type hint in `stream.py`

* Remove `Optional` in `select_users_txn`

* remove not needed return type in `__init__`

* Revert change in `get_stream_id_for_event_txn`

* Remove import from `Literal`
2022-04-12 11:54:00 +01:00
David Robertson 4e13743738
Poetry: select olddeps using `poetry` (#12407)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-04-12 10:50:11 +00:00
Erik Johnston 3ad74b63e5
Specify tls extra for Twisted dependency. (#12444)
* Specify `tls` extra for Twisted dependency.

It was already pulled in for us by `treq`, but we should be explicit
that we do use the `tls` functionality of Twisted directly.

* Mark `idna` as dev-dependency

This doesn't actually change anything, as `Twisted[tls]` will put it in
as a main dependency anyway.
2022-04-12 10:49:02 +00:00
David Robertson 5f8173dd80
Workaround pip bug installing latest treq and not-latest twisted from hashes (#12439)
The requirements file generated by `poetry export` isn't correctly processed by `pip install -r requirements.txt`. It contains twisted and treq, both pinned to 22.2.0.

When `pip` installs treq, it notices that `Twisted[tls]` is required. It then tries to acquire the latest twisted release, only to fail (because this hash isn't listed in the requirements file).From e.g. https://github.com/matrix-org/synapse/runs/5977154990?check_suite_focus=true

> ```
> #15 9.204 Collecting Twisted[tls]>=18.7.0
> #15 9.205 ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
> #15 9.205     Twisted[tls]>=18.7.0 from 38622ff95be907db1987c4d92edd09/Twisted-22.4.0-py3-none-any.whl#sha256=f9f7a91f94932477a9fc3b169d57f54f96c6e74a23d78d9ce54039a7f48928a2 (from treq==22.2.0->-r /synapse/requirements.txt (line 724))
> #15 ERROR: executor failed running [/bin/sh -c pip install --prefix="/install" --no-warn-script-location -r /synapse/requirements.txt]: exit code: 1
> ```

The underlying pip issue is https://github.com/pypa/pip/issues/9644. A comment notes that one can avoid this behaviour with by `pip install`ing with the `--no-deps` flag. Let us do so.

(At first glance, the problem looks like https://github.com/python-poetry/poetry/issues/5311, but that was a bug in `poetry install`; this is `poetry export`, whose behaviour is fine AFAICS).
2022-04-12 10:16:01 +01:00
Patrick Cloke ab3165efb7
Remove experimental configuration flag for MSC3666. (#12436) 2022-04-11 15:37:46 -04:00
Patrick Cloke 4586119f0b
Add missing type hints to config classes. (#12402) 2022-04-11 12:07:23 -04:00
Richard van der Hoff 214f3b7d21
Enable certificate checking during complement tests (#12435) 2022-04-11 16:35:41 +01:00
Patrick Cloke 772bad2562
Do not consider events by ignored users for bundled aggregations (#12235)
Consider the requester's ignored users when calculating the
bundled aggregations.

See #12285 / 4df10d3214
for corresponding changes for the `/relations` endpoint.
2022-04-11 10:09:57 -04:00
Richard van der Hoff 3cdf5a1386
Fix up healthcheck generation for workers docker image (#12405)
This wasn't quite generating the right thing.
2022-04-11 13:38:58 +00:00
David Robertson 961ee75a9b
Disallow untyped defs in synapse._scripts (#12422)
Of note: 

* No untyped defs in `register_new_matrix_user`

This one might be contraversial. `request_registration` has three
dependency-injection arguments used for testing. I'm removing the
injection of the `requests` module and using `unitest.mock.patch` in the
test cases instead.

Doing `reveal_type(requests)` and `reveal_type(requests.get)` before the
change:

```
synapse/_scripts/register_new_matrix_user.py:45: note: Revealed type is "Any"
synapse/_scripts/register_new_matrix_user.py:46: note: Revealed type is "Any"
```

And after:

```
synapse/_scripts/register_new_matrix_user.py:44: note: Revealed type is "types.ModuleType"
synapse/_scripts/register_new_matrix_user.py:45: note: Revealed type is "def (url: Union[builtins.str, builtins.bytes], params: Union[Union[_typeshed.SupportsItems[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]], Tuple[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]], typing.Iterable[Tuple[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]]], builtins.str, builtins.bytes], None] =, data: Union[Any, None] =, headers: Union[Any, None] =, cookies: Union[Any, None] =, files: Union[Any, None] =, auth: Union[Any, None] =, timeout: Union[Any, None] =, allow_redirects: builtins.bool =, proxies: Union[Any, None] =, hooks: Union[Any, None] =, stream: Union[Any, None] =, verify: Union[Any, None] =, cert: Union[Any, None] =, json: Union[Any, None] =) -> requests.models.Response"
```

* Drive-by comment in `synapse.storage.types`

* No untyped defs in `synapse_port_db`

This was by far the most painful. I'm happy to break this up into
smaller pieces for review if it's not managable as-is.
2022-04-11 12:41:55 +01:00
Richard van der Hoff 5f72ea1bde
Move complement setup stuff into the Synapse repo (#12404)
Fixes matrix-org/complement#330 (or it will, once we remove the old files).

It's not quite a lift-and-shift: I've also taken the opportunity to get rid of the custom CA that we used to use to sign the TLS certs, which has been superceded by the CA exposed by Complement.
2022-04-11 11:39:28 +01:00
reivilibre 85ca963c1a
Add Module API for reading and writing global account data. (#12391) 2022-04-11 10:05:43 +01:00
David Robertson 98ec375b26
CI: Fix the export-data job to run under poetry (#12418)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-04-08 19:18:54 +00:00
reivilibre e630722f11
Optimise `_update_client_ips_batch_txn` to batch together database operations. (#12252)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-04-08 15:29:13 +01:00
David Robertson 0cd182f296
Make `synapse._scripts` pass typechecks (#12421) 2022-04-08 14:00:12 +00:00
David Robertson dd5cc37aa4
Stop maintaining a list of lint targets in `lint.sh` (#12420) 2022-04-08 14:36:09 +01:00
David Robertson 95a038c106
Unify HTTP query parameter type hints (#12415)
* Pull out query param types to `synapse.http.types`
* Use QueryParams everywhere
* Simplify `encode_query_args`
* Add annotation which would have caught #12410
2022-04-08 13:06:51 +01:00
Jorge Florian 2e2d8cc2f9
Update the server notices user profile in room if changed. (#12115) 2022-04-08 07:51:27 -04:00
David Robertson 7851a2c62f
CI: Get the portdb job to run under poetry (#12412) 2022-04-08 12:00:05 +01:00
Jorge Florian 78e4d96a4d
Add missing type definitions for scripts in docker folder (#12280)
Signed-off-by: Jorge Florian <jafn28@gmail.com>
2022-04-08 10:10:58 +00:00
Erik Johnston 7732c4902c
Fix rejecting invites over federation (#12409)
Currently causes future incremental syncs to fail.

Broke by #12191
2022-04-07 15:54:16 +01:00
Erik Johnston 36af768c13
Fix fetching public rooms over federation (#12410)
Broke by #12364
2022-04-07 14:18:02 +00:00
Patrick Cloke 1a90c1e3af
Do not add groups to sync results if disabled. (#12408) 2022-04-07 09:29:56 -04:00
Erik Johnston d1cd96ce29
Add opentracing spans to calls to external cache (#12380) 2022-04-07 13:18:29 +01:00
David Robertson 3a7e97c7ad
Poetry: use locked environment in Docker images (#12385) 2022-04-07 11:43:31 +00:00
Patrick Cloke 0bcb651b3f
Support the v1 endpoint for `/relations`. (#12403)
Now that MSC2675 has passed FCP and the implementation is
compliant with the final version.
2022-04-07 07:08:23 -04:00
Eric Eastwood 350062661c
Clarify that we mark as outliers because we don't have any state for them (#12345)
As discussed at https://github.com/matrix-org/synapse/pull/12179#discussion_r837263852
2022-04-06 16:34:33 -05:00
Eric Eastwood f931c0602a
Update changelog to describe changes from #12370 relative to `develop` (#12401) 2022-04-06 14:41:15 -04:00
Patrick Cloke 6902e9ff2b
Remove docs for converting a legacy structured logging config. (#12392)
And update the upgrade notes to point at old versions of the documentation.
2022-04-06 13:41:14 -04:00
Patrick Cloke 05a37f4008
Remove support for the unstable identifier from MSC3288. (#12398) 2022-04-06 13:27:46 -04:00
David Robertson 2cf74cf2fc
Narrow the `importlib_metadata` dependency (#12400) 2022-04-06 17:37:56 +01:00
Richard van der Hoff 6fe757d69e
Fix `synapse_event_persisted_position` metric (#12390)
Fixes a bug introduced in #11417 where we would only included backfilled events
in `synapse_event_persisted_position`
2022-04-06 13:52:39 +00:00
Richard van der Hoff ae01a7edd3
Update type annotations for compatiblity with prometheus_client 0.14 (#12389)
Principally, `prometheus_client.REGISTRY.register` now requires its argument to
extend `prometheus_client.Collector`.

Additionally, `Gauge.set` is now annotated so that passing `Optional[int]`
causes an error.
2022-04-06 12:59:04 +00:00
Eric Eastwood 793d03e2c5
Generate historic pagination token for `/messages` when no `?from` token provided (#12370) 2022-04-06 11:40:28 +01:00
David Robertson 573cd0f92f
Add missing dependency on importlib_metadata (#12384) 2022-04-05 18:25:56 +00:00
David Robertson 7ec9b06303
Poetry: initial pyproject.toml and poetry.lock (#12381)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-04-05 18:40:56 +01:00
Sean Quah fd1e7d0fc2
Post 1.56.0 release tidy up (#12379) 2022-04-05 17:35:20 +01:00
Andrew Morgan 163fd686b5
Update mdbook to 0.4.17 (#12339)
Update mdbook (the tool used to render the documentation website) to version 0.4.17.
2022-04-05 17:21:58 +01:00
Sean Quah 79e7c2c426
Fix edge case where a `Linearizer` could get stuck (#12358)
Just after a task acquires a contended `Linearizer` lock, it sleeps.
If the task is cancelled during this sleep, we need to release the lock.

Signed-off-by: Sean Quah <seanq@element.io>
2022-04-05 17:19:16 +01:00
Sean Quah 31c1209c50
Make `StreamToken` and `RoomStreamToken` methods propagate cancellations (#12366)
`StreamToken.from_string` and `RoomStreamToken.parse` are both async
methods that could be cancelled. These methods must not replace
`CancelledError`s with `SynapseError`s.

Signed-off-by: Sean Quah <seanq@element.io>
2022-04-05 16:56:52 +01:00
Sean Quah 9c4c49991d
Update docstrings for `ReadWriteLock` tests (#12354)
Signed-off-by: Sean Quah <seanq@element.io>
2022-04-05 16:54:40 +01:00
Sean Quah 800ba87cc8
Refactor and convert `Linearizer` to async (#12357)
Refactor and convert `Linearizer` to async. This makes a `Linearizer`
cancellation bug easier to fix.

Also refactor to use an async context manager, which eliminates an
unlikely footgun where code that doesn't immediately use the context
manager could forget to release the lock.

Signed-off-by: Sean Quah <seanq@element.io>
2022-04-05 15:43:52 +01:00