5dcacdf6d1
Add support for ubuntu 21.10 "Impish Indri" ( #11024 )
...
* support ubuntu 21.10 indri
* add changelog
* update to correct codename
Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
2021-10-12 11:29:39 -07:00
9abc5f2a05
Merge remote-tracking branch 'origin/release-v1.45' into develop
2021-10-12 14:21:05 -04:00
84f5d83257
Add tests for `MediaFilePaths` ( #11057 )
2021-10-12 18:19:35 +01:00
8eaffe013c
Update `_wrap_in_base_path` type hints to preserve function arguments ( #11055 )
2021-10-12 18:19:21 +01:00
1db9282dfa
Fix formatting string when oEmbed errors occur. ( #11061 )
2021-10-12 17:15:42 +00:00
77ea03086c
Merge remote-tracking branch 'origin/release-v1.45' into develop
2021-10-12 14:41:13 +01:00
333d6f4e84
Fix race in `MultiWriterIdGenerator` ( #11045 )
...
The race allowed the current position to advance too far when stream IDs
are still being persisted.
This happened when it received a new stream ID from a remote write
between a new stream ID being allocated and it being added to the set of
unpersisted stream IDs.
Fixes #9424 .
2021-10-12 14:27:09 +01:00
5c35074d85
Reset global cache state before cache tests. ( #11036 )
...
This reverts #11019 and structures the code a bit more like it was before #10985 .
The global cache state must be reset before running the tests since other test
cases might have configured caching (and thus touched the global state).
2021-10-12 12:55:33 +00:00
36224e056a
Add type hints to `synapse.storage.databases.main.client_ips` ( #10972 )
2021-10-12 13:50:34 +01:00
a18c568516
Synapse 1.45.0rc1 (2021-10-12)
...
==============================
**Note:** We are aware of [a performance issue](https://github.com/matrix-org/synapse/issues/11049 ) introduced in Synapse v1.44.0, as well as [a bug](https://github.com/matrix-org/synapse/issues/11025 ) with the user directory when using application services. While this release candidate doesn't fix either of those issues, a second release candidate is expected to come out in a few days to address them.
Media storage providers module that read from Synapse's configuration need changes as of this version, see the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1450 ) for more information.
Features
--------
- Add [MSC3069](https://github.com/matrix-org/matrix-doc/pull/3069 ) support to `/account/whoami`. ([\#9655](https://github.com/matrix-org/synapse/issues/9655 ))
- Support autodiscovery of oEmbed previews. ([\#10822](https://github.com/matrix-org/synapse/issues/10822 ))
- Add a `user_may_send_3pid_invite` spam checker callback for modules to allow or deny 3PID invites. ([\#10894](https://github.com/matrix-org/synapse/issues/10894 ))
- Add a spam checker callback to allow or deny room joins. ([\#10910](https://github.com/matrix-org/synapse/issues/10910 ))
- Include an `update_synapse_database` script in the distribution. Contributed by @Fizzadar at Beeper. ([\#10954](https://github.com/matrix-org/synapse/issues/10954 ))
- Include exception information in JSON logging output. Contributed by @Fizzadar at Beeper. ([\#11028](https://github.com/matrix-org/synapse/issues/11028 ))
Bugfixes
--------
- Fix a minor bug in the response to `/_matrix/client/r0/voip/turnServer`. Contributed by @lukaslihotzki. ([\#10922](https://github.com/matrix-org/synapse/issues/10922 ))
- Fix a bug where empty `yyyy-mm-dd/` directories would be left behind in the media store's `url_cache_thumbnails/` directory. ([\#10924](https://github.com/matrix-org/synapse/issues/10924 ))
- Fix a bug introduced in Synapse v1.40.0 where the signature checks for room version 8 and 9 could be applied to earlier room versions in some situations. ([\#10927](https://github.com/matrix-org/synapse/issues/10927 ))
- Fix a long-standing bug wherein deactivated users still count towards the monthly active users limit. ([\#10947](https://github.com/matrix-org/synapse/issues/10947 ))
- Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state. ([\#10956](https://github.com/matrix-org/synapse/issues/10956 ))
- Fix a long-standing bug where rebuilding the user directory wouldn't exclude support and deactivated users. ([\#10960](https://github.com/matrix-org/synapse/issues/10960 ))
- Fix [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) `/batch_send` endpoint rejecting subsequent batches with unknown batch ID error in existing room versions from the room creator. ([\#10962](https://github.com/matrix-org/synapse/issues/10962 ))
- Fix a bug that could leak local users' per-room nicknames and avatars when the user directory is rebuilt. ([\#10981](https://github.com/matrix-org/synapse/issues/10981 ))
- Fix a long-standing bug where the remainder of a batch of user directory changes would be silently dropped if the server left a room early in the batch. ([\#10982](https://github.com/matrix-org/synapse/issues/10982 ))
- Correct a bugfix introduced in Synapse v1.44.0 that would catch the wrong error if a connection is lost before a response could be written to it. ([\#10995](https://github.com/matrix-org/synapse/issues/10995 ))
- Fix a long-standing bug where local users' per-room nicknames/avatars were visible to anyone who could see you in the user directory. ([\#11002](https://github.com/matrix-org/synapse/issues/11002 ))
- Fix a long-standing bug where a user's per-room nickname/avatar would overwrite their profile in the user directory when a room was made public. ([\#11003](https://github.com/matrix-org/synapse/issues/11003 ))
- Work around a regression, introduced in Synapse v1.39.0, that caused `SynapseError`s raised by the experimental third-party rules module callback `check_event_allowed` to be ignored. ([\#11042](https://github.com/matrix-org/synapse/issues/11042 ))
- Fix a bug in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) insertion events in rooms that could cause cross-talk/conflicts between batches. ([\#10877](https://github.com/matrix-org/synapse/issues/10877 ))
Improved Documentation
----------------------
- Change wording ("reference homeserver") in Synapse repository documentation. Contributed by @maxkratz. ([\#10971](https://github.com/matrix-org/synapse/issues/10971 ))
- Fix a dead URL in development documentation (SAML) and change wording from "Riot" to "Element". Contributed by @maxkratz. ([\#10973](https://github.com/matrix-org/synapse/issues/10973 ))
- Add additional content to the Welcome and Overview page of the documentation. ([\#10990](https://github.com/matrix-org/synapse/issues/10990 ))
- Update links to MSCs in documentation. Contributed by @dklimpel. ([\#10991](https://github.com/matrix-org/synapse/issues/10991 ))
Internal Changes
----------------
- Improve type hinting in `synapse.util`. ([\#10888](https://github.com/matrix-org/synapse/issues/10888 ))
- Add further type hints to `synapse.storage.util`. ([\#10892](https://github.com/matrix-org/synapse/issues/10892 ))
- Fix type hints to be compatible with an upcoming change to Twisted. ([\#10895](https://github.com/matrix-org/synapse/issues/10895 ))
- Update utility code to handle C implementations of frozendict. ([\#10902](https://github.com/matrix-org/synapse/issues/10902 ))
- Drop old functionality which maintained database compatibility with Synapse versions before v1.31. ([\#10903](https://github.com/matrix-org/synapse/issues/10903 ))
- Clean-up configuration helper classes for the `ServerConfig` class. ([\#10915](https://github.com/matrix-org/synapse/issues/10915 ))
- Use direct references to config flags. ([\#10916](https://github.com/matrix-org/synapse/issues/10916 ), [\#10959](https://github.com/matrix-org/synapse/issues/10959 ), [\#10985](https://github.com/matrix-org/synapse/issues/10985 ))
- Clean up some of the federation event authentication code for clarity. ([\#10926](https://github.com/matrix-org/synapse/issues/10926 ), [\#10940](https://github.com/matrix-org/synapse/issues/10940 ), [\#10986](https://github.com/matrix-org/synapse/issues/10986 ), [\#10987](https://github.com/matrix-org/synapse/issues/10987 ), [\#10988](https://github.com/matrix-org/synapse/issues/10988 ), [\#11010](https://github.com/matrix-org/synapse/issues/11010 ), [\#11011](https://github.com/matrix-org/synapse/issues/11011 ))
- Refactor various parts of the codebase to use `RoomVersion` objects instead of room version identifier strings. ([\#10934](https://github.com/matrix-org/synapse/issues/10934 ))
- Refactor user directory tests in preparation for upcoming changes. ([\#10935](https://github.com/matrix-org/synapse/issues/10935 ))
- Include the event id in the logcontext when handling PDUs received over federation. ([\#10936](https://github.com/matrix-org/synapse/issues/10936 ))
- Fix logged errors in unit tests. ([\#10939](https://github.com/matrix-org/synapse/issues/10939 ))
- Fix a broken test to ensure that consent configuration works during registration. ([\#10945](https://github.com/matrix-org/synapse/issues/10945 ))
- Add type hints to filtering classes. ([\#10958](https://github.com/matrix-org/synapse/issues/10958 ))
- Add type-hint to `HomeserverTestcase.setup_test_homeserver`. ([\#10961](https://github.com/matrix-org/synapse/issues/10961 ))
- Fix the test utility function `create_room_as` so that `is_public=True` will explicitly set the `visibility` parameter of room creation requests to `public`. Contributed by @AndrewFerr. ([\#10963](https://github.com/matrix-org/synapse/issues/10963 ))
- Make the release script more robust and transparent. ([\#10966](https://github.com/matrix-org/synapse/issues/10966 ))
- Refactor [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) `/batch_send` mega function into smaller handler functions. ([\#10974](https://github.com/matrix-org/synapse/issues/10974 ))
- Log stack traces when a missing opentracing span is detected. ([\#10983](https://github.com/matrix-org/synapse/issues/10983 ))
- Update GHA config to run tests against Python 3.10 and PostgreSQL 14. ([\#10992](https://github.com/matrix-org/synapse/issues/10992 ))
- Fix a long-standing bug where `ReadWriteLock`s could drop logging contexts on exit. ([\#10993](https://github.com/matrix-org/synapse/issues/10993 ))
- Add a `CODEOWNERS` file to automatically request reviews from the `@matrix-org/synapse-core` team on new pull requests. ([\#10994](https://github.com/matrix-org/synapse/issues/10994 ))
- Add further type hints to `synapse.state`. ([\#11004](https://github.com/matrix-org/synapse/issues/11004 ))
- Remove the deprecated `BaseHandler` object. ([\#11005](https://github.com/matrix-org/synapse/issues/11005 ))
- Bump mypy version for CI to 0.910, and pull in new type stubs for dependencies. ([\#11006](https://github.com/matrix-org/synapse/issues/11006 ))
- Fix CI to run the unit tests without optional deps. ([\#11017](https://github.com/matrix-org/synapse/issues/11017 ))
- Ensure that cache config tests do not share state. ([\#11019](https://github.com/matrix-org/synapse/issues/11019 ))
- Add additional type hints to `synapse.server_notices`. ([\#11021](https://github.com/matrix-org/synapse/issues/11021 ))
- Add additional type hints for `synapse.push`. ([\#11023](https://github.com/matrix-org/synapse/issues/11023 ))
- When installing the optional developer dependencies, also include the dependencies needed for type-checking and unit testing. ([\#11034](https://github.com/matrix-org/synapse/issues/11034 ))
- Remove unnecessary list comprehension from `synapse_port_db` to satisfy code style requirements. ([\#11043](https://github.com/matrix-org/synapse/issues/11043 ))
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAmFlaJYACgkQHgFcFF8Z
Fs0OBQ//QuIZHqwajqu6mVdcm5/GzYzerMVqcxByOsGX2Xeq0am1AnAG3MFH6TT/
f+e7T0ZIed7KTQFGKxyNX9nOi1RmQIaypBn66jt+tMzaCqMmlto3s0w2M5C+x4l4
lf9I0Shk/0wvrklc1GbmM89wB8eLmSz+UsrNK/ksMsevlP0K8GxA7HE3TrzDAth+
cLJOine4yfW4VlHus00u09E/2uILOE2wQ8V7YSpDMYOZc0bp0QEB5TGaab+4ESLz
JVLh1j3+dAmYPAAZMyWAp3PGN430NlNEwqomB1TYPdDRJ5ohlkWe22ACF6PD8eq+
TlnrANlo2CThZkTFQAecYkVPziETQv25BkGRho1IoBOfS8pTIDMoVwrrTc3okGfj
L6XlB1Rn+WauHXPq7EBJZnk9iw8qD0Pq8arMZeKYmQ1OMhUQmYTwBOj82fFU+jfu
a4v2rrMSVf3lmP9nAEHozGN0MAGsD0in+uvQuehdqVmKUbYUD0AWtHRZlO0ca4fH
BZxE2/CCJWaK/2IdH4I4S1l/AF4kdX0K0tcmAiwUKrm5jSC2Ww8BWeEqjwF9Bd+s
e1tF2pjw/VXjhAMmLIJVLEysb0yrw+QyZHWJ9fQP1KgaCcSwQjMgUBVrM3gcliIS
+/qjTl1QgkdTVHis6r5rdWBZrVJ/CeLFUz7pATEfoomf/EiEMHQ=
=VnsK
-----END PGP SIGNATURE-----
Merge tag 'v1.45.0rc1' into develop
Synapse 1.45.0rc1 (2021-10-12)
==============================
**Note:** We are aware of [a performance issue](https://github.com/matrix-org/synapse/issues/11049 ) introduced in Synapse v1.44.0, as well as [a bug](https://github.com/matrix-org/synapse/issues/11025 ) with the user directory when using application services. While this release candidate doesn't fix either of those issues, a second release candidate is expected to come out in a few days to address them.
Media storage providers module that read from Synapse's configuration need changes as of this version, see the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1450 ) for more information.
Features
--------
- Add [MSC3069](https://github.com/matrix-org/matrix-doc/pull/3069 ) support to `/account/whoami`. ([\#9655](https://github.com/matrix-org/synapse/issues/9655 ))
- Support autodiscovery of oEmbed previews. ([\#10822](https://github.com/matrix-org/synapse/issues/10822 ))
- Add a `user_may_send_3pid_invite` spam checker callback for modules to allow or deny 3PID invites. ([\#10894](https://github.com/matrix-org/synapse/issues/10894 ))
- Add a spam checker callback to allow or deny room joins. ([\#10910](https://github.com/matrix-org/synapse/issues/10910 ))
- Include an `update_synapse_database` script in the distribution. Contributed by @Fizzadar at Beeper. ([\#10954](https://github.com/matrix-org/synapse/issues/10954 ))
- Include exception information in JSON logging output. Contributed by @Fizzadar at Beeper. ([\#11028](https://github.com/matrix-org/synapse/issues/11028 ))
Bugfixes
--------
- Fix a minor bug in the response to `/_matrix/client/r0/voip/turnServer`. Contributed by @lukaslihotzki. ([\#10922](https://github.com/matrix-org/synapse/issues/10922 ))
- Fix a bug where empty `yyyy-mm-dd/` directories would be left behind in the media store's `url_cache_thumbnails/` directory. ([\#10924](https://github.com/matrix-org/synapse/issues/10924 ))
- Fix a bug introduced in Synapse v1.40.0 where the signature checks for room version 8 and 9 could be applied to earlier room versions in some situations. ([\#10927](https://github.com/matrix-org/synapse/issues/10927 ))
- Fix a long-standing bug wherein deactivated users still count towards the monthly active users limit. ([\#10947](https://github.com/matrix-org/synapse/issues/10947 ))
- Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state. ([\#10956](https://github.com/matrix-org/synapse/issues/10956 ))
- Fix a long-standing bug where rebuilding the user directory wouldn't exclude support and deactivated users. ([\#10960](https://github.com/matrix-org/synapse/issues/10960 ))
- Fix [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) `/batch_send` endpoint rejecting subsequent batches with unknown batch ID error in existing room versions from the room creator. ([\#10962](https://github.com/matrix-org/synapse/issues/10962 ))
- Fix a bug that could leak local users' per-room nicknames and avatars when the user directory is rebuilt. ([\#10981](https://github.com/matrix-org/synapse/issues/10981 ))
- Fix a long-standing bug where the remainder of a batch of user directory changes would be silently dropped if the server left a room early in the batch. ([\#10982](https://github.com/matrix-org/synapse/issues/10982 ))
- Correct a bugfix introduced in Synapse v1.44.0 that would catch the wrong error if a connection is lost before a response could be written to it. ([\#10995](https://github.com/matrix-org/synapse/issues/10995 ))
- Fix a long-standing bug where local users' per-room nicknames/avatars were visible to anyone who could see you in the user directory. ([\#11002](https://github.com/matrix-org/synapse/issues/11002 ))
- Fix a long-standing bug where a user's per-room nickname/avatar would overwrite their profile in the user directory when a room was made public. ([\#11003](https://github.com/matrix-org/synapse/issues/11003 ))
- Work around a regression, introduced in Synapse v1.39.0, that caused `SynapseError`s raised by the experimental third-party rules module callback `check_event_allowed` to be ignored. ([\#11042](https://github.com/matrix-org/synapse/issues/11042 ))
- Fix a bug in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) insertion events in rooms that could cause cross-talk/conflicts between batches. ([\#10877](https://github.com/matrix-org/synapse/issues/10877 ))
Improved Documentation
----------------------
- Change wording ("reference homeserver") in Synapse repository documentation. Contributed by @maxkratz. ([\#10971](https://github.com/matrix-org/synapse/issues/10971 ))
- Fix a dead URL in development documentation (SAML) and change wording from "Riot" to "Element". Contributed by @maxkratz. ([\#10973](https://github.com/matrix-org/synapse/issues/10973 ))
- Add additional content to the Welcome and Overview page of the documentation. ([\#10990](https://github.com/matrix-org/synapse/issues/10990 ))
- Update links to MSCs in documentation. Contributed by @dklimpel. ([\#10991](https://github.com/matrix-org/synapse/issues/10991 ))
Internal Changes
----------------
- Improve type hinting in `synapse.util`. ([\#10888](https://github.com/matrix-org/synapse/issues/10888 ))
- Add further type hints to `synapse.storage.util`. ([\#10892](https://github.com/matrix-org/synapse/issues/10892 ))
- Fix type hints to be compatible with an upcoming change to Twisted. ([\#10895](https://github.com/matrix-org/synapse/issues/10895 ))
- Update utility code to handle C implementations of frozendict. ([\#10902](https://github.com/matrix-org/synapse/issues/10902 ))
- Drop old functionality which maintained database compatibility with Synapse versions before v1.31. ([\#10903](https://github.com/matrix-org/synapse/issues/10903 ))
- Clean-up configuration helper classes for the `ServerConfig` class. ([\#10915](https://github.com/matrix-org/synapse/issues/10915 ))
- Use direct references to config flags. ([\#10916](https://github.com/matrix-org/synapse/issues/10916 ), [\#10959](https://github.com/matrix-org/synapse/issues/10959 ), [\#10985](https://github.com/matrix-org/synapse/issues/10985 ))
- Clean up some of the federation event authentication code for clarity. ([\#10926](https://github.com/matrix-org/synapse/issues/10926 ), [\#10940](https://github.com/matrix-org/synapse/issues/10940 ), [\#10986](https://github.com/matrix-org/synapse/issues/10986 ), [\#10987](https://github.com/matrix-org/synapse/issues/10987 ), [\#10988](https://github.com/matrix-org/synapse/issues/10988 ), [\#11010](https://github.com/matrix-org/synapse/issues/11010 ), [\#11011](https://github.com/matrix-org/synapse/issues/11011 ))
- Refactor various parts of the codebase to use `RoomVersion` objects instead of room version identifier strings. ([\#10934](https://github.com/matrix-org/synapse/issues/10934 ))
- Refactor user directory tests in preparation for upcoming changes. ([\#10935](https://github.com/matrix-org/synapse/issues/10935 ))
- Include the event id in the logcontext when handling PDUs received over federation. ([\#10936](https://github.com/matrix-org/synapse/issues/10936 ))
- Fix logged errors in unit tests. ([\#10939](https://github.com/matrix-org/synapse/issues/10939 ))
- Fix a broken test to ensure that consent configuration works during registration. ([\#10945](https://github.com/matrix-org/synapse/issues/10945 ))
- Add type hints to filtering classes. ([\#10958](https://github.com/matrix-org/synapse/issues/10958 ))
- Add type-hint to `HomeserverTestcase.setup_test_homeserver`. ([\#10961](https://github.com/matrix-org/synapse/issues/10961 ))
- Fix the test utility function `create_room_as` so that `is_public=True` will explicitly set the `visibility` parameter of room creation requests to `public`. Contributed by @AndrewFerr. ([\#10963](https://github.com/matrix-org/synapse/issues/10963 ))
- Make the release script more robust and transparent. ([\#10966](https://github.com/matrix-org/synapse/issues/10966 ))
- Refactor [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) `/batch_send` mega function into smaller handler functions. ([\#10974](https://github.com/matrix-org/synapse/issues/10974 ))
- Log stack traces when a missing opentracing span is detected. ([\#10983](https://github.com/matrix-org/synapse/issues/10983 ))
- Update GHA config to run tests against Python 3.10 and PostgreSQL 14. ([\#10992](https://github.com/matrix-org/synapse/issues/10992 ))
- Fix a long-standing bug where `ReadWriteLock`s could drop logging contexts on exit. ([\#10993](https://github.com/matrix-org/synapse/issues/10993 ))
- Add a `CODEOWNERS` file to automatically request reviews from the `@matrix-org/synapse-core` team on new pull requests. ([\#10994](https://github.com/matrix-org/synapse/issues/10994 ))
- Add further type hints to `synapse.state`. ([\#11004](https://github.com/matrix-org/synapse/issues/11004 ))
- Remove the deprecated `BaseHandler` object. ([\#11005](https://github.com/matrix-org/synapse/issues/11005 ))
- Bump mypy version for CI to 0.910, and pull in new type stubs for dependencies. ([\#11006](https://github.com/matrix-org/synapse/issues/11006 ))
- Fix CI to run the unit tests without optional deps. ([\#11017](https://github.com/matrix-org/synapse/issues/11017 ))
- Ensure that cache config tests do not share state. ([\#11019](https://github.com/matrix-org/synapse/issues/11019 ))
- Add additional type hints to `synapse.server_notices`. ([\#11021](https://github.com/matrix-org/synapse/issues/11021 ))
- Add additional type hints for `synapse.push`. ([\#11023](https://github.com/matrix-org/synapse/issues/11023 ))
- When installing the optional developer dependencies, also include the dependencies needed for type-checking and unit testing. ([\#11034](https://github.com/matrix-org/synapse/issues/11034 ))
- Remove unnecessary list comprehension from `synapse_port_db` to satisfy code style requirements. ([\#11043](https://github.com/matrix-org/synapse/issues/11043 ))
2021-10-12 12:33:41 +01:00
b8b905c4ea
Fix inconsistent behavior of `get_last_client_by_ip` ( #10970 )
...
Make `get_last_client_by_ip` return the same dictionary structure
regardless of whether the data has been persisted to the database.
This change will allow slightly cleaner type hints to be applied later
on.
2021-10-12 11:24:05 +01:00
6b18eb4430
Fix opentracing and Prometheus metrics for replication requests ( #10996 )
...
This commit fixes two bugs to do with decorators not instrumenting
`ReplicationEndpoint`'s `send_request` correctly. There are two
decorators on `send_request`: Prometheus' `Gauge.track_inprogress()`
and Synapse's `opentracing.trace`.
`Gauge.track_inprogress()` does not have any support for async
functions when used as a decorator. Since async functions behave like
regular functions that return coroutines, only the creation of the
coroutine was covered by the metric and none of the actual body of
`send_request`.
`Gauge.track_inprogress()` returns a regular, non-async function
wrapping `send_request`, which is the source of the next bug.
The `opentracing.trace` decorator would normally handle async functions
correctly, but since the wrapped `send_request` is a non-async function,
the decorator ends up suffering from the same issue as
`Gauge.track_inprogress()`: the opentracing span only measures the
creation of the coroutine and none of the actual function body.
Using `Gauge.track_inprogress()` as a context manager instead of a
decorator resolves both bugs.
2021-10-12 11:23:46 +01:00
8c5255b664
1.45.0rc1
2021-10-12 10:47:15 +01:00
406f7bfa17
Add an approximate difference method to StateFilters ( #10825 )
2021-10-12 10:44:59 +01:00
e0f11ae4a5
disallow-untyped-defs for synapse.push ( #11023 )
2021-10-11 17:42:10 +01:00
5e29d417fc
Include the requirements for [mypy,lint] in [dev] ( #11034 )
2021-10-11 16:34:31 +01:00
3828dd819b
Pass through `SynapseError`s that are raised from experimental `check_event_allowed` callback of the module API ( #11042 )
...
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-10-11 16:29:02 +01:00
4c838112dc
Remove unnecessary list comprehension in `synapse_port_db` to fix linting in CI ( #11043 )
2021-10-11 16:28:29 +01:00
b742cb2e4a
Release script improvements ( #10966 )
...
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-10-11 14:48:38 +01:00
a7d22c36db
Refactor MSC2716 `/batch_send` endpoint into separate handler functions ( #10974 )
2021-10-08 18:35:00 -05:00
1b112840d2
Autodiscover oEmbed endpoint from returned HTML ( #10822 )
...
Searches the returned HTML for an oEmbed endpoint using the
autodiscovery mechanism (`<link rel=...>`), and will request it
to generate the preview.
2021-10-08 14:14:42 -04:00
593eeac19e
Revert accidental push to develop.
2021-10-08 17:17:22 +01:00
d51a340019
Newsfile
...
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2021-10-08 17:12:40 +01:00
51a5da74cc
Annotate synapse.storage.util ( #10892 )
...
Also mark `synapse.streams` as having has no untyped defs
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2021-10-08 14:25:16 +00:00
797ee7812d
Relax `ignore-missing-imports` for modules that have stubs now and update mypy ( #11006 )
...
Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details.
We therefore pull in stub packages in setup.py
Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them.
The rest of this change consists of fixups to make the newer mypy + stubs pass CI.
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-10-08 14:49:41 +01:00
670a8d9a1e
Fix overwriting profile when making room public ( #11003 )
...
This splits apart `handle_new_user` into a function which adds an entry to the `user_directory` and a function which updates the room sharing tables. I plan to continue doing more of this kind of refactoring to clarify the implementation.
2021-10-08 12:52:48 +01:00
eb9ddc8c2e
Remove the deprecated BaseHandler. ( #11005 )
...
The shared ratelimit function was replaced with a dedicated
RequestRatelimiter class (accessible from the HomeServer
object).
Other properties were copied to each sub-class that inherited
from BaseHandler.
2021-10-08 07:44:43 -04:00
49a683d871
Fix long-standing bug where `ReadWriteLock` could drop logging contexts ( #10993 )
...
Use `PreserveLoggingContext()` to ensure that logging contexts are not
lost when exiting a read/write lock.
When exiting a read/write lock, callbacks on a `Deferred` are triggered
as a signal to any waiting coroutines. Any waiting coroutine that
becomes runnable is likely to follow the Synapse logging context rules
and will restore its own logging context, then either run to completion
or await another `Deferred`, resetting the logging context in the
process.
2021-10-08 12:27:16 +01:00
bb228f3523
Include exception in json logging ( #11028 )
2021-10-08 13:08:25 +02:00
0b4d5ce5e3
Fix CI to run the unit tests without optional deps ( #11017 )
...
This also turns off calculating code coverage, as we didn't use it and it was a lot of noise
2021-10-08 10:05:48 +01:00
e79ee48313
disallow-untyped-defs for synapse.server_notices ( #11021 )
2021-10-07 19:55:15 +01:00
7301019d48
Ensure each cache config test uses separate state. ( #11019 )
...
Hopefully this fixes these tests sometimes failing in CI.
2021-10-07 09:38:31 -04:00
e0bf34dada
Don't alter directory entries for local users when setting a per-room nickname ( #11002 )
...
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-10-07 13:26:11 +01:00
96fe77c254
Improve the logging in _auth_and_persist_outliers ( #11010 )
...
Include the event ids being peristed
2021-10-07 11:43:25 +00:00
86af6b2f0e
Add a comment in _process_received_pdu ( #11011 )
2021-10-07 12:20:03 +01:00
52aefd5086
Catch AttributeErrors when calling registerProducer ( #10995 )
...
Looks like the wrong exception type was caught in #10932 .
2021-10-07 10:37:10 +00:00
f563676c09
`disallow-untyped-defs` for `synapse.state` ( #11004 )
...
* `disallow-untyped-defs` for `synapse.state`
Much smaller than I was expecting!
2021-10-06 17:55:25 +00:00
e564bdd127
Add content to the Synapse documentation intro page ( #10990 )
...
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-10-06 17:09:35 +00:00
4e51621064
Add a spamchecker method to allow or deny 3pid invites ( #10894 )
...
This is in the context of creating new module callbacks that modules in https://github.com/matrix-org/synapse-dinsic can use, in an effort to reconcile the spam checker API in synapse-dinsic with the one in mainline.
Note that a module callback already exists for 3pid invites (https://matrix-org.github.io/synapse/develop/modules/third_party_rules_callbacks.html#check_threepid_can_be_invited ) but it doesn't check whether the sender of the invite is allowed to send it.
2021-10-06 17:18:13 +02:00
f4b1a9a527
Require direct references to configuration variables. ( #10985 )
...
This removes the magic allowing accessing configurable
variables directly from the config object. It is now required
that a specific configuration class is used (e.g. `config.foo`
must be replaced with `config.server.foo`).
2021-10-06 10:47:41 -04:00
829f2a82b0
Add a spamchecker callback to allow or deny room joins ( #10910 )
...
Co-authored-by: Erik Johnston <erik@matrix.org>
2021-10-06 14:32:16 +00:00
b0460936c8
Add the synapse-core team as code owners ( #10994 )
...
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2021-10-06 16:03:17 +02:00
370bca32e6
Don't drop user dir deltas when server leaves room ( #10982 )
...
Fix a long-standing bug where a batch of user directory changes would be
silently dropped if the server left a room early in the batch.
* Pull out `wait_for_background_update` in tests
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-10-06 12:56:45 +00:00
38b7db5885
Updated development doc on samling environment for testing. ( #10973 )
2021-10-06 07:20:41 -04:00
c80878d22a
Add `--run-background-updates` option to `update_database` script. ( #10954 )
...
Signed-off-by: Nick Barrett <nick@beeper.com>
2021-10-06 11:26:18 +01:00
f8d0f72b27
More types for synapse.util, part 1 ( #10888 )
...
The following modules now pass `disallow_untyped_defs`:
* synapse.util.caches.cached_call
* synapse.util.caches.lrucache
* synapse.util.caches.response_cache
* synapse.util.caches.stream_change_cache
* synapse.util.caches.ttlcache pass
* synapse.util.daemonize
* synapse.util.patch_inline_callbacks pass `no-untyped-defs`
* synapse.util.versionstring
Additional typing in synapse.util.metrics. Didn't get this to pass `no-untyped-defs`, think I'll need to watch #10847
2021-10-06 11:20:49 +01:00
6744273f0b
Remove "reference" wording according Synapse homeserver ( #10971 )
2021-10-06 10:05:07 +00:00
4f00432ce1
Fix potential leak of per-room profiles when the user dir is rebuilt. ( #10981 )
...
There are two steps to rebuilding the user directory:
1. a scan over rooms, followed by
2. a scan over local users.
The former reads avatars and display names from the `room_memberships`
table and therefore contains potentially private avatars and
display names. The latter reads from the the `profiles` table which only
contains public data; moreover it will overwrite any private profiles
that the rooms scan may have written to the user directory. This means
that the rebuild could leak private user while the rebuild was in
progress, only to later cover up the leaks once the rebuild had completed.
This change skips over local users when writing user_directory rows
when scanning rooms. Doing so means that it'll take longer for a rebuild
to make local users searchable, which is unfortunate. I think a future
PR can improve this by swapping the order of the two steps above. (And
indeed there's more to do here, e.g. copying from `profiles` without
going via Python.)
Small tidy-ups while I'm here:
* Remove duplicated code from test_initial. This was meant to be pulled into `purge_and_rebuild_user_dir`.
* Move `is_public` before updating sharing tables. No functional change; it's still before the first read of `is_public`.
* Don't bother creating a set from dict keys. Slightly nicer and makes the code simpler.
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-10-05 18:35:25 +01:00
392863fbf1
Fix logic flaw preventing tracking of MSC2716 events in existing room versions ( #10962 )
...
We correctly allowed using the MSC2716 batch endpoint for
the room creator in existing room versions but accidentally didn't track
the events because of a logic flaw.
This prevented you from connecting subsequent chunks together because it would
throw the unknown batch ID error.
We only want to process MSC2716 events when:
- The room version supports MSC2716
- Any room where the homeserver has the `msc2716_enabled` experimental feature enabled and the event is from the room creator
2021-10-05 11:51:57 -05:00
3a5b0cbe7a
Ensure that we reject events which use rejected events for auth ( #10956 )
...
When we consider whether to accept events, we should not accept those which
depend on rejected events for their auth events.
This (together with earlier changes such as
https://github.com/matrix-org/synapse/pull/10771 and
https://github.com/matrix-org/synapse/pull/10896 ) forms a partial fix to
https://github.com/matrix-org/synapse/issues/9595 . There still remain code
paths where we do not check the `auth_events` at all.
2021-10-05 13:23:29 +01:00
787af4a106
Host `cache_joined_hosts_for_event` to caller ( #10986 )
...
`_check_event_auth` is only called in two places, and only one of those sets
`send_on_behalf_of`. Warming the cache isn't really part of auth anyway, so
moving it out makes a lot more sense.
2021-10-05 13:01:41 +01:00
d099535deb
`_update_auth_events_and_context_for_auth`: add some comments ( #10987 )
...
Add some more comments about wtf is going on here.
2021-10-05 12:50:38 +01:00
cb88ed912b
`_check_event_auth`: move event validation earlier ( #10988 )
...
There's little point in doing a fancy state reconciliation dance if the event
itself is invalid.
Likewise, there's no point checking it again in `_check_for_soft_fail`.
2021-10-05 12:50:07 +01:00
6f6e956338
Run CI with Python 3.10 and Postgres 14 ( #10992 )
2021-10-05 12:43:04 +01:00
7036a7a60a
Update links to MSCs in documentation ( #10991 )
...
Based on matrix-doc switching from master -> main and
MSCs being merged.
2021-10-05 11:35:19 +00:00
660c8c1415
Log stack traces when a missing opentracing span is detected ( #10983 )
...
Make it easier to track down where opentracing spans are going missing
by including stack traces in the logs.
2021-10-05 12:23:25 +01:00
eda8c88b84
Add functionality to remove deactivated users from the monthly_active_users table ( #10947 )
...
* add test
* add function to remove user from monthly active table in deactivate code
* add function to remove user from monthly active table
* add changelog entry
* update changelog number
* requested changes
* update docstring on new function
* fix lint error
* Update synapse/storage/databases/main/monthly_active_users.py
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-10-04 08:34:42 -07:00
30f0240401
Make is_public Optional[bool] for create_room_as test util ( #10951 ) ( #10963 )
...
Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
2021-10-04 14:43:03 +00:00
730b40dd5e
Synapse 1.44.0rc3 (2021-10-04)
...
==============================
Bugfixes
--------
- Fix a bug introduced in Synapse v1.40.0 where changing a user's display name or avatar in a restricted room would cause an authentication error. ([\#10933](https://github.com/matrix-org/synapse/issues/10933 ))
- Fix `/admin/whois/{user_id}` endpoint, which was broken in v1.44.0rc1. ([\#10968](https://github.com/matrix-org/synapse/issues/10968 ))
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAmFbCRUACgkQHgFcFF8Z
Fs1n2Q/+P6WZmqNPEyPb4zgEhHgBZtAiYsS8i+6TBdu/viricfe3xmiifmKYyblV
C07a8Qu14OMgmXKYR8AY0HvLRx+zdhJFGuYa3I0+yY1GTlQWlf+OKnQtsbgdmJ6O
b8HbmGB1qJCJ0rbbWhCQvx/XXlqqMIATc6qJj3HTwOpWxNL8TymOFg6TGtb+rLkN
/s2NfxWPJqKMKTLVgZUVjkNGBBtJmu/+Ow3PYz7J9hEW69REONed/014wVgiWx+W
BJOoUj8dHAS5ZhdKWiSKAzl5A9FQyxUPDwkO44wsK5OIu+dVy4eF9HCMi0EP/9nT
G3VWwr3z/TA55foL8XdrIPdp1SFqJmIEwDLgaibISD1/8MoC15YzAkt4CoKYOsL6
EQtDDQal1BNSFZPITz7PWSFGOMgN3tKfMQUqCC+eagHvNfSxVH+J1zNg7ve2/24h
PbRU/tt4mJiPu7M2Ejj0EWNHyI2fp92ARzzAzQ0JlrPJe34Z/hAiqf7w4kjtJ7Ew
Lm890EAw2azo7RYU9xevkOsU2CEtLEnKsGMW8pJc9eRxUjRKfk/EW39dCQq9Myhu
uQFeYHALcn4vu9jGhGyoO9fJDKIxpM76h37Cwu7shg84Gp2ZwucSjwW2l2rZKiIS
YUeruLOavUueUZYTcyXxAqAAsH8z1hbKmnXIbNxFrcu+NOl+o84=
=HR3N
-----END PGP SIGNATURE-----
Merge tag 'v1.44.0rc3' into develop
Synapse 1.44.0rc3 (2021-10-04)
==============================
Bugfixes
--------
- Fix a bug introduced in Synapse v1.40.0 where changing a user's display name or avatar in a restricted room would cause an authentication error. ([\#10933](https://github.com/matrix-org/synapse/issues/10933 ))
- Fix `/admin/whois/{user_id}` endpoint, which was broken in v1.44.0rc1. ([\#10968](https://github.com/matrix-org/synapse/issues/10968 ))
2021-10-04 15:33:42 +01:00
2d2c6a41fe
1.44.0rc3
2021-10-04 14:57:40 +01:00
f7b034a24b
Consistently exclude from user_directory ( #10960 )
...
* Introduce `should_include_local_users_in_dir`
We exclude three kinds of local users from the user_directory tables. At
present we don't consistently exclude all three in the same places. This
commit introduces a new function to gather those exclusion conditions
together. Because we have to handle local and remote users in different
ways, I've made that function only consider the case of remote users.
It's the caller's responsibility to make the local versus remote
distinction clear and correct.
A test fixup is required. The test now hits a path which makes db
queries against the users table. The expected rows were missing, because
we were using a dummy user that hadn't actually been registered.
We also add new test cases to covert the exclusion logic.
----
By my reading this makes these changes:
* When an app service user registers or changes their profile, they will
_not_ be added to the user directory. (Previously only support and
deactivated users were excluded). This is consistent with the logic that
rebuilds the user directory. See also [the discussion
here](https://github.com/matrix-org/synapse/pull/10914#discussion_r716859548 ).
* When rebuilding the directory, exclude support and disabled users from
room sharing tables. Previously only appservice users were excluded.
* Exclude all three categories of local users when rebuilding the
directory. Previously `_populate_user_directory_process_users` didn't do
any exclusion.
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-10-04 11:45:51 +00:00
a0f48ee89d
Use direct references for configuration variables (part 7). ( #10959 )
2021-10-04 07:18:54 -04:00
d1cbad388f
Fix error in `get_user_ip_and_agents` when fetching from the database ( #10968 )
2021-10-01 17:22:13 +01:00
32072dcdac
Strip "join_authorised_via_users_server" from join events which do not need it. ( #10933 )
...
This fixes a "Event not signed by authorising server" error when
transition room member from join -> join, e.g. when updating a
display name or avatar URL for restricted rooms.
2021-10-01 11:39:17 -04:00
e46ac85d67
type-hint `HomeserverTestcase.setup_test_homeserver` ( #10961 )
...
* type-hint `HomeserverTestcase.setup_test_homeserver`
For better IDE completion. A small drive-by.
2021-10-01 12:22:47 +01:00
7e440520c9
Add type hints to filtering classes. ( #10958 )
2021-10-01 07:02:32 -04:00
9e5a429c8b
Clean-up registration tests ( #10945 )
...
Uses `override_config` and fixes test_auto_create_auto_join_where_no_consent
to properly configure auto-join rooms.
2021-09-30 14:06:02 -04:00
d1bf5f7c9d
Strip "join_authorised_via_users_server" from join events which do not need it. ( #10933 )
...
This fixes a "Event not signed by authorising server" error when
transition room member from join -> join, e.g. when updating a
display name or avatar URL for restricted rooms.
2021-09-30 11:13:59 -04:00
7d84d2523a
Fix errors in Synapse logs from unit tests. ( #10939 )
...
Fix some harmless errors from background processes (mostly
due to awaiting Mock objects) that occurred in the Synapse
logs during unit tests.
2021-09-30 11:03:29 -04:00
145cb6d08e
Fix getTurnServer response: return an integer ttl ( #10922 )
...
`ttl` must be an integer according to the OpenAPI spec:
https://github.com/matrix-org/matrix-doc/blob/old_master/data/api/client-server/voip.yaml#L70
True division (`/`) returns a float instead (`"ttl": 7200.0`).
Floor division (`//`) returns an integer, so the response is spec compliant.
Signed-off-by: Lukas Lihotzki <lukas@lihotzki.de>
2021-09-30 08:04:55 -04:00
29364145b2
Pass str to twisted's IReactorTCP ( #10895 )
...
This follows a correction made in twisted/twisted#1664 and should fix our Twisted Trial CI job.
Until that change is in a twisted release, we'll have to ignore the type
of the `host` argument. I've raised #10899 to remind us to review the
issue in a few months' time.
2021-09-30 12:51:47 +01:00
3412f5c8d8
1.44.0rc2
2021-09-30 12:40:24 +01:00
c4bf48ee6f
Fix event context for outliers in important MSC2716 spot ( #10938 )
...
Fix event context for outlier causing failures in all of the MSC2716
Complement tests.
The `EventContext.for_outlier` refactor happened in
https://github.com/matrix-org/synapse/pull/10883
and this spot was left out.
2021-09-30 11:34:44 +01:00
a03ed5e6ae
Fix issue causing sending presence to ASes to fail (due to incomplete type annotations) ( #10944 )
2021-09-30 11:06:47 +01:00
3aefc7b66d
Refactor user directory tests ( #10935 )
...
* Pull out GetUserDirectoryTables helper
* Don't rebuild the dir in tests that don't need it
In #10796 I changed registering a user to add directory entries under.
This means we don't have to force a directory regbuild in to tests of
the user directory search.
* Move test_initial to tests/storage
* Add type hints to both test_user_directory files
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-09-30 11:04:40 +01:00
428174f902
Split `event_auth.check` into two parts ( #10940 )
...
Broadly, the existing `event_auth.check` function has two parts:
* a validation section: checks that the event isn't too big, that it has the rught signatures, etc.
This bit is independent of the rest of the state in the room, and so need only be done once
for each event.
* an auth section: ensures that the event is allowed, given the rest of the state in the room.
This gets done multiple times, against various sets of room state, because it forms part of
the state res algorithm.
Currently, this is implemented with `do_sig_check` and `do_size_check` parameters, but I think
that makes everything hard to follow. Instead, we split the function in two and call each part
separately where it is needed.
2021-09-29 18:59:15 +01:00
a19aa8b162
Synapse 1.44.0rc1 (2021-09-29)
...
==============================
Features
--------
- Only allow the [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) `/batch_send?chunk_id=xxx` endpoint to connect to an already existing insertion event. ([\#10776](https://github.com/matrix-org/synapse/issues/10776 ))
- Improve oEmbed URL previews by processing the author name, photo, and video information. ([\#10814](https://github.com/matrix-org/synapse/issues/10814 ), [\#10819](https://github.com/matrix-org/synapse/issues/10819 ))
- Speed up responding with large JSON objects to requests. ([\#10868](https://github.com/matrix-org/synapse/issues/10868 ), [\#10905](https://github.com/matrix-org/synapse/issues/10905 ))
- Add a `user_may_create_room_with_invites` spam checker callback to allow modules to allow or deny a room creation request based on the invites and/or 3PID invites it includes. ([\#10898](https://github.com/matrix-org/synapse/issues/10898 ))
Bugfixes
--------
- Fix a long-standing bug that caused an `AssertionError` when purging history in certain rooms. Contributed by @Kokokokoka. ([\#10690](https://github.com/matrix-org/synapse/issues/10690 ))
- Fix a long-standing bug which caused deactivated users that were later reactivated to be missing from the user directory. ([\#10782](https://github.com/matrix-org/synapse/issues/10782 ))
- Fix a long-standing bug that caused unbanning a user by sending a membership event to fail. Contributed by @aaronraimist. ([\#10807](https://github.com/matrix-org/synapse/issues/10807 ))
- Fix a long-standing bug where logging contexts would go missing when federation requests time out. ([\#10810](https://github.com/matrix-org/synapse/issues/10810 ))
- Fix a long-standing bug causing an error in the deprecated `/initialSync` endpoint when using the undocumented `from` and `to` parameters. ([\#10827](https://github.com/matrix-org/synapse/issues/10827 ))
- Fix a bug causing the `remove_stale_pushers` background job to repeatedly fail and log errors. This bug affected Synapse servers that had been upgraded from version 1.28 or older and are using SQLite. ([\#10843](https://github.com/matrix-org/synapse/issues/10843 ))
- Fix a long-standing bug in Unicode support of the room search admin API breaking search for rooms with non-ASCII characters. ([\#10859](https://github.com/matrix-org/synapse/issues/10859 ))
- Fix a bug introduced in Synapse 1.37.0 which caused `knock` membership events which we sent to remote servers to be incorrectly stored in the local database. ([\#10873](https://github.com/matrix-org/synapse/issues/10873 ))
- Fix invalidating one-time key count cache after claiming keys. The bug was introduced in Synapse v1.41.0. Contributed by Tulir at Beeper. ([\#10875](https://github.com/matrix-org/synapse/issues/10875 ))
- Fix a long-standing bug causing application service users to be subject to MAU blocking if the MAU limit had been reached, even if configured not to be blocked. ([\#10881](https://github.com/matrix-org/synapse/issues/10881 ))
- Fix a long-standing bug which could cause events pulled over federation to be incorrectly rejected. ([\#10907](https://github.com/matrix-org/synapse/issues/10907 ))
- Fix a long-standing bug causing URL cache files to be stored in storage providers. Server admins may safely delete the `url_cache/` and `url_cache_thumbnails/` directories from any configured storage providers to reclaim space. ([\#10911](https://github.com/matrix-org/synapse/issues/10911 ))
- Fix a long-standing bug leading to race conditions when creating media store and config directories. ([\#10913](https://github.com/matrix-org/synapse/issues/10913 ))
Improved Documentation
----------------------
- Fix some crashes in the Module API example code, by adding JSON encoding/decoding. ([\#10845](https://github.com/matrix-org/synapse/issues/10845 ))
- Add developer documentation about experimental configuration flags. ([\#10865](https://github.com/matrix-org/synapse/issues/10865 ))
- Properly remove deleted files from GitHub pages when generating the documentation. ([\#10869](https://github.com/matrix-org/synapse/issues/10869 ))
Internal Changes
----------------
- Fix GitHub Actions config so we can run sytest on synapse from parallel branches. ([\#10659](https://github.com/matrix-org/synapse/issues/10659 ))
- Split out [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) meta events to their own fields in the `/batch_send` response. ([\#10777](https://github.com/matrix-org/synapse/issues/10777 ))
- Add missing type hints to REST servlets. ([\#10785](https://github.com/matrix-org/synapse/issues/10785 ), [\#10817](https://github.com/matrix-org/synapse/issues/10817 ))
- Simplify the internal logic which maintains the user directory database tables. ([\#10796](https://github.com/matrix-org/synapse/issues/10796 ))
- Use direct references to config flags. ([\#10812](https://github.com/matrix-org/synapse/issues/10812 ), [\#10885](https://github.com/matrix-org/synapse/issues/10885 ), [\#10893](https://github.com/matrix-org/synapse/issues/10893 ), [\#10897](https://github.com/matrix-org/synapse/issues/10897 ))
- Specify the type of token in generic "Invalid token" error messages. ([\#10815](https://github.com/matrix-org/synapse/issues/10815 ))
- Make `StateFilter` frozen so it is hashable. ([\#10816](https://github.com/matrix-org/synapse/issues/10816 ))
- Fix a long-standing bug where an `m.room.message` event containing a null byte would cause an internal server error. ([\#10820](https://github.com/matrix-org/synapse/issues/10820 ))
- Add type hints to the state database. ([\#10823](https://github.com/matrix-org/synapse/issues/10823 ))
- Opt out of cache expiry for `get_users_who_share_room_with_user`, to hopefully improve `/sync` performance when you
haven't synced recently. ([\#10826](https://github.com/matrix-org/synapse/issues/10826 ))
- Track cache eviction rates more finely in Prometheus's monitoring. ([\#10829](https://github.com/matrix-org/synapse/issues/10829 ))
- Add missing type hints to `synapse.handlers`. ([\#10831](https://github.com/matrix-org/synapse/issues/10831 ), [\#10856](https://github.com/matrix-org/synapse/issues/10856 ))
- Extend the Module API to let plug-ins check whether an ID is local and to access IP + User Agent data. ([\#10833](https://github.com/matrix-org/synapse/issues/10833 ))
- Factor out PNG image data to a constant to be used in several tests. ([\#10834](https://github.com/matrix-org/synapse/issues/10834 ))
- Add a test to ensure state events sent by modules get persisted correctly. ([\#10835](https://github.com/matrix-org/synapse/issues/10835 ))
- Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) fields and event types from `chunk` to `batch` to match the `/batch_send` endpoint. ([\#10838](https://github.com/matrix-org/synapse/issues/10838 ))
- Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) `/batch_send` query parameter from `?prev_event` to more obvious usage with `?prev_event_id`. ([\#10839](https://github.com/matrix-org/synapse/issues/10839 ))
- Add type hints to `synapse.http.site`. ([\#10867](https://github.com/matrix-org/synapse/issues/10867 ))
- Include outlier status when we log V2 or V3 events. ([\#10879](https://github.com/matrix-org/synapse/issues/10879 ))
- Break down Grafana's cache expiry time series based on reason for eviction, c.f. [\#10829](https://github.com/matrix-org/synapse/issues/10829 ). ([\#10880](https://github.com/matrix-org/synapse/issues/10880 ))
- Clean up some of the federation event authentication code for clarity. ([\#10883](https://github.com/matrix-org/synapse/issues/10883 ), [\#10884](https://github.com/matrix-org/synapse/issues/10884 ), [\#10896](https://github.com/matrix-org/synapse/issues/10896 ), [\#10901](https://github.com/matrix-org/synapse/issues/10901 ))
- Allow the `.` and `~` characters when creating registration tokens as per the change to [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231 ). ([\#10887](https://github.com/matrix-org/synapse/issues/10887 ))
- Clean up some unnecessary parentheses in places around the codebase. ([\#10889](https://github.com/matrix-org/synapse/issues/10889 ))
- Improve type hinting in the user directory code. ([\#10891](https://github.com/matrix-org/synapse/issues/10891 ))
- Update development testing script `test_postgresql.sh` to use a supported Python version and make re-runs quicker. ([\#10906](https://github.com/matrix-org/synapse/issues/10906 ))
- Document and summarize changes in schema version `61` – `64`. ([\#10917](https://github.com/matrix-org/synapse/issues/10917 ))
- Update release script to sign the newly created git tags. ([\#10925](https://github.com/matrix-org/synapse/issues/10925 ))
- Fix Debian builds due to `dh-virtualenv` no longer being able to build their docs. ([\#10931](https://github.com/matrix-org/synapse/issues/10931 ))
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEE8SRSDO7gYkSP4chELS76LzL74EcFAmFUY3UACgkQLS76LzL7
4EdDdg//egZvwUizHJAEWi5Vz029S2D3LN8Hl57TG+/KR+GgIJvOfgX9sgWDNsLA
4wTlsw6xhH9VcDzFhiAPrUTwhi4Gi80BZmnGXuR97yAE1z3oyS4IQg/g5LUXq9el
a/2JsK1Sau354TBMvLQgZsplqC+RPxE0k0MSTvEC+d3ft929jLXUftnI/nhGcDpX
7pP9RscVThkLLNC5Xcuu2ev4yPSrNNWhBeqz5J8CfbEW6Roej7a1d3WdehhoWJ5V
hwPOxOhL89lWwJ8uL/H5sDFMkxk+ua+IVQx8gJkQ1eRa3RIdFFSh1+exFtYwjIhI
Pd4wwUH7hGe1VIvmlVOoDZ1hka0RTjRLeJg39vq+yvWCBQW7syVE6sRUxTjeUvRe
7Cyz4sVG0g8uvWsF3YYva7S4SlYPPSj2Z+5/yL4RHlMZsPk/apmCRZBzaWlVEGur
XqTExi2rKzKsi2HZctHDWOeYF+G3c13FDNeXG+N2g/DfGKuKCTuhbdMm7LeIJ2L2
B8ki/nq3wc1FCJ0mkVlM+/UX3i3wXG+2UbOx29hU9yuEUIl4p4jyE1OjuH5l45TX
OWfW25BOGh5gYKvX8p9Kj4A9MIJ2YFRDFfcfF7g+S0ZA8KdUAjZkYBzfJtEzn2iK
uxVsSVdYOVqbbSwE5wto8WuOWu1op6RP3l1Wp92SoM5vqWjRmyA=
=SzDp
-----END PGP SIGNATURE-----
Merge tag 'v1.44.0rc1' into develop
Synapse 1.44.0rc1 (2021-09-29)
==============================
Features
--------
- Only allow the
[MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 )
`/batch_send?chunk_id=xxx` endpoint to connect to an already existing
insertion event.
([\#10776](https://github.com/matrix-org/synapse/issues/10776 ))
- Improve oEmbed URL previews by processing the author name, photo, and
video information.
([\#10814](https://github.com/matrix-org/synapse/issues/10814 ),
[\#10819](https://github.com/matrix-org/synapse/issues/10819 ))
- Speed up responding with large JSON objects to requests.
([\#10868](https://github.com/matrix-org/synapse/issues/10868 ),
[\#10905](https://github.com/matrix-org/synapse/issues/10905 ))
- Add a `user_may_create_room_with_invites` spam checker callback to
allow modules to allow or deny a room creation request based on the
invites and/or 3PID invites it includes.
([\#10898](https://github.com/matrix-org/synapse/issues/10898 ))
Bugfixes
--------
- Fix a long-standing bug that caused an `AssertionError` when purging
history in certain rooms. Contributed by @Kokokokoka.
([\#10690](https://github.com/matrix-org/synapse/issues/10690 ))
- Fix a long-standing bug which caused deactivated users that were later
reactivated to be missing from the user directory.
([\#10782](https://github.com/matrix-org/synapse/issues/10782 ))
- Fix a long-standing bug that caused unbanning a user by sending a
membership event to fail. Contributed by @aaronraimist.
([\#10807](https://github.com/matrix-org/synapse/issues/10807 ))
- Fix a long-standing bug where logging contexts would go missing when
federation requests time out.
([\#10810](https://github.com/matrix-org/synapse/issues/10810 ))
- Fix a long-standing bug causing an error in the deprecated
`/initialSync` endpoint when using the undocumented `from` and `to`
parameters.
([\#10827](https://github.com/matrix-org/synapse/issues/10827 ))
- Fix a bug causing the `remove_stale_pushers` background job to
repeatedly fail and log errors. This bug affected Synapse servers that
had been upgraded from version 1.28 or older and are using SQLite.
([\#10843](https://github.com/matrix-org/synapse/issues/10843 ))
- Fix a long-standing bug in Unicode support of the room search admin
API breaking search for rooms with non-ASCII characters.
([\#10859](https://github.com/matrix-org/synapse/issues/10859 ))
- Fix a bug introduced in Synapse 1.37.0 which caused `knock` membership
events which we sent to remote servers to be incorrectly stored in the
local database.
([\#10873](https://github.com/matrix-org/synapse/issues/10873 ))
- Fix invalidating one-time key count cache after claiming keys. The bug
was introduced in Synapse v1.41.0. Contributed by Tulir at Beeper.
([\#10875](https://github.com/matrix-org/synapse/issues/10875 ))
- Fix a long-standing bug causing application service users to be
subject to MAU blocking if the MAU limit had been reached, even if
configured not to be blocked.
([\#10881](https://github.com/matrix-org/synapse/issues/10881 ))
- Fix a long-standing bug which could cause events pulled over
federation to be incorrectly rejected.
([\#10907](https://github.com/matrix-org/synapse/issues/10907 ))
- Fix a long-standing bug causing URL cache files to be stored in
storage providers. Server admins may safely delete the `url_cache/` and
`url_cache_thumbnails/` directories from any configured storage
providers to reclaim space.
([\#10911](https://github.com/matrix-org/synapse/issues/10911 ))
- Fix a long-standing bug leading to race conditions when creating media
store and config directories.
([\#10913](https://github.com/matrix-org/synapse/issues/10913 ))
Improved Documentation
----------------------
- Fix some crashes in the Module API example code, by adding JSON
encoding/decoding.
([\#10845](https://github.com/matrix-org/synapse/issues/10845 ))
- Add developer documentation about experimental configuration flags.
([\#10865](https://github.com/matrix-org/synapse/issues/10865 ))
- Properly remove deleted files from GitHub pages when generating the
documentation.
([\#10869](https://github.com/matrix-org/synapse/issues/10869 ))
Internal Changes
----------------
- Fix GitHub Actions config so we can run sytest on synapse from
parallel branches.
([\#10659](https://github.com/matrix-org/synapse/issues/10659 ))
- Split out
[MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) meta
events to their own fields in the `/batch_send` response.
([\#10777](https://github.com/matrix-org/synapse/issues/10777 ))
- Add missing type hints to REST servlets.
([\#10785](https://github.com/matrix-org/synapse/issues/10785 ),
[\#10817](https://github.com/matrix-org/synapse/issues/10817 ))
- Simplify the internal logic which maintains the user directory
database tables.
([\#10796](https://github.com/matrix-org/synapse/issues/10796 ))
- Use direct references to config flags.
([\#10812](https://github.com/matrix-org/synapse/issues/10812 ),
[\#10885](https://github.com/matrix-org/synapse/issues/10885 ),
[\#10893](https://github.com/matrix-org/synapse/issues/10893 ),
[\#10897](https://github.com/matrix-org/synapse/issues/10897 ))
- Specify the type of token in generic "Invalid token" error messages.
([\#10815](https://github.com/matrix-org/synapse/issues/10815 ))
- Make `StateFilter` frozen so it is hashable.
([\#10816](https://github.com/matrix-org/synapse/issues/10816 ))
- Fix a long-standing bug where an `m.room.message` event containing a
null byte would cause an internal server error.
([\#10820](https://github.com/matrix-org/synapse/issues/10820 ))
- Add type hints to the state database.
([\#10823](https://github.com/matrix-org/synapse/issues/10823 ))
- Opt out of cache expiry for `get_users_who_share_room_with_user`, to
hopefully improve `/sync` performance when you
haven't synced recently.
([\#10826](https://github.com/matrix-org/synapse/issues/10826 ))
- Track cache eviction rates more finely in Prometheus's monitoring.
([\#10829](https://github.com/matrix-org/synapse/issues/10829 ))
- Add missing type hints to `synapse.handlers`.
([\#10831](https://github.com/matrix-org/synapse/issues/10831 ),
[\#10856](https://github.com/matrix-org/synapse/issues/10856 ))
- Extend the Module API to let plug-ins check whether an ID is local and
to access IP + User Agent data.
([\#10833](https://github.com/matrix-org/synapse/issues/10833 ))
- Factor out PNG image data to a constant to be used in several tests.
([\#10834](https://github.com/matrix-org/synapse/issues/10834 ))
- Add a test to ensure state events sent by modules get persisted
correctly.
([\#10835](https://github.com/matrix-org/synapse/issues/10835 ))
- Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 )
fields and event types from `chunk` to `batch` to match the
`/batch_send` endpoint.
([\#10838](https://github.com/matrix-org/synapse/issues/10838 ))
- Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 )
`/batch_send` query parameter from `?prev_event` to more obvious usage
with `?prev_event_id`.
([\#10839](https://github.com/matrix-org/synapse/issues/10839 ))
- Add type hints to `synapse.http.site`.
([\#10867](https://github.com/matrix-org/synapse/issues/10867 ))
- Include outlier status when we log V2 or V3 events.
([\#10879](https://github.com/matrix-org/synapse/issues/10879 ))
- Break down Grafana's cache expiry time series based on reason for
eviction, c.f.
[\#10829](https://github.com/matrix-org/synapse/issues/10829 ).
([\#10880](https://github.com/matrix-org/synapse/issues/10880 ))
- Clean up some of the federation event authentication code for clarity.
([\#10883](https://github.com/matrix-org/synapse/issues/10883 ),
[\#10884](https://github.com/matrix-org/synapse/issues/10884 ),
[\#10896](https://github.com/matrix-org/synapse/issues/10896 ),
[\#10901](https://github.com/matrix-org/synapse/issues/10901 ))
- Allow the `.` and `~` characters when creating registration tokens as
per the change to
[MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231 ).
([\#10887](https://github.com/matrix-org/synapse/issues/10887 ))
- Clean up some unnecessary parentheses in places around the codebase.
([\#10889](https://github.com/matrix-org/synapse/issues/10889 ))
- Improve type hinting in the user directory code.
([\#10891](https://github.com/matrix-org/synapse/issues/10891 ))
- Update development testing script `test_postgresql.sh` to use a
supported Python version and make re-runs quicker.
([\#10906](https://github.com/matrix-org/synapse/issues/10906 ))
- Document and summarize changes in schema version `61` – `64`.
([\#10917](https://github.com/matrix-org/synapse/issues/10917 ))
- Update release script to sign the newly created git tags.
([\#10925](https://github.com/matrix-org/synapse/issues/10925 ))
- Fix Debian builds due to `dh-virtualenv` no longer being able to build
their docs.
([\#10931](https://github.com/matrix-org/synapse/issues/10931 ))
F124520CEEE062448FE1C8442D2EFA2F32FBE047
<olivier@librepush.net>" [ultimate]
2021-09-29 14:23:39 +01:00
176aa55fd5
add event id to logcontext when handling incoming PDUs ( #10936 )
2021-09-29 11:59:43 +01:00
e32b9f44ee
Update installation instructions ( #10919 )
...
Various updates to the install docs.
2021-09-29 10:57:53 +00:00
94b620a5ed
Use direct references for configuration variables (part 6). ( #10916 )
2021-09-29 06:44:15 -04:00
8cef1ab2ac
Implement MSC3069: Guest support on whoami ( #9655 )
2021-09-29 11:32:45 +01:00
5279b9161b
Use `RoomVersion` objects ( #10934 )
...
Various refactors to use `RoomVersion` objects instead of room version identifiers.
2021-09-29 10:57:10 +01:00
2be0fde3d6
Fix empty `url_cache_thumbnails/yyyy-mm-dd/` directories being left behind ( #10924 )
2021-09-29 10:24:37 +01:00
9fd057b8c5
Ensure `(room_id, next_batch_id)` is unique to avoid cross-talk/conflicts between batches (MSC2716) ( #10877 )
...
Part of [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 )
Part of https://github.com/matrix-org/synapse/issues/10737
2021-09-28 21:23:16 -05:00
62800a8fe3
Add #10932 to release
2021-09-28 17:32:31 +01:00
0f007fe009
Update utility code to handle C implementations of frozendict ( #10902 )
...
* update _handle_frozendict to work with c implementations of frozen dict
* add changelog
* add clarifying comment to _handle_frozendict
2021-09-28 09:13:23 -07:00
8aaa4b7b5d
Drop backwards-compatibility support for "outlier" ( #10903 )
...
Before Synapse 1.31 (#9411 ), we relied on `outlier` being stored in the
`internal_metadata` column. We can now assume nobody will roll back their
deployment that far and drop the legacy support.
2021-09-28 15:25:36 +01:00
2622b28c5c
Inline `_check_event_auth` for outliers ( #10926 )
...
* Inline `_check_event_auth` for outliers
When we are persisting an outlier, most of `_check_event_auth` is redundant:
* `_update_auth_events_and_context_for_auth` does nothing, because the
`input_auth_events` are (now) exactly the event's auth_events,
which means that `missing_auth` is empty.
* we don't care about soft-fail, kicking guest users or `send_on_behalf_of`
for outliers
... so the only thing that matters is the auth itself, so let's just do that.
* `_auth_and_persist_fetched_events_inner`: de-async `prep`
`prep` no longer calls any `async` methods, so let's make it synchronous.
* Simplify `_check_event_auth`
We no longer need to support outliers here, which makes things rather simpler.
* changelog
* lint
2021-09-28 15:25:07 +01:00
37bb93d181
Fix exception responding to request that has been closed ( #10932 )
...
Introduced in #10905
2021-09-28 14:36:19 +01:00
eb2c7e51c4
Clean-up type hints in server config ( #10915 )
...
By using attrs instead of dicts to store configuration.
Also updates some of the attrs classes to use proper type
hints and auto_attribs.
2021-09-28 09:24:40 -04:00
c3ccad7785
Only do restricted join rules signature checks for room versions 8/9. ( #10927 )
...
Otherwise the presence of a (bogus, unused) field could cause
auth checks to fail.
2021-09-28 08:44:19 -04:00
3c50192d3f
1.44.0rc1
2021-09-28 13:42:21 +01:00
a8bbf08576
Fix debian package builds. ( #10931 )
...
This was due to dh-virtualenv builds being broken due to Shpinx removing
deprecated APIs.
2021-09-28 12:13:51 +01:00
707d5e4e48
Encode JSON responses on a thread in C, mk2 ( #10905 )
...
Currently we use `JsonEncoder.iterencode` to write JSON responses, which ensures that we don't block the main reactor thread when encoding huge objects. The downside to this is that `iterencode` falls back to using a pure Python encoder that is *much* less efficient and can easily burn a lot of CPU for huge responses. To fix this, while still ensuring we don't block the reactor loop, we encode the JSON on a threadpool using the standard `JsonEncoder.encode` functions, which is backed by a C library.
Doing so, however, requires `respond_with_json` to have access to the reactor, which it previously didn't. There are two ways of doing this:
1. threading through the reactor object, which is a bit fiddly as e.g. `DirectServeJsonResource` doesn't currently take a reactor, but is exposed to modules and so is a PITA to change; or
2. expose the reactor in `SynapseRequest`, which requires updating a bunch of servlet types.
I went with the latter as that is just a mechanical change, and I think makes sense as a request already has a reactor associated with it (via its http channel).
2021-09-28 09:37:58 +00:00
d37841787a
Sign the git tag in release script ( #10925 )
2021-09-27 15:39:49 +01:00
f7768f62cb
Avoid storing URL cache files in storage providers ( #10911 )
...
URL cache files are short-lived and it does not make sense to offload
them (eg. to the cloud) or back them up.
2021-09-27 12:55:27 +01:00
6c83c27107
Fix race conditions when creating media store and config directories ( #10913 )
2021-09-27 11:29:23 +01:00
d138187045
Document changes to schema version 61 - 64 ( #10917 )
...
As pointed out by @richvdh, https://github.com/matrix-org/synapse/pull/10838#discussion_r715424244
Retroactively summarize `61` - `64`
2021-09-24 17:09:12 -05:00
b10257e879
Add a spamchecker callback to allow or deny room creation based on invites ( #10898 )
...
This is in the context of creating new module callbacks that modules in https://github.com/matrix-org/synapse-dinsic can use, in an effort to reconcile the spam checker API in synapse-dinsic with the one in mainline.
This adds a callback that's fairly similar to user_may_create_room except it also allows processing based on the invites sent at room creation.
2021-09-24 16:38:23 +02:00
ea01d4c2de
Update postgresql testing script ( #10906 )
...
- Use sytest:bionic. Sytest:latest is two years old (do we want
CI to push out latest at all?) and comes with Python 3.5, which we
explictly no longer support. The script now runs under PostgreSQL 10
as a result.
- Advertise script in the docs
- Move pg testing script to scripts-dev directory
- Write to host as the script's exector, not root
A few changes to make it speedier to re-run the tests:
- Create blank DB in the container, not the script, so we don't have to
`initdb` each time
- Use a named volume to persist the tox environment, so we don't have to
fetch and install a bunch of packages from PyPI each time
Co-authored-by: reivilibre <olivier@librepush.net>
2021-09-24 14:27:09 +00:00
0420d4e6a5
Stop trying to auth/persist events whose auth events we do not have. ( #10907 )
2021-09-24 14:01:45 +01:00
bb7fdd821b
Use direct references for configuration variables (part 5). ( #10897 )
2021-09-24 07:25:21 -04:00
85551b7a85
Factor out common code for persisting fetched auth events ( #10896 )
...
* Factor more stuff out of `_get_events_and_persist`
It turns out that the event-sorting algorithm in `_get_events_and_persist` is
also useful in other circumstances. Here we move the current
`_auth_and_persist_fetched_events` to `_auth_and_persist_fetched_events_inner`,
and then factor the sorting part out to `_auth_and_persist_fetched_events`.
* `_get_remote_auth_chain_for_event`: remove redundant `outlier` assignment
`get_event_auth` returns events with the outlier flag already set, so this is
redundant (though we need to update a test where `get_event_auth` is mocked).
* `_get_remote_auth_chain_for_event`: move existing-event tests earlier
Move a couple of tests outside the loop. This is a bit inefficient for now, but
a future commit will make it better. It should be functionally identical.
* `_get_remote_auth_chain_for_event`: use `_auth_and_persist_fetched_events`
We can use the same codepath for persisting the events fetched as part of an
auth chain as for those fetched individually by `_get_events_and_persist` for
building the state at a backwards extremity.
* `_get_remote_auth_chain_for_event`: use a dict for efficiency
`_auth_and_persist_fetched_events` sorts the events itself, so we no longer
need to care about maintaining the ordering from `get_event_auth` (and no
longer need to sort by depth in `get_event_auth`).
That means that we can use a map, making it easier to filter out events we
already have, etc.
* changelog
* `_auth_and_persist_fetched_events`: improve docstring
2021-09-24 11:56:33 +01:00
261c9763c4
Simplify `_auth_and_persist_fetched_events` ( #10901 )
...
Combine the two loops over the list of events, and hence get rid of
`_NewEventInfo`. Also pass the event back alongside the context, so that it's
easier to process the result.
2021-09-24 11:56:13 +01:00
50022cff96
Add reactor to `SynapseRequest` and fix up types. ( #10868 )
2021-09-24 11:01:25 +01:00
fa74536384
Fix AuthBlocking check when requester is appservice ( #10881 )
...
If the MAU count had been reached, Synapse incorrectly blocked appservice users even though they've been explicitly configured not to be tracked (the default). This was due to bypassing the relevant if as it was chained behind another earlier hit if as an elif.
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2021-09-24 10:41:18 +01:00
7f3352743e
Improve typing in user_directory files ( #10891 )
...
* Improve typing in user_directory files
This makes the user_directory.py in storage pass most of mypy's
checks (including `no-untyped-defs`). Unfortunately that file is in the
tangled web of Store class inheritance so doesn't pass mypy at the moment.
The handlers directory has already been mypyed.
Co-authored-by: reivilibre <olivier@librepush.net>
2021-09-24 10:38:22 +01:00
e704cc2a48
In `_purge_history_txn`, ensure that txn.fetchall has elements before accessing rows ( #10690 )
...
This change adds a check for row existence before accessing row element, this should fix issue #10669
Signed-off-by: Vasya Boytsov vasiliy.boytsov@phystech.edu
2021-09-24 09:19:51 +00:00
90d9fc7505
Allow `.` and `~` chars in registration tokens ( #10887 )
...
Per updates to MSC3231 in order to use the same grammar
as other identifiers.
2021-09-23 17:58:12 +00:00
a7304adc7d
Factor out `_get_remote_auth_chain_for_event` from `_update_auth_events_and_context_for_auth` ( #10884 )
...
* Reload auth events from db after fetching and persisting
In `_update_auth_events_and_context_for_auth`, when we fetch the remote auth
tree and persist the returned events: load the missing events from the database
rather than using the copies we got from the remote server.
This is mostly in preparation for additional refactors, but does have an
advantage in that if we later get around to checking the rejected status, we'll
be able to make use of it.
* Factor out `_get_remote_auth_chain_for_event` from `_update_auth_events_and_context_for_auth`
* changelog
2021-09-23 17:34:33 +01:00
47854c71e9
Use direct references for configuration variables (part 4). ( #10893 )
2021-09-23 12:03:01 -04:00
a10988983a
Break down cache expiry reasons in grafana ( #10880 )
...
A follow-up to #10829
2021-09-23 14:45:32 +01:00
dcfd864970
Fix reactivated users not being added to the user directory ( #10782 )
...
Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com>
Co-authored-by: reivilibre <olivier@librepush.net>
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-09-23 12:02:13 +00:00
e584534403
Use direct references for some configuration variables (part 3) ( #10885 )
...
This avoids the overhead of searching through the various
configuration classes by directly referencing the class that
the attributes are in.
It also improves type hints since mypy can now resolve the
types of the configuration variables.
2021-09-23 07:13:34 -04:00
aa2c027792
Remove unnecessary parentheses around tuples returned from methods ( #10889 )
2021-09-23 11:59:07 +01:00
26f2bfedbf
Factor out a separate `EventContext.for_outlier` ( #10883 )
...
Constructing an EventContext for an outlier is actually really simple, and
there's no sense in going via an `async` method in the `StateHandler`.
This also means that we can resolve a bunch of FIXMEs.
2021-09-22 17:58:57 +01:00
f78b68a96b
Treat "\u0000" as "\u0020" for the purposes of message search (message indexing) ( #10820 )
...
* add test to check if null code points are being inserted
* add logic to detect and replace null code points before insertion into db
* lints
* add license to test
* change approach to null substitution
* add type hint for SearchEntry
* Add changelog entry
Signed-off-by: H.Shay <shaysquared@gmail.com>
* updated changelog
* update chanelog message
* remove duplicate changelog
* Update synapse/storage/databases/main/events.py remove extra space
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* rename and move test file, update tests, delete old test file
* fix typo in comments
* update _find_highlights_in_postgres to replace null byte with space
* replace null byte in sqlite search insertion
* beef up and reorganize test for this pr
* update changelog
* add type hints and update docstring
* check db engine directly vs using env variable
* refactor tests to be less repetetive
* move rplace logic into seperate function
* requested changes
* Fix typo.
* Update synapse/storage/databases/main/search.py
Co-authored-by: reivilibre <olivier@librepush.net>
* Update changelog.d/10820.misc
Co-authored-by: Aaron Raimist <aaron@raim.ist>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: reivilibre <olivier@librepush.net>
Co-authored-by: Aaron Raimist <aaron@raim.ist>
2021-09-22 08:25:26 -07:00
03db6701d5
Fix invalidating OTK count cache after claim ( #10875 )
...
The invalidation was missing in `_claim_e2e_one_time_key_returning`,
which is used on SQLite 3.24+ and Postgres. This could break e2ee if
nothing else happened to invalidate the caches before the keys ran out.
Signed-off-by: Tulir Asokan <tulir@beeper.com>
2021-09-22 15:31:05 +01:00
8f2a52766b
Ensure we mark sent knocks as outliers ( #10873 )
2021-09-22 15:20:18 +01:00
6fc8be9a1b
Include more information in oEmbed previews. ( #10819 )
...
* Improved titles (fall back to the author name if there's not title) and include the site name.
* Handle photo/video payloads.
* Include the original URL in the Open Graph response.
* Fix the expiration time (by properly converting from seconds to milliseconds).
2021-09-22 09:45:20 -04:00
9391de3f37
Fix /initialSync error due to unhashable `RoomStreamToken` ( #10827 )
...
The deprecated /initialSync endpoint maintains a cache of responses,
using parameter values as part of the cache key. When a `from` or `to`
parameter is specified, it gets converted into a `StreamToken`, which
contains a `RoomStreamToken` and forms part of the cache key.
`RoomStreamToken`s need to be made hashable for this to work.
2021-09-22 14:43:26 +01:00
52913d56a5
Add documentation for experimental feature flags. ( #10865 )
2021-09-22 13:41:42 +00:00
724aef9a87
Opt out of cache expiry for `get_users_who_share_room_with_user` ( #10826 )
...
* Allow LruCaches to opt out of time-based expiry
* Don't expire `get_users_who_share_room` & friends
2021-09-22 14:21:58 +01:00
80828eda06
Extend ModuleApi with the methods we'll need to reject spam based on …IP - resolves #10832 ( #10833 )
...
Extend ModuleApi with the methods we'll need to reject spam based on IP - resolves #10832
Signed-off-by: David Teller <davidt@element.io>
2021-09-22 13:09:43 +00:00
4ecf51812e
Include outlier status in `str(event)` for V2/V3 events ( #10879 )
...
I meant to do this before, in #10591 , but because I'm stupid I forgot to do it
for V2 and V3 events.
I've factored the common code out to `EventBase` to save us having two copies
of it.
This means that for `FrozenEvent` we replace `self.get("event_id", None)` with
`self.event_id`, which I think is safe. `get()` is an alias for
`self._dict.get()`, whereas `event_id()` is an `@property` method which looks
up `self._event_id`, which is populated during construction from the same
dict. We don't seem to rely on the fallback, because if the `event_id` key is
absent from the dict then construction of the `EventBase` object will
fail.
Long story short, the only way this could change behaviour is if
`event_dict["event_id"]` is changed *after* the `EventBase` object is
constructed without updating the `_event_id` field, or vice versa - either of
which would be very problematic anyway and the behavior of `str(event)` is the
least of our worries.
2021-09-22 12:30:59 +01:00
a2d7195e01
Track why we're evicting from caches ( #10829 )
...
So we can see distinguish between "evicting because the cache is too big" and "evicting because the cache entries haven't been recently used".
2021-09-22 10:59:52 +01:00
51e2db3598
Rename MSC2716 things from `chunk` to `batch` to match `/batch_send` endpoint ( #10838 )
...
See https://github.com/matrix-org/matrix-doc/pull/2716#discussion_r684574497
Dropping support for older MSC2716 room versions so we don't have to worry about
supporting both chunk and batch events.
2021-09-21 15:06:28 -05:00
4054dfa409
Add type hints for event streams. ( #10856 )
2021-09-21 13:34:26 -04:00
b25a494779
Add types to http.site ( #10867 )
2021-09-21 16:41:27 +00:00
ebd8baf61f
Clear our destination directories before copying files to GitHub pages. ( #10869 )
...
This should fix stale deleted files being still accessible.
2021-09-21 16:32:46 +00:00
ba7a91aea5
Refactor oEmbed previews ( #10814 )
...
The major change is moving the decision of whether to use oEmbed
further up the call-stack. This reverts the _download_url method to
being a "dumb" functionwhich takes a single URL and downloads it
(as it was before #7920 ).
This also makes more minor refactorings:
* Renames internal variables for clarity.
* Factors out shared code between the HTML and rich oEmbed
previews.
* Fixes tests to preview an oEmbed image.
2021-09-21 16:09:57 +00:00
2843058a8b
Test that state events sent by modules correctly end up in the room's state ( #10835 )
...
Test for #10830
Ideally the test would also make sure the new state event comes down sync, but this is probably good enough.
2021-09-21 17:40:20 +02:00
5fca3c8ae6
Allow Synapse Admin API's Room Search to accept non-ASCII characters ( #10859 )
...
* add tests for checking if room search works with non-ascii char
* change encoding on parse_string to UTF-8
* lints
* properly encode search term
* lints
* add changelog file
* update changelog number
* set changelog entry filetype to .bugfix
* Revert "set changelog entry filetype to .bugfix"
This reverts commit be8e5a3142
2021-09-21 08:04:35 -07:00
ee557b5375
Rename `/batch_send` query parameter from `?prev_event` to more obvious usage with `?prev_event_id` (MSC2716) ( #10839 )
...
As mentioned in https://github.com/matrix-org/matrix-doc/pull/2716#discussion_r705872887
and https://github.com/matrix-org/synapse/issues/10737
2021-09-21 14:10:01 +01:00
60453315bd
Always add local users to the user directory ( #10796 )
...
It's a simplification, but one that'll help make the user directory logic easier
to follow with the other changes upcoming. It's not strictly required for those
changes, but this will help simplify the resulting logic that listens for
`m.room.member` events and generally make the logic easier to follow.
This means the config option `search_all_users` ends up controlling the
search query only, and not the data we store. The cost of doing so is an
extra row in the `user_directory` and `user_directory_search` tables for
each local user which
- belongs to no public rooms
- belongs to no private rooms of size ≥ 2
I think the cost of this will be marginal (since they'll already have entries
in `users` and `profiles` anyway).
As a small upside, a homeserver whose directory was built with this
change can toggle `search_all_users` without having to rebuild their
directory.
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-09-21 12:02:34 +00:00
6a751ff5e0
Allow sending a membership event to unban a user ( #10807 )
...
* Allow membership event to unban user
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-09-21 10:23:34 +00:00
f455b0e420
GHA: reintroduce an env var for `$GITHUB_HEAD_REF` ( #10659 )
...
This should ensure GHA runs synapse against the same-named sytest branch
2021-09-20 17:35:16 +01:00
b3590614da
Require type hints in the handlers module. ( #10831 )
...
Adds missing type hints to methods in the synapse.handlers
module and requires all methods to have type hints there.
This also removes the unused construct_auth_difference method
from the FederationHandler.
2021-09-20 08:56:23 -04:00
437961744c
Fix remove_stale_pushers job on SQLite. ( #10843 )
2021-09-20 10:26:13 +01:00
6b6bb81b23
Fix #10837 by adding JSON encoding/decoding to the Module API example… ( #10845 )
2021-09-17 18:04:37 +01:00
bfb4b858a9
Create a constant for a small png image in tests. ( #10834 )
...
To avoid duplicating it between a few tests.
2021-09-16 12:01:14 -04:00
3eba047d38
Add type hints to state database module. ( #10823 )
2021-09-15 09:54:13 -04:00
b93259082c
Add missing type hints to non-client REST servlets. ( #10817 )
...
Including admin, consent, key, synapse, and media. All REST servlets
(the synapse.rest module) now require typed method definitions.
2021-09-15 08:45:32 -04:00
8c7a531e27
Use direct references for some configuration variables (part 2) ( #10812 )
2021-09-15 08:34:52 -04:00
145c006ef7
Verify `?chunk_id` actually corresponds to an insertion event that exists (MSC2716) ( #10776 )
2021-09-15 09:34:30 +01:00
1c555527b3
Split out `/batch_send` meta events to their own fields (MSC2716) ( #10777 )
2021-09-15 09:30:58 +01:00
8eb7cb2e0d
Make StateFilter frozen so we can hash it ( #10816 )
...
Also enables Mypy for related tests.
2021-09-14 16:35:53 +01:00
14b8c0476f
Prevent logging context going missing on federation request timeout ( #10810 )
...
In `MatrixFederationHttpClient._send_request()`, we make a HTTP request
using an `Agent`, wrap that request in a timeout and await the resulting
`Deferred`. On its own, the `Agent` performing the HTTP request
correctly stashes and restores the logging context while waiting.
The addition of the timeout introduces a path where the logging context
is not restored when execution resumes.
To address this, we wrap the timeout `Deferred` in a
`make_deferred_yieldable()` to stash the logging context and restore it
on completion of the `await`. However this is not sufficient, since by
the time we construct the timeout `Deferred`, the `Agent` has already
stashed and cleared the logging context when using
`make_deferred_yieldable()` to produce its `Deferred` for the request.
Hence, we wrap the `Agent` request in a `run_in_background()` to "fork"
and preserve the logging context so that we can stash and restore it
when `await`ing the timeout `Deferred`.
This approach is similar to the one used with `defer.gatherResults`.
Note that the code is still not fully correct. When a timeout occurs,
the request remains running in the background (existing behavior which
is nothing to do with the new call to `run_in_background`) and may
re-start the logging context after it has finished.
2021-09-14 13:01:30 +01:00
51e1b96d04
Synapse 1.43.0rc1 (2021-09-14)
...
This release drops support for the deprecated, unstable API for [MSC2858](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix ), as well as the undocumented `experimental.msc2858_enabled` config option. Client authors should update their clients to use the stable API, available since Synapse 1.30.
Features
--------
- Allow room creators to send historical events specified by [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) in existing room versions. ([\#10566](https://github.com/matrix-org/synapse/issues/10566 ))
- Add config option to use non-default manhole password and keys. ([\#10643](https://github.com/matrix-org/synapse/issues/10643 ))
- Skip final GC at shutdown to improve restart performance. ([\#10712](https://github.com/matrix-org/synapse/issues/10712 ))
- Allow configuration of the oEmbed URLs used for URL previews. ([\#10714](https://github.com/matrix-org/synapse/issues/10714 ), [\#10759](https://github.com/matrix-org/synapse/issues/10759 ))
- Prefer [room version 9](https://github.com/matrix-org/matrix-doc/pull/3375 ) for restricted rooms per the [room version capabilities](https://github.com/matrix-org/matrix-doc/pull/3244 ) API. ([\#10772](https://github.com/matrix-org/synapse/issues/10772 ))
Bugfixes
--------
- Fix a long-standing bug where room avatars were not included in email notifications. ([\#10658](https://github.com/matrix-org/synapse/issues/10658 ))
- Fix a bug where the ordering algorithm was skipping the `origin_server_ts` step in the spaces summary resulting in unstable room orderings. ([\#10730](https://github.com/matrix-org/synapse/issues/10730 ))
- Fix edge case when persisting events into a room where there are multiple events we previously hadn't calculated auth chains for (and hadn't marked as needing to be calculated). ([\#10743](https://github.com/matrix-org/synapse/issues/10743 ))
- Fix a bug which prevented calls to `/createRoom` that included the `room_alias_name` parameter from being handled by worker processes. ([\#10757](https://github.com/matrix-org/synapse/issues/10757 ))
- Fix a bug which prevented user registration via SSO to require consent tracking for SSO mapping providers that don't prompt for Matrix ID selection. Contributed by @AndrewFerr. ([\#10733](https://github.com/matrix-org/synapse/issues/10733 ))
- Only return the stripped state events for the `m.space.child` events in a room for the spaces summary from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946 ). ([\#10760](https://github.com/matrix-org/synapse/issues/10760 ))
- Properly handle room upgrades of spaces. ([\#10774](https://github.com/matrix-org/synapse/issues/10774 ))
- Fix a bug which generated invalid homeserver config when the `frontend_proxy` worker type was passed to the Synapse Worker-based Complement image. ([\#10783](https://github.com/matrix-org/synapse/issues/10783 ))
Improved Documentation
----------------------
- Minor fix to the `media_repository` developer documentation. Contributed by @cuttingedge1109. ([\#10556](https://github.com/matrix-org/synapse/issues/10556 ))
- Update the documentation to note that the `/spaces` and `/hierarchy` endpoints can be routed to workers. ([\#10648](https://github.com/matrix-org/synapse/issues/10648 ))
- Clarify admin API documentation on undoing room deletions. ([\#10735](https://github.com/matrix-org/synapse/issues/10735 ))
- Split up the modules documentation and add examples for module developers. ([\#10758](https://github.com/matrix-org/synapse/issues/10758 ))
- Correct 2 typographical errors in the [Log Contexts documentation](https://matrix-org.github.io/synapse/latest/log_contexts.html ). ([\#10795](https://github.com/matrix-org/synapse/issues/10795 ))
- Fix a wording mistake in the sample configuration. Contributed by @bramvdnheuvel:nltrix.net. ([\#10804](https://github.com/matrix-org/synapse/issues/10804 ))
Deprecations and Removals
-------------------------
- Remove the [unstable MSC2858 API](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix ), including the undocumented `experimental.msc2858_enabled` config option. The unstable API has been deprecated since Synapse 1.35. Client authors should update their clients to use the stable API introduced in Synapse 1.30 if they have not already done so. ([\#10693](https://github.com/matrix-org/synapse/issues/10693 ))
Internal Changes
----------------
- Add OpenTracing logging to help debug stuck messages (as described by issue [#9424 ](https://github.com/matrix-org/synapse/issues/9424 )). ([\#10704](https://github.com/matrix-org/synapse/issues/10704 ))
- Add type annotations to the `synapse.util` package. ([\#10601](https://github.com/matrix-org/synapse/issues/10601 ))
- Ensure `rooms.creator` field is always populated for easy lookup in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) usage later. ([\#10697](https://github.com/matrix-org/synapse/issues/10697 ))
- Add missing type hints to REST servlets. ([\#10707](https://github.com/matrix-org/synapse/issues/10707 ), [\#10728](https://github.com/matrix-org/synapse/issues/10728 ), [\#10736](https://github.com/matrix-org/synapse/issues/10736 ))
- Do not include rooms with unknown room versions in the spaces summary results. ([\#10727](https://github.com/matrix-org/synapse/issues/10727 ))
- Additional error checking for the `preset` field when creating a room. ([\#10738](https://github.com/matrix-org/synapse/issues/10738 ))
- Clean up some of the federation event authentication code for clarity. ([\#10744](https://github.com/matrix-org/synapse/issues/10744 ), [\#10745](https://github.com/matrix-org/synapse/issues/10745 ), [\#10746](https://github.com/matrix-org/synapse/issues/10746 ), [\#10771](https://github.com/matrix-org/synapse/issues/10771 ), [\#10773](https://github.com/matrix-org/synapse/issues/10773 ), [\#10781](https://github.com/matrix-org/synapse/issues/10781 ))
- Add an index to `presence_stream` to hopefully speed up startups a little. ([\#10748](https://github.com/matrix-org/synapse/issues/10748 ))
- Refactor event size checking code to simplify searching the codebase for the origins of certain error strings that are occasionally emitted. ([\#10750](https://github.com/matrix-org/synapse/issues/10750 ))
- Move tests relating to rooms having encryption out of the user directory tests. ([\#10752](https://github.com/matrix-org/synapse/issues/10752 ))
- Use `attrs` internally for the URL preview code & update documentation. ([\#10753](https://github.com/matrix-org/synapse/issues/10753 ))
- Minor speed ups when joining large rooms over federation. ([\#10754](https://github.com/matrix-org/synapse/issues/10754 ), [\#10755](https://github.com/matrix-org/synapse/issues/10755 ), [\#10756](https://github.com/matrix-org/synapse/issues/10756 ), [\#10780](https://github.com/matrix-org/synapse/issues/10780 ), [\#10784](https://github.com/matrix-org/synapse/issues/10784 ))
- Add a constant for `m.federate`. ([\#10775](https://github.com/matrix-org/synapse/issues/10775 ))
- Add a script to update the Debian changelog in a Docker container for systems that are not Debian-based. ([\#10778](https://github.com/matrix-org/synapse/issues/10778 ))
- Change the format of authenticated users in logs when a user is being puppeted by and admin user. ([\#10779](https://github.com/matrix-org/synapse/issues/10779 ))
- Remove fixed and flakey tests from the Sytest blacklist. ([\#10788](https://github.com/matrix-org/synapse/issues/10788 ))
- Improve internal details of the user directory code. ([\#10789](https://github.com/matrix-org/synapse/issues/10789 ))
- Use direct references to config flags. ([\#10798](https://github.com/matrix-org/synapse/issues/10798 ))
- Ensure the Rust reporter passes type checking with jaeger-client 4.7's type annotations. ([\#10799](https://github.com/matrix-org/synapse/issues/10799 ))
-----BEGIN PGP SIGNATURE-----
iQJHBAABCAAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmFAgvcTHGFuZHJld0Bh
bW9yZ2FuLnh5egAKCRCIhIgNLv5f9M6REACpRcTVbsp9Z3c6cajrShHWZr7o4z9y
9gjs38mELM7aysP1rcePsYWgsJtG66YCvmErEjuBGBKOSpuQCTRs+2fDC0m5bpia
Av+LzLyWM8lK2vaeg93xP5aBK7Nspkst3XW7Ststkcu25rZFkR/6gU1dE0+YXVbH
wlohqRGA085cZy2xGy+PalwuY8YWcMI2Vw2T+4pCxM7Awly6uK5ieghppKPZ4Xns
Z6rklnkrsf2yaCHzMZYq6TkCiZaJeXyGAxiBEPCrvOvBgTcfX53ZMa2YESOROgil
b9U2YUFa6d0V2vl1KZoSri/fafhY3LC1H+GJvxNeUFFOo1spKKOluNWPaxvZXQDk
+7MKdIzYp2UHyqn2F9rAf213RmGFvh3QDMxXkjUqlJby/mcxzMYsshTBPExcOR44
/XOOhrQRuOAbYttikXRldEta5R6oST9TKJ6ft0HZyYu0VFS2n5RM/HjPXfCZDPBk
0R6l1GjRf3YYYNu+9XZ1o1nZ49RgOHe3p+Ar5istzgbCL3/7nVo7lnCljldFz6ol
xQUdPCnGA0fOWins7o84gQw1fzvcbEHDlL/45VvwR6y5r3ZvbxFa17Vw1s1eKYhT
AglZvUhy8nXzYrasSagQ/a3xWMOcdpiyHyLCEHvpYdYWEbblYNmihivOrzffJsqd
DaRSoFyMrMI53A==
=fq25
-----END PGP SIGNATURE-----
Merge tag 'v1.43.0rc1' into develop
Synapse 1.43.0rc1 (2021-09-14)
This release drops support for the deprecated, unstable API for [MSC2858](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix ), as well as the undocumented `experimental.msc2858_enabled` config option. Client authors should update their clients to use the stable API, available since Synapse 1.30.
Features
--------
- Allow room creators to send historical events specified by [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) in existing room versions. ([\#10566](https://github.com/matrix-org/synapse/issues/10566 ))
- Add config option to use non-default manhole password and keys. ([\#10643](https://github.com/matrix-org/synapse/issues/10643 ))
- Skip final GC at shutdown to improve restart performance. ([\#10712](https://github.com/matrix-org/synapse/issues/10712 ))
- Allow configuration of the oEmbed URLs used for URL previews. ([\#10714](https://github.com/matrix-org/synapse/issues/10714 ), [\#10759](https://github.com/matrix-org/synapse/issues/10759 ))
- Prefer [room version 9](https://github.com/matrix-org/matrix-doc/pull/3375 ) for restricted rooms per the [room version capabilities](https://github.com/matrix-org/matrix-doc/pull/3244 ) API. ([\#10772](https://github.com/matrix-org/synapse/issues/10772 ))
Bugfixes
--------
- Fix a long-standing bug where room avatars were not included in email notifications. ([\#10658](https://github.com/matrix-org/synapse/issues/10658 ))
- Fix a bug where the ordering algorithm was skipping the `origin_server_ts` step in the spaces summary resulting in unstable room orderings. ([\#10730](https://github.com/matrix-org/synapse/issues/10730 ))
- Fix edge case when persisting events into a room where there are multiple events we previously hadn't calculated auth chains for (and hadn't marked as needing to be calculated). ([\#10743](https://github.com/matrix-org/synapse/issues/10743 ))
- Fix a bug which prevented calls to `/createRoom` that included the `room_alias_name` parameter from being handled by worker processes. ([\#10757](https://github.com/matrix-org/synapse/issues/10757 ))
- Fix a bug which prevented user registration via SSO to require consent tracking for SSO mapping providers that don't prompt for Matrix ID selection. Contributed by @AndrewFerr. ([\#10733](https://github.com/matrix-org/synapse/issues/10733 ))
- Only return the stripped state events for the `m.space.child` events in a room for the spaces summary from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946 ). ([\#10760](https://github.com/matrix-org/synapse/issues/10760 ))
- Properly handle room upgrades of spaces. ([\#10774](https://github.com/matrix-org/synapse/issues/10774 ))
- Fix a bug which generated invalid homeserver config when the `frontend_proxy` worker type was passed to the Synapse Worker-based Complement image. ([\#10783](https://github.com/matrix-org/synapse/issues/10783 ))
Improved Documentation
----------------------
- Minor fix to the `media_repository` developer documentation. Contributed by @cuttingedge1109. ([\#10556](https://github.com/matrix-org/synapse/issues/10556 ))
- Update the documentation to note that the `/spaces` and `/hierarchy` endpoints can be routed to workers. ([\#10648](https://github.com/matrix-org/synapse/issues/10648 ))
- Clarify admin API documentation on undoing room deletions. ([\#10735](https://github.com/matrix-org/synapse/issues/10735 ))
- Split up the modules documentation and add examples for module developers. ([\#10758](https://github.com/matrix-org/synapse/issues/10758 ))
- Correct 2 typographical errors in the [Log Contexts documentation](https://matrix-org.github.io/synapse/latest/log_contexts.html ). ([\#10795](https://github.com/matrix-org/synapse/issues/10795 ))
- Fix a wording mistake in the sample configuration. Contributed by @bramvdnheuvel:nltrix.net. ([\#10804](https://github.com/matrix-org/synapse/issues/10804 ))
Deprecations and Removals
-------------------------
- Remove the [unstable MSC2858 API](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix ), including the undocumented `experimental.msc2858_enabled` config option. The unstable API has been deprecated since Synapse 1.35. Client authors should update their clients to use the stable API introduced in Synapse 1.30 if they have not already done so. ([\#10693](https://github.com/matrix-org/synapse/issues/10693 ))
Internal Changes
----------------
- Add OpenTracing logging to help debug stuck messages (as described by issue [#9424 ](https://github.com/matrix-org/synapse/issues/9424 )). ([\#10704](https://github.com/matrix-org/synapse/issues/10704 ))
- Add type annotations to the `synapse.util` package. ([\#10601](https://github.com/matrix-org/synapse/issues/10601 ))
- Ensure `rooms.creator` field is always populated for easy lookup in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) usage later. ([\#10697](https://github.com/matrix-org/synapse/issues/10697 ))
- Add missing type hints to REST servlets. ([\#10707](https://github.com/matrix-org/synapse/issues/10707 ), [\#10728](https://github.com/matrix-org/synapse/issues/10728 ), [\#10736](https://github.com/matrix-org/synapse/issues/10736 ))
- Do not include rooms with unknown room versions in the spaces summary results. ([\#10727](https://github.com/matrix-org/synapse/issues/10727 ))
- Additional error checking for the `preset` field when creating a room. ([\#10738](https://github.com/matrix-org/synapse/issues/10738 ))
- Clean up some of the federation event authentication code for clarity. ([\#10744](https://github.com/matrix-org/synapse/issues/10744 ), [\#10745](https://github.com/matrix-org/synapse/issues/10745 ), [\#10746](https://github.com/matrix-org/synapse/issues/10746 ), [\#10771](https://github.com/matrix-org/synapse/issues/10771 ), [\#10773](https://github.com/matrix-org/synapse/issues/10773 ), [\#10781](https://github.com/matrix-org/synapse/issues/10781 ))
- Add an index to `presence_stream` to hopefully speed up startups a little. ([\#10748](https://github.com/matrix-org/synapse/issues/10748 ))
- Refactor event size checking code to simplify searching the codebase for the origins of certain error strings that are occasionally emitted. ([\#10750](https://github.com/matrix-org/synapse/issues/10750 ))
- Move tests relating to rooms having encryption out of the user directory tests. ([\#10752](https://github.com/matrix-org/synapse/issues/10752 ))
- Use `attrs` internally for the URL preview code & update documentation. ([\#10753](https://github.com/matrix-org/synapse/issues/10753 ))
- Minor speed ups when joining large rooms over federation. ([\#10754](https://github.com/matrix-org/synapse/issues/10754 ), [\#10755](https://github.com/matrix-org/synapse/issues/10755 ), [\#10756](https://github.com/matrix-org/synapse/issues/10756 ), [\#10780](https://github.com/matrix-org/synapse/issues/10780 ), [\#10784](https://github.com/matrix-org/synapse/issues/10784 ))
- Add a constant for `m.federate`. ([\#10775](https://github.com/matrix-org/synapse/issues/10775 ))
- Add a script to update the Debian changelog in a Docker container for systems that are not Debian-based. ([\#10778](https://github.com/matrix-org/synapse/issues/10778 ))
- Change the format of authenticated users in logs when a user is being puppeted by and admin user. ([\#10779](https://github.com/matrix-org/synapse/issues/10779 ))
- Remove fixed and flakey tests from the Sytest blacklist. ([\#10788](https://github.com/matrix-org/synapse/issues/10788 ))
- Improve internal details of the user directory code. ([\#10789](https://github.com/matrix-org/synapse/issues/10789 ))
- Use direct references to config flags. ([\#10798](https://github.com/matrix-org/synapse/issues/10798 ))
- Ensure the Rust reporter passes type checking with jaeger-client 4.7's type annotations. ([\#10799](https://github.com/matrix-org/synapse/issues/10799 ))
2021-09-14 12:59:53 +01:00
b996782df5
Convert media repo's FileInfo to attrs. ( #10785 )
...
This is mostly an internal change, but improves type hints in the
media code.
2021-09-14 07:09:38 -04:00
Hillery Shay