bfb4b858a9
Create a constant for a small png image in tests. ( #10834 )
...
To avoid duplicating it between a few tests.
2021-09-16 12:01:14 -04:00
3eba047d38
Add type hints to state database module. ( #10823 )
2021-09-15 09:54:13 -04:00
b93259082c
Add missing type hints to non-client REST servlets. ( #10817 )
...
Including admin, consent, key, synapse, and media. All REST servlets
(the synapse.rest module) now require typed method definitions.
2021-09-15 08:45:32 -04:00
8c7a531e27
Use direct references for some configuration variables (part 2) ( #10812 )
2021-09-15 08:34:52 -04:00
145c006ef7
Verify `?chunk_id` actually corresponds to an insertion event that exists (MSC2716) ( #10776 )
2021-09-15 09:34:30 +01:00
1c555527b3
Split out `/batch_send` meta events to their own fields (MSC2716) ( #10777 )
2021-09-15 09:30:58 +01:00
8eb7cb2e0d
Make StateFilter frozen so we can hash it ( #10816 )
...
Also enables Mypy for related tests.
2021-09-14 16:35:53 +01:00
14b8c0476f
Prevent logging context going missing on federation request timeout ( #10810 )
...
In `MatrixFederationHttpClient._send_request()`, we make a HTTP request
using an `Agent`, wrap that request in a timeout and await the resulting
`Deferred`. On its own, the `Agent` performing the HTTP request
correctly stashes and restores the logging context while waiting.
The addition of the timeout introduces a path where the logging context
is not restored when execution resumes.
To address this, we wrap the timeout `Deferred` in a
`make_deferred_yieldable()` to stash the logging context and restore it
on completion of the `await`. However this is not sufficient, since by
the time we construct the timeout `Deferred`, the `Agent` has already
stashed and cleared the logging context when using
`make_deferred_yieldable()` to produce its `Deferred` for the request.
Hence, we wrap the `Agent` request in a `run_in_background()` to "fork"
and preserve the logging context so that we can stash and restore it
when `await`ing the timeout `Deferred`.
This approach is similar to the one used with `defer.gatherResults`.
Note that the code is still not fully correct. When a timeout occurs,
the request remains running in the background (existing behavior which
is nothing to do with the new call to `run_in_background`) and may
re-start the logging context after it has finished.
2021-09-14 13:01:30 +01:00
51e1b96d04
Synapse 1.43.0rc1 (2021-09-14)
...
This release drops support for the deprecated, unstable API for [MSC2858](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix ), as well as the undocumented `experimental.msc2858_enabled` config option. Client authors should update their clients to use the stable API, available since Synapse 1.30.
Features
--------
- Allow room creators to send historical events specified by [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) in existing room versions. ([\#10566](https://github.com/matrix-org/synapse/issues/10566 ))
- Add config option to use non-default manhole password and keys. ([\#10643](https://github.com/matrix-org/synapse/issues/10643 ))
- Skip final GC at shutdown to improve restart performance. ([\#10712](https://github.com/matrix-org/synapse/issues/10712 ))
- Allow configuration of the oEmbed URLs used for URL previews. ([\#10714](https://github.com/matrix-org/synapse/issues/10714 ), [\#10759](https://github.com/matrix-org/synapse/issues/10759 ))
- Prefer [room version 9](https://github.com/matrix-org/matrix-doc/pull/3375 ) for restricted rooms per the [room version capabilities](https://github.com/matrix-org/matrix-doc/pull/3244 ) API. ([\#10772](https://github.com/matrix-org/synapse/issues/10772 ))
Bugfixes
--------
- Fix a long-standing bug where room avatars were not included in email notifications. ([\#10658](https://github.com/matrix-org/synapse/issues/10658 ))
- Fix a bug where the ordering algorithm was skipping the `origin_server_ts` step in the spaces summary resulting in unstable room orderings. ([\#10730](https://github.com/matrix-org/synapse/issues/10730 ))
- Fix edge case when persisting events into a room where there are multiple events we previously hadn't calculated auth chains for (and hadn't marked as needing to be calculated). ([\#10743](https://github.com/matrix-org/synapse/issues/10743 ))
- Fix a bug which prevented calls to `/createRoom` that included the `room_alias_name` parameter from being handled by worker processes. ([\#10757](https://github.com/matrix-org/synapse/issues/10757 ))
- Fix a bug which prevented user registration via SSO to require consent tracking for SSO mapping providers that don't prompt for Matrix ID selection. Contributed by @AndrewFerr. ([\#10733](https://github.com/matrix-org/synapse/issues/10733 ))
- Only return the stripped state events for the `m.space.child` events in a room for the spaces summary from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946 ). ([\#10760](https://github.com/matrix-org/synapse/issues/10760 ))
- Properly handle room upgrades of spaces. ([\#10774](https://github.com/matrix-org/synapse/issues/10774 ))
- Fix a bug which generated invalid homeserver config when the `frontend_proxy` worker type was passed to the Synapse Worker-based Complement image. ([\#10783](https://github.com/matrix-org/synapse/issues/10783 ))
Improved Documentation
----------------------
- Minor fix to the `media_repository` developer documentation. Contributed by @cuttingedge1109. ([\#10556](https://github.com/matrix-org/synapse/issues/10556 ))
- Update the documentation to note that the `/spaces` and `/hierarchy` endpoints can be routed to workers. ([\#10648](https://github.com/matrix-org/synapse/issues/10648 ))
- Clarify admin API documentation on undoing room deletions. ([\#10735](https://github.com/matrix-org/synapse/issues/10735 ))
- Split up the modules documentation and add examples for module developers. ([\#10758](https://github.com/matrix-org/synapse/issues/10758 ))
- Correct 2 typographical errors in the [Log Contexts documentation](https://matrix-org.github.io/synapse/latest/log_contexts.html ). ([\#10795](https://github.com/matrix-org/synapse/issues/10795 ))
- Fix a wording mistake in the sample configuration. Contributed by @bramvdnheuvel:nltrix.net. ([\#10804](https://github.com/matrix-org/synapse/issues/10804 ))
Deprecations and Removals
-------------------------
- Remove the [unstable MSC2858 API](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix ), including the undocumented `experimental.msc2858_enabled` config option. The unstable API has been deprecated since Synapse 1.35. Client authors should update their clients to use the stable API introduced in Synapse 1.30 if they have not already done so. ([\#10693](https://github.com/matrix-org/synapse/issues/10693 ))
Internal Changes
----------------
- Add OpenTracing logging to help debug stuck messages (as described by issue [#9424 ](https://github.com/matrix-org/synapse/issues/9424 )). ([\#10704](https://github.com/matrix-org/synapse/issues/10704 ))
- Add type annotations to the `synapse.util` package. ([\#10601](https://github.com/matrix-org/synapse/issues/10601 ))
- Ensure `rooms.creator` field is always populated for easy lookup in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) usage later. ([\#10697](https://github.com/matrix-org/synapse/issues/10697 ))
- Add missing type hints to REST servlets. ([\#10707](https://github.com/matrix-org/synapse/issues/10707 ), [\#10728](https://github.com/matrix-org/synapse/issues/10728 ), [\#10736](https://github.com/matrix-org/synapse/issues/10736 ))
- Do not include rooms with unknown room versions in the spaces summary results. ([\#10727](https://github.com/matrix-org/synapse/issues/10727 ))
- Additional error checking for the `preset` field when creating a room. ([\#10738](https://github.com/matrix-org/synapse/issues/10738 ))
- Clean up some of the federation event authentication code for clarity. ([\#10744](https://github.com/matrix-org/synapse/issues/10744 ), [\#10745](https://github.com/matrix-org/synapse/issues/10745 ), [\#10746](https://github.com/matrix-org/synapse/issues/10746 ), [\#10771](https://github.com/matrix-org/synapse/issues/10771 ), [\#10773](https://github.com/matrix-org/synapse/issues/10773 ), [\#10781](https://github.com/matrix-org/synapse/issues/10781 ))
- Add an index to `presence_stream` to hopefully speed up startups a little. ([\#10748](https://github.com/matrix-org/synapse/issues/10748 ))
- Refactor event size checking code to simplify searching the codebase for the origins of certain error strings that are occasionally emitted. ([\#10750](https://github.com/matrix-org/synapse/issues/10750 ))
- Move tests relating to rooms having encryption out of the user directory tests. ([\#10752](https://github.com/matrix-org/synapse/issues/10752 ))
- Use `attrs` internally for the URL preview code & update documentation. ([\#10753](https://github.com/matrix-org/synapse/issues/10753 ))
- Minor speed ups when joining large rooms over federation. ([\#10754](https://github.com/matrix-org/synapse/issues/10754 ), [\#10755](https://github.com/matrix-org/synapse/issues/10755 ), [\#10756](https://github.com/matrix-org/synapse/issues/10756 ), [\#10780](https://github.com/matrix-org/synapse/issues/10780 ), [\#10784](https://github.com/matrix-org/synapse/issues/10784 ))
- Add a constant for `m.federate`. ([\#10775](https://github.com/matrix-org/synapse/issues/10775 ))
- Add a script to update the Debian changelog in a Docker container for systems that are not Debian-based. ([\#10778](https://github.com/matrix-org/synapse/issues/10778 ))
- Change the format of authenticated users in logs when a user is being puppeted by and admin user. ([\#10779](https://github.com/matrix-org/synapse/issues/10779 ))
- Remove fixed and flakey tests from the Sytest blacklist. ([\#10788](https://github.com/matrix-org/synapse/issues/10788 ))
- Improve internal details of the user directory code. ([\#10789](https://github.com/matrix-org/synapse/issues/10789 ))
- Use direct references to config flags. ([\#10798](https://github.com/matrix-org/synapse/issues/10798 ))
- Ensure the Rust reporter passes type checking with jaeger-client 4.7's type annotations. ([\#10799](https://github.com/matrix-org/synapse/issues/10799 ))
-----BEGIN PGP SIGNATURE-----
iQJHBAABCAAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmFAgvcTHGFuZHJld0Bh
bW9yZ2FuLnh5egAKCRCIhIgNLv5f9M6REACpRcTVbsp9Z3c6cajrShHWZr7o4z9y
9gjs38mELM7aysP1rcePsYWgsJtG66YCvmErEjuBGBKOSpuQCTRs+2fDC0m5bpia
Av+LzLyWM8lK2vaeg93xP5aBK7Nspkst3XW7Ststkcu25rZFkR/6gU1dE0+YXVbH
wlohqRGA085cZy2xGy+PalwuY8YWcMI2Vw2T+4pCxM7Awly6uK5ieghppKPZ4Xns
Z6rklnkrsf2yaCHzMZYq6TkCiZaJeXyGAxiBEPCrvOvBgTcfX53ZMa2YESOROgil
b9U2YUFa6d0V2vl1KZoSri/fafhY3LC1H+GJvxNeUFFOo1spKKOluNWPaxvZXQDk
+7MKdIzYp2UHyqn2F9rAf213RmGFvh3QDMxXkjUqlJby/mcxzMYsshTBPExcOR44
/XOOhrQRuOAbYttikXRldEta5R6oST9TKJ6ft0HZyYu0VFS2n5RM/HjPXfCZDPBk
0R6l1GjRf3YYYNu+9XZ1o1nZ49RgOHe3p+Ar5istzgbCL3/7nVo7lnCljldFz6ol
xQUdPCnGA0fOWins7o84gQw1fzvcbEHDlL/45VvwR6y5r3ZvbxFa17Vw1s1eKYhT
AglZvUhy8nXzYrasSagQ/a3xWMOcdpiyHyLCEHvpYdYWEbblYNmihivOrzffJsqd
DaRSoFyMrMI53A==
=fq25
-----END PGP SIGNATURE-----
Merge tag 'v1.43.0rc1' into develop
Synapse 1.43.0rc1 (2021-09-14)
This release drops support for the deprecated, unstable API for [MSC2858](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix ), as well as the undocumented `experimental.msc2858_enabled` config option. Client authors should update their clients to use the stable API, available since Synapse 1.30.
Features
--------
- Allow room creators to send historical events specified by [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) in existing room versions. ([\#10566](https://github.com/matrix-org/synapse/issues/10566 ))
- Add config option to use non-default manhole password and keys. ([\#10643](https://github.com/matrix-org/synapse/issues/10643 ))
- Skip final GC at shutdown to improve restart performance. ([\#10712](https://github.com/matrix-org/synapse/issues/10712 ))
- Allow configuration of the oEmbed URLs used for URL previews. ([\#10714](https://github.com/matrix-org/synapse/issues/10714 ), [\#10759](https://github.com/matrix-org/synapse/issues/10759 ))
- Prefer [room version 9](https://github.com/matrix-org/matrix-doc/pull/3375 ) for restricted rooms per the [room version capabilities](https://github.com/matrix-org/matrix-doc/pull/3244 ) API. ([\#10772](https://github.com/matrix-org/synapse/issues/10772 ))
Bugfixes
--------
- Fix a long-standing bug where room avatars were not included in email notifications. ([\#10658](https://github.com/matrix-org/synapse/issues/10658 ))
- Fix a bug where the ordering algorithm was skipping the `origin_server_ts` step in the spaces summary resulting in unstable room orderings. ([\#10730](https://github.com/matrix-org/synapse/issues/10730 ))
- Fix edge case when persisting events into a room where there are multiple events we previously hadn't calculated auth chains for (and hadn't marked as needing to be calculated). ([\#10743](https://github.com/matrix-org/synapse/issues/10743 ))
- Fix a bug which prevented calls to `/createRoom` that included the `room_alias_name` parameter from being handled by worker processes. ([\#10757](https://github.com/matrix-org/synapse/issues/10757 ))
- Fix a bug which prevented user registration via SSO to require consent tracking for SSO mapping providers that don't prompt for Matrix ID selection. Contributed by @AndrewFerr. ([\#10733](https://github.com/matrix-org/synapse/issues/10733 ))
- Only return the stripped state events for the `m.space.child` events in a room for the spaces summary from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946 ). ([\#10760](https://github.com/matrix-org/synapse/issues/10760 ))
- Properly handle room upgrades of spaces. ([\#10774](https://github.com/matrix-org/synapse/issues/10774 ))
- Fix a bug which generated invalid homeserver config when the `frontend_proxy` worker type was passed to the Synapse Worker-based Complement image. ([\#10783](https://github.com/matrix-org/synapse/issues/10783 ))
Improved Documentation
----------------------
- Minor fix to the `media_repository` developer documentation. Contributed by @cuttingedge1109. ([\#10556](https://github.com/matrix-org/synapse/issues/10556 ))
- Update the documentation to note that the `/spaces` and `/hierarchy` endpoints can be routed to workers. ([\#10648](https://github.com/matrix-org/synapse/issues/10648 ))
- Clarify admin API documentation on undoing room deletions. ([\#10735](https://github.com/matrix-org/synapse/issues/10735 ))
- Split up the modules documentation and add examples for module developers. ([\#10758](https://github.com/matrix-org/synapse/issues/10758 ))
- Correct 2 typographical errors in the [Log Contexts documentation](https://matrix-org.github.io/synapse/latest/log_contexts.html ). ([\#10795](https://github.com/matrix-org/synapse/issues/10795 ))
- Fix a wording mistake in the sample configuration. Contributed by @bramvdnheuvel:nltrix.net. ([\#10804](https://github.com/matrix-org/synapse/issues/10804 ))
Deprecations and Removals
-------------------------
- Remove the [unstable MSC2858 API](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix ), including the undocumented `experimental.msc2858_enabled` config option. The unstable API has been deprecated since Synapse 1.35. Client authors should update their clients to use the stable API introduced in Synapse 1.30 if they have not already done so. ([\#10693](https://github.com/matrix-org/synapse/issues/10693 ))
Internal Changes
----------------
- Add OpenTracing logging to help debug stuck messages (as described by issue [#9424 ](https://github.com/matrix-org/synapse/issues/9424 )). ([\#10704](https://github.com/matrix-org/synapse/issues/10704 ))
- Add type annotations to the `synapse.util` package. ([\#10601](https://github.com/matrix-org/synapse/issues/10601 ))
- Ensure `rooms.creator` field is always populated for easy lookup in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716 ) usage later. ([\#10697](https://github.com/matrix-org/synapse/issues/10697 ))
- Add missing type hints to REST servlets. ([\#10707](https://github.com/matrix-org/synapse/issues/10707 ), [\#10728](https://github.com/matrix-org/synapse/issues/10728 ), [\#10736](https://github.com/matrix-org/synapse/issues/10736 ))
- Do not include rooms with unknown room versions in the spaces summary results. ([\#10727](https://github.com/matrix-org/synapse/issues/10727 ))
- Additional error checking for the `preset` field when creating a room. ([\#10738](https://github.com/matrix-org/synapse/issues/10738 ))
- Clean up some of the federation event authentication code for clarity. ([\#10744](https://github.com/matrix-org/synapse/issues/10744 ), [\#10745](https://github.com/matrix-org/synapse/issues/10745 ), [\#10746](https://github.com/matrix-org/synapse/issues/10746 ), [\#10771](https://github.com/matrix-org/synapse/issues/10771 ), [\#10773](https://github.com/matrix-org/synapse/issues/10773 ), [\#10781](https://github.com/matrix-org/synapse/issues/10781 ))
- Add an index to `presence_stream` to hopefully speed up startups a little. ([\#10748](https://github.com/matrix-org/synapse/issues/10748 ))
- Refactor event size checking code to simplify searching the codebase for the origins of certain error strings that are occasionally emitted. ([\#10750](https://github.com/matrix-org/synapse/issues/10750 ))
- Move tests relating to rooms having encryption out of the user directory tests. ([\#10752](https://github.com/matrix-org/synapse/issues/10752 ))
- Use `attrs` internally for the URL preview code & update documentation. ([\#10753](https://github.com/matrix-org/synapse/issues/10753 ))
- Minor speed ups when joining large rooms over federation. ([\#10754](https://github.com/matrix-org/synapse/issues/10754 ), [\#10755](https://github.com/matrix-org/synapse/issues/10755 ), [\#10756](https://github.com/matrix-org/synapse/issues/10756 ), [\#10780](https://github.com/matrix-org/synapse/issues/10780 ), [\#10784](https://github.com/matrix-org/synapse/issues/10784 ))
- Add a constant for `m.federate`. ([\#10775](https://github.com/matrix-org/synapse/issues/10775 ))
- Add a script to update the Debian changelog in a Docker container for systems that are not Debian-based. ([\#10778](https://github.com/matrix-org/synapse/issues/10778 ))
- Change the format of authenticated users in logs when a user is being puppeted by and admin user. ([\#10779](https://github.com/matrix-org/synapse/issues/10779 ))
- Remove fixed and flakey tests from the Sytest blacklist. ([\#10788](https://github.com/matrix-org/synapse/issues/10788 ))
- Improve internal details of the user directory code. ([\#10789](https://github.com/matrix-org/synapse/issues/10789 ))
- Use direct references to config flags. ([\#10798](https://github.com/matrix-org/synapse/issues/10798 ))
- Ensure the Rust reporter passes type checking with jaeger-client 4.7's type annotations. ([\#10799](https://github.com/matrix-org/synapse/issues/10799 ))
2021-09-14 12:59:53 +01:00
b996782df5
Convert media repo's FileInfo to attrs. ( #10785 )
...
This is mostly an internal change, but improves type hints in the
media code.
2021-09-14 07:09:38 -04:00
d725e0956f
1.43.0rc1
2021-09-14 11:47:11 +01:00
319b8b6bef
Name the type of token in "Invalid token" messages ( #10815 )
...
I had one of these error messages yesterday and assumed it was an
invalid auth token (because that was an HTTP query parameter in the
test) I was working on. In fact, it was an invalid next batch token for
syncing.
2021-09-14 11:25:05 +01:00
01c88a09cd
Use direct references for some configuration variables ( #10798 )
...
Instead of proxying through the magic getter of the RootConfig
object. This should be more performant (and is more explicit).
2021-09-13 13:07:12 -04:00
9f111075e8
Fix copy-paste error in the password section of the sample-config. ( #10804 )
2021-09-13 08:58:34 -04:00
003846d68a
Use the BaseReporter super-class for _WrappedRustReporter. ( #10799 )
...
This fixes mypy errors with jaeger-client >= 4.7.0 and should be a no-op
for versions before that.
2021-09-13 08:54:01 -04:00
524b8ead77
Add types to synapse.util. ( #10601 )
2021-09-10 17:03:18 +01:00
ceab5a4bfa
Fix 2 typos in docs/log_contexts.md ( #10795 )
2021-09-10 16:33:36 +01:00
63f28e4a0c
Handle room upgrades for spaces ( #10774 )
...
By copying the `room_type` field of the create event and
migrating any non-empty `m.space.child` events to the
new room that is created.
2021-09-10 07:30:05 -04:00
318162f5de
Easy refactors of the user directory ( #10789 )
...
No functional changes here. This came out as I was working to tackle #5677
2021-09-10 10:54:38 +01:00
c6f5fb5477
Remove fixed and flakey tests from the sytest blacklist ( #10788 )
...
We want to blacklist only known, consistent failures. We should deflake tests rather than ignoring them.
2021-09-10 10:53:04 +01:00
0c0da36a68
Ask consent on SSO registration with default mxid ( #10733 )
...
Fixes #10732 : consent flow skipped during SSO user registration if username is left at default
Signed-off-by: Andrew Ferrazzutti fair@miscworks.net
2021-09-10 10:36:45 +01:00
7f0565e029
Don't needlessly batch in `add_event_to_cache` ( #10784 )
...
We've already batched up the events previously, and assume in other
places in the events.py file that we have. Removing this makes it easier
to adjust the batch sizes in one place.
2021-09-10 10:16:52 +01:00
273b6861f2
Remove unstable MSC2858 API, including `experimental.msc2858_enabled` config option ( #10693 )
...
Signed-off-by: Sean Quah <seanq@element.io>
2021-09-09 17:59:59 +01:00
a621ba0259
Prefer room v9 for restricted rooms. ( #10772 )
...
Hint to clients via the room capabilities API (MSC3244) that
room version 9 should be preferred for creating a room with
restricted join rules (instead of room version 8).
2021-09-09 07:37:33 -04:00
abedf7d77f
Get rid of `_auth_and_persist_event` ( #10781 )
...
This is only called in two places, and the code seems much clearer without it.
2021-09-08 19:03:08 +01:00
03caba6577
Improve the modules doc ( #10758 )
...
* Split up the documentation in several files rather than one huge one
* Add examples for each callback category
* Other niceties like fixing https://github.com/matrix-org/synapse/issues/10632
* Add titles to callbacks so they're easier to find in the navigation panels and link to
2021-09-08 17:14:54 +00:00
01df612e1e
Fix frontend_proxy jinja script in docker workers ( #10783 )
2021-09-08 17:24:53 +01:00
5154afc00d
Add a script to update the debian changelog for non-Debian systems ( #10778 )
...
When releasing 1.42.0 with @Azrenbeth and talking with @clokep yesterday I realised doing the dch incantations related to releasing Synapse wasn't trivial on eg a macOS system, so this is a script to run in a Debian container to make things a bit easier.
2021-09-08 17:03:25 +02:00
1fdf2cf8e8
Document that /spaces & /hierarchy can be routed to workers. ( #10648 )
...
Also refactors some of the registration of endpoints on workers.
2021-09-08 11:02:31 -04:00
74f01e11c9
Skip handling of push actions for outlier events ( #10780 )
...
Outlier events don't ever have push actions associated with them, so we
can skip some expensive queries during event persistence.
2021-09-08 15:18:35 +01:00
0288e6033b
Add a constant for m.federate. ( #10775 )
2021-09-08 10:00:43 -04:00
66d72b7e17
Change logging of puppeted requests to better differentiate users ( #10779 )
...
This used to be a comma and got accidentally changed to a period in #9654 , but a pipe character is more easier to parse visually.
2021-09-08 12:59:15 +00:00
580a15e039
Request JSON for oEmbed requests (and ignore XML only providers). ( #10759 )
...
This adds the format to the request arguments / URL to
ensure that JSON data is returned (which is all that
Synapse supports).
This also adds additional error checking / filtering to the
configuration file to ignore XML-only providers.
2021-09-08 07:17:52 -04:00
aacdce8fc0
Add some assertions about outliers ( #10773 )
...
I think I have finally teased apart the codepaths which handle outliers, and those that handle non-outliers.
Let's add some assertions to demonstrate my newfound knowledge.
2021-09-08 10:41:13 +01:00
5724883ac2
Persist auth events before the events that rely on them ( #10771 )
...
If we're persisting an event E which has auth_events A1, A2, then we ought to make sure that we correctly auth
and persist A1 and A2, before we blindly accept E.
This PR does part of that - it persists the auth events first - but it does not fully solve the problem, because we
still don't check that the auth events weren't rejected.
2021-09-08 10:37:50 +01:00
89ba834818
Use attrs internally for the URL preview code & add documentation. ( #10753 )
2021-09-07 13:10:34 +00:00
a23f3abb9b
Return stripped m.space.child events via the space summary. ( #10760 )
...
The full event content cannot be trusted from this API (as no auth
chain, etc.) is processed over federation. Returning the full event
content was a bug as MSC2946 specifies that only the stripped
state should be returned.
This also avoids calculating aggregations / annotations which go
unused.
2021-09-07 08:43:54 -04:00
f30c9745ab
Underscore-prefix private fields in `FederationEventHandler` ( #10746 )
2021-09-07 11:15:51 +01:00
f1c6b76418
Add logging to help debug #9424 ( #10704 )
2021-09-06 16:08:25 +01:00
6e895366ea
Add config option to use non-default manhole password and keys ( #10643 )
2021-09-06 16:08:03 +01:00
b298de780a
Stop using BaseHandler in `FederationEventHandler` ( #10745 )
...
It's now only used in a couple of places, so we can drop it altogether.
2021-09-06 14:49:33 +01:00
40a1fddd1b
Allow `room_alias_name` parameter to be handled by /createRoom calls on workers ( #10757 )
2021-09-06 14:37:15 +01:00
7bb3673f37
Ease searching for M_TOO_LARGE-related error codes ( #10750 )
2021-09-06 14:35:56 +01:00
e1641b46d1
Doc: Clarifying undoing room shutdowns ( #10480 )
2021-09-06 14:24:31 +01:00
56e2a30634
Move `maybe_kick_guest_users` out of `BaseHandler` ( #10744 )
...
This is part of my ongoing war against BaseHandler. I've moved kick_guest_users into RoomMemberHandler (since it calls out to that handler anyway), and split maybe_kick_guest_users into the two places it is called.
2021-09-06 12:17:16 +01:00
5e9b382505
Pull out encrypted_by_default tests from user_directory tests ( #10752 )
2021-09-06 11:37:54 +01:00
2ca0d64854
Speed up persisting redacted events ( #10756 )
2021-09-06 10:14:07 +01:00
1ca70fd312
Allow room creator to send MSC2716 related events in existing room versions ( #10566 )
...
* Allow room creator to send MSC2716 related events in existing room versions
Discussed at https://github.com/matrix-org/matrix-doc/pull/2716/#discussion_r682474869
Restoring `get_create_event_for_room_txn` from,
44bb3f0cf5https://matrix.to/#/!UytJQHLQYfvYWsGrGY:jki.re/$p2fKESoFst038x6pOOmsY0C49S2gLKMr0jhNMz_JJz0?via=jki.re&via=matrix.org ), my plan is to use `rooms.creator`. But currently, we don't fill in `creator` for remote rooms when a user is invited to a room for example. So we need to add some code to fill in `creator` wherever we add to the `rooms` table. And also add a background update to fill in the rows missing `creator` (we can use the same logic that `get_create_event_for_room_txn` is doing by looking in the state events to get the `creator`).
>
> https://github.com/matrix-org/synapse/pull/10566#issuecomment-901616642
* Remove and switch away from get_create_event_for_room_txn
* Fix no create event being found because no state events persisted yet
* Fix and add tests for rooms creator bg update
* Populate rooms.creator field for easy lookup
Part of https://github.com/matrix-org/synapse/pull/10566
- Fill in creator whenever we insert into the rooms table
- Add background update to backfill any missing creator values
* Add changelog
* Fix usage
* Remove extra delta already included in #10697
* Don't worry about setting creator for invite
* Only iterate over rows missing the creator
See https://github.com/matrix-org/synapse/pull/10697#discussion_r695940898
* Use constant to fetch room creator field
See https://github.com/matrix-org/synapse/pull/10697#discussion_r696803029
* More protection from other random types
See https://github.com/matrix-org/synapse/pull/10697#discussion_r696806853
* Move new background update to end of list
See https://github.com/matrix-org/synapse/pull/10697#discussion_r696814181
* Fix query casing
* Fix ambiguity iterating over cursor instead of list
Fix `psycopg2.ProgrammingError: no results to fetch` error
when tests run with Postgres.
```
SYNAPSE_POSTGRES=1 SYNAPSE_TEST_LOG_LEVEL=INFO python -m twisted.trial tests.storage.databases.main.test_room
```
---
We use `txn.fetchall` because it will return the results as a
list or an empty list when there are no results.
Docs:
> `cursor` objects are iterable, so, instead of calling explicitly fetchone() in a loop, the object itself can be used:
>
> https://www.psycopg.org/docs/cursor.html#cursor-iterable
And I'm guessing iterating over a raw cursor does something weird when there are no results.
---
Test CI failure: https://github.com/matrix-org/synapse/pull/10697/checks?check_run_id=3468916530
```
tests.test_visibility.FilterEventsForServerTestCase.test_large_room
===============================================================================
[FAIL]
Traceback (most recent call last):
File "/home/runner/work/synapse/synapse/tests/storage/databases/main/test_room.py", line 85, in test_background_populate_rooms_creator_column
self.get_success(
File "/home/runner/work/synapse/synapse/tests/unittest.py", line 500, in get_success
return self.successResultOf(d)
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/trial/_synctest.py", line 700, in successResultOf
self.fail(
twisted.trial.unittest.FailTest: Success result expected on <Deferred at 0x7f4022f3eb50 current result: None>, found failure result instead:
Traceback (most recent call last):
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 701, in errback
self._startRunCallbacks(fail)
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 764, in _startRunCallbacks
self._runCallbacks()
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 1751, in gotResult
current_context.run(_inlineCallbacks, r, gen, status)
--- <exception caught here> ---
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 1657, in _inlineCallbacks
result = current_context.run(
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/home/runner/work/synapse/synapse/synapse/storage/background_updates.py", line 224, in do_next_background_update
await self._do_background_update(desired_duration_ms)
File "/home/runner/work/synapse/synapse/synapse/storage/background_updates.py", line 261, in _do_background_update
items_updated = await update_handler(progress, batch_size)
File "/home/runner/work/synapse/synapse/synapse/storage/databases/main/room.py", line 1399, in _background_populate_rooms_creator_column
end = await self.db_pool.runInteraction(
File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 686, in runInteraction
result = await self.runWithConnection(
File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 791, in runWithConnection
return await make_deferred_yieldable(
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/home/runner/work/synapse/synapse/tests/server.py", line 425, in <lambda>
d.addCallback(lambda x: function(*args, **kwargs))
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection
compat.reraise(excValue, excTraceback)
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction
return function(*args, **kwargs)
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/compat.py", line 404, in reraise
raise exception.with_traceback(traceback)
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection
result = func(conn, *args, **kw)
File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 786, in inner_func
return func(db_conn, *args, **kwargs)
File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 554, in new_transaction
r = func(cursor, *args, **kwargs)
File "/home/runner/work/synapse/synapse/synapse/storage/databases/main/room.py", line 1375, in _background_populate_rooms_creator_column_txn
for room_id, event_json in txn:
psycopg2.ProgrammingError: no results to fetch
```
* Move code not under the MSC2716 room version underneath an experimental config option
See https://github.com/matrix-org/synapse/pull/10566#issuecomment-906437909
* Add ordering to rooms creator background update
See https://github.com/matrix-org/synapse/pull/10697#discussion_r696815277
* Add comment to better document constant
See https://github.com/matrix-org/synapse/pull/10697#discussion_r699674458
* Use constant field
2021-09-04 00:58:49 -05:00
92b6ac31b2
Speed up MultiWriterIdGenerator when lots of IDs are in flight. ( #10755 )
2021-09-03 18:23:46 +01:00
924276f482
Add a partial index to `presence_stream` to speed up startups ( #10748 )
...
Signed-off-by: Sean Quah <seanq@element.io>
2021-09-03 17:16:56 +01:00
0eae330a26
Use `execute_values` more in PostgreSQL ( #10754 )
...
`execute_values` is a faster version of `execute_batch`.
2021-09-03 15:35:49 +00:00
2cb85bdf75
Raise an error if an unknown preset is used to create a room. ( #10738 )
...
Raises a 400 error instead of a 500 if an unknown preset is passed
from a client to create a room.
2021-09-03 13:46:18 +00:00
ecbfa4fe4f
Additional type hints for client REST servlets (part 5) ( #10736 )
...
Additionally this enforce type hints on all function signatures inside
of the synapse.rest.client package.
2021-09-03 09:22:22 -04:00
f58d202e3f
Fix bug with reusing 'txn' when persisting event. ( #10743 )
...
This will only happen when a server has multiple out of band membership
events in a single room.
2021-09-03 10:59:25 +01:00
00640ee71a
Fix documentation of directory name for remote thumbnails ( #10556 )
2021-09-02 14:07:53 +01:00
c586d6803a
Ignore rooms with unknown room versions in the spaces summary. ( #10727 )
...
This avoids breaking the entire endpoint if a room with
an unsupported room version is encountered.
2021-09-01 17:01:08 +00:00
6258730ebe
Consider the `origin_server_ts` of the `m.space.child` event when ordering rooms. ( #10730 )
...
This updates the ordering of the returned events from the spaces
summary API to that defined in MSC2946 (which updates MSC1772).
Previously a step was skipped causing ordering to be inconsistent with
clients.
2021-09-01 12:59:52 -04:00
d1f1b46c2c
Additional type hints for client REST servlets (part 4) ( #10728 )
2021-09-01 11:59:32 -04:00
dc75fb7f05
Populate `rooms.creator` field for easy lookup ( #10697 )
...
Part of https://github.com/matrix-org/synapse/pull/10566
- Fill in creator whenever we insert into the rooms table
- Add background update to backfill any missing creator values
2021-09-01 16:27:58 +01:00
e059094119
Synapse 1.42.0rc1 (2021-09-01)
...
==============================
Server administrators are reminded to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420).
Features
--------
- Add support for [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231 ): Token authenticated registration. Users can be required to submit a token during registration to authenticate themselves. Contributed by Callum Brown. ([\#10142](https://github.com/matrix-org/synapse/issues/10142 ))
- Add support for [MSC3283](https://github.com/matrix-org/matrix-doc/pull/3283 ): Expose `enable_set_displayname` in capabilities. ([\#10452](https://github.com/matrix-org/synapse/issues/10452 ))
- Port the `PresenceRouter` module interface to the new generic interface. ([\#10524](https://github.com/matrix-org/synapse/issues/10524 ))
- Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946 ). ([\#10613](https://github.com/matrix-org/synapse/issues/10613 ), [\#10725](https://github.com/matrix-org/synapse/issues/10725 ))
Bugfixes
--------
- Validate new `m.room.power_levels` events. Contributed by @aaronraimist. ([\#10232](https://github.com/matrix-org/synapse/issues/10232 ))
- Display an error on User-Interactive Authentication fallback pages when authentication fails. Contributed by Callum Brown. ([\#10561](https://github.com/matrix-org/synapse/issues/10561 ))
- Remove pushers when deleting an e-mail address from an account. Pushers for old unlinked emails will also be deleted. ([\#10581](https://github.com/matrix-org/synapse/issues/10581 ), [\#10734](https://github.com/matrix-org/synapse/issues/10734 ))
- Reject Client-Server `/keys/query` requests which provide `device_ids` incorrectly. ([\#10593](https://github.com/matrix-org/synapse/issues/10593 ))
- Rooms with unsupported room versions are no longer returned via `/sync`. ([\#10644](https://github.com/matrix-org/synapse/issues/10644 ))
- Enforce the maximum length for per-room display names and avatar URLs. ([\#10654](https://github.com/matrix-org/synapse/issues/10654 ))
- Fix a bug which caused the `synapse_user_logins_total` Prometheus metric not to be correctly initialised on restart. ([\#10677](https://github.com/matrix-org/synapse/issues/10677 ))
- Improve `ServerNoticeServlet` to avoid duplicate requests and add unit tests. ([\#10679](https://github.com/matrix-org/synapse/issues/10679 ))
- Fix long-standing issue which caused an error when a thumbnail is requested and there are multiple thumbnails with the same quality rating. ([\#10684](https://github.com/matrix-org/synapse/issues/10684 ))
- Fix a regression introduced in v1.41.0 which affected the performance of concurrent fetches of large sets of events, in extreme cases causing the process to hang. ([\#10703](https://github.com/matrix-org/synapse/issues/10703 ))
- Fix a regression introduced in Synapse 1.41 which broke email transmission on Systems using older versions of the Twisted library. ([\#10713](https://github.com/matrix-org/synapse/issues/10713 ))
Improved Documentation
----------------------
- Add documentation on how to connect Django with Synapse using OpenID Connect and django-oauth-toolkit. Contributed by @HugoDelval. ([\#10192](https://github.com/matrix-org/synapse/issues/10192 ))
- Advertise https://matrix-org.github.io/synapse documentation in the `README` and `CONTRIBUTING` files. ([\#10595](https://github.com/matrix-org/synapse/issues/10595 ))
- Fix some of the titles not rendering in the OpenID Connect documentation. ([\#10639](https://github.com/matrix-org/synapse/issues/10639 ))
- Minor clarifications to the documentation for reverse proxies. ([\#10708](https://github.com/matrix-org/synapse/issues/10708 ))
- Remove table of contents from the top of installation and contributing documentation pages. ([\#10711](https://github.com/matrix-org/synapse/issues/10711 ))
Deprecations and Removals
-------------------------
- Remove deprecated Shutdown Room and Purge Room Admin API. ([\#8830](https://github.com/matrix-org/synapse/issues/8830 ))
Internal Changes
----------------
- Improve type hints for the proxy agent and SRV resolver modules. Contributed by @dklimpel. ([\#10608](https://github.com/matrix-org/synapse/issues/10608 ))
- Clean up some of the federation event authentication code for clarity. ([\#10614](https://github.com/matrix-org/synapse/issues/10614 ), [\#10615](https://github.com/matrix-org/synapse/issues/10615 ), [\#10624](https://github.com/matrix-org/synapse/issues/10624 ), [\#10640](https://github.com/matrix-org/synapse/issues/10640 ))
- Add a comment asking developers to leave a reason when bumping the database schema version. ([\#10621](https://github.com/matrix-org/synapse/issues/10621 ))
- Remove not needed database updates in modify user admin API. ([\#10627](https://github.com/matrix-org/synapse/issues/10627 ))
- Convert room member storage tuples to `attrs` classes. ([\#10629](https://github.com/matrix-org/synapse/issues/10629 ), [\#10642](https://github.com/matrix-org/synapse/issues/10642 ))
- Use auto-attribs for the attrs classes used in sync. ([\#10630](https://github.com/matrix-org/synapse/issues/10630 ))
- Make `backfill` and `get_missing_events` use the same codepath. ([\#10645](https://github.com/matrix-org/synapse/issues/10645 ))
- Improve the performance of the `/hierarchy` API (from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946 )) by caching responses received over federation. ([\#10647](https://github.com/matrix-org/synapse/issues/10647 ))
- Run a nightly CI build against Twisted trunk. ([\#10651](https://github.com/matrix-org/synapse/issues/10651 ), [\#10672](https://github.com/matrix-org/synapse/issues/10672 ))
- Do not print out stack traces for network errors when fetching data over federation. ([\#10662](https://github.com/matrix-org/synapse/issues/10662 ))
- Simplify tests for device admin rest API. ([\#10664](https://github.com/matrix-org/synapse/issues/10664 ))
- Add missing type hints to REST servlets. ([\#10665](https://github.com/matrix-org/synapse/issues/10665 ), [\#10666](https://github.com/matrix-org/synapse/issues/10666 ), [\#10674](https://github.com/matrix-org/synapse/issues/10674 ))
- Flatten the `tests.synapse.rests` package by moving the contents of `v1` and `v2_alpha` into the parent. ([\#10667](https://github.com/matrix-org/synapse/issues/10667 ))
- Update `complement.sh` to rebuild the base Docker image when run with workers. ([\#10686](https://github.com/matrix-org/synapse/issues/10686 ))
- Split the event-processing methods in `FederationHandler` into a separate `FederationEventHandler`. ([\#10692](https://github.com/matrix-org/synapse/issues/10692 ))
- Remove unused `compare_digest` function. ([\#10706](https://github.com/matrix-org/synapse/issues/10706 ))
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEE8SRSDO7gYkSP4chELS76LzL74EcFAmEvd3cACgkQLS76LzL7
4EcF0hAAvOjDeH2hn0eI4YX03/BTCesJ5pCTbEJeC9Hsui3SbWamyYarH0Ae0gM6
BP9x5/XkCdsZhKT2OJedtZQUQ0cUws5FdJP87NnE1m7KBW9djbTxmkue3JKaHNTT
ajvEoLe8UjOFbjvyMc+188uWVgtKQ6KLcetwllTpYY3FJ3+yk9CQP1ydYzwfHcHe
hXW3b20WoRdAmH0B27mj1oQzCeXclbj1kCthcBcxH5RytA8bBc1itBtpAP2FFaeS
Ml6paoXSs0wZaXXL5T1MhQtcVOOG5Yh7EPsemdyfqkwhIk9QtV3nmGfC7K5iEeq0
DldTgBNE0wLO+ycdgEZ2oCvitCLT4WDoKkVqKPA91wpuvCMeK5lnDdqbFUSN3hfd
l8Oz+ydhjsCx7+pBBopnsyGrPIHNaN/vySYGF3lPa3ANKS2J0IVlAaJ59bZqzg2C
vzdyMBCnn0fhnXMkADPLnIYqtebvC09JlPBRGo5BKmwmlM8Ea1dLQ+Lcy5vXalcw
+GaYw3U805c03EvsWQxWchjwq6IwMeP06YpK/1ruMjlrCWYJp3V7f/4hXjlTdYm5
OEWc/kHOnlLfH2cA4aS0sV5WsnEuftotQIzBXuXWw5hSMdFmH7eAviPg0vXgfeA/
i0FOkOgEYR8O3RMh4Cip8+h44t8SPNjYUFJK9ip3DmNateeUCH0=
=XxMc
-----END PGP SIGNATURE-----
Merge tag 'v1.42.0rc1' into develop
Synapse 1.42.0rc1 (2021-09-01)
==============================
Server administrators are reminded to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420).
Features
--------
- Add support for [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231 ): Token authenticated registration. Users can be required to submit a token during registration to authenticate themselves. Contributed by Callum Brown. ([\#10142](https://github.com/matrix-org/synapse/issues/10142 ))
- Add support for [MSC3283](https://github.com/matrix-org/matrix-doc/pull/3283 ): Expose `enable_set_displayname` in capabilities. ([\#10452](https://github.com/matrix-org/synapse/issues/10452 ))
- Port the `PresenceRouter` module interface to the new generic interface. ([\#10524](https://github.com/matrix-org/synapse/issues/10524 ))
- Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946 ). ([\#10613](https://github.com/matrix-org/synapse/issues/10613 ), [\#10725](https://github.com/matrix-org/synapse/issues/10725 ))
Bugfixes
--------
- Validate new `m.room.power_levels` events. Contributed by @aaronraimist. ([\#10232](https://github.com/matrix-org/synapse/issues/10232 ))
- Display an error on User-Interactive Authentication fallback pages when authentication fails. Contributed by Callum Brown. ([\#10561](https://github.com/matrix-org/synapse/issues/10561 ))
- Remove pushers when deleting an e-mail address from an account. Pushers for old unlinked emails will also be deleted. ([\#10581](https://github.com/matrix-org/synapse/issues/10581 ), [\#10734](https://github.com/matrix-org/synapse/issues/10734 ))
- Reject Client-Server `/keys/query` requests which provide `device_ids` incorrectly. ([\#10593](https://github.com/matrix-org/synapse/issues/10593 ))
- Rooms with unsupported room versions are no longer returned via `/sync`. ([\#10644](https://github.com/matrix-org/synapse/issues/10644 ))
- Enforce the maximum length for per-room display names and avatar URLs. ([\#10654](https://github.com/matrix-org/synapse/issues/10654 ))
- Fix a bug which caused the `synapse_user_logins_total` Prometheus metric not to be correctly initialised on restart. ([\#10677](https://github.com/matrix-org/synapse/issues/10677 ))
- Improve `ServerNoticeServlet` to avoid duplicate requests and add unit tests. ([\#10679](https://github.com/matrix-org/synapse/issues/10679 ))
- Fix long-standing issue which caused an error when a thumbnail is requested and there are multiple thumbnails with the same quality rating. ([\#10684](https://github.com/matrix-org/synapse/issues/10684 ))
- Fix a regression introduced in v1.41.0 which affected the performance of concurrent fetches of large sets of events, in extreme cases causing the process to hang. ([\#10703](https://github.com/matrix-org/synapse/issues/10703 ))
- Fix a regression introduced in Synapse 1.41 which broke email transmission on Systems using older versions of the Twisted library. ([\#10713](https://github.com/matrix-org/synapse/issues/10713 ))
Improved Documentation
----------------------
- Add documentation on how to connect Django with Synapse using OpenID Connect and django-oauth-toolkit. Contributed by @HugoDelval. ([\#10192](https://github.com/matrix-org/synapse/issues/10192 ))
- Advertise https://matrix-org.github.io/synapse documentation in the `README` and `CONTRIBUTING` files. ([\#10595](https://github.com/matrix-org/synapse/issues/10595 ))
- Fix some of the titles not rendering in the OpenID Connect documentation. ([\#10639](https://github.com/matrix-org/synapse/issues/10639 ))
- Minor clarifications to the documentation for reverse proxies. ([\#10708](https://github.com/matrix-org/synapse/issues/10708 ))
- Remove table of contents from the top of installation and contributing documentation pages. ([\#10711](https://github.com/matrix-org/synapse/issues/10711 ))
Deprecations and Removals
-------------------------
- Remove deprecated Shutdown Room and Purge Room Admin API. ([\#8830](https://github.com/matrix-org/synapse/issues/8830 ))
Internal Changes
----------------
- Improve type hints for the proxy agent and SRV resolver modules. Contributed by @dklimpel. ([\#10608](https://github.com/matrix-org/synapse/issues/10608 ))
- Clean up some of the federation event authentication code for clarity. ([\#10614](https://github.com/matrix-org/synapse/issues/10614 ), [\#10615](https://github.com/matrix-org/synapse/issues/10615 ), [\#10624](https://github.com/matrix-org/synapse/issues/10624 ), [\#10640](https://github.com/matrix-org/synapse/issues/10640 ))
- Add a comment asking developers to leave a reason when bumping the database schema version. ([\#10621](https://github.com/matrix-org/synapse/issues/10621 ))
- Remove not needed database updates in modify user admin API. ([\#10627](https://github.com/matrix-org/synapse/issues/10627 ))
- Convert room member storage tuples to `attrs` classes. ([\#10629](https://github.com/matrix-org/synapse/issues/10629 ), [\#10642](https://github.com/matrix-org/synapse/issues/10642 ))
- Use auto-attribs for the attrs classes used in sync. ([\#10630](https://github.com/matrix-org/synapse/issues/10630 ))
- Make `backfill` and `get_missing_events` use the same codepath. ([\#10645](https://github.com/matrix-org/synapse/issues/10645 ))
- Improve the performance of the `/hierarchy` API (from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946 )) by caching responses received over federation. ([\#10647](https://github.com/matrix-org/synapse/issues/10647 ))
- Run a nightly CI build against Twisted trunk. ([\#10651](https://github.com/matrix-org/synapse/issues/10651 ), [\#10672](https://github.com/matrix-org/synapse/issues/10672 ))
- Do not print out stack traces for network errors when fetching data over federation. ([\#10662](https://github.com/matrix-org/synapse/issues/10662 ))
- Simplify tests for device admin rest API. ([\#10664](https://github.com/matrix-org/synapse/issues/10664 ))
- Add missing type hints to REST servlets. ([\#10665](https://github.com/matrix-org/synapse/issues/10665 ), [\#10666](https://github.com/matrix-org/synapse/issues/10666 ), [\#10674](https://github.com/matrix-org/synapse/issues/10674 ))
- Flatten the `tests.synapse.rests` package by moving the contents of `v1` and `v2_alpha` into the parent. ([\#10667](https://github.com/matrix-org/synapse/issues/10667 ))
- Update `complement.sh` to rebuild the base Docker image when run with workers. ([\#10686](https://github.com/matrix-org/synapse/issues/10686 ))
- Split the event-processing methods in `FederationHandler` into a separate `FederationEventHandler`. ([\#10692](https://github.com/matrix-org/synapse/issues/10692 ))
- Remove unused `compare_digest` function. ([\#10706](https://github.com/matrix-org/synapse/issues/10706 ))
2021-09-01 14:58:14 +01:00
d9069388f3
Correctly include room avatars in email notifications ( #10658 )
...
Judging by the template, this was intended ages ago, but we never
actually passed an avatar URL to the template. So let's provide one.
Closes #1546 .
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-09-01 13:48:41 +01:00
f8bf83b811
Skip the final GC on shutdown to improve restart times ( #10712 )
...
Use `gc.freeze()` on exit to exclude all existing objects from the final GC.
In testing, this sped up shutdown by up to a few seconds.
`gc.freeze()` runs in constant time, so there is little chance of performance
regression.
Signed-off-by: Sean Quah <seanq@element.io>
2021-09-01 11:55:31 +01:00
6b2aca473a
1.42.0rc1
2021-09-01 11:47:24 +01:00
3693ea61f5
Fix iteration in _remove_deleted_email_pushers background job. ( #10734 )
2021-09-01 09:13:01 +00:00
e2481dbe93
Allow configuration of the oEmbed URLs. ( #10714 )
...
This adds configuration options (under an `oembed` section) to
configure which URLs are matched to use oEmbed for URL
previews.
2021-08-31 18:37:07 -04:00
287918e2d4
Additional type hints for the client REST servlets (part 3). ( #10707 )
2021-08-31 17:22:29 +00:00
78e590d473
Move the sessions delta to the latest schema version. ( #10725 )
...
This was erroneously put under schema version 62 instead of 63.
2021-08-31 16:38:43 +00:00
5d9e7e0c71
Merge branch 'master' into develop
2021-08-31 14:09:48 +01:00
8c26f16c76
Fix up unit tests ( #10723 )
...
These were broken in an incorrect merge of GHSA-jj53-8fmw-f2w2 (cb35df9
2021-08-31 12:56:22 +01:00
46ff99ef95
Advertise matrix-org.github.io/synapse docs ( #10595 )
...
Point to the book where possible, and use hyperlinks to github to refer to files not included in the book.
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-08-31 11:29:27 +01:00
e3abc0a5cc
Merge remote-tracking branch 'origin/release-v1.41' into develop
2021-08-27 16:35:02 +01:00
8f98260552
Fix incompatibility with Twisted < 21. ( #10713 )
...
Turns out that the functionality added in #10546 to skip TLS was incompatible
with older Twisted versions, so we need to be a bit more inventive.
Also, add a test to (hopefully) not break this in future. Sadly, testing TLS is
really hard.
2021-08-27 16:33:41 +01:00
54aa7047eb
Removed page summaries from the top of installation and contributing doc pages ( #10711 )
...
- Removed page summaries from CONTRIBUTING and installation pages as
this information was already in the table of contents on the right hand side
- Fixed some broken links in CONTRIBUTING
- Added margin-right tag for when table of contents is being shown
(otherwise the text in the page sometimes overlaps with it)
2021-08-27 14:19:17 +00:00
051ddac53b
Clarifications to reverse_proxy.md ( #10708 )
...
* Update reverse_proxy.md
* Create 10708.doc
2021-08-27 12:54:21 +01:00
029b7ad7b9
Remove unused `compare_digest` function. ( #10706 )
2021-08-27 07:08:02 -04:00
e62cdbef1a
Improve ServerNoticeServlet to avoid duplicate requests ( #10679 )
...
Fixes : #9544
2021-08-27 09:16:40 +00:00
c4fa4f37cb
Fix perf of fetching the same events many times. ( #10703 )
...
The code to deduplicate repeated fetches of the same set of events was
N^2 (over the number of events requested), which could lead to a process
being completely wedged.
The main fix is to deduplicate the returned deferreds so we only await
on a deferred once rather than many times. Seperately, when handling the
returned events from the defrered we only add the events we care about
to the event map to be returned (so that we don't pay the price of
inserting extraneous events into the dict).
2021-08-27 09:15:50 +00:00
1800aabfc2
Split `FederationHandler` in half ( #10692 )
...
The idea here is to take anything to do with incoming events and move it out to a separate handler, as a way of making FederationHandler smaller.
2021-08-26 21:41:44 +01:00
96715d7633
Make `backfill` and `get_missing_events` use the same codepath ( #10645 )
...
Given that backfill and get_missing_events are basically the same thing, it's somewhat crazy that we have entirely separate code paths for them. This makes backfill use the existing get_missing_events code, and then clears up all the unused code.
2021-08-26 18:34:57 +01:00
40f619eaa5
Validate new m.room.power_levels events ( #10232 )
...
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-08-26 17:07:58 +01:00
ad17fbd20e
Remove pushers when deleting 3pid from account ( #10581 )
...
When a user deletes an email from their account it will
now also remove all pushers for that email and that user
(even if these pushers were created by a different client)
2021-08-26 13:53:57 +01:00
1aa0dad021
Additional type hints for REST servlets (part 2). ( #10674 )
...
Applies the changes from #10665 to additional modules.
2021-08-26 11:53:52 +00:00
5548fe0978
Cache the result of fetching the room hierarchy over federation. ( #10647 )
2021-08-26 07:16:53 -04:00
b45cc1530b
Make a note to leave a summary when one is bumping the schema version ( #10621 )
...
I found this easy to miss (and evidently, it looks like it was missed for schema version 62).
2021-08-25 17:00:44 +01:00
882539e423
Ensure the base Docker image is rebuilt when running complement with workers. ( #10686 )
...
We now always rebuild the matrixdotorg/synapse image, then
build the matrixdotorg/synapse-workers image on top of it.
2021-08-25 10:18:23 -04:00
7367473f96
Fix error when selecting between thumbnails with the same quality ( #10684 )
...
Fixes #10318
2021-08-25 09:51:08 +00:00
6f77a3d433
1.41.0
2021-08-24 15:31:55 +01:00
d12ba52f17
Persist room hierarchy pagination sessions to the database. ( #10613 )
2021-08-24 08:14:03 -04:00
15db8b7c7f
Correctly initialise the `synapse_user_logins` metric. ( #10677 )
...
Fix a bug where the prometheus metrics for SSO logins wouldn't be initialised
until the first user logged in with a given auth provider.
2021-08-24 09:17:51 +00:00
86415f162d
doc: add django-oauth-toolkit to oidc doc ( #10192 )
...
Signed-off-by: Hugo Delval <hugo.delval@gmail.com>
2021-08-23 17:12:36 +00:00
0c1d6f65d7
Enforce the max length for per-room display names / avatar URLs. ( #10654 )
...
To match the maximum lengths allowed for profile data.
2021-08-23 11:25:33 -04:00
3e83f97154
Fix the titles in the OIDC documentation ( #10639 )
...
* Fix the titles in the OIDC documentation
Having them as links broke the table-of-contents rendering in mdbook.
Plus there's no reason for only some of the provider titles to be links.
* Changelog
* Add link to google idp docs
2021-08-23 14:58:31 +01:00
2efc838f05
Avoid duplicate issues from Twisted trunk failures ( #10672 )
...
Setting `update_existing: true` in the `create-an-issue` GitHub Action
will avoid opening duplicate issues if an open issue already exists with
an identical title.
If no open issues match the title, then a new issue will be created.
This helps avoid spamming our issue tracker should there be a failure
when testing against Twisted's trunk.
This PR also pins the SHA of the `create-an-issue` action to mitigate
the risk of a malicious actor gaining access to JasonEtco's account.
See GitHub's page on security hardening third party actions for more:
https://docs.github.com/en/actions/learn-github-actions/security-hardening-for-github-actions#using-third-party-actions
Signed-off-by: Dan Callahan <danc@element.io>
2021-08-23 13:06:49 +00:00
bd7d398b05
Additional type hints for the sync REST servlet. ( #10666 )
2021-08-23 08:14:42 -04:00
2af6d31b78
Addtional type hints for the REST servlets. ( #10665 )
2021-08-23 08:14:17 -04:00
31dac7ffee
Do not include stack traces for known exceptions when trying multiple federation destinations. ( #10662 )
2021-08-23 08:00:25 -04:00
4db65f911a
Run a nightly CI build against Twisted trunk. ( #10651 )
...
This creates a GHA workflow which runs at 8am every day, and runs mypy, trial and sytest against Twisted's current trunk. If any of the jobs fail, it opens an issue.
2021-08-23 11:12:45 +01:00
947dbbdfd1
Implement MSC3231: Token authenticated registration ( #10142 )
...
Signed-off-by: Callum Brown <callum@calcuode.com>
This is part of my GSoC project implementing [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231 ).
2021-08-21 22:14:43 +01:00
ecd823d766
Flatten tests/rest/client/{v1,v2_alpha} too ( #10667 )
2021-08-20 17:50:44 +01:00
f499dc38bc
Simplify tests for the device admin rest API. ( #10664 )
...
By replacing duplicated code with parameterized tests and
avoiding unnecessary dumping of JSON data.
2021-08-20 15:43:26 +00:00
7862d704fd
Follow-up: format changelog, add licence ( #10593 )
...
Merged before approval; these comments from @clokep on that PR.
2021-08-20 16:33:52 +01:00
ee3b2ac59a
Validate device_keys for C-S /keys/query requests ( #10593 )
...
* Validate device_keys for C-S /keys/query requests
Closes #10354
A small, not particularly critical fix. I'm interested in seeing if we
can find a more systematic approach though. #8445 is the place for any discussion.
2021-08-20 15:47:03 +01:00
5cda75fede
Set room version 8 as preferred for restricted rooms. ( #10571 )
2021-08-20 07:17:50 -04:00
e81d62009e
Split `on_receive_pdu` in half ( #10640 )
...
Here we split on_receive_pdu into two functions (on_receive_pdu and process_pulled_event), rather than having both cases in the same method. There's a tiny bit of overlap, but not that much.
2021-08-19 17:05:12 +00:00
50af1efe4b
Extract `_resolve_state_at_missing_prevs` ( #10624 )
...
This is a follow-up to #10615 : it takes the code that constructs the state at a backwards extremity, and extracts it to a separate method.
2021-08-19 17:31:40 +01:00
000aa89be6
Do not include rooms with an unknown room version in a sync response. ( #10644 )
...
A user will still see this room if it is in a local cache, but it will
not reappear if clearing the cache and reloading.
2021-08-19 11:12:55 -04:00
b5fef6054a
Support MSC3283: Expose `enable_set_displayname` in capabilities ( #10452 )
2021-08-19 09:40:40 +00:00
220f901229
Remove not needed database updates in modify user admin API ( #10627 )
2021-08-19 10:25:05 +01:00
0c3565da4c
Additional type hints for the proxy agent and SRV resolver modules. ( #10608 )
2021-08-18 13:53:20 -04:00
78a70a2e0b
Merge branch 'release-v1.41' into develop
2021-08-18 17:02:47 +01:00
d9856d9150
Fix weakref_slot parameter for room member storage attrs. ( #10642 )
...
Follow-up to #10629 which set it to true, not false.
2021-08-18 15:00:37 +00:00
49cb7eae97
1.41.0rc1
2021-08-18 15:52:11 +01:00
bec01c0758
Convert room member storage tuples to attrs. ( #10629 )
...
Instead of using namedtuples. This helps with asserting type hints
and code completion.
2021-08-18 09:22:07 -04:00
3692f7fd33
Mount /_synapse/admin/v1/users/{userId}/media admin API on media workers only ( #10628 )
...
Co-authored-by: Patrick Cloke <patrickc@matrix.org>
2021-08-18 13:25:12 +01:00
6e613a10d0
Display an error page during failure of fallback UIA. ( #10561 )
2021-08-18 08:13:35 -04:00
eea2873595
fix broken link to upgrade notes ( #10631 )
2021-08-18 12:38:37 +01:00
964f29cb6f
Refactor `on_receive_pdu` code ( #10615 )
...
* drop room pdu linearizer sooner
No point holding onto it while we recheck the db
* move out `missing_prevs` calculation
we're going to need `missing_prevs` whatever we do, so we may as well calculate
it eagerly and just update it if it gets outdated.
* Add another `if missing_prevs` condition
this should be a no-op, since all the code inside the block already checks `if
missing_prevs`
* reorder if conditions
This shouldn't change the logic at all.
* Push down `min_depth` read
No point reading it from the database unless we're going to use it.
* Collect the sent_to_us_directly code together
Move the remaining `sent_to_us_directly` code inside the `if
sent_to_us_directly` block.
* Properly separate the `not sent_to_us_directly` branch
Since the only way this second block is now reachable is if we
*didn't* go into the `sent_to_us_directly` branch, we can replace it with a
simple `else`.
* changelog
2021-08-18 12:36:22 +01:00
6a5f8fbcda
Use auto-attribs for attrs classes for sync. ( #10630 )
2021-08-18 07:27:32 -04:00
5581dd7bf7
Allow modules to run looping call on all instances ( #10638 )
...
By default the calls only ran on the worker configured to run background
tasks.
2021-08-18 10:21:11 +00:00
430241a1e9
Remove deprecated Shutdown Room and Purge Room Admin API ( #8830 )
2021-08-17 21:19:13 +01:00
703e3a9e85
Allow /createRoom to be run on workers ( #10564 )
...
Fixes https://github.com/matrix-org/synapse/issues/7867
2021-08-17 14:33:16 +01:00
1a9f531c79
Port the PresenceRouter module interface to the new generic interface ( #10524 )
...
Port the PresenceRouter module interface to the new generic interface introduced in v1.37.0
2021-08-17 13:22:45 +00:00
84469bdac7
Remove the unused public_room_list_stream ( #10565 )
...
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-08-17 14:02:50 +01:00
c8132f4a31
Build debs for bookworm ( #10612 )
2021-08-17 13:48:59 +01:00
5639759980
Centralise the custom template directory ( #10596 )
...
Several configuration sections are using separate settings for custom template directories, which can be confusing. This PR adds a new top-level configuration for a custom template directory which is then used for every module. The only exception is the consent templates, since the consent template directory require a specific hierarchy, so it's probably better that it stays separate from everything else.
2021-08-17 14:45:24 +02:00
c4cf0c0473
Attempt to pull from the legacy spaces summary API over federation. ( #10583 )
...
If the new /hierarchy API does not exist on all destinations,
fallback to querying the /spaces API and translating the results.
This is a backwards compatibility hack since not all of the
federated homeservers will update at the same time.
2021-08-17 08:19:12 -04:00
272b89d547
Stop setting the outlier flag for things that aren't ( #10614 )
...
Marking things as outliers to inhibit pushes is a sledgehammer to crack a
nut. Move the test further down the stack so that we just inhibit the thing we
want.
2021-08-17 13:13:42 +01:00
5f7b1e1f27
Make `PeriodicallyFlushingMemoryHandler` the default logging handler. ( #10518 )
2021-08-17 13:13:11 +01:00
642a42edde
Flatten the synapse.rest.client package ( #10600 )
2021-08-17 11:57:58 +00:00
b62eba7705
Always list fallback key types in /sync ( #10623 )
2021-08-17 12:32:25 +01:00
3bcd525b46
Allow to edit `external_ids` by Edit User admin API ( #10598 )
...
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-08-17 11:56:11 +01:00
58f0d97275
update links to schema doc ( #10620 )
2021-08-17 10:45:35 +00:00
ae2714c1f3
Allow using several custom template directories ( #10587 )
...
Allow using several directories in read_templates.
2021-08-17 10:23:14 +00:00
a933c2c7d8
Add an admin API to check if a username is available ( #10578 )
...
This adds a new API GET /_synapse/admin/v1/username_available?username=foo to check if a username is available. It is the counterpart to https://matrix.org/docs/spec/client_server/r0.6.0#get-matrix-client-r0-register-available , except that it works even if registration is disabled.
2021-08-17 10:52:38 +01:00
19e51b14d2
Manhole: wrap coroutines in `defer.ensureDeferred` automatically ( #10602 )
2021-08-16 18:11:48 +01:00
0db8cab72c
Update CONTRIBUTING.md to fix index links and SyTest instructions ( #10599 )
...
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2021-08-16 18:09:47 +01:00
5af83efe8d
Validate the max_rooms_per_space parameter to ensure it is non-negative. ( #10611 )
2021-08-16 12:01:30 -04:00
0ace38b7b3
Experimental support for MSC3266 Room Summary API. ( #10394 )
2021-08-16 14:49:12 +00:00
87b62f8bb2
Split `synapse.federation.transport.server` into multiple files. ( #10590 )
2021-08-16 10:14:31 -04:00
2d9ca4ca77
Clean up some logging in the federation event handler ( #10591 )
...
* Include outlier status in `str(event)`
In places where we log event objects, knowing whether or not you're dealing
with an outlier is super useful.
* Remove duplicated logging in get_missing_events
When we process events received from get_missing_events, we log them twice
(once in `_get_missing_events_for_pdu`, and once in `on_receive_pdu`). Reduce
the duplication by removing the logging in `on_receive_pdu`, and ensuring the
call sites do sensible logging.
* log in `on_receive_pdu` when we already have the event
* Log which prev_events we are missing
* changelog
2021-08-16 13:19:02 +01:00
7de445161f
Support federation in the new spaces summary API (MSC2946). ( #10569 )
2021-08-16 08:06:17 -04:00
a3a7514570
Handle string read receipt data ( #10606 )
...
* Handle string read receipt data
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Test that we handle string read receipt data
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Add changelog for #10606
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Add docs
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Ignore malformed RRs
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Only surround hidden = ...
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Remove unnecessary argument
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Update changelog.d/10606.bugfix
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-08-16 12:22:38 +01:00
d1f43b731c
Update the Synapse Grafana dashboard ( #10570 )
2021-08-16 12:57:09 +02:00
c8d54be44c
Move /batch_send to /v2_alpha directory (MSC2716) ( #10576 )
...
* Move /batch_send to /v2_alpha directory
As pointed out by @erikjohnston,
https://github.com/matrix-org/synapse/pull/10552#discussion_r685836624
2021-08-13 14:37:24 -05:00
c12b5577f2
Fix a harmless exception when the staged events queue is empty. ( #10592 )
2021-08-13 11:49:06 +00:00
d2ad397d3c
Stop building a debian package for Groovy Gorilla ( #10588 )
2021-08-12 16:50:18 +01:00
4a76d01ff7
Merge pull request #10573 from DMRobertson/dmr/goodbye-buildkite
...
Remove references to BuildKite in favour of GitHub Actions
2021-08-12 13:05:23 +01:00
98a3355d9a
Update the pagination parameter name based on MSC2946 review. ( #10579 )
2021-08-11 15:44:45 -04:00
915b37e5ef
Admin API to delete media for a specific user ( #10558 )
2021-08-11 19:29:59 +00:00
3ebb6694f0
Allow requesting the summary of a space which is joinable. ( #10580 )
...
As opposed to only allowing the summary of spaces which the user is
already in or has world-readable visibility.
This makes the logic consistent with whether a space/room is returned
as part of a space and whether a space summary can start at a space.
2021-08-11 15:04:51 -04:00
Patrick Cloke