Commit Graph

1045 Commits

Author SHA1 Message Date
Patrick Cloke d9f092285b
Remove federation client code for groups. (#12563) 2022-05-27 07:13:58 -04:00
Sean Quah 053ca5f3ca Synapse 1.60.0rc2 (2022-05-27)
==============================
 
 This release of Synapse adds a unique index to the `state_group_edges` table, in
 order to prevent accidentally introducing duplicate information (for example,
 because a database backup was restored multiple times). If your Synapse database
 already has duplicate rows in this table, this could fail with an error and
 require manual remediation.
 
 Additionally, the signature of the `check_event_for_spam` module callback has changed.
 The previous signature has been deprecated and remains working for now. Module authors
 should update their modules to use the new signature where possible.
 
 See [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1600)
 for more details.
 
 Features
 --------
 
 - Add an option allowing users to use their password to reauthenticate for privileged actions even though password login is disabled. ([\#12883](https://github.com/matrix-org/synapse/issues/12883))
 
 Bugfixes
 --------
 
 - Explicitly close `ijson` coroutines once we are done with them, instead of leaving the garbage collector to close them. ([\#12875](https://github.com/matrix-org/synapse/issues/12875))
 
 Internal Changes
 ----------------
 
 - Improve URL previews by not including the content of media tags in the generated description. ([\#12887](https://github.com/matrix-org/synapse/issues/12887))
 -----BEGIN PGP SIGNATURE-----
 
 iQGzBAABCgAdFiEEWMTnW8Z8khaaf90R+84KzgcyGG8FAmKQqMcACgkQ+84Kzgcy
 GG9Z2Av+N+b/fvaB3D56UkFqTW/xLmCEyri65njcXU8625bWiLSPM6hssmyJB1FA
 xlc2RBKr8QxlnHRS/v31wDtONC8YZ2O3fyzYPFfY1fF5Ul7Kg3XCzLeUH4/j1/Ar
 5bqriDqaN9FQ/6QJybShXlA4l7lY1Fs0C4P23jDBgqfKjnlToeVLqhVA70dDaFu/
 ir+vVprKCkQI1iqnYXwIxGRmgBzLWGoVqQFGbSI6hugGwXpGIyX7+2I+0v8tI6vA
 SZ99vLFWcvnd6DJTyBhIeV22Ff4qA7eQsyPvSrMETdsaZmrxGlG+t332HNCgplv8
 gv2gUpJL0br++5WTAX+nRc7HpfKo/74vKeTktqPmlvFP8kUOg+PbzmoJFUu21PhA
 rnq5TzgsPHK0dqBhM1RC2vtOiJ5v3ZBqzJJzSRXl6lsFpWxxFmwesEcIDAYS0Nmh
 QoJb7/L8cPCHksHvZM76bzB465tSH9NhuFYZQoLGHcpxa0kYekrdlYasP8U0FU7L
 nF3C0Pgw
 =D3F+
 -----END PGP SIGNATURE-----

Merge tag 'v1.60.0rc2' into develop

Synapse 1.60.0rc2 (2022-05-27)
==============================

This release of Synapse adds a unique index to the `state_group_edges` table, in
order to prevent accidentally introducing duplicate information (for example,
because a database backup was restored multiple times). If your Synapse database
already has duplicate rows in this table, this could fail with an error and
require manual remediation.

Additionally, the signature of the `check_event_for_spam` module callback has changed.
The previous signature has been deprecated and remains working for now. Module authors
should update their modules to use the new signature where possible.

See [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1600)
for more details.

Features
--------

- Add an option allowing users to use their password to reauthenticate for privileged actions even though password login is disabled. ([\#12883](https://github.com/matrix-org/synapse/issues/12883))

Bugfixes
--------

- Explicitly close `ijson` coroutines once we are done with them, instead of leaving the garbage collector to close them. ([\#12875](https://github.com/matrix-org/synapse/issues/12875))

Internal Changes
----------------

- Improve URL previews by not including the content of media tags in the generated description. ([\#12887](https://github.com/matrix-org/synapse/issues/12887))
2022-05-27 12:07:18 +01:00
Sean Quah bb7a637765
Close `ijson` coroutines ourselves instead of letting the GC close them (#12875)
Hopefully this means that exceptions raised due to truncated JSON
get a sensible logging context and stack.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-05-27 11:03:05 +01:00
Patrick Cloke 1885ee0113
Remove unstable APIs for /hierarchy. (#12851)
Removes the unstable endpoint as well as a duplicated field
which was modified during stabilization.
2022-05-26 07:10:28 -04:00
Patrick Cloke b5707ceaba
Avoid attempting to delete push actions for remote users. (#12879)
Remote users will never have push actions, so we can avoid a database
round-trip/transaction completely.
2022-05-26 07:09:16 -04:00
Richard van der Hoff 1b338476af
Allow bigger responses to `/federation/v1/state` (#12877)
* Refactor HTTP response size limits

Rather than passing a separate `max_response_size` down the stack, make it an
attribute of the `parser`.

* Allow bigger responses on `federation/v1/state`

`/state` can return huge responses, so we need to handle that.
2022-05-25 22:24:28 +01:00
Patrick Cloke a8db8c6eba
Remove user-visible groups/communities code (#12553)
Makes it so that groups/communities no longer exist from a user-POV. E.g. we remove:

* All API endpoints (including Client-Server, Server-Server, and admin).
* Documented configuration options (and the experimental flag, which is now unused).
* Special handling during room upgrades.
* The `groups` section of the `/sync` response.
2022-05-25 07:53:40 -04:00
David Teller 28199e9357
Uniformize spam-checker API, part 2: check_event_for_spam (#12808)
Signed-off-by: David Teller <davidt@element.io>
2022-05-23 17:27:39 +00:00
Jess Porter a608ac847b
add SpamChecker callback for silently dropping inbound federated events (#12744)
Signed-off-by: jesopo <github@lolnerd.net>
2022-05-23 16:36:21 +00:00
Hubert Chathi 8afb7b55d0
Make handling of federation Authorization header (more) compliant with RFC7230 (#12774)
The main differences are:
- values with delimiters (such as colons) should be quoted, so always
  quote the origin, since it could contain a colon followed by a port
  number
- should allow more than one space after "X-Matrix"
- quoted values with backslash-escaped characters should be unescaped
- names should be case insensitive
2022-05-18 11:19:30 +01:00
Dirk Klimpel 6edefef602
Add some type hints to datastore (#12717) 2022-05-17 15:29:06 +01:00
Sean Quah 6ee61b9052
Complain if a federation endpoint has the `@cancellable` flag (#12705)
`BaseFederationServlet` wraps its endpoints in a bunch of async code
that has not been vetted for compatibility with cancellation.
Fail CI if a `@cancellable` flag is applied to a federation endpoint.

Signed-off-by: Sean Quah <seanq@element.io>
2022-05-11 14:52:26 +01:00
Val Lorentz bf0c3ca20a
Fix inconsistent spelling of 'M_UNRECOGNIZED'. (#12665) 2022-05-09 20:29:07 +00:00
DeepBlueV7.X a377a43386
Support MSC3266 room summaries over federation (#11507)
Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
2022-05-05 15:25:00 +01:00
Richard van der Hoff d66d68f917
Add extra debug logging to federation sender (#12614)
... in order to debug some problems we've been having with certain events not
being sent when expected.
2022-05-03 16:32:40 +01:00
Richard van der Hoff db2edf5a65
Exclude OOB memberships from the federation sender (#12570)
As the comment says, there is no need to process such events, and indeed we
need to avoid doing so.

Fixes #12509.
2022-05-03 12:47:56 +00:00
David Robertson 6463244375
Remove unused `# type: ignore`s (#12531)
Over time we've begun to use newer versions of mypy, typeshed, stub
packages---and of course we've improved our own annotations. This makes
some type ignore comments no longer necessary. I have removed them.

There was one exception: a module that imports `select.epoll`. The
ignore is redundant on Linux, but I've kept it ignored for those of us
who work on the source tree using not-Linux. (#11771)

I'm more interested in the config line which enforces this. I want
unused ignores to be reported, because I think it's useful feedback when
annotating to know when you've fixed a problem you had to previously
ignore.

* Installing extras before typechecking

Lacking an easy way to install all extras generically, let's bite the bullet and
make install the hand-maintained `all` extra before typechecking.

Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to
the release/v1 branch.
2022-04-27 14:03:44 +01:00
Jan Christian Grünhage a1f87f57ff
Implement MSC3383: include destination in X-Matrix auth header (#11398)
Co-authored-by: Jan Christian Grünhage <jan.christian@gruenhage.xyz>
Co-authored-by: Marcus Hoffmann <bubu@bubu1.eu>
2022-04-19 16:23:53 +01:00
Richard van der Hoff b121a3ad2b
Back out implementation of MSC2314 (#12474)
MSC2314 has now been closed, so we're backing out its implementation, which
originally happened in #6176.

Unfortunately it's not a direct revert, as that PR mixed in a bunch of
unrelated changes to tests etc.
2022-04-19 11:17:29 +00:00
Patrick Cloke 4bdbebccb9
Remove the unstable event field for `/send_join` per MSC3083. (#12395)
This was missed when initially stabilising room version 8 and was
left in as a compatibility shim. Most homeservers have upgraded
to a version which expects the proper field name, and the failure
mode is reasonable (a user on an older server may have to attempt
joining the room twice with an obscure error message the first time).
2022-04-12 11:27:45 -04:00
David Robertson 95a038c106
Unify HTTP query parameter type hints (#12415)
* Pull out query param types to `synapse.http.types`
* Use QueryParams everywhere
* Simplify `encode_query_args`
* Add annotation which would have caught #12410
2022-04-08 13:06:51 +01:00
Erik Johnston 36af768c13
Fix fetching public rooms over federation (#12410)
Broke by #12364
2022-04-07 14:18:02 +00:00
Sean Quah 800ba87cc8
Refactor and convert `Linearizer` to async (#12357)
Refactor and convert `Linearizer` to async. This makes a `Linearizer`
cancellation bug easier to fix.

Also refactor to use an async context manager, which eliminates an
unlikely footgun where code that doesn't immediately use the context
manager could forget to release the lock.

Signed-off-by: Sean Quah <seanq@element.io>
2022-04-05 15:43:52 +01:00
reivilibre 42d8710f38
Fix a spec compliance issue where requests to the `/publicRooms` federation API would specify `limit` as a string. (#12364) 2022-04-05 12:45:36 +01:00
Richard van der Hoff 38adf14998
Enhance logging for inbound federation events (#12301)
It is currently rather hard to see which rooms are causing inbound federation
traffic. Add the room id to the logs.
2022-03-25 14:44:57 +00:00
Richard van der Hoff afa17f0eab
Return a 404 from `/state` for an outlier (#12087)
* Replace `get_state_for_pdu` with  `get_state_ids_for_pdu` and `get_events_as_list`.
* Return a 404 from `/state` and `/state_ids` for an outlier
2022-03-21 11:23:32 +00:00
Patrick Cloke 54f674f7a9
Deprecate the groups/communities endpoints and add an experimental configuration flag. (#12200) 2022-03-12 13:23:37 -05:00
Patrick Cloke 3e4af36bc8
Rename get_tcp_replication to get_replication_command_handler. (#12192)
Since the object it returns is a ReplicationCommandHandler.

This is clean-up from adding support to Redis where the command handler
was added as an additional layer of abstraction from the TCP protocol.
2022-03-10 13:01:56 +00:00
Erik Johnston 423cca9efe
Spread out sending device lists to remote hosts (#12132) 2022-03-04 11:48:15 +00:00
Patrick Cloke 1103c5fe8a
Check if instances are lists, not sequences. (#12128)
As a str is a sequence, the checks were not granular
enough and would allow lists or strings, when only
lists were valid.
2022-03-02 13:18:51 +00:00
Patrick Cloke 7754af24ab
Remove the unstable `/spaces` endpoint. (#12073)
...and various code supporting it.

The /spaces endpoint was from an old version of MSC2946 and included
both a Client-Server and Server-Server API. Note that the unstable
/hierarchy endpoint (from the final version of MSC2946) is not yet
removed.
2022-02-28 18:33:00 +00:00
David Robertson 5565f454e1
Actually fix bad debug logging rejecting device list & signing key transactions (#12098) 2022-02-28 14:10:36 +00:00
Patrick Cloke 9e83521af8
Properly failover for unknown endpoints from Conduit/Dendrite. (#12077)
Before this fix, a legitimate 404 from a federation endpoint (e.g. due
to an unknown room) would be treated as an unknown endpoint. This
could cause unnecessary federation traffic.
2022-02-28 07:52:44 -05:00
Richard van der Hoff e24ff8ebe3
Remove `HomeServer.get_datastore()` (#12031)
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.

Part of #11733
2022-02-23 11:04:02 +00:00
Brendan Abolivier 250104d357
Implement account status endpoints (MSC3720) (#12001)
See matrix-org/matrix-doc#3720

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-02-22 15:10:10 +00:00
Richard van der Hoff 7273011f60
Faster joins: Support for calling `/federation/v1/state` (#12013)
This is an endpoint that we have server-side support for, but no client-side support. It's going to be useful for resyncing partial-stated rooms, so let's introduce it.
2022-02-22 12:17:10 +00:00
Richard van der Hoff 3070af4809
remote join processing: get create event from state, not auth_chain (#12039)
A follow-up to #12005, in which I apparently missed that there are a bunch of other places that assume the create event is in the auth chain.
2022-02-21 19:27:35 +00:00
Richard van der Hoff a85dde3445
Minor typing fixes (#12034)
These started failing in
https://github.com/matrix-org/synapse/pull/12031... I'm a bit mystified by how
they ever worked.
2022-02-21 18:37:04 +00:00
Richard van der Hoff da0e9f8efd
Faster joins: parse msc3706 fields in send_join response (#12011)
Part of my work on #11249: add code to handle the new fields added in MSC3706.
2022-02-17 16:11:59 +00:00
David Robertson 4ae956c8bb
Use version string helper from matrix-common (#11979)
* Require latest matrix-common
* Use the common function
2022-02-14 13:12:22 +00:00
Richard van der Hoff 63c46349c4
Implement MSC3706: partial state in `/send_join` response (#11967)
* Make `get_auth_chain_ids` return a Set

It has a set internally, and a set is often useful where it gets used, so let's
avoid converting to an intermediate list.

* Minor refactors in `on_send_join_request`

A little bit of non-functional groundwork

* Implement MSC3706: partial state in /send_join response
2022-02-12 10:44:16 +00:00
Richard van der Hoff 964f5b9324
Improve opentracing for federation requests (#11870)
The idea here is to set the parent span for incoming federation requests to the
*outgoing* span on the other end. That means that you can see (most of) the
full end-to-end flow when you have a process that includes federation requests.

However, in order not to lose information, we still want a link to the
`incoming-federation-request` span from the servlet, so we have to create
another span to do exactly that.
2022-02-03 12:29:16 +00:00
David Robertson dd7f825118
Fix losing incoming EDUs if debug logging enabled (#11890)
* Fix losing incoming EDUs if debug logging enabled

Fixes #11889. Homeservers should only be affected if the
`synapse.8631_debug` logger was enabled for DEBUG mode.

I am not sure if this merits a bugfix release: I think the logging can
be disabled in config if anyone is affected? But it is still pretty bad.
2022-02-02 16:25:17 +00:00
Dirk Klimpel 0d6cfea9b8
Add admin API to reset connection timeouts for remote server (#11639)
* Fix get federation status of destination if no error occured
2022-01-25 12:06:29 +00:00
David Robertson f160fe18e3
Debug for device lists updates (#11760)
Debug for #8631.

I'm having a hard time tracking down what's going wrong in that issue.
In the reported example, I could see server A sending federation traffic
to server B and all was well. Yet B reports out-of-sync device updates
from A.

I couldn't see what was _in_ the events being sent from A to B. So I
have added some crude logging to track

- when we have updates to send to a remote HS
- the edus we actually accumulate to send
- when a federation transaction includes a device list update edu
- when such an EDU is received

This is a bit of a sledgehammer.
2022-01-20 13:38:44 +00:00
Sean Quah af13a3be29
Fix a bug that corrupted the cache of federated space hierarchies (#11775)
`FederationClient.get_room_hierarchy()` caches its return values, so
refactor the code to avoid modifying the returned room summary.
2022-01-20 11:03:42 +00:00
Richard van der Hoff 251b5567ec
Remove `log_function` and its uses (#11761)
I've never found this terribly useful. I think it was added in the early days
of Synapse, without much thought as to what would actually be useful to log,
and has just been cargo-culted ever since.

Rather, it tends to clutter up debug logs with useless information.
2022-01-18 13:06:04 +00:00
Patrick Cloke 10a88ba91c
Use auto_attribs/native type hints for attrs classes. (#11692) 2022-01-13 13:49:28 +00:00
Shay 70ce9aea71
Strip unauthorized fields from `unsigned` object in events received over federation (#11530)
* add some tests to verify we are stripping unauthorized fields out of unsigned

* add function to strip unauthorized fields from the unsigned object of event

* newsfragment

* update newsfragment number

* add check to on_send_membership_event

* refactor tests

* fix lint error

* slightly refactor tests and add some comments

* slight refactor

* refactor tests

* fix import error

* slight refactor

* remove unsigned filtration code from synapse/handlers/federation_event.py

* lint

* move unsigned filtering code to event base

* refactor tests

* update newsfragment

* requested changes

* remove unused retun values
2022-01-06 09:09:30 -08:00
Richard van der Hoff 0fb3dd0830
Refactor the way we set `outlier` (#11634)
* `_auth_and_persist_outliers`: mark persisted events as outliers

Mark any events that get persisted via `_auth_and_persist_outliers` as, well,
outliers.

Currently this will be a no-op as everything will already be flagged as an
outlier, but I'm going to change that.

* `process_remote_join`: stop flagging as outlier

The events are now flagged as outliers later on, by `_auth_and_persist_outliers`.

* `send_join`: remove `outlier=True`

The events created here are returned in the result of `send_join` to
`FederationHandler.do_invite_join`. From there they are passed into
`FederationEventHandler.process_remote_join`, which passes them to
`_auth_and_persist_outliers`... which sets the `outlier` flag.

* `get_event_auth`: remove `outlier=True`

stop flagging the events returned by `get_event_auth` as outliers. This method
is only called by `_get_remote_auth_chain_for_event`, which passes the results
into `_auth_and_persist_outliers`, which will flag them as outliers.

* `_get_remote_auth_chain_for_event`: remove `outlier=True`

we pass all the events into `_auth_and_persist_outliers`, which will now flag
the events as outliers.

* `_check_sigs_and_hash_and_fetch`: remove unused `outlier` parameter

This param is now never set to True, so we can remove it.

* `_check_sigs_and_hash_and_fetch_one`: remove unused `outlier` param

This is no longer set anywhere, so we can remove it.

* `get_pdu`: remove unused `outlier` parameter

... and chase it down into `get_pdu_from_destination_raw`.

* `event_from_pdu_json`: remove redundant `outlier` param

This is never set to `True`, so can be removed.

* changelog

* update docstring
2022-01-05 12:26:11 +00:00