Commit Graph

3110 Commits

Author SHA1 Message Date
Brendan Abolivier f141af4c79
Update copyright 2019-11-05 14:52:38 +00:00
Brendan Abolivier 1dffa78701
Filter events_before and events_after in /context requests
While the current version of the spec doesn't say much about how this endpoint uses filters (see https://github.com/matrix-org/matrix-doc/issues/2338), the current implementation is that some fields of an EventFilter apply (the ones that are used when running the SQL query) and others don't (the ones that are used by the filter itself) because we don't call event_filter.filter(...). This seems counter-intuitive and probably not what we want so this commit fixes it.
2019-11-05 12:21:59 +00:00
Andrew Morgan c2203bea57 Re-add docstring, with caveats detailed 2019-11-04 18:17:11 +00:00
Andrew Morgan 0287d033ee Transfer upgraded rooms on groups 2019-11-04 18:08:50 +00:00
Brendan Abolivier 09957ce0e4
Implement per-room message retention policies 2019-11-04 17:09:22 +00:00
Erik Johnston 6a0092d371 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history 2019-11-04 13:29:35 +00:00
Richard van der Hoff c6516adbe0
Factor out an _AsyncEventContextImpl (#6298)
The intention here is to make it clearer which fields we can expect to be
populated when: notably, that the _event_type etc aren't used for the
synchronous impl of EventContext.
2019-11-01 16:19:09 +00:00
Richard van der Hoff 1cb84c6486
Support for routing outbound HTTP requests via a proxy (#6239)
The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy.

The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`.

The proxy will then be used for
 * push
 * url previews
 * phone-home stats
 * recaptcha validation
 * CAS auth validation

It will *not* be used for:
 * Application Services
 * Identity servers
 * Outbound federation
 * In worker configurations, connections from workers to masters

Fixes #4198.
2019-11-01 14:07:44 +00:00
Andrew Morgan fe1f2b4520
Remove last usages of deprecated logging.warn method (#6314) 2019-11-01 12:03:44 +00:00
Andrew Morgan ace947e8da
Depublish a room from the public rooms list when it is upgraded (#6232) 2019-11-01 10:28:09 +00:00
Hubert Chathi 6f4bc6d01d Merge branch 'develop' into cross-signing_federation 2019-10-31 22:38:21 -04:00
Amber Brown 020add5099
Update black to 19.10b0 (#6304)
* update version of black and also fix the mypy config being overridden
2019-11-01 02:43:24 +11:00
Erik Johnston cd581338cf Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history 2019-10-31 15:19:26 +00:00
Erik Johnston dfe0cd71b6
Merge pull request #6294 from matrix-org/erikj/add_state_storage
Add StateGroupStorage interface
2019-10-31 16:17:53 +01:00
Andrew Morgan 54fef094b3
Remove usage of deprecated logger.warn method from codebase (#6271)
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
2019-10-31 10:23:24 +00:00
Erik Johnston 7c8c97e635 Split purge API into events vs state 2019-10-30 15:23:37 +00:00
Erik Johnston 69f0054ce6 Port to use state storage 2019-10-30 14:46:54 +00:00
Hubert Chathi 7d7eac61be Merge branch 'develop' into cross-signing_federation 2019-10-30 10:17:10 -04:00
Hubert Chathi bc32f102cd black 2019-10-30 10:07:36 -04:00
Hubert Chathi d78b1e339d apply changes as a result of PR review 2019-10-30 10:01:53 -04:00
Erik Johnston ec6de1cc7d Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_persistence_store 2019-10-30 13:37:04 +00:00
Andrew Morgan a2276d4d3c
Fix log line that was printing undefined value (#6278) 2019-10-30 11:28:48 +00:00
Erik Johnston 2c35ffead2 Port receipt and read markers to async/wait 2019-10-29 15:08:22 +00:00
Richard van der Hoff 172f264ed3
Improve signature checking on some federation APIs (#6262)
Make sure that we check that events sent over /send_join, /send_leave, and
/invite, are correctly signed and come from the expected servers.
2019-10-28 12:43:23 +00:00
Erik Johnston f666d6f5d7 Remove repeated calls to config.stats_enabled.
Turns out that fetching variables from the config object is expensive,
so doing it once at startup avoids unnecessary work.
2019-10-25 10:28:36 +01:00
Hubert Chathi ff05c9b760 don't error if federation query doesn't have cross-signing keys 2019-10-24 21:46:11 -04:00
Erik Johnston 3ca4c7c516 Use new EventPersistenceStore 2019-10-23 16:15:03 +01:00
Hubert Chathi 480eac30eb black 2019-10-22 22:37:16 -04:00
Hubert Chathi 404e8c8532 vendor-prefix the EDU name until MSC1756 is merged into the spec 2019-10-22 22:33:23 -04:00
Hubert Chathi 1fabf82d50 update to work with newer code, and fix formatting 2019-10-22 21:44:58 -04:00
Hubert Chathi 41ad35b523 add missing param 2019-10-22 19:06:29 -04:00
Hubert Chathi cfdb84422d make black happy 2019-10-22 19:06:06 -04:00
Hubert Chathi 8d3542a64e implement federation parts of cross-signing 2019-10-22 19:04:35 -04:00
Hubert Chathi 2761731634
Merge pull request #5726 from matrix-org/uhoreg/e2e_cross-signing2-part2
Cross-signing [3/4] -- uploading signatures edition
2019-10-22 17:05:06 +01:00
Richard van der Hoff 93eaeec75a
Remove Auth.check method (#6217)
This method was somewhat redundant, and confusing.
2019-10-18 19:43:36 +02:00
Hubert Chathi 36adfaedab Merge branch 'develop' into cross-signing_sig_upload 2019-10-18 18:34:42 +01:00
Hubert Chathi 125eb45e19 fix doc strings 2019-10-18 16:56:16 +01:00
Hubert Chathi 0a9d22808c Merge branch 'uhoreg/e2e_cross-signing_merged' into develop 2019-10-18 15:57:40 +01:00
Richard van der Hoff 2b22faded7 Merge remote-tracking branch 'origin/develop' into rav/event_auth/1 2019-10-18 12:35:13 +01:00
Hubert Chathi cfc28325a6 Merge branch 'develop' into uhoreg/e2e_cross-signing_merged 2019-10-18 11:39:40 +01:00
Richard van der Hoff 80003dfcd5 Merge remote-tracking branch 'origin/develop' into rav/event_auth/1 2019-10-18 10:11:40 +01:00
Erik Johnston 5859a5c569 Fix presence timeouts when synchrotron restarts. (#6212)
* Fix presence timeouts when synchrotron restarts.

Handling timeouts would fail if there was an external process that had
timed out, e.g. a synchrotron restarting. This was due to a couple of
variable name typoes.

Fixes #3715.
2019-10-18 06:42:26 +01:00
Richard van der Hoff 1ba359a11f rip out some unreachable code
The only possible rejection reason is AUTH_ERROR, so all of this is unreachable.
2019-10-17 21:45:59 +01:00
Hubert Chathi 06fc66c81e
Merge pull request #6193 from matrix-org/uhoreg/interpret_device_key_in_storage
make storage layer in charge of interpreting the device key data
2019-10-11 15:19:06 -04:00
Hubert Chathi 691dd67fcd
Merge pull request #6189 from matrix-org/uhoreg/e2e_backup_optional_version
make version optional in body of e2e backup version update
2019-10-11 10:11:59 -04:00
Hubert Chathi 7a0dce9259 make sure we actually return something 2019-10-10 20:31:30 -04:00
Hubert Chathi 4908fb3b30 make storage layer in charge of interpreting the device key data 2019-10-10 19:15:30 -04:00
werner291 b5b03b7079 Add domain validation when creating room with list of invitees (#6121) 2019-10-10 13:05:48 +01:00
Richard van der Hoff a139420a3c
Fix races in room stats (and other) updates. (#6187)
Hopefully this will fix the occasional failures we were seeing in the room directory.

The problem was that events are not necessarily persisted (and `current_state_delta_stream` updated) in the same order as their stream_id. So for instance current_state_delta 9 might be persisted *before* current_state_delta 8. Then, when the room stats saw stream_id 9, it assumed it had done everything up to 9, and never came back to do stream_id 8.

We can solve this easily by only processing up to the stream_id where we know all events have been persisted.
2019-10-10 11:29:01 +01:00
Andrew Morgan da815c1f69 Move tag/push rules room upgrade checking ealier (#6155)
It turns out that _local_membership_update doesn't run when you join a new, remote room. It only runs if you're joining a room that your server already knows about. This would explain #4703 and #5295 and why the transfer would work in testing and some rooms, but not others. This would especially hit single-user homeservers.

The check has been moved to right after the room has been joined, and works much more reliably. (Though it may still be a bit awkward of a place).
2019-10-10 10:06:45 +01:00
Hubert Chathi 4535a07f4a make version optional in body of e2e backup version update
to agree with latest version of the MSC
2019-10-09 17:54:03 -04:00
Andrew Morgan ea7d938bca Remove unused public room list timeout param (#6179)
* Remove unused public room list timeout param

* Add changelog
2019-10-08 13:51:25 +01:00
Erik Johnston 2cb7466993
Merge pull request #6161 from matrix-org/erikj/dont_regen_user_id_on_failure
Don't regenerate numeric user ID if registration fails.
2019-10-07 16:56:23 +01:00
Richard van der Hoff 276ae5c63e
add some logging to the rooms stats updates, to try to track down a flaky test (#6167) 2019-10-07 14:41:39 +01:00
Brendan Abolivier ae0b78cb1f
Merge pull request #6147 from matrix-org/babolivier/3pid-invite-revoked
Don't 500 when trying to exchange a revoked 3PID invite
2019-10-04 12:09:05 +01:00
Brendan Abolivier 21d51ab598
Typo 2019-10-04 11:21:24 +01:00
Brendan Abolivier 4676732ca0
Lint 2019-10-04 11:18:28 +01:00
Brendan Abolivier 81d51ce48b
Incorporate review 2019-10-04 11:16:19 +01:00
Erik Johnston ab8a64772b Remove unused variable 2019-10-03 17:42:32 +01:00
Erik Johnston 693156aaf4 Don't regenerate numeric user ID if registration fails.
This causes huge amounts of DB IO if registrations start to fail e.g.
because the DB is struggling with IO.
2019-10-03 17:33:54 +01:00
Brendan Abolivier 6527fa18c1 Add test case 2019-10-03 11:24:36 +01:00
Erik Johnston 03cf4385e0 Fix public room list pagination.
We incorrectly used `room_id` as to bound the result set, even though we
order by `joined_members, room_id`, leading to incorrect results after
pagination.
2019-10-02 15:11:17 +01:00
Erik Johnston a5166e4d5f
Land improved room list based on room stats (#6019)
Use room_stats and room_state for room directory search
2019-10-02 14:08:35 +01:00
Brendan Abolivier 972c9f65d7
Lint 2019-10-02 12:17:46 +01:00
Andrew Morgan 2a1470cd05
Fix yields and copy instead of move push rules on room upgrade (#6144)
Copy push rules during a room upgrade from the old room to the new room, instead of deleting them from the old room.

For instance, we've defined upgrading of a room multiple times to be possible, and push rules won't be transferred on the second upgrade if they're deleted during the first.

Also fix some missing yields that probably broke things quite a bit.
2019-10-02 12:04:22 +01:00
Brendan Abolivier 5705ecaec6
Don't 500 code when trying to exchange a revoked 3PID invite
While this is not documented in the spec (but should be), Riot (and other clients) revoke 3PID invites by sending a m.room.third_party_invite event with an empty ({}) content to the room's state.
When the invited 3PID gets associated with a MXID, the identity server (which doesn't know about revocations) sends down to the MXID's homeserver all of the undelivered invites it has for this 3PID. The homeserver then tries to talk to the inviting homeserver in order to exchange these invite for m.room.member events.
When one of the invite is revoked, the inviting homeserver responds with a 500 error because it tries to extract a 'display_name' property from the content, which is empty. This might cause the invited server to consider that the server is down and not try to exchange other, valid invites (or at least delay it).

This fix handles the case of revoked invites by avoiding trying to fetch a 'display_name' from the original invite's content, and letting the m.room.member event fail the auth rules (because, since the original invite's content is empty, it doesn't have public keys), which results in sending a 403 with the correct error message to the invited server.
2019-10-02 11:19:43 +01:00
Brendan Abolivier 3e42d47a5a
Incorporate review 2019-09-27 16:15:01 +01:00
Brendan Abolivier af92110c46
Update synapse/handlers/deactivate_account.py
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2019-09-27 16:12:15 +01:00
Brendan Abolivier bbe2a0f339
Update synapse/handlers/deactivate_account.py
Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2019-09-27 16:10:36 +01:00
Brendan Abolivier 873fe7883c
Lint 2019-09-27 15:21:03 +01:00
Brendan Abolivier 72a2708ac6
Fixup and add some logging 2019-09-27 15:13:39 +01:00
Brendan Abolivier 5257a2fb1c
Reject pending invites on deactivation 2019-09-27 14:49:53 +01:00
Andrew Morgan 8c27bc8b60
Move lookup-related functions from RoomMemberHandler to IdentityHandler (#5978)
Just to have all the methods that make calls to identity services in one place.
2019-09-27 10:36:20 +01:00
Neil Johnson 034db2ba21 Fix dummy event insertion consent bug (#6053)
Fixes #5905
2019-09-26 11:47:53 +01:00
Neil Johnson 77dc7093a7 Threepid validity checks on msisdns should not be dependent on 'threepid_behaviour_email'. (#6104)
Fixes #6103
2019-09-25 12:29:35 +01:00
Richard van der Hoff 990928abde
Stop advertising unsupported flows for registration (#6107)
If email or msisdn verification aren't supported, let's stop advertising them
for registration.

Fixes #6100.
2019-09-25 12:10:26 +01:00
Richard van der Hoff 2cd98812ba
Refactor the user-interactive auth handling (#6105)
Pull the checkers out to their own classes, rather than having them lost in a
massive 1000-line class which does everything.

This is also preparation for some more intelligent advertising of flows, as per #6100
2019-09-25 11:33:03 +01:00
Hubert Chathi c3635c9459 make isort happy 2019-09-24 16:21:03 -04:00
Hubert Chathi f4b6d43ec3 add some comments 2019-09-24 16:19:54 -04:00
Hubert Chathi 39864f45ec drop some logger lines to debug 2019-09-24 15:26:45 -04:00
Hubert Chathi 26113fb7de make changes based on PR feedback 2019-09-24 14:12:20 -04:00
Richard van der Hoff 9b7c4f4078 Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work 2019-09-24 17:03:50 +01:00
J. Ryan Stinnett 40fb00f5b7
Add sid to next_link for email validation (#6097) 2019-09-24 14:39:50 +01:00
Richard van der Hoff ed8b92f0d2 Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work 2019-09-24 12:57:32 +01:00
Andrew Morgan 50776261e1 Add submit_url response parameter to msisdn /requestToken (#6079)
Second part of solving #6076
Fixes #6076

We return a submit_url parameter on calls to POST */msisdn/requestToken so that clients know where to submit token information to.
2019-09-23 21:21:03 +01:00
Andrew Morgan e08ea43463 Use the federation blacklist for requests to untrusted Identity Servers (#6000)
Uses a SimpleHttpClient instance equipped with the federation_ip_range_blacklist list for requests to identity servers provided by user input. Does not use a blacklist when contacting identity servers specified by account_threepid_delegates. The homeserver trusts the latter and we don't want to prevent homeserver admins from specifying delegates that are on internal IP addresses.

Fixes #5935
2019-09-23 20:23:20 +01:00
Andrew Morgan 2c99c63453 Add POST submit_token endpoint for MSISDN (#6078)
First part of solving #6076
2019-09-23 17:49:00 +01:00
Andrew Morgan 30af161af2 Implement MSC2290 (#6043)
Implements MSC2290. This PR adds two new endpoints, /unstable/account/3pid/add and /unstable/account/3pid/bind. Depending on the progress of that MSC the unstable prefix may go away.

This PR also removes the blacklist on some 3PID tests which occurs in #6042, as the corresponding Sytest PR changes them to use the new endpoints.

Finally, it also modifies the account deactivation code such that it doesn't just try to deactivate 3PIDs that were bound to the user's account, but any 3PIDs that were bound through the homeserver on that user's account.
2019-09-23 16:50:27 +01:00
Andrew Morgan 885a4726b7 Return timeout error to user for identity server calls (#6073) 2019-09-23 14:37:23 +01:00
Andrew Morgan df3401a71d
Allow HS to send emails when adding an email to the HS (#6042) 2019-09-20 15:21:30 +01:00
Richard van der Hoff 366dc7d2d4 Merge commit '33757bad1' into rav/saml_mapping_work 2019-09-20 11:21:40 +01:00
Richard van der Hoff 33757bad19 More better logging 2019-09-20 11:20:02 +01:00
Richard van der Hoff b65327ff66 Merge branch 'develop' into rav/saml_mapping_work 2019-09-19 18:13:31 +01:00
Richard van der Hoff 7423fade92 better logging 2019-09-19 17:16:50 +01:00
Andrew Morgan 6670bd4072
v2 3PID Invites (part of MSC2140) (#5979)
3PID invites require making a request to an identity server to check that the invited 3PID has an Matrix ID linked, and if so, what it is.

These requests are being made on behalf of a user. The user will supply an identity server and an access token for that identity server. The homeserver will then forward this request with the access token (using an `Authorization` header) and, if the given identity server doesn't support v2 endpoints, will fall back to v1 (which doesn't require any access tokens).

Requires: ~~#5976~~
2019-09-17 18:05:13 +01:00
Erik Johnston 70c52821ce Fix race condition in room stats. (#6029)
Broke in #5971

Basically the bug is that if get_current_state_deltas returns no new updates and we then take the max pos, its possible that we miss an update that happens in between the two calls. (e.g. get_current_state_deltas looks up to stream pos 5, then an event persists and so getting the max stream pos returns 6, meaning that next time we check for things with a stream pos bigger than 6)
2019-09-17 12:41:23 +01:00
Richard van der Hoff a8ac40445c Record mappings from saml users in an external table
We want to assign unique mxids to saml users based on an incrementing
suffix. For that to work, we need to record the allocated mxid in a separate
table.
2019-09-13 16:01:46 +01:00
Andrew Morgan 9fc71dc5ee
Use the v2 Identity Service API for lookups (MSC2134 + MSC2140) (#5976)
This is a redo of https://github.com/matrix-org/synapse/pull/5897 but with `id_access_token` accepted.

Implements [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134) plus Identity Service v2 authentication ala [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140).

Identity lookup-related functions were also moved from `RoomMemberHandler` to `IdentityHandler`.
2019-09-11 16:02:42 +01:00
Erik Johnston cbcbfe64a2
Merge pull request #6015 from matrix-org/erikj/ratelimit_admin_redaction
Allow use of different ratelimits for admin redactions.
2019-09-11 15:39:38 +01:00
Andrew Morgan 9c555f37e3
Add note about extra arg to send_membership_event, remove arg in remote_reject_invite (#6009)
Some small fixes to `room_member.py` found while doing other PRs.

1. Add requester to the base `_remote_reject_invite` method.
2. `send_membership_event`'s docstring was out of date and took in a `remote_room_hosts` arg that was not used and no calling function provided.
2019-09-11 14:23:24 +01:00