Commit Graph

528 Commits

Author SHA1 Message Date
Erik Johnston ea7b3c4b1b Remove unused ReplicationLayer 2018-03-13 11:00:04 +00:00
Erik Johnston 265b993b8a Split replication layer into two 2018-03-13 10:55:47 +00:00
Erik Johnston e05bf34117 Move property setting from ReplicationLayer to FederationBase 2018-03-13 10:51:30 +00:00
Erik Johnston c3f79c9da5 Split out edu/query registration to a separate class 2018-03-13 10:24:27 +00:00
Matthew Hodgson ab9f844aaf
Add federation_domain_whitelist option (#2820)
Add federation_domain_whitelist

gives a way to restrict which domains your HS is allowed to federate with.
useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network
2018-01-22 19:11:18 +01:00
Richard van der Hoff a027c2af8d Metrics for events processed in appservice and fed sender
More metrics I wished I'd had
2018-01-15 18:23:24 +00:00
Richard van der Hoff bd91857028 Check missing fields in event_from_pdu_json
Return a 400 rather than a 500 when somebody messes up their send_join
2017-12-30 18:40:19 +00:00
Richard van der Hoff 3079f80d4a Factor out `event_from_pdu_json`
turns out we have two copies of this, and neither needs to be an instance
method
2017-12-30 18:40:19 +00:00
Richard van der Hoff 65abc90fb6 federation_server: clean up imports 2017-12-30 18:40:19 +00:00
Richard van der Hoff a7b726ad18 federation_client: clean up imports 2017-12-30 18:40:19 +00:00
Richard van der Hoff d4fb4f7c52 Clear logcontext before starting fed txn queue runner
These processes take a long time compared to the request, so there is lots of
"Entering|Restoring dead context" in the logs. Let's try to shut it up a bit.
2017-11-28 15:26:14 +00:00
Richard van der Hoff 7e6fa29cb5 Remove preserve_context_over_{fn, deferred}
Both of these functions ae known to leak logcontexts. Replace the remaining
calls to them and kill them off.
2017-11-14 11:22:42 +00:00
Erik Johnston 82e4bfb53d Add brackets 2017-11-09 10:06:42 +00:00
Erik Johnston e8814410ef Have an explicit API to update room config 2017-11-08 16:13:27 +00:00
Erik Johnston 94ff2cda73
Revert "Modify group room association API to allow modification of is_public" 2017-11-08 15:43:34 +00:00
Luke Barnard 207fabbc6a Update docs for updating room group association 2017-11-01 09:35:15 +00:00
Luke Barnard 13b3d7b4a0 Flake8 2017-10-31 17:20:11 +00:00
Luke Barnard 20fe347906 Modify group room association API to allow modification of is_public
also includes renamings to make things more consistent.
2017-10-31 17:04:28 +00:00
Erik Johnston 2a7e9faeec Do logcontexts outside ResponseCache 2017-10-25 15:21:08 +01:00
Erik Johnston 39dc52157d Merge branch 'develop' of github.com:matrix-org/synapse into erikj/group_fed_update_profile 2017-10-24 09:16:20 +01:00
Richard van der Hoff eaaabc6c4f replace 'except:' with 'except Exception:'
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Erik Johnston ce6d4914f4 Correctly wire in update group profile over federation 2017-10-23 15:21:24 +01:00
Erik Johnston 011d03a0f6 Fix typo 2017-10-19 11:22:48 +01:00
Erik Johnston 9ab859f27b Fix typo in group attestation handling 2017-10-19 10:55:52 +01:00
Richard van der Hoff 582bd19ee9 Fix 500 error when we get an error handling a PDU
FederationServer doesn't have a send_failure (and nor does its subclass,
ReplicationLayer), so this was failing.

I'm not really sure what the idea behind send_failure is, given (a) we don't do
anything at the other end with it except log it, and (b) we also send back the
failure via the transaction response. I suspect there's a whole lot of dead
code around it, but for now I'm just removing the broken bit.
2017-10-17 20:52:40 +01:00
Luke Barnard 85f5674e44 Delint 2017-10-16 15:52:17 +01:00
Luke Barnard 2c5972f87f Implement GET /groups/$groupId/invited_users 2017-10-16 15:31:11 +01:00
Richard van der Hoff 8dd0c85ac5 Merge pull request #2529 from matrix-org/rav/fix_transaction_failure_handling
log pdu_failures from incoming transactions
2017-10-11 17:29:14 +01:00
Erik Johnston 271f5601f3 Fix typo in invite to group 2017-10-11 14:45:33 +01:00
Richard van der Hoff b75d443caf log pdu_failures from incoming transactions
... even if we have no EDUs.

This appears to have been introduced in
476899295f.
2017-10-11 14:36:13 +01:00
Erik Johnston 535cc49f27 Merge pull request #2466 from matrix-org/erikj/groups_merged
Initial Group Implementation
2017-10-11 13:20:07 +01:00
Richard van der Hoff 6a6cc27aee fed server: process PDUs for different rooms in parallel
With luck, this will give a real-time improvement when there are many rooms and
the server ends up calling out to fetch missing events.
2017-10-09 18:30:31 +01:00
Richard van der Hoff 4c7c4d4061 Fed server: use a linearizer for ongoing transactions
We don't want to process the same transaction multiple times concurrently, so
use a linearizer.
2017-10-09 18:30:10 +01:00
Richard van der Hoff ba5b9b80a5 fed server: refactor on_incoming_transaction
Move as much as possible to after the have_responded check, and reduce the
number of times we iterate over the pdu list.
2017-10-09 18:10:53 +01:00
Richard van der Hoff a6e3222fe5 Fed server: Move origin-check code to _handle_received_pdu
The response-building code expects there to be an entry in the `results` list
for each entry in the pdu_list, so the early `continue` was messing this
up. That doesn't really matter, because all that the federation client does is
log any errors, but it's pretty poor form.
2017-10-09 17:53:32 +01:00
Richard van der Hoff 01bbacf3c4 Fix up logcontext handling in (federation) TransactionQueue
Avoid using preserve_context_over_function, which has problems with respect to
logcontexts.
2017-10-06 22:39:25 +01:00
David Baker 27955056e0 Merge branch 'develop' into erikj/groups_merged 2017-10-02 16:20:41 +01:00
David Baker 6cd5fcd536 Make the spam checker a module 2017-09-26 19:20:23 +01:00
Erik Johnston 17b8e2bd02 Add remove room API 2017-09-26 15:52:41 +01:00
Richard van der Hoff 6de74ea6d7 Fix logcontexts in _check_sigs_and_hashes 2017-09-20 01:32:42 +01:00
Richard van der Hoff fcf2c0fd1a Remove redundant `preserve_fn`
preserve_fn is a no-op unless the wrapped function returns a
Deferred. verify_json_objects_for_server returns a list, so this is doing
nothing.
2017-09-20 01:32:42 +01:00
Richard van der Hoff 5ed109d59f PoC for filtering spammy events (#2456)
Demonstration of how you might add some hooks to filter out spammy events.
2017-09-19 12:20:11 +01:00
Erik Johnston ef8e578677 Add bulk group publicised lookup API 2017-08-09 13:36:22 +01:00
Erik Johnston 6f443a74cf Add update group profile API 2017-07-20 09:46:33 +01:00
Erik Johnston 14a34f12d7 Comments 2017-07-18 17:28:42 +01:00
Erik Johnston 3431ec55dc Comments 2017-07-18 17:23:50 +01:00
Erik Johnston 94ecd871a0 Fix typos 2017-07-18 16:38:54 +01:00
Erik Johnston 12ed4ee48e Correctly parse query params 2017-07-18 15:33:09 +01:00
Erik Johnston e5ea6dd021 Add client apis 2017-07-18 14:37:06 +01:00
Erik Johnston cccfcfa7b9 Comments 2017-07-18 10:35:18 +01:00
Erik Johnston 2f9eafdd36 Add local group server support 2017-07-17 12:03:49 +01:00
Erik Johnston cb3aee8219 Ensure category and role ids are non-null 2017-07-14 14:06:55 +01:00
Erik Johnston 8575e3160f Comments 2017-07-13 13:52:41 +01:00
Erik Johnston 7a39da8cc6 Add summary APIs to federation 2017-07-13 11:13:19 +01:00
Erik Johnston a62406aaa5 Add group summary APIs 2017-07-12 11:36:15 +01:00
Erik Johnston fe4e885f54 Add federation API for adding room to group 2017-07-11 14:35:07 +01:00
Erik Johnston 6322fbbd41 Comment 2017-07-11 11:52:03 +01:00
Erik Johnston b8ca494ee9 Initial group server implementation 2017-07-10 15:44:15 +01:00
Erik Johnston 5e49a57ecc Separate federation servlet into different lists 2017-07-05 14:32:24 +01:00
Erik Johnston 6e2a7ee1bc Remove spurious log lines 2017-06-07 11:05:17 +01:00
Erik Johnston dfbda5e025 Faster cache for get_joined_hosts 2017-05-25 17:24:44 +01:00
Erik Johnston ec5c4499f4 Make presence use cached users/hosts in room 2017-05-16 16:01:43 +01:00
Richard van der Hoff de042b3b88 Do some logging when one-time-keys get claimed
might help us figure out if https://github.com/vector-im/riot-web/issues/3868
has happened.
2017-05-09 19:04:56 +01:00
Erik Johnston 27c28eaa27 Merge pull request #2190 from matrix-org/erikj/mark_remote_as_back_more
Always mark remotes as up if we receive a signed request from them
2017-05-05 14:08:12 +01:00
Erik Johnston 310b1ccdc1 Use preserve_fn and add logs 2017-05-05 13:41:19 +01:00
Erik Johnston db7d0c3127 Always mark remotes as up if we receive a signed request from them 2017-05-05 10:34:53 +01:00
Erik Johnston 7166854f41 Add cache for get_current_hosts_in_room 2017-05-02 10:36:35 +01:00
Richard van der Hoff 91b3981800 Try harder when sending leave events
When we're rejecting invites, ignore the backoff data, so that we have a better
chance of not getting the room out of sync.
2017-04-21 01:50:36 +01:00
Richard van der Hoff 4903ccf159 Fix some lies, and other clarifications, in docstrings
The documentation on get_json has been wrong ever since the very first commit
to synapse...
2017-04-21 01:31:09 +01:00
Erik Johnston 247c736b9b Merge pull request #2115 from matrix-org/erikj/dedupe_federation_repl
Reduce federation replication traffic
2017-04-12 11:07:13 +01:00
Paul Evans 8fbc0d29ee Merge pull request #2121 from matrix-org/paul/sent-transactions-metric
Add a counter metric for successfully-sent transactions
2017-04-12 11:04:31 +01:00
Erik Johnston 26ae5178a4 Add some comments 2017-04-12 10:36:29 +01:00
Erik Johnston 1745069543 Comment 2017-04-12 10:17:10 +01:00
Erik Johnston c7ddb5ef7a Reuse get_interested_parties 2017-04-12 10:16:26 +01:00
Paul "LeoNerd" Evans 11dbceb761 Add a counter metric for successfully-sent transactions 2017-04-11 17:16:12 +01:00
Erik Johnston a8c8e4efd4 Comment 2017-04-11 15:35:49 +01:00
Erik Johnston 2be8a281d2 Comments 2017-04-11 15:28:24 +01:00
Erik Johnston 6308ac45b0 Move get_interested_remotes back to presence handler 2017-04-11 15:19:26 +01:00
Erik Johnston b9b72bc6e2 Comments 2017-04-11 15:15:34 +01:00
Erik Johnston 84fbb80c8f Use generators 2017-04-10 16:55:56 +01:00
Erik Johnston 40453b3f84 Dedupe KeyedEdu and Devices federation repl traffic 2017-04-10 16:49:51 +01:00
Erik Johnston 29574fd5b3 Reduce federation presence replication traffic
This is mainly done by moving the calculation of where to send presence
updates from the presence handler to the transaction queue, so we only
need to send the presence event (and not the destinations) across the
replication connection. Before we were duplicating by sending the full
state across once per destination.
2017-04-10 16:48:30 +01:00
Erik Johnston 0018491af2 Rename variable 2017-04-10 12:44:43 +01:00
Erik Johnston 8c5f03cec7 Revert to sending the same data type as before 2017-04-10 10:07:18 +01:00
Erik Johnston f8434db549 Change name 2017-04-10 10:03:07 +01:00
Erik Johnston ab904caf33 Comments 2017-04-10 10:02:17 +01:00
Erik Johnston a828a64b75 Comment 2017-04-07 11:54:03 +01:00
Erik Johnston d4d176e5d0 Add logging 2017-04-07 11:51:28 +01:00
Erik Johnston 449d1297ca Fix up federation SendQueue and document types 2017-04-07 11:48:33 +01:00
Erik Johnston f10ce8944b Don't double json encode federation replication data 2017-04-05 11:10:28 +01:00
Erik Johnston 62b89daac6 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/repl_tcp_server 2017-04-04 09:46:16 +01:00
Richard van der Hoff 30bcbf775a Accept join events from all servers
Make sure that we accept join events from any server, rather than just the
origin server, to make the federation join dance work correctly.

(Fixes #1893).
2017-04-03 15:58:07 +01:00
Erik Johnston 11880103b1 Make federation send queue take the current position 2017-03-30 12:54:36 +01:00
Erik Johnston 85be3dde81 Bail early if remote wouldn't be retried (#2064)
* Bail early if remote wouldn't be retried

* Don't always return true

* Just use get_retry_limiter

* Spelling
2017-03-29 11:48:27 +01:00
Erik Johnston c5b0bdd542 Merge pull request #2067 from matrix-org/erikj/notify_on_fed
Notify on new federation traffic
2017-03-29 11:41:37 +01:00
Erik Johnston 35b4aa04be Notify on new federation traffic 2017-03-27 14:07:47 +01:00
Erik Johnston 2a28b79e04 Batch sending of device list pokes 2017-03-24 14:44:49 +00:00
Richard van der Hoff 5a16cb4bf0 Ignore backoff history for invites, aliases, and roomdirs
Add a param to the federation client which lets us ignore historical backoff
data for federation queries, and set it for a handful of operations.
2017-03-23 12:23:22 +00:00
Richard van der Hoff 4bd597d9fc push federation retry limiter down to matrixfederationclient
rather than having to instrument everywhere we make a federation call,
make the MatrixFederationHttpClient manage the retry limiter.
2017-03-23 09:28:46 +00:00
Richard van der Hoff 29ed09e80a Fix assertion to stop transaction queue getting wedged
... and update some docstrings to correctly reflect the types being used.

get_new_device_msgs_for_remote can return a long under some circumstances,
which was being stored in last_device_list_stream_id_by_dest, and was then
upsetting things on the next loop.
2017-03-15 12:16:55 +00:00