11a67b7c9d
Merge pull request #3093 from matrix-org/rav/response_cache_wrap
...
Refactor ResponseCache usage
2018-04-20 11:31:17 +01:00
0c280d4d99
Reinstate linearizer for federation_server.on_context_state_request
2018-04-20 11:10:04 +01:00
b1dfbc3c40
Refactor store.have_events
...
It turns out that most of the time we were calling have_events, we were only
using half of the result. Replace have_events with have_seen_events and
get_rejection_reasons, so that we can see what's going on a bit more clearly.
2018-04-20 10:25:56 +01:00
1f4b498b73
Add some comments
2018-04-18 00:15:36 +01:00
36c59ce669
Use six.itervalues in some places
...
There's more where that came from
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 20:39:43 +02:00
78a9698650
fix federation_domain_whitelist
...
we were checking the wrong server_name on inbound requests
2018-04-13 15:47:43 +01:00
25b0ba30b1
revert last to PR properly
2018-04-13 15:46:37 +01:00
f8d46cad3c
correctly auth inbound federation_domain_whitelist reqs
2018-04-13 15:41:52 +01:00
d3347ad485
Revert "Use sortedcontainers instead of blist"
...
This reverts commit 9fbe70a7dc
2018-04-13 11:16:43 +01:00
b78395b7fe
Refactor ResponseCache usage
...
Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a
(get, set) pair, and then use it throughout the codebase.
This will be largely non-functional, but does include the following functional
changes:
* federation_server.on_context_state_request: drops use of _server_linearizer
which looked redundant and could cause incorrect cache misses by yielding
between the get and the set.
* RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks
* the wrap function includes some logging. I'm hoping this won't be too noisy
on production.
2018-04-12 13:02:15 +01:00
d5c74b9f6c
Merge pull request #3092 from matrix-org/rav/response_cache_metrics
...
Add metrics for ResponseCache
2018-04-12 12:59:36 +01:00
f67e906e18
Set all metrics at the same time
2018-04-12 11:18:19 +01:00
4dae4a97ed
Track last processed event received_ts
2018-04-11 14:27:09 +01:00
92e34615c5
Track where event stream processing have gotten up to
2018-04-11 12:13:40 +01:00
233699c42e
Merge pull request #2760 from Valodim/pypy
...
Synapse on PyPy
2018-04-11 11:20:01 +01:00
b3384232a0
Add metrics for ResponseCache
2018-04-10 23:14:47 +01:00
9daf82278f
Merge pull request #3078 from matrix-org/erikj/federation_sender
...
Send federation events concurrently
2018-04-10 16:43:48 +01:00
a060dfa132
Use run_in_background instead
2018-04-10 14:25:11 +01:00
1246d23710
Preserve log contexts correctly
2018-04-10 12:04:32 +01:00
d49cbf712f
Log event ID on exception
2018-04-10 12:03:41 +01:00
11d2609da7
Ensure slashes are escaped
2018-04-10 11:24:40 +01:00
dab87b84a3
URL quote path segments over federation
2018-04-10 11:16:08 +01:00
9fbe70a7dc
Use sortedcontainers instead of blist
...
This commit drop-in replaces blist with SortedContainers. They are
written in pure python so work with pypy, but perform as good as
native implementations, at least in a couple benchmarks:
http://www.grantjenks.com/docs/sortedcontainers/performance.html
2018-04-10 11:29:51 +02:00
6e025a97b4
Handle all events in a room correctly
2018-04-09 16:02:48 +01:00
11974f3787
Send federation events concurrently
2018-04-09 11:47:10 +01:00
145d14656b
Handle exceptions in get_hosts_for_room when sending events over federation
2018-04-09 11:47:01 +01:00
112c2253e2
pep8
2018-04-06 15:43:27 +01:00
f8d1917fce
Fix federation client `set_group_joinable` typo
2018-04-06 15:43:27 +01:00
b370fe61c0
Implement group join API
2018-04-06 15:43:27 +01:00
1d71f484d4
use PUT instead of POST for federating groups/m.join_policy
2018-04-06 12:54:09 +02:00
104c0bc1d5
Use "/settings/" (plural)
2018-04-05 14:07:16 +01:00
eb8d8d6f57
Use join_policy API instead of joinable
...
The API is now under
/groups/$group_id/setting/m.join_policy
and expects a JSON blob of the shape
```json
{
"m.join_policy": {
"type": "invite"
}
}
```
where "invite" could alternatively be "open".
2018-04-03 16:16:40 +01:00
32260baa41
pep8
2018-03-28 14:29:42 +01:00
79452edeee
Add joinability for groups
...
Adds API to set the 'joinable' flag, and corresponding flag in the
table.
2018-03-28 14:03:37 +01:00
95cb401ae0
Merge pull request #2978 from matrix-org/erikj/refactor_replication_layer
...
Remove ReplicationLayer and user Client/Server directly
2018-03-13 15:45:08 +00:00
56e709857c
Merge pull request #2979 from matrix-org/erikj/no_handlers
...
Don't build handlers on workers unnecessarily
2018-03-13 13:46:38 +00:00
cea462e285
s/replication_server/federation_server
2018-03-13 13:22:21 +00:00
9a2d9b4789
Merge pull request #2977 from matrix-org/erikj/replication_move_props
...
Move property setting from ReplicationLayer to base classes
2018-03-13 11:45:25 +00:00
f43b6d6d9b
Fix docstring types
2018-03-13 11:29:35 +00:00
ea7b3c4b1b
Remove unused ReplicationLayer
2018-03-13 11:00:04 +00:00
265b993b8a
Split replication layer into two
2018-03-13 10:55:47 +00:00
e05bf34117
Move property setting from ReplicationLayer to FederationBase
2018-03-13 10:51:30 +00:00
c3f79c9da5
Split out edu/query registration to a separate class
2018-03-13 10:24:27 +00:00
ab9f844aaf
Add federation_domain_whitelist option ( #2820 )
...
Add federation_domain_whitelist
gives a way to restrict which domains your HS is allowed to federate with.
useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network
2018-01-22 19:11:18 +01:00
a027c2af8d
Metrics for events processed in appservice and fed sender
...
More metrics I wished I'd had
2018-01-15 18:23:24 +00:00
bd91857028
Check missing fields in event_from_pdu_json
...
Return a 400 rather than a 500 when somebody messes up their send_join
2017-12-30 18:40:19 +00:00
3079f80d4a
Factor out `event_from_pdu_json`
...
turns out we have two copies of this, and neither needs to be an instance
method
2017-12-30 18:40:19 +00:00
65abc90fb6
federation_server: clean up imports
2017-12-30 18:40:19 +00:00
a7b726ad18
federation_client: clean up imports
2017-12-30 18:40:19 +00:00
d4fb4f7c52
Clear logcontext before starting fed txn queue runner
...
These processes take a long time compared to the request, so there is lots of
"Entering|Restoring dead context" in the logs. Let's try to shut it up a bit.
2017-11-28 15:26:14 +00:00
7e6fa29cb5
Remove preserve_context_over_{fn, deferred}
...
Both of these functions ae known to leak logcontexts. Replace the remaining
calls to them and kill them off.
2017-11-14 11:22:42 +00:00
82e4bfb53d
Add brackets
2017-11-09 10:06:42 +00:00
e8814410ef
Have an explicit API to update room config
2017-11-08 16:13:27 +00:00
94ff2cda73
Revert "Modify group room association API to allow modification of is_public"
2017-11-08 15:43:34 +00:00
207fabbc6a
Update docs for updating room group association
2017-11-01 09:35:15 +00:00
13b3d7b4a0
Flake8
2017-10-31 17:20:11 +00:00
20fe347906
Modify group room association API to allow modification of is_public
...
also includes renamings to make things more consistent.
2017-10-31 17:04:28 +00:00
2a7e9faeec
Do logcontexts outside ResponseCache
2017-10-25 15:21:08 +01:00
39dc52157d
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/group_fed_update_profile
2017-10-24 09:16:20 +01:00
eaaabc6c4f
replace 'except:' with 'except Exception:'
...
what could possibly go wrong
2017-10-23 15:52:32 +01:00
ce6d4914f4
Correctly wire in update group profile over federation
2017-10-23 15:21:24 +01:00
011d03a0f6
Fix typo
2017-10-19 11:22:48 +01:00
9ab859f27b
Fix typo in group attestation handling
2017-10-19 10:55:52 +01:00
582bd19ee9
Fix 500 error when we get an error handling a PDU
...
FederationServer doesn't have a send_failure (and nor does its subclass,
ReplicationLayer), so this was failing.
I'm not really sure what the idea behind send_failure is, given (a) we don't do
anything at the other end with it except log it, and (b) we also send back the
failure via the transaction response. I suspect there's a whole lot of dead
code around it, but for now I'm just removing the broken bit.
2017-10-17 20:52:40 +01:00
85f5674e44
Delint
2017-10-16 15:52:17 +01:00
2c5972f87f
Implement GET /groups/$groupId/invited_users
2017-10-16 15:31:11 +01:00
8dd0c85ac5
Merge pull request #2529 from matrix-org/rav/fix_transaction_failure_handling
...
log pdu_failures from incoming transactions
2017-10-11 17:29:14 +01:00
271f5601f3
Fix typo in invite to group
2017-10-11 14:45:33 +01:00
b75d443caf
log pdu_failures from incoming transactions
...
... even if we have no EDUs.
This appears to have been introduced in
476899295f
2017-10-11 14:36:13 +01:00
535cc49f27
Merge pull request #2466 from matrix-org/erikj/groups_merged
...
Initial Group Implementation
2017-10-11 13:20:07 +01:00
6a6cc27aee
fed server: process PDUs for different rooms in parallel
...
With luck, this will give a real-time improvement when there are many rooms and
the server ends up calling out to fetch missing events.
2017-10-09 18:30:31 +01:00
4c7c4d4061
Fed server: use a linearizer for ongoing transactions
...
We don't want to process the same transaction multiple times concurrently, so
use a linearizer.
2017-10-09 18:30:10 +01:00
ba5b9b80a5
fed server: refactor on_incoming_transaction
...
Move as much as possible to after the have_responded check, and reduce the
number of times we iterate over the pdu list.
2017-10-09 18:10:53 +01:00
a6e3222fe5
Fed server: Move origin-check code to _handle_received_pdu
...
The response-building code expects there to be an entry in the `results` list
for each entry in the pdu_list, so the early `continue` was messing this
up. That doesn't really matter, because all that the federation client does is
log any errors, but it's pretty poor form.
2017-10-09 17:53:32 +01:00
01bbacf3c4
Fix up logcontext handling in (federation) TransactionQueue
...
Avoid using preserve_context_over_function, which has problems with respect to
logcontexts.
2017-10-06 22:39:25 +01:00
27955056e0
Merge branch 'develop' into erikj/groups_merged
2017-10-02 16:20:41 +01:00
6cd5fcd536
Make the spam checker a module
2017-09-26 19:20:23 +01:00
17b8e2bd02
Add remove room API
2017-09-26 15:52:41 +01:00
6de74ea6d7
Fix logcontexts in _check_sigs_and_hashes
2017-09-20 01:32:42 +01:00
fcf2c0fd1a
Remove redundant `preserve_fn`
...
preserve_fn is a no-op unless the wrapped function returns a
Deferred. verify_json_objects_for_server returns a list, so this is doing
nothing.
2017-09-20 01:32:42 +01:00
5ed109d59f
PoC for filtering spammy events ( #2456 )
...
Demonstration of how you might add some hooks to filter out spammy events.
2017-09-19 12:20:11 +01:00
ef8e578677
Add bulk group publicised lookup API
2017-08-09 13:36:22 +01:00
6f443a74cf
Add update group profile API
2017-07-20 09:46:33 +01:00
14a34f12d7
Comments
2017-07-18 17:28:42 +01:00
3431ec55dc
Comments
2017-07-18 17:23:50 +01:00
94ecd871a0
Fix typos
2017-07-18 16:38:54 +01:00
12ed4ee48e
Correctly parse query params
2017-07-18 15:33:09 +01:00
e5ea6dd021
Add client apis
2017-07-18 14:37:06 +01:00
cccfcfa7b9
Comments
2017-07-18 10:35:18 +01:00
2f9eafdd36
Add local group server support
2017-07-17 12:03:49 +01:00
cb3aee8219
Ensure category and role ids are non-null
2017-07-14 14:06:55 +01:00
8575e3160f
Comments
2017-07-13 13:52:41 +01:00
7a39da8cc6
Add summary APIs to federation
2017-07-13 11:13:19 +01:00
a62406aaa5
Add group summary APIs
2017-07-12 11:36:15 +01:00
fe4e885f54
Add federation API for adding room to group
2017-07-11 14:35:07 +01:00
6322fbbd41
Comment
2017-07-11 11:52:03 +01:00
b8ca494ee9
Initial group server implementation
2017-07-10 15:44:15 +01:00
5e49a57ecc
Separate federation servlet into different lists
2017-07-05 14:32:24 +01:00
6e2a7ee1bc
Remove spurious log lines
2017-06-07 11:05:17 +01:00
dfbda5e025
Faster cache for get_joined_hosts
2017-05-25 17:24:44 +01:00
ec5c4499f4
Make presence use cached users/hosts in room
2017-05-16 16:01:43 +01:00
de042b3b88
Do some logging when one-time-keys get claimed
...
might help us figure out if https://github.com/vector-im/riot-web/issues/3868
has happened.
2017-05-09 19:04:56 +01:00
27c28eaa27
Merge pull request #2190 from matrix-org/erikj/mark_remote_as_back_more
...
Always mark remotes as up if we receive a signed request from them
2017-05-05 14:08:12 +01:00
310b1ccdc1
Use preserve_fn and add logs
2017-05-05 13:41:19 +01:00
db7d0c3127
Always mark remotes as up if we receive a signed request from them
2017-05-05 10:34:53 +01:00
7166854f41
Add cache for get_current_hosts_in_room
2017-05-02 10:36:35 +01:00
91b3981800
Try harder when sending leave events
...
When we're rejecting invites, ignore the backoff data, so that we have a better
chance of not getting the room out of sync.
2017-04-21 01:50:36 +01:00
4903ccf159
Fix some lies, and other clarifications, in docstrings
...
The documentation on get_json has been wrong ever since the very first commit
to synapse...
2017-04-21 01:31:09 +01:00
247c736b9b
Merge pull request #2115 from matrix-org/erikj/dedupe_federation_repl
...
Reduce federation replication traffic
2017-04-12 11:07:13 +01:00
8fbc0d29ee
Merge pull request #2121 from matrix-org/paul/sent-transactions-metric
...
Add a counter metric for successfully-sent transactions
2017-04-12 11:04:31 +01:00
26ae5178a4
Add some comments
2017-04-12 10:36:29 +01:00
1745069543
Comment
2017-04-12 10:17:10 +01:00
c7ddb5ef7a
Reuse get_interested_parties
2017-04-12 10:16:26 +01:00
11dbceb761
Add a counter metric for successfully-sent transactions
2017-04-11 17:16:12 +01:00
a8c8e4efd4
Comment
2017-04-11 15:35:49 +01:00
2be8a281d2
Comments
2017-04-11 15:28:24 +01:00
6308ac45b0
Move get_interested_remotes back to presence handler
2017-04-11 15:19:26 +01:00
b9b72bc6e2
Comments
2017-04-11 15:15:34 +01:00
84fbb80c8f
Use generators
2017-04-10 16:55:56 +01:00
40453b3f84
Dedupe KeyedEdu and Devices federation repl traffic
2017-04-10 16:49:51 +01:00
29574fd5b3
Reduce federation presence replication traffic
...
This is mainly done by moving the calculation of where to send presence
updates from the presence handler to the transaction queue, so we only
need to send the presence event (and not the destinations) across the
replication connection. Before we were duplicating by sending the full
state across once per destination.
2017-04-10 16:48:30 +01:00
0018491af2
Rename variable
2017-04-10 12:44:43 +01:00
8c5f03cec7
Revert to sending the same data type as before
2017-04-10 10:07:18 +01:00
f8434db549
Change name
2017-04-10 10:03:07 +01:00
ab904caf33
Comments
2017-04-10 10:02:17 +01:00
a828a64b75
Comment
2017-04-07 11:54:03 +01:00
d4d176e5d0
Add logging
2017-04-07 11:51:28 +01:00
449d1297ca
Fix up federation SendQueue and document types
2017-04-07 11:48:33 +01:00
f10ce8944b
Don't double json encode federation replication data
2017-04-05 11:10:28 +01:00
62b89daac6
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/repl_tcp_server
2017-04-04 09:46:16 +01:00
30bcbf775a
Accept join events from all servers
...
Make sure that we accept join events from any server, rather than just the
origin server, to make the federation join dance work correctly.
(Fixes #1893 ).
2017-04-03 15:58:07 +01:00
11880103b1
Make federation send queue take the current position
2017-03-30 12:54:36 +01:00
85be3dde81
Bail early if remote wouldn't be retried ( #2064 )
...
* Bail early if remote wouldn't be retried
* Don't always return true
* Just use get_retry_limiter
* Spelling
2017-03-29 11:48:27 +01:00
c5b0bdd542
Merge pull request #2067 from matrix-org/erikj/notify_on_fed
...
Notify on new federation traffic
2017-03-29 11:41:37 +01:00
35b4aa04be
Notify on new federation traffic
2017-03-27 14:07:47 +01:00
2a28b79e04
Batch sending of device list pokes
2017-03-24 14:44:49 +00:00
5a16cb4bf0
Ignore backoff history for invites, aliases, and roomdirs
...
Add a param to the federation client which lets us ignore historical backoff
data for federation queries, and set it for a handful of operations.
2017-03-23 12:23:22 +00:00
4bd597d9fc
push federation retry limiter down to matrixfederationclient
...
rather than having to instrument everywhere we make a federation call,
make the MatrixFederationHttpClient manage the retry limiter.
2017-03-23 09:28:46 +00:00
29ed09e80a
Fix assertion to stop transaction queue getting wedged
...
... and update some docstrings to correctly reflect the types being used.
get_new_device_msgs_for_remote can return a long under some circumstances,
which was being stored in last_device_list_stream_id_by_dest, and was then
upsetting things on the next loop.
2017-03-15 12:16:55 +00:00
29235901b8
Move FederationServer._handle_new_pdu to FederationHandler
...
Unfortunately this significantly increases the size of the already-rather-big
FederationHandler, but the code fits more naturally here, and it paves the way
for the tighter integration that I need between handling incoming PDUs and
doing the join dance.
Other than renaming the existing `FederationHandler.on_receive_pdu` to
`_process_received_pdu` to make way for it, this just consists of the move, and
replacing `self.handler` with `self` and `self` with `self.replication_layer`.
2017-03-09 16:20:13 +00:00
e8b1721290
Move sig check out of _handle_new_pdu
...
When we receive PDUs via `get_missing_events`, we have already checked their
sigs, so there is no need to do it again.
2017-03-09 15:50:44 +00:00
3406333a58
Factor _get_missing_events_for_pdu out of _handle_new_pdu
...
This should be functionally identical: it just seeks to improve readability by
reducing indentation.
2017-03-09 15:50:44 +00:00
45d173a59a
Fix docstring
2017-03-09 15:50:29 +00:00
0c4cf9372b
Fix a race in transaction queue
...
It was theoretically possible for a PDU to get queued and not sent for ages. On
closer inspection I think there were bigger problems elsewhere, but we might as
well fix this since it's easy.
2017-02-20 16:46:25 +00:00
df4ecff5a9
Correctly raise exceptions for ratelimitng. Ratelimit on 401
2017-02-01 15:42:19 +00:00
ae7a132f38
Better handle 404 response for federation /send/
2017-01-31 13:40:09 +00:00
c974116f19
Implement device key caching over federation
2017-01-26 16:07:24 +00:00
51e9fe36e4
Fix up sending of m.device_list_update edus
2017-01-25 16:55:21 +00:00
2367c5568c
Add basic implementation of local device list changes
2017-01-25 14:27:27 +00:00
fc7cae8aa3
Merge pull request #1824 from matrix-org/erikj/retry_host_log
...
Lower the not retrying host log line to debug
2017-01-18 11:07:51 +00:00
f878f64f43
Lower the not retrying host log line to debug
2017-01-17 17:20:39 +00:00
e6153e1bd1
Fix couple of federation state bugs
2017-01-17 13:22:34 +00:00
f7085ac84f
Name linearizer's for better logs
2017-01-09 17:17:10 +00:00
4304e7e593
do the discard check in the right place to avoid grabbing dependent events
2017-01-07 03:44:18 +00:00
e10c527930
Discard PDUs from invalid origins due to #1753 in 0.18.[56]
2017-01-07 02:13:14 +00:00
468749c9fc
fix comment
2017-01-05 12:00:11 +00:00
eedf400d05
limit total timeout for get_missing_events to 10s
2017-01-05 11:58:15 +00:00
f784980d2b
Only send events that originate on this server.
...
Or events that are sent via the federation "send_join" API.
This should match the behaviour from before v0.18.5 and #1635 landed.
2017-01-05 11:26:30 +00:00
e02bdaf08b
Get the destinations from the state from before the event
...
Rather than the state after then event.
2017-01-04 15:17:15 +00:00
b6b67715ed
Send ALL membership events to the server that was affected.
...
Send all membership changes to the server that was affected.
This ensures that if the last member of a room on a server
was kicked or banned they get told about it.
2017-01-04 13:56:20 +00:00
62ce3034f3
s/aquire/acquire/g
2016-12-30 20:04:44 +00:00
0aff09f6c9
Add more useful logging when we block fetching events
2016-12-30 20:00:44 +00:00
cc50b1ae53
Remove fallback from get_missing_events.
...
get_missing_events used to fallback to fetching the missing events
individually requesting from every server in the room, one by one.e
This could be unacceptably slow, possibly causing #1732
2016-12-30 18:13:15 +00:00
f32fb65552
Add new API appservice specific public room list
2016-12-06 16:12:27 +00:00
aaecffba3a
Correctly handle 500's and 429 on federation
2016-11-24 15:04:49 +00:00
feec718265
Shuffle receipt handler around so that worker apps don't need to load it
2016-11-23 15:14:24 +00:00
4c79a63fd7
Explicit federation ack
2016-11-23 10:40:44 +00:00
51e89709aa
Comments
2016-11-21 17:59:39 +00:00
88d85ebae1
Add some metrics
2016-11-21 17:36:05 +00:00
50934ce460
Comments
2016-11-21 16:55:23 +00:00
9687e039e7
Remove explicit calls to send_pdu
2016-11-21 14:48:51 +00:00
524d61bf7e
Fix tests
2016-11-21 11:53:02 +00:00
7c9cdb2245
Store federation stream positions in the database
2016-11-21 11:33:08 +00:00
f8ee66250a
Handle sending events and device messages over federation
2016-11-17 15:48:04 +00:00
ed787cf09e
Hook up the send queue and create a federation sender worker
2016-11-16 17:34:44 +00:00
1587b5a033
Add initial cut of federation send queue
2016-11-16 14:47:52 +00:00
59ef517e6b
Use new federation_sender DI
2016-11-16 14:47:52 +00:00
847d5db1d1
Add transaction queue and transport layer to DI
2016-11-16 14:47:52 +00:00
daec6fc355
Move logic into transaction_queue
2016-11-16 14:47:52 +00:00
0e830d3770
Rename transaction queue functions to send_*
2016-11-16 14:47:52 +00:00
22578545a0
Time out typing over federation
2016-09-23 14:00:52 +01:00
23b6701a28
Support filtering remote room lists
2016-09-16 10:24:15 +01:00
5810cffd33
Pass since/from parameters over federation
2016-09-15 10:36:19 +01:00
76b09c29b0
Merge pull request #1110 from matrix-org/markjh/e2e_timeout
...
Add a timeout parameter for end2end key queries.
2016-09-13 10:50:45 +01:00
949c2c5435
Add a timeout parameter for end2end key queries.
...
Add a timeout parameter for controlling how long synapse will wait
for responses from remote servers. For servers that fail include how
they failed to make it easier to debug.
Fetch keys from different servers in parallel rather than in series.
Set the default timeout to 10s.
2016-09-12 18:17:09 +01:00
706b5d76ed
Fix backfill when cannot find an event.
...
`get_pdu` can succeed but return None.
2016-09-12 14:59:51 +01:00
af4701b311
Fix incorrect attribute name
2016-09-09 17:36:56 +01:00
464ffd1b5e
Comment
2016-09-09 17:17:23 +01:00
327425764e
Add edu.type as part of key. Remove debug logging
2016-09-09 17:13:30 +01:00
52b2318777
Clobber EDUs in send queue
2016-09-09 15:59:08 +01:00
ab80d5e0a9
Drop replication log levels
2016-09-09 14:56:50 +01:00
a6c6750166
Check if destination is ready for retry earlier
2016-09-09 13:46:05 +01:00
4598682b43
Fix tightloop on sending transaction
2016-09-09 13:12:53 +01:00
d2688d7f03
Correctly guard against multiple concurrent transactions
2016-09-09 11:44:36 +01:00
b390756150
Update last_device_stream_id_by_dest if there is nothing to send
2016-09-09 11:00:15 +01:00
43954d000e
Add a new method to enqueue the device messages rather than sending a dummy EDU
2016-09-07 16:10:51 +01:00
cb98ac261b
Move the check for federated device_messages.
...
Move the check into _attempt_new_transaction.
Only delete messages if there were messages to delete.
2016-09-07 15:39:13 +01:00
31a07d2335
Add stream change caches for device messages
2016-09-07 15:27:07 +01:00
d4a35ada28
Send device messages over federation
2016-09-06 18:16:20 +01:00
2854ee2a52
Only pull out IDs from DB for /state_ids/ request
2016-09-02 10:53:36 +01:00
f4778d4cd9
Merge branch 'erikj/pdu_check' of github.com:matrix-org/synapse into develop
2016-09-01 14:40:36 +01:00
516a272aca
Ensure we only return a validated pdu in get_pdu
2016-09-01 10:55:02 +01:00
bed10f9880
Use state handler instead of get_users_in_room/get_joined_hosts
2016-08-26 14:54:30 +01:00
9219139351
Preserve some logcontexts
2016-08-24 11:58:40 +01:00
c315922b5f
PEP8
2016-08-10 16:34:10 +01:00
ca8abfbf30
Clean up TransactionQueue
2016-08-10 16:24:16 +01:00
5aeadb7414
Merge pull request #999 from matrix-org/erikj/measure_more
...
Measure federation send transaction resources
2016-08-10 14:16:14 +01:00
487bc49bf8
Don't stop on 4xx series errors
2016-08-10 13:39:12 +01:00
ea8c4094db
Also pull out rejected events
2016-08-10 13:26:13 +01:00
f91df1f761
Store if we fail to fetch an event from a destination
2016-08-10 11:31:46 +01:00
3bc9629be5
Measure federation send transaction resources
2016-08-10 10:56:38 +01:00
f5deaff424
Merge pull request #991 from matrix-org/erikj/retry_make
...
Retry joining via other servers if first one failed. Fix some other bugs.
2016-08-05 18:21:27 +01:00
5f360182c6
Fix a couple of python bugs
2016-08-05 18:08:32 +01:00
46453bfc2f
Retry joining via other servers if first one failed
2016-08-05 18:02:03 +01:00
24f36469bc
Add federation /version API
2016-08-05 16:36:07 +01:00
597c79be10
Change the way we specify if we require auth or not
2016-08-05 16:17:04 +01:00
2d4de61fb7
Fix typo
2016-08-05 10:48:56 +01:00
fccadb7719
Check if we already have the events returned by /state/
2016-08-05 10:43:47 +01:00
1515d1b581
Fallback to /state/ on both 400 and 404
2016-08-05 10:24:23 +01:00
a2b7102eea
Tweak integrity error recovery to work as intended
2016-08-04 20:38:08 -07:00
257c41cc2e
Fix typos.
2016-08-04 14:05:45 +01:00
b4e2290d89
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_ids_api
2016-08-04 14:04:35 +01:00
e3ee63578f
Tidy up get_events
2016-08-04 14:01:18 +01:00
edb33eb163
Rename fields to _ids
2016-08-03 17:19:15 +01:00
bcc9cda8ca
Fix copy + paste fails
2016-08-03 17:17:26 +01:00
4c56bedee3
Actually call get_room_state
2016-08-03 15:04:29 +01:00
520ee9bd2c
Fix syntax error
2016-08-03 15:03:15 +01:00
a60a2eaa02
Comment
2016-08-03 14:52:43 +01:00
e3a720217a
Add /state_ids federation API
...
The new API only returns the event_ids for the state, as most
requesters will already have the vast majority of the events already.
2016-08-03 14:47:37 +01:00
921f17f938
Merge branch 'develop' into rav/refactor_device_query
2016-08-03 11:12:47 +01:00
1efee2f52b
E2E keys: Make federation query share code with client query
...
Refactor the e2e query handler to separate out the local query, and then make
the federation handler use it.
2016-08-02 18:12:00 +01:00
c9154b970c
Don't double wrap 200
2016-08-02 16:45:53 +01:00
b3d5c4ad9d
Fix response cache
2016-08-02 16:42:21 +01:00
248e6770ca
Cache federation state responses
2016-07-21 10:30:12 +01:00
8f4a9bbc16
Linearize some federation endpoints based on (origin, room_id)
2016-06-17 16:43:45 +01:00
9ba2bf1570
Merge pull request #878 from matrix-org/erikj/ujson
...
Disable responding with canonical json for federation
2016-06-17 16:22:12 +01:00
120c238705
Disable responding with canonical json for federation
2016-06-17 16:10:37 +01:00
2884712ca7
Only re-sign our own events
2016-06-17 14:47:33 +01:00
d41a1a91d3
Linearize fetching of gaps on incoming events
...
This potentially stops the server from doing multiple requests for the
same data.
2016-06-15 15:16:14 +01:00
d88faf92d1
Fix up federation PublicRoomList
2016-06-08 14:39:31 +01:00
958c968d02
Merge pull request #856 from matrix-org/erikj/fed_pub_rooms
...
Enable auth on /publicRoom endpoints
2016-06-08 14:36:09 +01:00
1fd6eb695d
Enable auth on federation PublicRoomList
2016-06-08 14:15:18 +01:00
17aab5827a
Add some logging for when servers ask for missing events
2016-06-08 11:55:31 +01:00
6ecb2ca4ec
pep8
2016-06-01 09:48:55 +01:00
963e3ed282
Apparently I am not permitted to have two blank lines here
2016-05-31 17:22:53 +01:00
d240796ded
Basic, un-cached support for secondary_directory_servers
2016-05-31 17:20:07 +01:00
70ecb415f5
Fix c+p fail
2016-05-31 12:00:54 +01:00
e1625d62a8
Add federation room list servlet
2016-05-31 11:55:57 +01:00
f6ebaf4a32
Run transaction queue on reactor
...
This ensures that any CPU work that happens doesn't block message
sending.
2016-05-09 10:10:06 +01:00
9c272da05f
Add an openidish mechanism for proving to third parties that you own a given user_id
2016-05-05 13:42:44 +01:00
62b51b8452
Fix typo in event_auth servlet path
2016-04-29 12:00:51 +01:00
914f1eafac
Lower timeout for make_membership_event
...
Calls to make_membership_event are done in response to client requests,
and so should not be retried over long timeframes.
2016-04-15 11:22:23 +01:00
acdfef7b14
Intern all the things
2016-03-23 16:25:54 +00:00
5244c0b48e
Remove unused backfilled parameter from persist_event
2016-03-21 18:06:08 +00:00
9adf0e92bc
Catch exceptions from EDU handling
2016-03-18 15:12:50 +00:00
3c5f25507b
Yield on EDU handling
2016-03-18 13:55:16 +00:00
e9c1cabac2
Use parse_json_object_from_request to parse JSON out of request bodies
2016-03-11 16:41:03 +00:00
ed61a49169
Add profile information to invites
2016-03-04 14:35:02 +00:00
577951b032
Allow third_party_signed to be specified on /join
2016-02-23 15:11:25 +00:00
2c1fbea531
Fix up logcontexts
2016-02-08 14:26:45 +00:00
d83d004ccd
Fix flake8 warnings for new flake8
2016-02-02 17:18:50 +00:00
9959d9ece8
Remove redundated BaseHomeServer
2016-01-26 13:52:29 +00:00
6c28ac260c
copyrights
2016-01-07 04:26:29 +00:00
14d7acfad4
Host /unstable and /r0 versions of r0 APIs
2015-12-01 17:34:32 +00:00
cf4ef5f3c7
Only retry federation requests for a long time for background requests
2015-11-17 18:26:50 +00:00
aca6e5bf46
Don't complain if /make_join response lacks 'prev_state' list (SYN-517)
2015-11-13 17:27:25 +00:00
c0b3554401
Fix missing profile data in federation joins
...
There was a regression where we stopped including profile data in
initial joins for rooms joined over federation.
2015-11-12 16:19:55 +00:00
2cebe53545
Exchange 3pid invites for m.room.member invites
2015-11-05 16:43:19 +00:00
97d792b28f
Don't rearrange transaction_queue
2015-11-03 16:31:08 +00:00
7ce264ce5f
Fix broken cache for getting retry times. This meant we retried remote destinations way more frequently than we should
2015-11-03 16:24:03 +00:00
137fafce4e
Allow rejecting invites
...
This is done by using the same /leave flow as you would use if you had
already accepted the invite and wanted to leave.
2015-10-20 11:58:58 +01:00
643b5fcdc8
Look for keys on the right objects
2015-10-15 13:10:30 +01:00
0c38e8637f
Remove unnecessary class-wrapping
2015-10-13 18:00:38 +01:00
17dffef5ec
Move event contents into third_party_layout field
2015-10-13 15:48:12 +01:00
fcd9ba8802
Fix lint errors
2015-10-06 10:13:05 -05:00
5b3e9713dd
Implement third party identifier invites
2015-10-01 17:49:52 +01:00
8899df13bf
Merge pull request #208 from matrix-org/markjh/end-to-end-key-federation
...
Federation for end-to-end key requests.
2015-08-18 09:12:54 +01:00
0cceb2ac92
Add a few strategic new lines to break up the on_query_client_keys and on_claim_client_keys methods in federation_server.py
2015-08-13 17:27:46 +01:00
2df8dd9b37
Move all the caches into their own package, synapse.util.caches
2015-08-11 18:00:59 +01:00
2da3b1e60b
Get the end-to-end key federation working
2015-07-24 18:26:46 +01:00
62c010283d
Add federation support for end-to-end key requests
2015-07-23 16:03:38 +01:00
b5f55a1d85
Implement bulk verify_signed_json API
2015-06-26 10:39:34 +01:00
cee69441d3
Log more when we have processed the request
2015-06-15 17:11:44 +01:00
8047fd2434
Merge pull request #176 from matrix-org/erikj/backfill_auth
...
Improve backfill.
2015-06-03 17:25:37 +01:00
3483b78d1a
Log where a request came from in federation
2015-06-02 18:15:13 +01:00
d3ded420b1
Rephrase log line
2015-06-02 16:30:52 +01:00
22716774d5
Don't about JSON when warning about content tampering
2015-06-02 16:30:52 +01:00
6eadbfbea0
Remove redundant for loop
2015-05-22 16:12:20 +01:00
74b7de83ec
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/backfill_fixes
2015-05-22 16:10:42 +01:00
36317f3dad
Merge pull request #156 from matrix-org/erikj/join_perf
...
Make joining #matrix:matrix.org over federation quicker
2015-05-22 16:09:54 +01:00
284f55a7fb
Add doc strings
2015-05-22 15:18:04 +01:00
b21d015c55
Log origin and stats of incoming transactions
2015-05-22 14:44:25 +01:00
e70e8e053e
Add txn_id to some log lines
2015-05-22 14:44:02 +01:00
1b446a5d85
Log less lines at INFO level, but include more helpful information
2015-05-22 14:29:57 +01:00
c2633907c5
Merge branch 'erikj/join_perf' of github.com:matrix-org/synapse into erikj/backfill_fixes
2015-05-21 14:58:47 +01:00
ebfdd2eb5b
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/join_perf
2015-05-21 14:54:52 +01:00
20814fabdd
Actually fetch state for new backwards extremeties when backfilling.
2015-05-20 11:59:02 +01:00
3a653515ec
Add None check
2015-05-19 15:27:09 +01:00
5b1631a4a9
Add a timeout param to get_event
2015-05-19 14:53:32 +01:00
d5cea26d45
Remove pointless newline
2015-05-18 10:16:45 +01:00
c71176858b
Newline, remove debug logging
2015-05-18 10:11:14 +01:00
f8bd4de87d
Remove debug logging
2015-05-18 09:58:03 +01:00
a2c4f3f150
Fix daedlock
2015-05-15 10:54:04 +01:00
95dedb866f
Unwrap defer.gatherResults failures
2015-05-12 13:14:29 +01:00
476899295f
Change the way we do logging contexts so that they survive divergences
2015-05-08 16:32:18 +01:00
0cf7e480b4
And use buffer(...) there as well
2015-05-05 18:20:01 +01:00
977338a7af
Use buffer(...) when inserting into bytea column
2015-05-05 18:12:53 +01:00
3d5a955e08
Missed events are not outliers
2015-05-05 17:36:57 +01:00
1942382246
Don't log enqueue_
2015-05-01 16:14:25 +01:00
23c639ff32
Split a storage function in two so that we don't have to do extra work.
2015-05-01 10:17:19 +01:00
c178e4e6ca
Add missing servlet to list
2015-03-19 17:48:21 +00:00
128cf2daf7
Appease pep8
2015-03-12 16:24:51 +00:00
c782e893ec
Neater metrics from TransactionQueue
2015-03-12 16:24:51 +00:00
2e4f0b2bd7
Replace the @metrics.counted annotations in federation with specifically-written counters and distributions
2015-03-12 16:24:51 +00:00
1748605c5d
Count incoming HTTP requests per servlet that responds
2015-03-12 16:24:51 +00:00
b0cf867319
Use _ instead of . as a metric namespacing separator, for Prometheus
2015-03-12 16:24:51 +00:00
f9478e475b
Rename Metrics' "keys" to "labels"
2015-03-12 16:24:51 +00:00
094803cf82
Put vector gauges on transaction queue pending PDU and EDU dicts
2015-03-12 16:24:50 +00:00
9470412316
Initial attempt at sprinkling some @metrics.counted decorations around the federation code
2015-03-12 16:24:50 +00:00
120b689284
Delete pointless (and unreachable) __init__ method from FederationClient
2015-03-12 16:24:50 +00:00
abaf47bbb6
Fix bug in logging.
2015-03-10 10:28:29 +00:00
98b867f7b7
Fix bug in logging.
2015-03-10 10:16:09 +00:00
019422ebba
Merge pull request #101 from matrix-org/neaten-federation-servlets
...
Neaten federation servlets
2015-03-09 17:39:06 +00:00
d79d91a4a7
Appease pep8
2015-03-05 20:55:40 +00:00
5eab2549ab
Append a $ on PATH at registration time, meaning each PATH attribute doesn't need it
2015-03-05 20:36:05 +00:00
7644cb79b2
Slightly neater(?) arrangement of authentication wrapper for HTTP servlet methods
2015-03-05 20:33:16 +00:00
ba8ac996f9
Remove the dead 'rate_limit_origin' method from TransportLayerServer
2015-03-05 19:43:17 +00:00
a901ed16b5
Move federation API responding code out of weird mix of lambdas into Servlet-style methods on instances
2015-03-05 19:10:57 +00:00
9708f49abf
Docs
2015-03-05 16:35:16 +00:00
96fee64421
Remove unecessary check
2015-03-05 16:31:47 +00:00
39aa968a76
Respect min_depth argument
2015-03-05 16:31:32 +00:00
6dfd8c73fc
Docs.
2015-03-05 16:31:13 +00:00
ae702d161a
Handle if get_missing_pdu returns 400 or not all events.
2015-03-05 16:08:02 +00:00
0a036944bd
Merge branch 'develop' of github.com:matrix-org/synapse into batched_get_pdu
2015-03-02 13:53:30 +00:00
4195e55ccc
Merge branch 'develop' of github.com:matrix-org/synapse into federation_rate_limit
2015-03-02 13:39:22 +00:00
29481690c5
If we're yielding don't add errback
2015-03-02 11:50:43 +00:00
23d9bd1d74
Process transactions serially.
...
Since the events received in a transaction are ordered, later events
might depend on earlier events and so we shouldn't blindly process them
in parellel.
2015-03-02 11:39:57 +00:00
9d9b230501
Make the federation server ratelimiting configurable.
2015-03-02 11:33:45 +00:00
0554d07082
Move federation rate limiting out of transport layer
2015-02-27 15:41:52 +00:00
9dc9118e55
Document FederationRateLimiter
2015-02-27 15:16:47 +00:00
93d90765c4
Initial implementation of federation server rate limiting
2015-02-26 16:15:26 +00:00
59362454dd
Must update pending_transactions map before yield'ing
2015-02-26 15:47:35 +00:00
db215b7e00
Implement and use new batched get missing pdu
2015-02-23 13:58:02 +00:00
0ac2a79faa
Initial stab at implementing a batched get_missing_pdus request
2015-02-19 17:24:14 +00:00
a7925259a1
Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.7.1
2015-02-18 13:57:55 +00:00
446ef58992
Add errback to all deferreds in transaction_queue
2015-02-18 12:03:26 +00:00
2462aacd77
Restrict the destinations that synapse can talk to
2015-02-18 11:52:51 +00:00
b68e4a729f
Discard destination 'localhost'
2015-02-18 11:32:39 +00:00
47d3ff4cf8
Don't send failure to self
2015-02-18 11:30:37 +00:00
e482541e1d
Fix pyflakes
2015-02-18 10:44:22 +00:00
Richard van der Hoff