Commit Graph

599 Commits

Author SHA1 Message Date
Erik Johnston 88d85ebae1 Add some metrics 2016-11-21 17:36:05 +00:00
Erik Johnston 50934ce460 Comments 2016-11-21 16:55:23 +00:00
Erik Johnston 9687e039e7 Remove explicit calls to send_pdu 2016-11-21 14:48:51 +00:00
Erik Johnston 524d61bf7e Fix tests 2016-11-21 11:53:02 +00:00
Erik Johnston 7c9cdb2245 Store federation stream positions in the database 2016-11-21 11:33:08 +00:00
Erik Johnston f8ee66250a Handle sending events and device messages over federation 2016-11-17 15:48:04 +00:00
Erik Johnston ed787cf09e Hook up the send queue and create a federation sender worker 2016-11-16 17:34:44 +00:00
Erik Johnston 1587b5a033 Add initial cut of federation send queue 2016-11-16 14:47:52 +00:00
Erik Johnston 59ef517e6b Use new federation_sender DI 2016-11-16 14:47:52 +00:00
Erik Johnston 847d5db1d1 Add transaction queue and transport layer to DI 2016-11-16 14:47:52 +00:00
Erik Johnston daec6fc355 Move logic into transaction_queue 2016-11-16 14:47:52 +00:00
Erik Johnston 0e830d3770 Rename transaction queue functions to send_* 2016-11-16 14:47:52 +00:00
Erik Johnston 22578545a0 Time out typing over federation 2016-09-23 14:00:52 +01:00
Erik Johnston 23b6701a28 Support filtering remote room lists 2016-09-16 10:24:15 +01:00
Erik Johnston 5810cffd33 Pass since/from parameters over federation 2016-09-15 10:36:19 +01:00
Mark Haines 76b09c29b0 Merge pull request #1110 from matrix-org/markjh/e2e_timeout
Add a timeout parameter for end2end key queries.
2016-09-13 10:50:45 +01:00
Mark Haines 949c2c5435 Add a timeout parameter for end2end key queries.
Add a timeout parameter for controlling how long synapse will wait
for responses from remote servers. For servers that fail include how
they failed to make it easier to debug.

Fetch keys from different servers in parallel rather than in series.

Set the default timeout to 10s.
2016-09-12 18:17:09 +01:00
Erik Johnston 706b5d76ed Fix backfill when cannot find an event.
`get_pdu` can succeed but return None.
2016-09-12 14:59:51 +01:00
Erik Johnston af4701b311 Fix incorrect attribute name 2016-09-09 17:36:56 +01:00
Erik Johnston 464ffd1b5e Comment 2016-09-09 17:17:23 +01:00
Erik Johnston 327425764e Add edu.type as part of key. Remove debug logging 2016-09-09 17:13:30 +01:00
Erik Johnston 52b2318777 Clobber EDUs in send queue 2016-09-09 15:59:08 +01:00
Erik Johnston ab80d5e0a9 Drop replication log levels 2016-09-09 14:56:50 +01:00
Erik Johnston a6c6750166 Check if destination is ready for retry earlier 2016-09-09 13:46:05 +01:00
Erik Johnston 4598682b43 Fix tightloop on sending transaction 2016-09-09 13:12:53 +01:00
Erik Johnston d2688d7f03 Correctly guard against multiple concurrent transactions 2016-09-09 11:44:36 +01:00
Erik Johnston b390756150 Update last_device_stream_id_by_dest if there is nothing to send 2016-09-09 11:00:15 +01:00
Mark Haines 43954d000e Add a new method to enqueue the device messages rather than sending a dummy EDU 2016-09-07 16:10:51 +01:00
Mark Haines cb98ac261b Move the check for federated device_messages.
Move the check into _attempt_new_transaction.
Only delete messages if there were messages to delete.
2016-09-07 15:39:13 +01:00
Mark Haines 31a07d2335 Add stream change caches for device messages 2016-09-07 15:27:07 +01:00
Mark Haines d4a35ada28 Send device messages over federation 2016-09-06 18:16:20 +01:00
Erik Johnston 2854ee2a52 Only pull out IDs from DB for /state_ids/ request 2016-09-02 10:53:36 +01:00
Erik Johnston f4778d4cd9 Merge branch 'erikj/pdu_check' of github.com:matrix-org/synapse into develop 2016-09-01 14:40:36 +01:00
Erik Johnston 516a272aca Ensure we only return a validated pdu in get_pdu 2016-09-01 10:55:02 +01:00
Erik Johnston bed10f9880 Use state handler instead of get_users_in_room/get_joined_hosts 2016-08-26 14:54:30 +01:00
Erik Johnston 9219139351 Preserve some logcontexts 2016-08-24 11:58:40 +01:00
Erik Johnston c315922b5f PEP8 2016-08-10 16:34:10 +01:00
Erik Johnston ca8abfbf30 Clean up TransactionQueue 2016-08-10 16:24:16 +01:00
Erik Johnston 5aeadb7414 Merge pull request #999 from matrix-org/erikj/measure_more
Measure federation send transaction resources
2016-08-10 14:16:14 +01:00
Erik Johnston 487bc49bf8 Don't stop on 4xx series errors 2016-08-10 13:39:12 +01:00
Erik Johnston ea8c4094db Also pull out rejected events 2016-08-10 13:26:13 +01:00
Erik Johnston f91df1f761 Store if we fail to fetch an event from a destination 2016-08-10 11:31:46 +01:00
Erik Johnston 3bc9629be5 Measure federation send transaction resources 2016-08-10 10:56:38 +01:00
Erik Johnston f5deaff424 Merge pull request #991 from matrix-org/erikj/retry_make
Retry joining via other servers if first one failed. Fix some other bugs.
2016-08-05 18:21:27 +01:00
Erik Johnston 5f360182c6 Fix a couple of python bugs 2016-08-05 18:08:32 +01:00
Erik Johnston 46453bfc2f Retry joining via other servers if first one failed 2016-08-05 18:02:03 +01:00
Erik Johnston 24f36469bc Add federation /version API 2016-08-05 16:36:07 +01:00
Erik Johnston 597c79be10 Change the way we specify if we require auth or not 2016-08-05 16:17:04 +01:00
Erik Johnston 2d4de61fb7 Fix typo 2016-08-05 10:48:56 +01:00
Erik Johnston fccadb7719 Check if we already have the events returned by /state/ 2016-08-05 10:43:47 +01:00
Erik Johnston 1515d1b581 Fallback to /state/ on both 400 and 404 2016-08-05 10:24:23 +01:00
Benjamin Saunders a2b7102eea Tweak integrity error recovery to work as intended 2016-08-04 20:38:08 -07:00
Erik Johnston 257c41cc2e Fix typos. 2016-08-04 14:05:45 +01:00
Erik Johnston b4e2290d89 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_ids_api 2016-08-04 14:04:35 +01:00
Erik Johnston e3ee63578f Tidy up get_events 2016-08-04 14:01:18 +01:00
Erik Johnston edb33eb163 Rename fields to _ids 2016-08-03 17:19:15 +01:00
Erik Johnston bcc9cda8ca Fix copy + paste fails 2016-08-03 17:17:26 +01:00
Erik Johnston 4c56bedee3 Actually call get_room_state 2016-08-03 15:04:29 +01:00
Erik Johnston 520ee9bd2c Fix syntax error 2016-08-03 15:03:15 +01:00
Erik Johnston a60a2eaa02 Comment 2016-08-03 14:52:43 +01:00
Erik Johnston e3a720217a Add /state_ids federation API
The new API only returns the event_ids for the state, as most
requesters will already have the vast majority of the events already.
2016-08-03 14:47:37 +01:00
Mark Haines 921f17f938 Merge branch 'develop' into rav/refactor_device_query 2016-08-03 11:12:47 +01:00
Richard van der Hoff 1efee2f52b E2E keys: Make federation query share code with client query
Refactor the e2e query handler to separate out the local query, and then make
the federation handler use it.
2016-08-02 18:12:00 +01:00
Erik Johnston c9154b970c Don't double wrap 200 2016-08-02 16:45:53 +01:00
Erik Johnston b3d5c4ad9d Fix response cache 2016-08-02 16:42:21 +01:00
Erik Johnston 248e6770ca Cache federation state responses 2016-07-21 10:30:12 +01:00
Erik Johnston 8f4a9bbc16 Linearize some federation endpoints based on (origin, room_id) 2016-06-17 16:43:45 +01:00
Erik Johnston 9ba2bf1570 Merge pull request #878 from matrix-org/erikj/ujson
Disable responding with canonical json for federation
2016-06-17 16:22:12 +01:00
Erik Johnston 120c238705 Disable responding with canonical json for federation 2016-06-17 16:10:37 +01:00
Erik Johnston 2884712ca7 Only re-sign our own events 2016-06-17 14:47:33 +01:00
Erik Johnston d41a1a91d3 Linearize fetching of gaps on incoming events
This potentially stops the server from doing multiple requests for the
same data.
2016-06-15 15:16:14 +01:00
Erik Johnston d88faf92d1 Fix up federation PublicRoomList 2016-06-08 14:39:31 +01:00
Erik Johnston 958c968d02 Merge pull request #856 from matrix-org/erikj/fed_pub_rooms
Enable auth on /publicRoom endpoints
2016-06-08 14:36:09 +01:00
Erik Johnston 1fd6eb695d Enable auth on federation PublicRoomList 2016-06-08 14:15:18 +01:00
Erik Johnston 17aab5827a Add some logging for when servers ask for missing events 2016-06-08 11:55:31 +01:00
David Baker 6ecb2ca4ec pep8 2016-06-01 09:48:55 +01:00
David Baker 963e3ed282 Apparently I am not permitted to have two blank lines here 2016-05-31 17:22:53 +01:00
David Baker d240796ded Basic, un-cached support for secondary_directory_servers 2016-05-31 17:20:07 +01:00
David Baker 70ecb415f5 Fix c+p fail 2016-05-31 12:00:54 +01:00
David Baker e1625d62a8 Add federation room list servlet 2016-05-31 11:55:57 +01:00
Erik Johnston f6ebaf4a32 Run transaction queue on reactor
This ensures that any CPU work that happens doesn't block message
sending.
2016-05-09 10:10:06 +01:00
Mark Haines 9c272da05f Add an openidish mechanism for proving to third parties that you own a given user_id 2016-05-05 13:42:44 +01:00
Erik Johnston 62b51b8452 Fix typo in event_auth servlet path 2016-04-29 12:00:51 +01:00
Erik Johnston 914f1eafac Lower timeout for make_membership_event
Calls to make_membership_event are done in response to client requests,
and so should not be retried over long timeframes.
2016-04-15 11:22:23 +01:00
Erik Johnston acdfef7b14 Intern all the things 2016-03-23 16:25:54 +00:00
Mark Haines 5244c0b48e Remove unused backfilled parameter from persist_event 2016-03-21 18:06:08 +00:00
Erik Johnston 9adf0e92bc Catch exceptions from EDU handling 2016-03-18 15:12:50 +00:00
Erik Johnston 3c5f25507b Yield on EDU handling 2016-03-18 13:55:16 +00:00
Mark Haines e9c1cabac2 Use parse_json_object_from_request to parse JSON out of request bodies 2016-03-11 16:41:03 +00:00
Erik Johnston ed61a49169 Add profile information to invites 2016-03-04 14:35:02 +00:00
Daniel Wagner-Hall 577951b032 Allow third_party_signed to be specified on /join 2016-02-23 15:11:25 +00:00
Erik Johnston 2c1fbea531 Fix up logcontexts 2016-02-08 14:26:45 +00:00
Daniel Wagner-Hall d83d004ccd Fix flake8 warnings for new flake8 2016-02-02 17:18:50 +00:00
Erik Johnston 9959d9ece8 Remove redundated BaseHomeServer 2016-01-26 13:52:29 +00:00
Matthew Hodgson 6c28ac260c copyrights 2016-01-07 04:26:29 +00:00
Daniel Wagner-Hall 14d7acfad4 Host /unstable and /r0 versions of r0 APIs 2015-12-01 17:34:32 +00:00
Erik Johnston cf4ef5f3c7 Only retry federation requests for a long time for background requests 2015-11-17 18:26:50 +00:00
Paul "LeoNerd" Evans aca6e5bf46 Don't complain if /make_join response lacks 'prev_state' list (SYN-517) 2015-11-13 17:27:25 +00:00
Erik Johnston c0b3554401 Fix missing profile data in federation joins
There was a regression where we stopped including profile data in
initial joins for rooms joined over federation.
2015-11-12 16:19:55 +00:00
Daniel Wagner-Hall 2cebe53545 Exchange 3pid invites for m.room.member invites 2015-11-05 16:43:19 +00:00
Erik Johnston 97d792b28f Don't rearrange transaction_queue 2015-11-03 16:31:08 +00:00
Erik Johnston 7ce264ce5f Fix broken cache for getting retry times. This meant we retried remote destinations way more frequently than we should 2015-11-03 16:24:03 +00:00
Daniel Wagner-Hall 137fafce4e Allow rejecting invites
This is done by using the same /leave flow as you would use if you had
already accepted the invite and wanted to leave.
2015-10-20 11:58:58 +01:00
Daniel Wagner-Hall 643b5fcdc8 Look for keys on the right objects 2015-10-15 13:10:30 +01:00
Daniel Wagner-Hall 0c38e8637f Remove unnecessary class-wrapping 2015-10-13 18:00:38 +01:00
Daniel Wagner-Hall 17dffef5ec Move event contents into third_party_layout field 2015-10-13 15:48:12 +01:00
Daniel Wagner-Hall fcd9ba8802 Fix lint errors 2015-10-06 10:13:05 -05:00
Daniel Wagner-Hall 5b3e9713dd Implement third party identifier invites 2015-10-01 17:49:52 +01:00
Mark Haines 8899df13bf Merge pull request #208 from matrix-org/markjh/end-to-end-key-federation
Federation for end-to-end key requests.
2015-08-18 09:12:54 +01:00
Mark Haines 0cceb2ac92 Add a few strategic new lines to break up the on_query_client_keys and on_claim_client_keys methods in federation_server.py 2015-08-13 17:27:46 +01:00
Erik Johnston 2df8dd9b37 Move all the caches into their own package, synapse.util.caches 2015-08-11 18:00:59 +01:00
Mark Haines 2da3b1e60b Get the end-to-end key federation working 2015-07-24 18:26:46 +01:00
Mark Haines 62c010283d Add federation support for end-to-end key requests 2015-07-23 16:03:38 +01:00
Erik Johnston b5f55a1d85 Implement bulk verify_signed_json API 2015-06-26 10:39:34 +01:00
Erik Johnston cee69441d3 Log more when we have processed the request 2015-06-15 17:11:44 +01:00
Erik Johnston 8047fd2434 Merge pull request #176 from matrix-org/erikj/backfill_auth
Improve backfill.
2015-06-03 17:25:37 +01:00
Erik Johnston 3483b78d1a Log where a request came from in federation 2015-06-02 18:15:13 +01:00
Erik Johnston d3ded420b1 Rephrase log line 2015-06-02 16:30:52 +01:00
Erik Johnston 22716774d5 Don't about JSON when warning about content tampering 2015-06-02 16:30:52 +01:00
Erik Johnston 6eadbfbea0 Remove redundant for loop 2015-05-22 16:12:20 +01:00
Erik Johnston 74b7de83ec Merge branch 'develop' of github.com:matrix-org/synapse into erikj/backfill_fixes 2015-05-22 16:10:42 +01:00
Mark Haines 36317f3dad Merge pull request #156 from matrix-org/erikj/join_perf
Make joining #matrix:matrix.org over federation quicker
2015-05-22 16:09:54 +01:00
Erik Johnston 284f55a7fb Add doc strings 2015-05-22 15:18:04 +01:00
Erik Johnston b21d015c55 Log origin and stats of incoming transactions 2015-05-22 14:44:25 +01:00
Erik Johnston e70e8e053e Add txn_id to some log lines 2015-05-22 14:44:02 +01:00
Erik Johnston 1b446a5d85 Log less lines at INFO level, but include more helpful information 2015-05-22 14:29:57 +01:00
Erik Johnston c2633907c5 Merge branch 'erikj/join_perf' of github.com:matrix-org/synapse into erikj/backfill_fixes 2015-05-21 14:58:47 +01:00
Erik Johnston ebfdd2eb5b Merge branch 'develop' of github.com:matrix-org/synapse into erikj/join_perf 2015-05-21 14:54:52 +01:00
Erik Johnston 20814fabdd Actually fetch state for new backwards extremeties when backfilling. 2015-05-20 11:59:02 +01:00
Erik Johnston 3a653515ec Add None check 2015-05-19 15:27:09 +01:00
Erik Johnston 5b1631a4a9 Add a timeout param to get_event 2015-05-19 14:53:32 +01:00
Erik Johnston d5cea26d45 Remove pointless newline 2015-05-18 10:16:45 +01:00
Erik Johnston c71176858b Newline, remove debug logging 2015-05-18 10:11:14 +01:00
Erik Johnston f8bd4de87d Remove debug logging 2015-05-18 09:58:03 +01:00
Erik Johnston a2c4f3f150 Fix daedlock 2015-05-15 10:54:04 +01:00
Erik Johnston 95dedb866f Unwrap defer.gatherResults failures 2015-05-12 13:14:29 +01:00
Erik Johnston 476899295f Change the way we do logging contexts so that they survive divergences 2015-05-08 16:32:18 +01:00
Erik Johnston 0cf7e480b4 And use buffer(...) there as well 2015-05-05 18:20:01 +01:00
Erik Johnston 977338a7af Use buffer(...) when inserting into bytea column 2015-05-05 18:12:53 +01:00
Erik Johnston 3d5a955e08 Missed events are not outliers 2015-05-05 17:36:57 +01:00
Erik Johnston 1942382246 Don't log enqueue_ 2015-05-01 16:14:25 +01:00
Erik Johnston 23c639ff32 Split a storage function in two so that we don't have to do extra work. 2015-05-01 10:17:19 +01:00
Erik Johnston c178e4e6ca Add missing servlet to list 2015-03-19 17:48:21 +00:00
Paul "LeoNerd" Evans 128cf2daf7 Appease pep8 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans c782e893ec Neater metrics from TransactionQueue 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans 2e4f0b2bd7 Replace the @metrics.counted annotations in federation with specifically-written counters and distributions 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans 1748605c5d Count incoming HTTP requests per servlet that responds 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans b0cf867319 Use _ instead of . as a metric namespacing separator, for Prometheus 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans f9478e475b Rename Metrics' "keys" to "labels" 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans 094803cf82 Put vector gauges on transaction queue pending PDU and EDU dicts 2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans 9470412316 Initial attempt at sprinkling some @metrics.counted decorations around the federation code 2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans 120b689284 Delete pointless (and unreachable) __init__ method from FederationClient 2015-03-12 16:24:50 +00:00
Erik Johnston abaf47bbb6 Fix bug in logging. 2015-03-10 10:28:29 +00:00
Erik Johnston 98b867f7b7 Fix bug in logging. 2015-03-10 10:16:09 +00:00
Erik Johnston 019422ebba Merge pull request #101 from matrix-org/neaten-federation-servlets
Neaten federation servlets
2015-03-09 17:39:06 +00:00
Paul "LeoNerd" Evans d79d91a4a7 Appease pep8 2015-03-05 20:55:40 +00:00
Paul "LeoNerd" Evans 5eab2549ab Append a $ on PATH at registration time, meaning each PATH attribute doesn't need it 2015-03-05 20:36:05 +00:00
Paul "LeoNerd" Evans 7644cb79b2 Slightly neater(?) arrangement of authentication wrapper for HTTP servlet methods 2015-03-05 20:33:16 +00:00
Paul "LeoNerd" Evans ba8ac996f9 Remove the dead 'rate_limit_origin' method from TransportLayerServer 2015-03-05 19:43:17 +00:00
Paul "LeoNerd" Evans a901ed16b5 Move federation API responding code out of weird mix of lambdas into Servlet-style methods on instances 2015-03-05 19:10:57 +00:00
Erik Johnston 9708f49abf Docs 2015-03-05 16:35:16 +00:00
Erik Johnston 96fee64421 Remove unecessary check 2015-03-05 16:31:47 +00:00
Erik Johnston 39aa968a76 Respect min_depth argument 2015-03-05 16:31:32 +00:00
Erik Johnston 6dfd8c73fc Docs. 2015-03-05 16:31:13 +00:00
Erik Johnston ae702d161a Handle if get_missing_pdu returns 400 or not all events. 2015-03-05 16:08:02 +00:00
Erik Johnston 0a036944bd Merge branch 'develop' of github.com:matrix-org/synapse into batched_get_pdu 2015-03-02 13:53:30 +00:00
Erik Johnston 4195e55ccc Merge branch 'develop' of github.com:matrix-org/synapse into federation_rate_limit 2015-03-02 13:39:22 +00:00
Erik Johnston 29481690c5 If we're yielding don't add errback 2015-03-02 11:50:43 +00:00
Erik Johnston 23d9bd1d74 Process transactions serially.
Since the events received in a transaction are ordered, later events
might depend on earlier events and so we shouldn't blindly process them
in parellel.
2015-03-02 11:39:57 +00:00
Erik Johnston 9d9b230501 Make the federation server ratelimiting configurable. 2015-03-02 11:33:45 +00:00
Erik Johnston 0554d07082 Move federation rate limiting out of transport layer 2015-02-27 15:41:52 +00:00
Erik Johnston 9dc9118e55 Document FederationRateLimiter 2015-02-27 15:16:47 +00:00
Erik Johnston 93d90765c4 Initial implementation of federation server rate limiting 2015-02-26 16:15:26 +00:00
Erik Johnston 59362454dd Must update pending_transactions map before yield'ing 2015-02-26 15:47:35 +00:00
Erik Johnston db215b7e00 Implement and use new batched get missing pdu 2015-02-23 13:58:02 +00:00
Erik Johnston 0ac2a79faa Initial stab at implementing a batched get_missing_pdus request 2015-02-19 17:24:14 +00:00
Erik Johnston a7925259a1 Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.7.1 2015-02-18 13:57:55 +00:00
Erik Johnston 446ef58992 Add errback to all deferreds in transaction_queue 2015-02-18 12:03:26 +00:00
Mark Haines 2462aacd77 Restrict the destinations that synapse can talk to 2015-02-18 11:52:51 +00:00
Erik Johnston b68e4a729f Discard destination 'localhost' 2015-02-18 11:32:39 +00:00
Erik Johnston 47d3ff4cf8 Don't send failure to self 2015-02-18 11:30:37 +00:00
Erik Johnston e482541e1d Fix pyflakes 2015-02-18 10:44:22 +00:00
Erik Johnston 1be67eca8a Merge branch 'keyclient_retry_scheme' of github.com:matrix-org/synapse into develop 2015-02-18 10:34:40 +00:00
Erik Johnston fe672a04f7 Merge pull request #77 from matrix-org/failures
Failures
2015-02-18 10:29:29 +00:00
Erik Johnston 08f804208b Merge pull request #79 from matrix-org/get_pdu_limiting
Get pdu limiting
2015-02-18 10:29:10 +00:00
Erik Johnston ec847059f3 Rename _fail_fetch_pdu_cache to _get_pdu_cache 2015-02-18 10:14:10 +00:00
Erik Johnston 9371019133 Try to only back off if we think we failed to connect to the remote 2015-02-17 18:13:34 +00:00
Erik Johnston 2b8f1a956c Add per server retry limiting.
Factor out the pre destination retry logic from TransactionQueue so it
can be reused in both get_pdu and crypto.keyring
2015-02-17 17:20:56 +00:00
Erik Johnston 676e8ee78a Remove debug raise 2015-02-17 15:22:45 +00:00
Erik Johnston fa6c93bd26 Merge branch 'consumeErrors' of github.com:matrix-org/synapse into develop 2015-02-17 15:18:17 +00:00
Erik Johnston c02da58a9d Merge branch 'develop' of github.com:matrix-org/synapse into failures 2015-02-17 15:15:07 +00:00
Erik Johnston 659ead082f Format the response of transaction request in a nicer way 2015-02-17 15:11:44 +00:00
Erik Johnston c82e26ad4b Actually respond with JSON to incoming transaction 2015-02-17 13:24:13 +00:00
Erik Johnston 02bfa889de Handle recieving failures in transactions 2015-02-17 13:13:14 +00:00
Erik Johnston 72a4de2ce6 Use consumeErrors=True on all DeferredLists.
This is so that the DeferredLists actually consume the error instead of
propogating down the non-existent errback chain. This should reduce the
number of unhandled errors we are seeing.
2015-02-17 10:07:01 +00:00
Erik Johnston 0194e71e99 Merge branch 'develop' of github.com:matrix-org/synapse into get_pdu_limiting 2015-02-17 09:48:23 +00:00
Erik Johnston baa5b9a975 Cache results of get_pdu. 2015-02-16 18:02:39 +00:00
Erik Johnston 91fc5eef1d Mark old events as outliers.
This is to fix the issue where if a remote server sends an event
that references a really "old" event, then the local server will pull
that in and send to all clients.

We decide if an event is old if its depth is less than the minimum depth
of the room.
2015-02-16 14:27:40 +00:00
Erik Johnston 6138584651 Don't return anything from _handle_new_pdu, since we ignore the return value anyway 2015-02-16 14:08:02 +00:00
Erik Johnston 789251afa7 Fix logging 2015-02-12 19:29:43 +00:00
Erik Johnston 58d848adc0 Parrellize fetching of events 2015-02-12 18:35:36 +00:00
Erik Johnston 963256638d Correctly handle all the places that can throw exceptions 2015-02-12 18:17:11 +00:00
Erik Johnston c52e8d395b Merge pull request #61 from matrix-org/timeout-federation-requests
Timeout federation requests
2015-02-11 17:10:33 +00:00
Erik Johnston ef276e8770 Fix so timing out connections to actually work. 2015-02-11 16:48:05 +00:00
Erik Johnston ddb816cf60 Don't unfreeze when using FreezeEvent.get_dict, as we are using a JSONEncoder that understands FrozenDict 2015-02-11 15:44:28 +00:00
Erik Johnston 4ebbaf0d43 Blunty replace json with simplejson 2015-02-11 14:23:10 +00:00
Erik Johnston c8e1da930d Log all the exits from _attempt_new_transaction 2015-02-10 17:30:46 +00:00
Erik Johnston 697ab75a34 Sign auth_chains when returned by /state/ requests 2015-02-10 15:46:24 +00:00
Erik Johnston e9c85a4d5a Connection errors in twisted aren't RuntimeErrors 2015-02-05 13:50:15 +00:00
Erik Johnston e1515c3e91 Pass through list of room hosts from room alias query to federation so that it can retry against different room hosts 2015-02-05 13:44:42 +00:00
Erik Johnston ae46f10fc5 Apply sanity to the transport client interface. Convert 'make_join' and 'send_join' to accept iterables of destinations 2015-02-04 16:28:12 +00:00
Erik Johnston ff78eded01 Retry make_join 2015-02-04 13:55:10 +00:00
Erik Johnston 3c39f42a05 New line 2015-02-03 16:14:19 +00:00
Erik Johnston 9bace3a367 Actually, the old prune_event function was non-deterministic, so no point keeping it around :( 2015-02-03 15:32:17 +00:00
Erik Johnston 8dae5c8108 Remove unused imports 2015-02-03 15:01:12 +00:00
Erik Johnston 7b810e136e Add new FederationBase 2015-02-03 15:00:42 +00:00
Erik Johnston 0dd3aea319 Keep around the old (buggy) version of the prune_event function so that we can use it to check signatures for events on old servers 2015-02-03 14:58:30 +00:00
Erik Johnston 0f48e22ef6 PEP8 2015-02-03 10:43:29 +00:00
Erik Johnston 40c6fe1b81 Don't bother requesting PDUs with bad signatures from the same server 2015-02-02 17:06:37 +00:00
Erik Johnston 941f59101b Don't fail an entire request if one of the returned events fails a signature check. If an event does fail a signature check, look in the local database and request it from the originator. 2015-02-02 16:56:01 +00:00
Erik Johnston 776ac820f9 Briefly doc structure of query_auth API. 2015-01-30 15:58:28 +00:00
Erik Johnston a70a801184 Fix bug where we superfluously asked for current state. Change API of /query_auth/ so that we don't duplicate events in the response. 2015-01-30 13:34:01 +00:00
Erik Johnston c1d860870b Fix regression where we no longer correctly handled the case of gaps in our event graph 2015-01-30 10:48:47 +00:00
Erik Johnston 78015948a7 Initial implementation of auth conflict resolution 2015-01-29 16:52:33 +00:00
Erik Johnston 0ef5bfd6a9 Start implementing auth conflict res 2015-01-28 16:16:53 +00:00
Erik Johnston c92d64a6c3 Make it the responsibility of the replication layer to check signature and hashes. 2015-01-26 14:33:11 +00:00
Erik Johnston 7b88619241 Split up replication_layer module into client, server and transaction queue 2015-01-26 10:45:24 +00:00
Erik Johnston ca65a9d03e Split out TransactionQueue from replication layer 2015-01-22 16:37:08 +00:00
Mark Haines 5fed042640 Finish renaming "context" to "room_id" in federation codebase 2015-01-16 19:01:03 +00:00
Mark Haines 2408c4b0a4 Fold _do_request_for_transaction into the methods that called it since it was a trivial wrapper around client.get_json 2015-01-16 19:01:03 +00:00
Mark Haines 602684eac5 Split transport layer into client and server parts 2015-01-16 19:01:03 +00:00
Matrix 0529a7e2e9 Add some logging for when we are sending transactions. 2015-01-06 14:06:25 +00:00
Mark Haines adb04b1e57 Update copyright notices 2015-01-06 13:21:39 +00:00
Mark Haines 041ac476a5 Supply auth_chain along with current state in '/state/', fetch auth events from a remote server if we are missing some of them 2014-12-18 18:47:13 +00:00
Erik Johnston 52f99243ab Use is_outlier() so that we don't get AttributeError 2014-12-16 18:33:50 +00:00
Erik Johnston 3c77d13aa5 Kill off synapse.api.events.* 2014-12-16 11:29:05 +00:00
Erik Johnston aae8a37e63 Merge branch 'develop' of github.com:matrix-org/synapse into events_refactor 2014-12-10 13:18:40 +00:00
Erik Johnston 95aa903ffa Try and figure out how and why signatures are being changed. 2014-12-10 11:37:47 +00:00
Erik Johnston b8d30899b1 Code style. 2014-12-10 10:16:09 +00:00
Matthew Hodgson 71da2bed55 plateau retries after 1h 2014-12-10 00:18:44 +00:00
Matthew Hodgson faf12b64f8 add errbacks to enqueue_pdu deferreds; change logging for failed federation sends to warn rather than exception 2014-12-10 00:12:51 +00:00
Matthew Hodgson 2b1acb7671 squidge to 79 columns as per pep8 2014-12-10 00:03:55 +00:00
Matthew Hodgson 8529fba02d fix a million stupid bugs and make it actually work 2014-12-08 19:34:51 +00:00
Erik Johnston ee3df06183 More bug fixes 2014-12-08 14:50:48 +00:00
Erik Johnston d044121168 Various typos and bug fixes. 2014-12-08 09:08:26 +00:00
Matthew Hodgson 9c43b258ec actually reset retry schedule if we can successfuly talk to it 2014-12-08 00:17:12 +00:00
Matthew Hodgson 5cd43d4b9f fix stupid syntax thinkos 2014-12-07 23:44:16 +00:00
Matthew Hodgson aed62a3583 track replication destination health, and perform exponential back-off when sending transactions. does *not* yet retry transactions, but drops them on the floor if waiting for a server to recover. 2014-12-07 02:26:07 +00:00
Erik Johnston c31dba86ec Convert rest and handlers to use new event structure 2014-12-04 15:50:01 +00:00
Erik Johnston 5d7c9ab789 Begin converting things to use the new Event structure 2014-12-04 11:27:59 +00:00
Erik Johnston d06dfc70b0 Fix bug where did not always resolve all the deferreds in _attempt_new_transaction 2014-12-03 11:50:23 +00:00
Erik Johnston 07699b5871 Change the way we get missing auth and state events 2014-11-27 14:31:43 +00:00
Erik Johnston 3598c11c8d Correctly handle the case where we get an event for an unknown room, which turns out we are actually in 2014-11-26 10:41:08 +00:00
Erik Johnston 64fc859dac Fix bugs in invite/join dances.
We now do more implement more of the auth on the events so that we
don't reject valid events.
2014-11-25 17:59:49 +00:00
Erik Johnston 4961a4fab1 Mark the auth events as possible outlier 2014-11-24 13:55:49 +00:00
Erik Johnston 2bca242fdc Ask for any auth events that we don't have 2014-11-24 13:46:41 +00:00
Erik Johnston 4bd0ab76c6 We don't always want to Auth get_persisted_pdu 2014-11-24 12:56:17 +00:00
Mark Haines db9ce032a4 Fix pep8 codestyle warnings 2014-11-20 17:26:36 +00:00
Mark Haines 32090aee16 Add a few missing yields, Move deferred lists inside PreserveLoggingContext because they don't interact well with the logging contexts 2014-11-20 16:24:00 +00:00
Erik Johnston 4eada9a908 Fix backfill request 2014-11-19 17:22:37 +00:00
Mark Haines c5eabe3143 replace user_id with sender 2014-11-19 16:38:40 +00:00
Mark Haines 428581dd05 SYN-144: Remove bad keys from pdu json objects, convert age_ts to age
for all pdus sent.
2014-11-18 19:20:25 +00:00
Mark Haines db7e8b5619 SYN-141: Decode the query params as UTF-8 2014-11-18 17:17:57 +00:00
Mark Haines cb4b6c844a Merge PDUs and Events into one object 2014-11-14 21:25:02 +00:00
Mark Haines 8d8a133c89 SYN-103: Remove "origin" and "destination" keys from edus 2014-11-13 15:49:03 +00:00
Erik Johnston 3db0efa69f Fix pyflake warnings and add a FIXME comment to deal with auth_chains received when joining 2014-11-12 11:27:02 +00:00
Erik Johnston 6fea478d2e Fix bugs with invites/joins across federatiom.
Both in terms of auth and not trying to fetch missing PDUs for invites,
joins etc.
2014-11-12 11:24:11 +00:00
Erik Johnston 092979b8cc Fix bugs which broke federation due to changes in function signatures. 2014-11-11 14:19:13 +00:00
Erik Johnston 5d439b127b PEP8 2014-11-10 13:46:44 +00:00
Erik Johnston c46088405a Remove useless comments 2014-11-10 13:39:33 +00:00
Erik Johnston 003668cfaa Add auth to the various server-server APIs 2014-11-10 13:37:24 +00:00
Erik Johnston 6447db063a Fix backfill to work. Add auth to backfill request 2014-11-10 11:59:51 +00:00
Erik Johnston 1c06806f90 Finish redaction algorithm. 2014-11-10 10:21:32 +00:00
Erik Johnston 02c3b1c9e2 Add '/event_auth/' federation api 2014-11-07 15:35:53 +00:00
Erik Johnston d2fb2b8095 Implement invite part of invite join dance 2014-11-07 13:41:00 +00:00
Erik Johnston 328dab2463 Remove /context/ request 2014-11-07 11:40:38 +00:00
Erik Johnston 3b4dec442d Return auth chain when handling send_join 2014-11-07 11:22:12 +00:00
Erik Johnston a5a4ef3fd7 Fix bug in replication 2014-11-04 15:16:43 +00:00
Erik Johnston fc7b2b11a2 PEP8 2014-11-04 15:09:34 +00:00
Erik Johnston 440cbd5235 Add support for sending failures 2014-11-04 14:17:55 +00:00
Erik Johnston d7412c4df1 Remove unused interface 2014-11-04 14:16:19 +00:00
Erik Johnston aa76bf39ab Remove unused imports 2014-11-04 14:14:02 +00:00
Erik Johnston 68698e0ac8 Fix bugs in generating event signatures and hashing 2014-11-03 17:51:42 +00:00
Erik Johnston ad6eacb3e9 Rename PDU fields to match that of events. 2014-11-03 13:06:58 +00:00
Erik Johnston d59aa6af25 For now, don't store txn -> pdu mappings. 2014-11-03 11:35:19 +00:00
Erik Johnston 2f39dc19a2 Remove more references to dead PDU tables 2014-10-31 14:27:14 +00:00
Erik Johnston bfa36a72b9 Remove PDU tables. 2014-10-31 14:00:32 +00:00
Erik Johnston 21fe249d62 Actually don't store any PDUs 2014-10-31 10:47:34 +00:00
Erik Johnston 841df4da71 Don't store any PDUs 2014-10-31 09:59:59 +00:00
Erik Johnston f2de2d644a Move the impl of backfill to use events. 2014-10-31 09:59:02 +00:00
Erik Johnston 12ce441e67 Convert event ids to be of the form :example.com 2014-10-30 17:00:11 +00:00
Erik Johnston e7858b6d7e Start filling out and using new events tables 2014-10-29 16:59:24 +00:00
Erik Johnston 2d1dfb3b34 Begin implementing all the PDU storage stuff in Events land 2014-10-28 16:42:35 +00:00
Erik Johnston ad9226eeec Merge branch 'event_signing' of github.com:matrix-org/synapse into federation_authorization
Conflicts:
	synapse/storage/__init__.py
2014-10-27 11:58:32 +00:00
Mark Haines 5e2236f9ff fix pyflakes warnings 2014-10-27 11:19:15 +00:00
Mark Haines acb2d171e8 Merge branch 'develop' into event_signing 2014-10-27 11:14:11 +00:00
Erik Johnston bb4a20174c Merge branch 'develop' of github.com:matrix-org/synapse into federation_authorization
Conflicts:
	synapse/federation/transport.py
	synapse/handlers/message.py
2014-10-27 10:20:44 +00:00
Erik Johnston ac9345b47a Check that we have auth headers and fail nicely 2014-10-17 21:00:58 +01:00
Erik Johnston cd198dfea8 More log lines. 2014-10-17 20:58:47 +01:00
Mark Haines 8afbece683 Remove signatures from pdu when computing hashes to use for prev pdus, make sure is_state is a boolean. 2014-10-17 19:41:32 +01:00