Commit Graph

2514 Commits

Author SHA1 Message Date
Erik Johnston b21d015c55 Log origin and stats of incoming transactions 2015-05-22 14:44:25 +01:00
Erik Johnston e70e8e053e Add txn_id to some log lines 2015-05-22 14:44:02 +01:00
Erik Johnston 1b446a5d85 Log less lines at INFO level, but include more helpful information 2015-05-22 14:29:57 +01:00
Erik Johnston 59a0682f3e Enable changing the interface the metrics listener binds to 2015-05-22 13:13:07 +01:00
Mark Haines b6adfc59f5 Invalidate the get_latest_event_ids_in_room cache when deleting from event_forward_extremities 2015-05-22 13:01:03 +01:00
Mark Haines f43544eecc Make the appservice use 'users_in_room' rather than get_room_members since it is cached 2015-05-22 11:01:28 +01:00
Mark Haines a04cde613e Add a cache for get_push rules for user, fix cache invalidation 2015-05-22 10:39:45 +01:00
Erik Johnston 4429e720ae Merge branch 'master' of github.com:matrix-org/synapse into develop 2015-05-22 10:33:00 +01:00
Erik Johnston f8c2cd129d Bump version 2015-05-21 17:03:30 +01:00
Mark Haines 2043527b9b Don't try to use a txn when not in one, remove spurious debug logging 2015-05-21 16:53:03 +01:00
Mark Haines 53447e9cd3 Add caches for things requested by the pushers 2015-05-21 16:41:39 +01:00
Mark Haines d61ce3f670 Add a cache for get_current_state with state_key 2015-05-21 16:41:39 +01:00
Erik Johnston a910984b58 Actually return something from lambda 2015-05-21 15:58:41 +01:00
Erik Johnston e309b1045d Sort backfill events 2015-05-21 15:57:35 +01:00
Erik Johnston 0180bfe4aa Remove dead code 2015-05-21 15:53:41 +01:00
Erik Johnston 1f3d1d85a9 Only get non-state 2015-05-21 15:52:29 +01:00
Erik Johnston 39a3340f73 Skip events we've already seen 2015-05-21 15:48:56 +01:00
Erik Johnston ae3bff3491 Correctly prepopulate queue 2015-05-21 15:46:07 +01:00
Erik Johnston dc085ddf8c Don't prepopulate event_results 2015-05-21 15:44:05 +01:00
Erik Johnston 73d23c6ae8 Don't readd things that are already in event_results 2015-05-21 15:40:22 +01:00
Erik Johnston 6189d8e54d PriorityQueue gives lowest first 2015-05-21 15:38:08 +01:00
Erik Johnston 115ef3ddac Correctly capture Queue.Empty exception 2015-05-21 15:37:43 +01:00
Erik Johnston 4fb858d90a Merge branch 'develop' of github.com:matrix-org/synapse into erikj/backfill_fixes 2015-05-21 15:25:54 +01:00
Mark Haines 88f1ea36ce Oops, get_rooms_for_user returns a namedtuple, not a room_id 2015-05-21 15:23:58 +01:00
Erik Johnston c2633907c5 Merge branch 'erikj/join_perf' of github.com:matrix-org/synapse into erikj/backfill_fixes 2015-05-21 14:58:47 +01:00
Erik Johnston ebfdd2eb5b Merge branch 'develop' of github.com:matrix-org/synapse into erikj/join_perf 2015-05-21 14:54:52 +01:00
Erik Johnston a551c5dad7 Merge pull request #155 from matrix-org/erikj/perf
Bulk and batch retrieval of events.
2015-05-21 14:54:40 +01:00
Erik Johnston 27e4b45c06 s/for events/for requests for events/ 2015-05-21 14:52:23 +01:00
Erik Johnston ac5f2bf9db s/for events/for requests for events/ 2015-05-21 14:50:57 +01:00
Erik Johnston 80a167b1f0 Add comments 2015-05-21 11:19:04 +01:00
Mark Haines 7ae8afb7ef Removed unused 'is_visible' method 2015-05-20 14:48:11 +01:00
Erik Johnston 9118a92862 Split up _get_events into defer and txn versions 2015-05-20 13:27:16 +01:00
Mark Haines e01b825cc9 Clean up the presence_list checking logic a bit 2015-05-20 13:21:59 +01:00
Erik Johnston ab45e12d31 Make not return a deferred _get_event_from_row_txn 2015-05-20 13:07:19 +01:00
Erik Johnston f407cbd2f1 PEP8 2015-05-20 13:02:01 +01:00
Erik Johnston 227f8ef031 Split out _get_event_from_row back into defer and _txn version 2015-05-20 13:00:57 +01:00
Erik Johnston 2bc60c55af Fix _get_backfill_events to return events in the correct order 2015-05-20 12:57:00 +01:00
Erik Johnston 20814fabdd Actually fetch state for new backwards extremeties when backfilling. 2015-05-20 11:59:02 +01:00
Erik Johnston 9084cdd70f Ensure event_results is a set 2015-05-19 16:34:31 +01:00
Erik Johnston 3a653515ec Add None check 2015-05-19 15:27:09 +01:00
Erik Johnston aa729349dd Fix event_backwards_extrem insertion to ignore outliers 2015-05-19 15:27:00 +01:00
Erik Johnston 5b1631a4a9 Add a timeout param to get_event 2015-05-19 14:53:32 +01:00
Erik Johnston 291cba284b Handle the case when things return empty but non none things 2015-05-19 14:42:46 +01:00
Erik Johnston 253f76a0a5 Don't always hit get_server_verify_key_v1_direct 2015-05-19 14:42:38 +01:00
Erik Johnston 6837c5edab Handle the case when things return empty but non none things 2015-05-19 14:27:11 +01:00
Erik Johnston 7223129916 Don't apply new room join hack if depth > 5 2015-05-19 14:16:08 +01:00
Erik Johnston 5ae4a84211 Don't always hit get_server_verify_key_v1_direct 2015-05-19 13:43:34 +01:00
Erik Johnston 118a760719 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/join_perf 2015-05-19 13:20:29 +01:00
David Baker 19505e0392 Disable GZip encoding on static file resources as per comment 2015-05-19 13:20:25 +01:00
Erik Johnston d3e09f12d0 SYN-383: Actually, we expect this value to be a dict 2015-05-19 13:12:41 +01:00
Erik Johnston 677be13ffc Revert accidental commit 2015-05-19 13:12:28 +01:00
Erik Johnston 350b88656a SYN-383: Actually, we expect this value to be a dict 2015-05-19 13:01:57 +01:00
Erik Johnston 9de94d5a4d Merge branch 'develop' of github.com:matrix-org/synapse into erikj/join_perf 2015-05-19 12:50:17 +01:00
Erik Johnston 2b7120e233 SYN-383: Handle the fact the server might not have signed things 2015-05-19 12:49:38 +01:00
Erik Johnston 8b256a7296 Don't reuse var names 2015-05-19 11:58:22 +01:00
Erik Johnston 62ccc6d95f Don't reuse var names 2015-05-19 11:58:04 +01:00
Erik Johnston 01858bcbf2 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/join_perf 2015-05-19 11:56:35 +01:00
Erik Johnston 2aeee2a905 SYN-383: Fix parsing of verify_keys and catching of _DefGen_Return 2015-05-19 11:56:18 +01:00
Erik Johnston 5e7883ec19 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/join_perf 2015-05-19 10:50:43 +01:00
Mark Haines c6a03c46e6 SYN-383: Extract the response list from 'server_keys' in the response JSON as it might work better than iterating over the top level dict 2015-05-19 10:23:02 +01:00
Mark Haines e4c65b338d Speed up the get_pagination_rows as well 2015-05-18 18:21:06 +01:00
Mark Haines 99914ec9f8 Merge pull request #152 from matrix-org/notifier_performance
Notifier performance
2015-05-18 17:49:59 +01:00
Erik Johnston ef910a0358 Do work in parellel when joining a room 2015-05-18 17:17:04 +01:00
Mark Haines 591c4bf223 Cache the most recent serial for each room 2015-05-18 16:21:51 +01:00
Mark Haines e1150cac4b Move updating the serial and state of the presence cache into a single function 2015-05-18 15:46:37 +01:00
Erik Johnston 165eb2dbe6 Comments and shuffle of functions 2015-05-18 15:18:41 +01:00
Mark Haines 880fb46de0 Merge branch 'notifier_performance' into markjh/presence_performance 2015-05-18 14:33:58 +01:00
Erik Johnston 65878a2319 Remove unused metric 2015-05-18 14:06:30 +01:00
Mark Haines ad31fa3040 Don't bother sorting by the room_stream_ids, it shouldn't matter which order they are notified in 2015-05-18 14:04:58 +01:00
Erik Johnston 4d1b6f4ad1 Remove rejected events if we don't want rejected events 2015-05-18 14:03:46 +01:00
Mark Haines 0b0033c40b Merge branch 'develop' into notifier_performance 2015-05-18 13:50:01 +01:00
Mark Haines 755def8083 Add more doc string, reduce C+P boilerplate for getting room list 2015-05-18 13:46:47 +01:00
Mark Haines 1e90715a3d Make sure the notifier stream token goes forward when it is updated. Sort the pending events by the correct room_stream_id 2015-05-18 13:17:36 +01:00
Erik Johnston 131bdf9bb1 Merge branch 'erikj/events_move' of github.com:matrix-org/synapse into erikj/perf 2015-05-18 10:23:37 +01:00
Erik Johnston 10f1bdb9a2 Move get_events functions to storage.events 2015-05-18 10:21:40 +01:00
Erik Johnston d5cea26d45 Remove pointless newline 2015-05-18 10:16:45 +01:00
Erik Johnston c71176858b Newline, remove debug logging 2015-05-18 10:11:14 +01:00
Erik Johnston f8bd4de87d Remove debug logging 2015-05-18 09:58:03 +01:00
Erik Johnston c3b37abdfd PEP8 2015-05-15 16:59:58 +01:00
Erik Johnston 6c74fd62a0 Revert limiting of fetching, it didn't help perf. 2015-05-15 16:45:35 +01:00
Erik Johnston 9ff7f66a2b init j 2015-05-15 16:36:03 +01:00
Erik Johnston 70f272f71c Don't completely drain the list 2015-05-15 16:34:17 +01:00
Erik Johnston 8763dd80ef Don't fetch prev_content for current_state 2015-05-15 15:33:01 +01:00
Erik Johnston 807229f2f2 Err, defer.gatherResults ftw 2015-05-15 15:20:29 +01:00
Erik Johnston acb12cc811 Make store.get_current_state fetch events asyncly 2015-05-15 15:20:05 +01:00
Erik Johnston d62dee7eae Remove more debug logging 2015-05-15 15:06:37 +01:00
Erik Johnston 0f29cfabc3 Remove debug logging 2015-05-15 14:06:42 +01:00
Erik Johnston e275a9c0d9 preserve log context 2015-05-15 11:54:51 +01:00
Erik Johnston aa32bd38e4 Add a wait 2015-05-15 11:35:04 +01:00
Erik Johnston 372d4c6d7b Srsly. Don't use closures. Baaaaaad 2015-05-15 11:26:00 +01:00
Erik Johnston 575ec91d82 Correctly pass through params 2015-05-15 11:15:10 +01:00
Mark Haines 10be983f2c Merge pull request #153 from matrix-org/markjh/presence_docstring
Add some doc strings for presence.
2015-05-15 11:11:47 +01:00
Mark Haines 415b158ce2 More whitespace 2015-05-15 11:09:47 +01:00
Erik Johnston de01438a57 Sort out error handling 2015-05-15 11:00:50 +01:00
Erik Johnston a2c4f3f150 Fix daedlock 2015-05-15 10:54:04 +01:00
Mark Haines 0a4330cd5d Add some missed argument types, cleanup the whitespace a bit 2015-05-14 17:48:12 +01:00
Mark Haines 47ec693e29 More doc-strings 2015-05-14 17:07:02 +01:00
Erik Johnston 1d566edb81 Remove race condition 2015-05-14 16:54:35 +01:00
David Baker 6e1ad283cf Support gzip encoding for client, client v2 and web client resources (SYN-176). 2015-05-14 16:39:19 +01:00
Erik Johnston ef3d8754f5 Call from right thread 2015-05-14 15:41:55 +01:00
Erik Johnston 142934084a Count and loop 2015-05-14 15:40:21 +01:00
Erik Johnston 96c5b9f87c Don't start up more fetch_events 2015-05-14 15:36:04 +01:00
Erik Johnston 7cd6a6f6cf Awful idea for speeding up fetching of events 2015-05-14 15:34:02 +01:00
Mark Haines c5d1b4986b Remove unused arguments and doc PresenceHandler.push_update_to_clients 2015-05-14 14:59:31 +01:00
Erik Johnston 7f4105a5c9 Turn off preemptive transactions 2015-05-14 14:51:06 +01:00
Erik Johnston f4d58deba1 PEP8 2015-05-14 14:45:42 +01:00
Erik Johnston 386b7330d2 Move from _base to events 2015-05-14 14:45:22 +01:00
Mark Haines 0ad1c67234 Add some doc-strings to notifier 2015-05-14 14:35:07 +01:00
Erik Johnston 7d6a1dae31 Jump out early 2015-05-14 14:27:58 +01:00
Erik Johnston 656223fbd3 Actually, we probably want to run this in a transaction 2015-05-14 14:26:35 +01:00
David Baker 67800f7626 Treat setting your display name to the empty string as removing it (SYN-186). 2015-05-14 14:19:59 +01:00
Erik Johnston 2f7f8e1c2b Preemptively jump into a transaction if we ask for get_prev_content 2015-05-14 14:17:36 +01:00
Mark Haines 4770cec7bc Merge pull request #150 from matrix-org/notifier_unify
Make v1 and v2 client APIs interact with the notifier in the same way.
2015-05-14 14:16:59 +01:00
Erik Johnston e1e9f0c5b2 loop -> gatherResults 2015-05-14 13:58:49 +01:00
Erik Johnston ab78a8926e Err, we probably want a bigger limit 2015-05-14 13:47:16 +01:00
Erik Johnston f6f902d459 Move fetching of events into their own transactions 2015-05-14 13:45:48 +01:00
Erik Johnston cdb3757942 Refactor _get_events 2015-05-14 13:31:55 +01:00
David Baker 92e1c8983d Disallow whitespace in aliases here too 2015-05-14 13:21:55 +01:00
David Baker 0c894e1ebd Throw error when creating room if alias contains whitespace #SYN-335 2015-05-14 13:11:28 +01:00
Mark Haines 084c365c3a Use the current token when timing out a notifier, make sure the user_id is a string in on_new_user_event 2015-05-14 12:03:26 +01:00
David Baker c37a6e151f Make shared secret registration work again 2015-05-14 12:03:13 +01:00
Erik Johnston 36ea26c5c0 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/perf 2015-05-14 12:01:38 +01:00
David Baker 7c549dd557 Add ID generator for push_rules_enable to #resolve SYN-378 2015-05-14 11:44:03 +01:00
Mark Haines 899d4675dd Merge branch 'notifier_unify' into notifier_performance 2015-05-14 11:36:44 +01:00
Mark Haines 243c56e725 Merge branch 'develop' into notifier_unify 2015-05-14 11:36:23 +01:00
Mark Haines 3edd2d5c93 Fix v2 sync, update the last_notified_ms only if there was an active listener 2015-05-14 11:25:30 +01:00
Erik Johnston 4f1d984e56 Add index on events 2015-05-13 17:22:26 +01:00
Mark Haines 5e0c533672 Fix metric counter 2015-05-13 17:20:28 +01:00
Erik Johnston 968b01a91a Actually use async method 2015-05-13 17:02:46 +01:00
Erik Johnston 4071f29653 Fetch events from events_id in their own transactions 2015-05-13 16:59:41 +01:00
Mark Haines f1b83d88a3 Discard unused NotifierUserStreams 2015-05-13 16:54:02 +01:00
Erik Johnston a988361aea Typo 2015-05-13 15:44:15 +01:00
Erik Johnston 8888982db3 Don't insert None 2015-05-13 15:43:32 +01:00
Mark Haines 9af432257d Don't set a timer if there's already a result to return 2015-05-13 15:42:13 +01:00
Erik Johnston cf706cc6ef Don't return None 2015-05-13 15:31:25 +01:00
Erik Johnston 5971d240d4 Limit batch size 2015-05-13 15:26:49 +01:00
Erik Johnston ca4f458787 Fetch events in bulk 2015-05-13 15:13:42 +01:00
Mark Haines df6db5c802 Don't bother checking for new events from a source if the stream token hasn't advanced for that source 2015-05-13 15:08:24 +01:00
Erik Johnston 6edff11a88 Don't fetch redaction and rejection stuff for each event, so we can use index only scan 2015-05-13 14:39:05 +01:00
Mark Haines 63878c0379 Don't bother checking for updates if the stream token hasn't advanced for a user 2015-05-13 13:42:21 +01:00
Erik Johnston 02590c3e1d Temp turn off checking for rejections and redactions 2015-05-13 11:31:28 +01:00
Erik Johnston 619a21812b defer.gatherResults loop 2015-05-13 11:29:03 +01:00
Erik Johnston fec4485e28 Batch fetching of events for state groups 2015-05-13 11:22:42 +01:00
Erik Johnston 409bcc76bd Load events for state group seperately 2015-05-13 11:13:31 +01:00
Mark Haines cffe6057fb Merge branch 'notifier_unify' into notifier_performance
Conflicts:
	synapse/notifier.py
2015-05-12 16:37:50 +01:00
Erik Johnston 80fd2b574c Don't talk to yourself when backfilling 2015-05-12 16:19:46 +01:00
Erik Johnston e122685978 You need to call contextmanager 2015-05-12 16:12:37 +01:00
Mark Haines d7b3ac46f8 Revert "Improvement to performance of presence event stream handling" 2015-05-12 15:44:21 +01:00
Mark Haines 4429e4bf24 Merge branch 'develop' into notifier_unify
Conflicts:
	synapse/notifier.py
2015-05-12 15:31:26 +01:00
Mark Haines ec07dba29e Merge pull request #143 from matrix-org/erikj/SYN-375
SYN-375 - Lots of unhandled deferred exceptions.
2015-05-12 15:25:54 +01:00
Mark Haines c167cbc9fd Merge pull request #147 from matrix-org/presence-performance
Improvement to performance of presence event stream handling
2015-05-12 15:24:54 +01:00
Mark Haines a6fb2aa2a5 Merge pull request #144 from matrix-org/erikj/logging_context
Preserving logging contexts
2015-05-12 15:23:50 +01:00
Mark Haines 1fce36b111 Merge pull request #149 from matrix-org/erikj/backfill
Backfill support
2015-05-12 15:20:32 +01:00
Erik Johnston 8b28209c60 Err, delete the right stuff 2015-05-12 15:02:53 +01:00
Erik Johnston 30c72d377e Newlines 2015-05-12 14:47:40 +01:00
Erik Johnston e4eddf9b36 We do actually want to delete rows out of event_backward_extremities 2015-05-12 14:47:23 +01:00
Erik Johnston c1779a79bc Fix up _handle_prev_events to not try to insert duplicate rows 2015-05-12 14:41:50 +01:00
Erik Johnston 74850d7f75 Do state groups persistence /after/ checking if we have already persisted the event 2015-05-12 14:14:58 +01:00
Erik Johnston 07a1223156 s/backfil/backfill/ 2015-05-12 14:09:54 +01:00
Erik Johnston 0d31ad5101 Typos everywhere 2015-05-12 14:02:01 +01:00
Erik Johnston a0dfffb33c And another typo. 2015-05-12 14:00:31 +01:00
Erik Johnston 6e5ac4a28f Err, gatherResults doesn't take a dict... 2015-05-12 13:58:14 +01:00
Erik Johnston 8022b27fc2 Make distributer.fire work as it did 2015-05-12 13:14:48 +01:00
Erik Johnston 95dedb866f Unwrap defer.gatherResults failures 2015-05-12 13:14:29 +01:00
Mark Haines 78672a9fd5 Merge branch 'notifier_unify' into notifier_performance 2015-05-12 13:11:54 +01:00
Erik Johnston da6a7bbdde Merge branch 'develop' of github.com:matrix-org/synapse into erikj/logging_context 2015-05-12 13:10:42 +01:00
Mark Haines 2551b6645d Update the end_token correctly, otherwise the token doesn't advance and the client gets duplicate events 2015-05-12 11:54:18 +01:00
Mark Haines 5e4ba463b7 Merge branch 'develop' into notifier_unify 2015-05-12 11:41:53 +01:00
Mark Haines 5002056b16 SYN-377: Make sure that the StreamIdGenerator.get_next.__exit__ is called from the main thread after the transaction completes, not from database thread before the transaction completes. 2015-05-12 11:20:40 +01:00
Mark Haines 5c75adff95 Add a NotifierUserStream to hold all the notification listeners for a user 2015-05-12 11:00:37 +01:00
Erik Johnston 367382b575 Handle the case where the other side is unreachable when backfilling 2015-05-12 10:35:45 +01:00
Erik Johnston 4df11b5039 Make get_current_token accept a direction parameter, which tells whether the source whether we want a token for going 'forwards' or 'backwards' 2015-05-12 10:28:10 +01:00
Erik Johnston 84e6b4001f Initial hack at wiring together pagination and backfill 2015-05-11 18:01:31 +01:00
Erik Johnston 17653a5dfe Move storage.stream._StreamToken to types.RoomStreamToken 2015-05-11 18:01:01 +01:00
Mark Haines e269c511f6 Don't bother passing the events to the notifier since it isn't using them 2015-05-11 15:01:51 +01:00
Mark Haines 5e3b254dc8 Use wait_for_events to implement 'get_events' 2015-05-11 14:37:33 +01:00
Erik Johnston 79b7154454 Merge pull request #146 from matrix-org/erikj/push_rules_fixes
Fix 500 on push rule updates.
2015-05-11 11:33:47 +01:00
Erik Johnston 4ef556f650 Bump version 2015-05-11 11:31:04 +01:00
Erik Johnston b036596b75 Prefer to use _simple_*. 2015-05-11 11:24:01 +01:00
Erik Johnston cd525c0f5a push_rules table expects an 'id' field 2015-05-11 11:24:01 +01:00
Erik Johnston de87541862 Bump version 2015-05-10 10:51:08 +01:00
Erik Johnston 22d2f498fa Fix push rule bug: can't insert bool into small int column 2015-05-10 10:50:51 +01:00
Erik Johnston 2236ef6c92 Fix up leak. Add warnings. 2015-05-08 19:53:34 +01:00
Erik Johnston da1aa07db5 Add some docs 2015-05-08 16:52:49 +01:00
Erik Johnston 4ac1941592 PEP8 2015-05-08 16:33:01 +01:00
Erik Johnston 476899295f Change the way we do logging contexts so that they survive divergences 2015-05-08 16:32:18 +01:00
Erik Johnston fca28d243e Change the way we create observers to deferreds so that we don't get spammed by 'unhandled errors' 2015-05-08 16:28:08 +01:00
Erik Johnston 0cd1401f8d Bump version 2015-05-08 16:11:51 +01:00
Erik Johnston 724bb1e7d9 Merge branch 'master' of github.com:matrix-org/synapse into develop 2015-05-08 16:11:19 +01:00
Mark Haines 1c7912751e Drop the old table not the new table 2015-05-08 16:04:32 +01:00
Mark Haines 9d36eb4eab Rename unique constraint 2015-05-08 16:01:55 +01:00
Mark Haines b0f71db3ff Remove unsigned 2015-05-08 15:59:51 +01:00
Mark Haines 84e1cacea4 Bump schema version 2015-05-08 15:58:14 +01:00
Mark Haines 6538d445e8 Make the timestamps in server_keys_json bigints 2015-05-08 15:55:17 +01:00
Erik Johnston 126d562576 Bump version 2015-05-08 13:29:37 +01:00
Paul "LeoNerd" Evans 45543028bb Use the presence cachemap ordering to early-abort the iteration loop 2015-05-07 22:40:10 +01:00
Paul "LeoNerd" Evans f683b5de47 Store presence cachemap in an ordered dict, so that the newer serials will be at the end 2015-05-07 21:27:53 +01:00
Erik Johnston db0dca2f6f Merge branch 'master' of github.com:matrix-org/synapse into develop 2015-05-07 19:21:00 +01:00
Erik Johnston a151693a3b Bump syweb version 2015-05-07 18:01:46 +01:00
Mark Haines dfa98f911b revert accidental bcrypt gensalt round reduction from loadtesting 2015-05-07 17:45:42 +01:00