Commit Graph

298 Commits

Author SHA1 Message Date
Neil Johnson e92fb00f32 sync auth blocking 2018-08-08 17:54:49 +01:00
Matthew Hodgson a75231b507
Deduplicate redundant lazy-loaded members (#3331)
* attempt at deduplicating lazy-loaded members

as per the proposal; we can deduplicate redundant lazy-loaded members
which are sent in the same sync sequence. we do this heuristically
rather than requiring the client to somehow tell us which members it
has chosen to cache, by instead caching the last N members sent to
a client, and not sending them again.  For now we hardcode N to 100.
Each cache for a given (user,device) tuple is in turn cached for up to
X minutes (to avoid the caches building up).  For now we hardcode X to 30.

* add include_redundant_members filter option & make it work

* remove stale todo

* add tests for _get_some_state_from_cache

* incorporate review
2018-07-26 22:51:30 +01:00
Matthew Hodgson 1a01a5b964 clarify comment on p_ids 2018-07-24 14:03:15 +01:00
Matthew Hodgson eb1d911ab7 rather than adding ll_ids, remove them from p_ids 2018-07-24 13:40:49 +01:00
Matthew Hodgson cd241d6bda incorporate more review 2018-07-24 12:39:40 +01:00
Matthew Hodgson 254fb430d1 incorporate review 2018-07-23 19:21:20 +01:00
Matthew Hodgson bcaec2915a incorporate review 2018-07-19 19:03:50 +01:00
Matthew Hodgson 924eb34d94 add a filtered_types param to limit filtering to specific types 2018-07-19 18:32:02 +01:00
Matthew Hodgson be3adfc331 merge develop pydoc for _get_state_for_groups 2018-07-19 11:26:04 +01:00
Amber Brown 49af402019 run isort 2018-07-09 16:09:20 +10:00
Matthew Hodgson 6eb861b67f typo 2018-06-25 17:37:16 +01:00
Matthew Hodgson 28f09fcdd5 Merge branch 'develop' into matthew/filter_members 2018-06-04 00:09:17 +03:00
Matthew Hodgson 5f6122fe10 more comments 2018-06-04 00:08:52 +03:00
Amber Brown c936a52a9e
Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307) 2018-05-31 19:03:47 +10:00
Richard van der Hoff 34b85df7f5 Update some comments and docstrings in SyncHandler 2018-05-29 22:31:18 +01:00
Matthew Hodgson a6c8f7c875 add pydoc 2018-05-29 01:09:55 +01:00
Matthew Hodgson 7a6df013cc merge develop 2018-05-29 00:25:22 +01:00
Adrian Tschira 933bf2dd35 replace some iteritems with six
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:59:26 +02:00
Erik Johnston d26bec8a43 Add comment to sync as to why code path is split 2018-05-09 15:56:07 +01:00
Erik Johnston e2accd7f1d Refactor sync APIs to reuse pagination API
The sync API often returns events in a topological rather than stream
ordering, e.g. when the user joined the room or on initial sync. When
this happens we can reuse existing pagination storage functions.
2018-05-09 13:43:39 +01:00
Erik Johnston c4af4c24ca Refactor get_recent_events_for_room return type
There is no reason to return a tuple of tokens when the last token is
always the token passed as an argument. Changing it makes it consistent
with other storage APIs
2018-05-09 11:55:34 +01:00
Richard van der Hoff 8dc4a6144b
Merge pull request #3107 from NotAFile/py3-bool-nonzero
add __bool__ alias to __nonzero__ methods
2018-04-20 15:43:39 +01:00
Adrian Tschira f63ff73c7f add __bool__ alias to __nonzero__ methods
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 20:40:47 +02:00
Richard van der Hoff b78395b7fe Refactor ResponseCache usage
Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a
(get, set) pair, and then use it throughout the codebase.

This will be largely non-functional, but does include the following functional
changes:

* federation_server.on_context_state_request: drops use of _server_linearizer
  which looked redundant and could cause incorrect cache misses by yielding
  between the get and the set.
* RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks
* the wrap function includes some logging. I'm hoping this won't be too noisy
  on production.
2018-04-12 13:02:15 +01:00
Richard van der Hoff b3384232a0 Add metrics for ResponseCache 2018-04-10 23:14:47 +01:00
Matthew Hodgson b2f2282947 make lazy_load_members configurable in filters 2018-03-19 01:15:13 +00:00
Matthew Hodgson 478af0f720 reshuffle todo & comments 2018-03-19 01:00:12 +00:00
Matthew Hodgson 366f730bf6 only get member state IDs for incremental syncs if we're filtering 2018-03-18 21:40:43 +00:00
Matthew Hodgson fc5397fdf5 remove debug 2018-03-16 01:44:55 +00:00
Matthew Hodgson 4f0493c850 fix tsm search again 2018-03-16 01:43:37 +00:00
Matthew Hodgson f7dcc404f2 add state_ids for timeline entries 2018-03-16 01:37:53 +00:00
Matthew Hodgson 5b3b3aada8 simplify timeline_start_members 2018-03-16 01:17:34 +00:00
Matthew Hodgson 3bc5bd2d22 make incr syncs work 2018-03-16 00:52:04 +00:00
Matthew Hodgson 4d0cfef6ee add copyright to nudge CI 2018-03-14 00:02:20 +00:00
Matthew Hodgson c9d72e4571 oops 2018-03-13 23:46:45 +00:00
Matthew Hodgson ccca02846d make it work 2018-03-13 22:31:41 +00:00
Matthew Hodgson 14a9d2f73d ensure we always include the members for a given timeline block 2018-03-13 22:03:42 +00:00
Matthew Hodgson 8713365265 typos 2018-03-11 20:10:25 +00:00
Matthew Hodgson 9b334b3f97 WIP experiment in lazyloading room members 2018-03-11 20:01:41 +00:00
Erik Johnston a56d54dcb7 Fix up log message 2018-03-07 11:55:31 +00:00
Erik Johnston 02a1296ad6 Fix typo 2018-03-07 11:55:31 +00:00
Erik Johnston 8cb44da4aa Fix race in sync when joining room
The race happens when the user joins a room at the same time as doing a
sync. We fetch the current token and then get the rooms the user is in.
If the join happens after the current token, but before we get the rooms
we end up sending down a partial room entry in the sync.

This is fixed by looking at the stream ordering of the membership
returned by get_rooms_for_user, and handling the case when that stream
ordering is after the current token.
2018-03-07 11:55:31 +00:00
Erik Johnston d4ffe61d4f Remove ability for AS users to call /events and /sync
This functionality has been deprecated for a while as well as being
broken for a while. Instead of fixing it lets just remove it entirely.

See: https://github.com/matrix-org/matrix-doc/issues/1144
2018-03-05 15:44:46 +00:00
Erik Johnston 2a7e9faeec Do logcontexts outside ResponseCache 2017-10-25 15:21:08 +01:00
David Baker 27955056e0 Merge branch 'develop' into erikj/groups_merged 2017-10-02 16:20:41 +01:00
Erik Johnston e3edca3b5d Refactor to speed up incremental syncs 2017-09-25 17:35:39 +01:00
Erik Johnston 3166ed55b2 Fix device list when rejoining room (#2461) 2017-09-22 14:44:17 +01:00
Erik Johnston 93e504d04e Ensure that creator of group sees group down /sync 2017-09-19 11:08:16 +01:00
Erik Johnston b5feaa5a49 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/groups_merged 2017-09-19 11:07:45 +01:00
Erik Johnston 2d1b7955ae Don't filter out current state events from timeline 2017-09-18 17:13:03 +01:00
Erik Johnston 3a0cee28d6 Actually hook leave notifs up 2017-09-14 11:49:37 +01:00
Erik Johnston 473700f016 Get left rooms 2017-09-13 15:13:41 +01:00
Erik Johnston 9ce866ed4f In sync handle device lists for newly joined/left rooms 2017-09-12 16:44:26 +01:00
Erik Johnston 69ef4987a6 Add left section to /keys/changes 2017-09-08 14:44:36 +01:00
Erik Johnston 53cc8ad35a Send down device list change notif when member leaves/rejoins room 2017-09-07 15:08:39 +01:00
Erik Johnston d5e32c843f Correctly add joins to correct segment 2017-07-24 13:31:26 +01:00
Erik Johnston 139fe30f47 Remember to cast to bool 2017-07-20 16:47:35 +01:00
Erik Johnston c544188ee3 Add groups to sync stream 2017-07-20 16:36:42 +01:00
Erik Johnston e9aec001f4 Use less DB for device list handling in sync 2017-07-12 10:30:10 +01:00
Erik Johnston b4f59c7e27 Add count of one time keys to sync stream 2017-05-19 15:47:55 +01:00
Erik Johnston da146657c9 Comments 2017-03-16 13:04:07 +00:00
Erik Johnston a158c36a8a Comment 2017-03-16 11:57:45 +00:00
Erik Johnston 6957bfdca6 Don't recreate so many sets 2017-03-16 11:54:26 +00:00
Erik Johnston 2ccf3b241c Implement no op for room stream in sync 2017-03-16 11:06:41 +00:00
Erik Johnston 6c82de5100 Format presence events on the edges instead of reformatting them multiple times 2017-03-15 14:27:34 +00:00
Erik Johnston 64d62e41b8 Noop repated delete device inbox calls from /sync 2017-03-10 10:36:43 +00:00
Erik Johnston a597994fb6 Measure new device list stuff 2017-02-02 18:36:17 +00:00
Erik Johnston 6d6591880e Wake sync up for device changes 2017-02-01 15:15:16 +00:00
Erik Johnston 252b503fc8 Hook device list updates to replication 2017-01-27 14:31:35 +00:00
Erik Johnston 2367c5568c Add basic implementation of local device list changes 2017-01-25 14:27:27 +00:00
Richard van der Hoff edc6a1e4f9 Add some logging for syncing to_device events
Attempt to track down the loss of to_device events
(https://github.com/vector-im/riot-web/issues/2711 etc).
2016-12-15 18:16:10 +00:00
Erik Johnston be14c24cea Fix rare notifier bug where listeners dont timeout
There was a race condition that caused the notifier to 'miss' the
timeout notification, since there were no other checks for the timeout
this caused listeners to get stuck in a loop until something happened.
2016-12-09 15:43:18 +00:00
Kegan Dougal 83bcdcee61 Return early on /sync code paths if a '*' filter is used
This is currently very conservative in that it only does this if there is no
`since` token. This limits the risk to clients likely to be doing one-off
syncs (like bridges), but does mean that normal human clients won't benefit
from the time savings here. If the savings are large enough, I would consider
generalising this to just check the filter.
2016-11-22 16:38:35 +00:00
Patrik Oldsberg 9bfc617791 storage/appservice: make appservice methods only relying on the cache synchronous 2016-10-06 15:24:59 +02:00
Mark Haines 65fa37ac5e Only return new device messages in /sync 2016-09-02 15:50:37 +01:00
Erik Johnston c8cd87b21b Comment about message deletion 2016-08-30 11:23:26 +01:00
Erik Johnston 1ee6285905 Fix check 2016-08-30 11:17:46 +01:00
Erik Johnston 68e1a872fd Noop get_new_messages_for_device if token hasn't changed 2016-08-30 10:58:46 +01:00
Erik Johnston 55fc17cf4b Merge pull request #1049 from matrix-org/erikj/presence_users_in_room
Use state handler instead of get_users_in_room/get_joined_hosts
2016-08-30 10:50:37 +01:00
Erik Johnston bed10f9880 Use state handler instead of get_users_in_room/get_joined_hosts 2016-08-26 14:54:30 +01:00
Mark Haines 4bbef62124 Merge remote-tracking branch 'origin/develop' into markjh/direct_to_device 2016-08-26 14:35:31 +01:00
Erik Johnston 778fa85f47 Make sync not pull out full state 2016-08-25 18:59:44 +01:00
Mark Haines e993925279 Add store-and-forward direct-to-device messaging 2016-08-25 17:35:37 +01:00
Erik Johnston afbf6b33fc defer.returnValue must not be called within Measure 2016-08-19 18:23:44 +01:00
Erik Johnston 248e6770ca Cache federation state responses 2016-07-21 10:30:12 +01:00
Erik Johnston 6a0afa582a Load push rules in storage layer, so that they get cached 2016-06-03 11:10:00 +01:00
Mark Haines 661a540dd1 Deduplicate presence entries in sync (#818) 2016-06-02 15:20:28 +01:00
Erik Johnston faad233ea6 Change short circuit path 2016-05-24 14:27:19 +01:00
Erik Johnston 6900303997 Don't send down all ephemeral events 2016-05-24 11:44:55 +01:00
Erik Johnston 1c5ed2a19b Only work out newly_joined_users for incremental sync 2016-05-24 11:21:34 +01:00
Erik Johnston b08ad0389e Only include non-offline presence in initial sync 2016-05-24 11:15:05 +01:00
Erik Johnston be2c677386 Spell builder correctly 2016-05-24 10:53:03 +01:00
Erik Johnston 79bea8ab9a Inline function. Make load_filtered_recents private 2016-05-24 10:22:24 +01:00
Erik Johnston 84f94e4cbb Add comments 2016-05-24 10:14:53 +01:00
Erik Johnston 137e6a4557 Shuffle things room 2016-05-24 09:50:55 +01:00
Erik Johnston c0c79ef444 Add back concurrently_execute 2016-05-23 18:21:27 +01:00
Erik Johnston b5605dfecc Refactor SyncHandler 2016-05-23 18:08:18 +01:00
Mark Haines f68eea808a Move SyncHandler out of the Handlers object 2016-05-16 20:19:26 +01:00
Mark Haines 53e171f345 Merge branch 'markjh/liberate_presence_handler' into markjh/liberate_sync_handler 2016-05-16 20:08:32 +01:00
Mark Haines 80cb9becd8 Remove get_joined_rooms_for_user from RoomMemberHandler 2016-05-16 20:06:55 +01:00