Commit Graph

210 Commits

Author SHA1 Message Date
Erik Johnston cb53ce9d64
Refactor state group lookup to reduce DB hits (#4011)
Currently when fetching state groups from the data store we make two
hits two the database: once for members and once for non-members (unless
request is filtered to one or the other). This adds needless load to the
datbase, so this PR refactors the lookup to make only a single database
hit.
2018-10-25 17:49:55 +01:00
Richard van der Hoff ae6ad4cf41
docstrings and unittests for storage.state (#3958)
I spent ages trying to figure out how I was going mad...
2018-09-27 11:22:25 +01:00
Matthew Hodgson bb81e78ec6
Split the state_group_cache in two (#3726)
Splits the state_group_cache in two.

One half contains normal state events; the other contains member events.

The idea is that the lazyloading common case of: "I want a subset of member events plus all of the other state" can be accomplished efficiently by splitting the cache into two, and asking for "all events" from the non-members cache, and "just these keys" from the members cache.  This means we can avoid having to make DictionaryCache aware of these sort of complicated queries, whilst letting LL requests benefit from the caching.

Previously we were unable to sensibly use the caching and had to pull all state from the DB irrespective of the filtering, which made things slow.  Hopefully fixes https://github.com/matrix-org/synapse/issues/3720.
2018-08-22 00:56:37 +02:00
Matthew Hodgson 3f543dc021 initial cut at a room summary API (#3574) 2018-08-16 09:46:50 +01:00
Matthew Hodgson 2f78f432c4 speed up /members and add at= and membership params (#3568) 2018-08-15 16:35:22 +01:00
Richard van der Hoff 0ca459ea33 Basic support for room versioning
This is the first tranche of support for room versioning. It includes:
 * setting the default room version in the config file
 * new room_version param on the createRoom API
 * storing the version of newly-created rooms in the m.room.create event
 * fishing the version of existing rooms out of the m.room.create event
2018-08-03 16:08:32 +01:00
Matthew Hodgson bc7944e6d2 switch missing_types to be a bool 2018-07-25 23:36:31 +01:00
Matthew Hodgson 7d9fb88617 incorporate more review. 2018-07-25 16:33:50 +01:00
Matthew Hodgson cb5c37a57c handle the edge case for _get_some_state_from_cache where types is [] 2018-07-24 20:34:45 +01:00
Matthew Hodgson e22700c3dd consider non-filter_type types as wildcards, thus missing from the state-group-cache 2018-07-24 13:59:07 +01:00
Matthew Hodgson cd241d6bda incorporate more review 2018-07-24 12:39:40 +01:00
Matthew Hodgson efcdacad7d handle case where types is [] on postgres correctly 2018-07-23 22:41:05 +01:00
Matthew Hodgson 254fb430d1 incorporate review 2018-07-23 19:21:20 +01:00
Matthew Hodgson 2f558300cc fix thinkos; unbreak tests 2018-07-19 19:22:27 +01:00
Matthew Hodgson bcaec2915a incorporate review 2018-07-19 19:03:50 +01:00
Matthew Hodgson 924eb34d94 add a filtered_types param to limit filtering to specific types 2018-07-19 18:32:02 +01:00
Matthew Hodgson be3adfc331 merge develop pydoc for _get_state_for_groups 2018-07-19 11:26:04 +01:00
Amber Brown 49af402019 run isort 2018-07-09 16:09:20 +10:00
Richard van der Hoff 43e02c409d Disable partial state group caching for wildcard lookups
When _get_state_for_groups is given a wildcard filter, just do a complete
lookup. Hopefully this will give us the best of both worlds by not filling up
the ram if we only need one or two keys, but also making the cache still work
for the federation reader usecase.
2018-06-22 11:52:07 +01:00
Richard van der Hoff 240f192523
Merge pull request #3382 from matrix-org/rav/optimise_state_groups
Optimise state_group_cache update
2018-06-22 11:20:20 +01:00
Richard van der Hoff eb32b2ca20 Optimise state_group_cache update
(1) matrix-org-hotfixes has removed the intern calls; let's do the same here.
(2) remove redundant iteritems() so we can used an optimised db update.
2018-06-11 22:56:11 +01:00
Matthew Hodgson c96d882a02 Merge branch 'develop' into matthew/filter_members 2018-06-10 12:26:14 +03:00
Richard van der Hoff d3a8c9c55e Fix sql error in _get_state_groups_from_groups
If this was called with a `(type, None)` entry in types (which is supposed to
return all state of type `type`), it would explode with a sql error.
2018-06-06 14:19:01 +01:00
Erik Johnston 042eedfa2b Add hacky cache factor override system 2018-06-04 15:39:28 +01:00
Matthew Hodgson 28f09fcdd5 Merge branch 'develop' into matthew/filter_members 2018-06-04 00:09:17 +03:00
Amber Brown c936a52a9e
Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307) 2018-05-31 19:03:47 +10:00
Matthew Hodgson a6c8f7c875 add pydoc 2018-05-29 01:09:55 +01:00
Matthew Hodgson 9f77001e27 pep8 2018-03-14 00:07:47 +00:00
Matthew Hodgson d144ed6ffb
fix bug #2926 (loading all state for a given type from the DB if the state_key is None) (#2990)
Fixes a regression that had crept in where the caching layer upholds requests for loading state which is filtered by type (but not by state_key), but the DB layer itself would interpret a missing state_key as a request to filter by null state_key rather than returning all state_keys.
2018-03-13 22:36:04 +00:00
Matthew Hodgson ccca02846d make it work 2018-03-13 22:31:41 +00:00
Matthew Hodgson f0f9a0605b remove comment now #2969 is fixed 2018-03-13 22:12:15 +00:00
Matthew Hodgson 12350e3f9a merge proper fix to bug 2969 2018-03-13 22:11:58 +00:00
Matthew Hodgson afbf4d3dcc typoe 2018-03-13 19:48:04 +00:00
Matthew Hodgson 865377a70d disable optimisation for searching for state groups
when type filter includes wildcards on state_key
2018-03-13 19:46:04 +00:00
Matthew Hodgson b2aba9e430 build where_clause sanely 2018-03-13 18:13:44 +00:00
Matthew Hodgson 52f7e23c72 PR feedbackz 2018-03-13 18:07:55 +00:00
Matthew Hodgson 1b1c137771 fix bug #2926 2018-03-13 17:52:52 +00:00
Matthew Hodgson fdedcd1f4d correctly handle None state_keys
and fix include_other_types thinko
2018-03-12 01:39:06 +00:00
Matthew Hodgson 97c0496cfa fix sqlite where clause 2018-03-12 00:27:06 +00:00
Matthew Hodgson 8713365265 typos 2018-03-11 20:10:25 +00:00
Matthew Hodgson 9b334b3f97 WIP experiment in lazyloading room members 2018-03-11 20:01:41 +00:00
Erik Johnston 106906a65e Don't serialize current state over replication 2018-02-15 13:53:18 +00:00
Erik Johnston fd1601c596 Fix state group storage bug in workers
We needed to move `_count_state_group_hops_txn` to the
StateGroupWorkerStore.
2018-02-15 11:04:32 +00:00
Erik Johnston 3d33eef6fc
Store state groups separately from events (#2784)
* Split state group persist into seperate storage func

* Add per database engine code for state group id gen

* Move store_state_group to StateReadStore

This allows other workers to use it, and so resolve state.

* Hook up store_state_group

* Fix tests

* Rename _store_mult_state_groups_txn

* Rename StateGroupReadStore

* Remove redundant _have_persisted_state_group_txn

* Update comments

* Comment compute_event_context

* Set start val for state_group_id_seq

... otherwise we try to recreate old state groups

* Update comments

* Don't store state for outliers

* Update comment

* Update docstring as state groups are ints
2018-02-06 14:31:24 +00:00
Richard van der Hoff 35a4b63240 Pull out bits of StateStore to a mixin
... so that we don't need to secretly gut-wrench it for use in the slaved
stores. I haven't done the other stores yet, but we should. I'm tired of the
workers breaking every time we tweak the stores because I forgot to gut-wrench
the right method.

fixes https://github.com/matrix-org/synapse/issues/2655.
2017-11-14 11:43:58 +00:00
Richard van der Hoff 4dd1bfa8c1 Revert "Revert "move _state_group_cache to statestore""
We're going to fix this properly on this branch, so that the _state_group_cache
can end up in StateGroupReadStore.

This reverts commit ab335edb02.
2017-11-14 11:43:58 +00:00
Richard van der Hoff 6cfee09be9 Make __init__ consitstent across Store heirarchy
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
2017-11-13 10:46:07 +00:00
Erik Johnston ab335edb02 Revert "move _state_group_cache to statestore"
This reverts commit f5cf3638e9.
2017-11-13 10:05:33 +00:00
Richard van der Hoff f5cf3638e9 move _state_group_cache to statestore
this is internal to statestore, so let's keep it there.
2017-11-07 16:43:00 +00:00
Erik Johnston 5946aa0877 Prefill forward extrems and event to state groups 2017-06-29 15:38:48 +01:00