Commit Graph

1941 Commits

Author SHA1 Message Date
Erik Johnston ac5491f563 Select distinct devices from DB
Otherwise we might pull out tonnes of duplicate user_ids and this can
make synapse sad.
2017-03-06 11:10:14 +00:00
Erik Johnston da52d3af31 Fix up 2017-03-03 15:29:13 +00:00
Erik Johnston d766343668 Add index to device_lists_stream 2017-03-01 15:56:30 +00:00
Erik Johnston 3365117151 Clobber old device list stream entries 2017-03-01 10:21:30 +00:00
Erik Johnston d638a7484b Merge pull request #1959 from matrix-org/erikj/intern_once
Intern table column names once
2017-02-28 15:15:16 +00:00
Erik Johnston b84907bdbb Intern table column names once 2017-02-28 14:38:16 +00:00
Erik Johnston e4919b9329 Add stream_id index to device_lists_outbound_pokes
As this is used for replication streaming
2017-02-28 11:19:06 +00:00
Erik Johnston 8a12b6f1eb Fix up txn name 2017-02-28 10:15:50 +00:00
Erik Johnston a41dce8f8a Remove needless check 2017-02-27 18:54:43 +00:00
Erik Johnston c0d6045776 It should be all 2017-02-27 18:45:24 +00:00
Erik Johnston 49f4bc4709 Don't fetch current state in common case
Currently we fetch the list of current state events whenever we send
something in a room. This is overkill for the common case of persisting
a simple chain of non-state events, so lets handle that case specially.
2017-02-27 18:33:41 +00:00
Erik Johnston fc5d876dba Merge pull request #1954 from matrix-org/erikj/cache_device2
Cache get_user_devices_from_cache
2017-02-27 16:36:40 +00:00
Erik Johnston f58dbb02a6 Cache get_user_devices_from_cache 2017-02-27 16:22:12 +00:00
Erik Johnston 1a4f8022e6 Strip newlines from SQL queries 2017-02-23 11:15:31 +00:00
Erik Johnston b2d20e94fa Remove lock from rotate notifs 2017-02-22 14:24:02 +00:00
Erik Johnston b7442c3e2b Store looping call 2017-02-21 13:59:25 +00:00
Erik Johnston 699be7d1be Fix up notif rotation 2017-02-18 14:42:39 +00:00
David Baker 4aa29508af Use TEXT rather than VARCHAR
While we're changing anyway
2017-02-17 10:51:49 +00:00
David Baker b4017539d4 Make the pushers lang field column longer
To accommodate things like zh-Hans-CN

Fixes https://github.com/vector-im/riot-ios/issues/1031
2017-02-17 10:42:57 +00:00
Erik Johnston 138e030cfe Comment 2017-02-16 15:03:36 +00:00
Erik Johnston 502ae6c663 Comment 2017-02-16 14:47:11 +00:00
Erik Johnston e6acf0c399 Store the default push actions in a more efficient manner 2017-02-16 14:40:24 +00:00
Erik Johnston 04eca2589d Merge pull request #1916 from matrix-org/erikj/push_actions_delete
Aggregate event push actions
2017-02-16 15:28:58 +01:00
Richard van der Hoff 7dcbcca68c Merge pull request #1921 from matrix-org/rav/fix_key_changes
Fix bugs in the /keys/changes api
2017-02-15 11:25:16 +00:00
Richard van der Hoff fc2f29c1d0 Fix bugs in the /keys/changes api
* `get_forward_extremeties_for_room` takes a numeric `stream_ordering`. We were
  passing a `RoomStreamToken`, which meant that it returned the *current*
  extremities, rather than those corresponding to the `from_token`. However:
* `get_state_ids_for_events` required a second ('types') parameter; this meant
  that a `TypeError` was thrown and we ended up acting as though there was *no*
  prev state.
* `get_state_ids_for_events` actually returns a map from event_id to state
  dictionary - just looking up the state keys in it again meant that we acted
  as though there was no prev state. We now check if each member's state has
  changed since *any* of the extremities.

Also add/fix some comments.
2017-02-14 13:59:50 +00:00
Erik Johnston ce3c8df6df Less aggressive timers 2017-02-14 13:41:24 +00:00
Erik Johnston 095b45c165 Aggregate event push actions 2017-02-14 13:39:41 +00:00
Erik Johnston 795f8e3fe7 Merge pull request #1873 from matrix-org/erikj/delete_push_actions
Be more agressive about purging old room event_push_actions
2017-02-14 14:29:04 +01:00
Erik Johnston 9e617cd4c2 Cache get_presence storage 2017-02-13 13:50:03 +00:00
Erik Johnston 52cd019a54 Make None check explicit 2017-02-08 16:04:29 +00:00
Erik Johnston f20cd34858 Merge pull request #1892 from matrix-org/erikj/rejection_fwd_extrem
Ignore new rejected events when working out forward extremities.
2017-02-08 16:59:06 +01:00
Erik Johnston 7723b4caa4 Ignore new rejected events when working out forward extremeties. 2017-02-08 14:48:06 +00:00
Erik Johnston af6da6db2d Merge pull request #1784 from morteza-araby/user-admin
Administration functionalities
2017-02-06 16:21:10 +01:00
Erik Johnston 38258a0976 Bump cache sizes for common membership queries 2017-02-02 18:45:55 +00:00
Erik Johnston 832e9c52ca Comment 2017-02-02 13:09:56 +00:00
Erik Johnston 54a79c1d37 Make presence.get_new_events a bit faster
We do this by caching the set of users a user shares rooms with.
2017-02-02 13:07:18 +00:00
Morteza Araby 2849d3f29d admin,storage: added more administrator functionalities
administrators can now:
 - Set displayname of users
 - Update user avatars
 - Search for users by user_id
 - Browse all users in a paginated API
 - Reset user passwords
 - Deactivate users

Helpers for doing paginated queries has also been added to storage

Signed-off-by: Morteza Araby <morteza.araby@ericsson.com>
2017-02-02 14:02:26 +01:00
Erik Johnston fac3c03087 Be more agressive about purging old room event_push_actions 2017-02-01 18:27:24 +00:00
Erik Johnston fbfe44bb4d Doc args 2017-02-01 17:52:57 +00:00
Erik Johnston d61a04583e Comment 2017-02-01 17:35:23 +00:00
Erik Johnston 7e919bdbd0 Include newly joined users in /keys/changes API 2017-02-01 17:33:16 +00:00
Erik Johnston ebfaff84c9 Merge pull request #1870 from matrix-org/erikj/cache_get_all_new_events
Add a small cache get_all_new_events
2017-02-01 13:22:02 +00:00
Erik Johnston 62f6b86ba7 Merge pull request #1868 from matrix-org/erikj/replication_cache
Only invalidate membership caches based on the cache stream
2017-02-01 13:12:30 +00:00
Erik Johnston f6124311fd Add m.room.member type to query 2017-02-01 11:59:17 +00:00
Erik Johnston 88a4d54883 Merge pull request #1867 from matrix-org/erikj/member_index
Add an index to make membership queries faster
2017-02-01 11:44:27 +00:00
Erik Johnston 368c88c487 Add a small cache get_all_new_events 2017-02-01 10:50:44 +00:00
Erik Johnston 5deaf9e30b Up get_latest_event_ids_in_room cache 2017-02-01 10:39:41 +00:00
Erik Johnston 458b6f4733 Only invalidate membership caches based on the cache stream
Before we completely invalidated get_users_in_room whenever we updated
any current_state_events table. This was way too aggressive.
2017-01-31 16:09:03 +00:00
Erik Johnston 21b7375778 Add an index to make membership queries faster 2017-01-31 15:15:57 +00:00
Erik Johnston ac001dabdc Merge pull request #1864 from matrix-org/erikj/device_list_fixes
Fix clearing out old device list outbound pokes
2017-01-31 13:35:35 +00:00
Erik Johnston bfb3d255b1 Merge pull request #1862 from matrix-org/erikj/presence_update
Use DB cache of joined users for presence
2017-01-31 13:23:24 +00:00
Erik Johnston ab55794b6f Fix deletion of old sent devices correctly 2017-01-31 13:22:41 +00:00
Erik Johnston d3169e8d28 Only fetch with row ts and count > 1 2017-01-31 11:20:03 +00:00
Erik Johnston 05b9f48ee5 Fix clearing out old device list outbound pokes 2017-01-31 10:08:55 +00:00
Erik Johnston 4b3403ca9b Stream cache invalidations for room membership storage functions 2017-01-30 17:28:22 +00:00
Erik Johnston 1c13c9f6b6 Don't have such a large cache 2017-01-30 17:12:14 +00:00
Erik Johnston c7a26b7c32 Fix unit tests 2017-01-30 17:11:24 +00:00
Erik Johnston c2c9a78db9 Noop device key changes if they're the same 2017-01-30 16:55:04 +00:00
Erik Johnston e75a779d9e Fix query 2017-01-30 16:38:20 +00:00
Erik Johnston 828db669ec Use get_users_in_room and declare it iterable 2017-01-30 16:37:22 +00:00
Erik Johnston 3670025e64 Rename func 2017-01-30 14:11:31 +00:00
Erik Johnston d360c97ae1 Clear out old destination pokes. 2017-01-30 10:14:37 +00:00
Erik Johnston 76100203ab Always use the latest stream_id, sent or unsent 2017-01-30 10:14:25 +00:00
Erik Johnston d1e1fd6210 Add ts column to device_lists_outbound_pokes 2017-01-27 15:23:48 +00:00
Erik Johnston 252b503fc8 Hook device list updates to replication 2017-01-27 14:31:35 +00:00
Erik Johnston 84a35f32c7 Comment 2017-01-27 10:35:12 +00:00
Erik Johnston c517a19c2d Comment 2017-01-27 10:33:26 +00:00
Erik Johnston 738a2867c8 SQL param ordering 2017-01-27 10:31:29 +00:00
Erik Johnston 755adff0e4 User if rather than for 2017-01-27 10:31:06 +00:00
Erik Johnston 888c59c955 Better name 2017-01-27 10:29:47 +00:00
Erik Johnston f25a4a4692 Remove unused param 2017-01-27 10:27:39 +00:00
Erik Johnston 31aca5589c Fix on sqlite: use left rather than outer join 2017-01-26 16:55:50 +00:00
Erik Johnston 76d40f4904 Handle users leaving rooms 2017-01-26 16:39:33 +00:00
Erik Johnston fbfad76c03 Add comments 2017-01-26 16:33:21 +00:00
Erik Johnston c974116f19 Implement device key caching over federation 2017-01-26 16:07:24 +00:00
Erik Johnston 51e9fe36e4 Fix up sending of m.device_list_update edus 2017-01-25 16:55:21 +00:00
Erik Johnston 2367c5568c Add basic implementation of local device list changes 2017-01-25 14:27:27 +00:00
Erik Johnston fdf2a31a51 Typo 2017-01-23 16:14:14 +00:00
Erik Johnston c77b24c092 Refactor to calculate state delta outside transaction 2017-01-23 14:51:33 +00:00
Erik Johnston 5d2134d485 Comments 2017-01-20 17:13:24 +00:00
Erik Johnston a55fa2047f Insert delta of current_state_events to be more efficient 2017-01-20 17:10:18 +00:00
Erik Johnston d0897dead5 Spelling 2017-01-20 15:05:11 +00:00
Erik Johnston 567aa35b67 Update all call sites after rename 2017-01-20 14:40:31 +00:00
Erik Johnston f2f40e64a9 Comments 2017-01-20 14:38:13 +00:00
Erik Johnston 4c6a31cd6e Calculate the forward extremeties once 2017-01-20 14:28:53 +00:00
Erik Johnston 09eb08f910 Derive current_state_events from state groups 2017-01-20 11:52:51 +00:00
Erik Johnston 4ec1cf49e2 Lower loading events log to DEBUG 2017-01-17 17:28:32 +00:00
Erik Johnston d906206049 Increase state_group_cache_size 2017-01-17 11:31:08 +00:00
Erik Johnston 897f8752da Up cache max entries for state 2017-01-16 15:08:17 +00:00
Erik Johnston 01521299c7 Increase cache size limit 2017-01-16 11:56:51 +00:00
Erik Johnston 2fae34bd2c Optionally measure size of cache by sum of length of values 2017-01-13 17:46:17 +00:00
Erik Johnston f0325a9ccc Merge pull request #1793 from matrix-org/erikj/change_device_inbox_index
Change device_inbox stream index to include user
2017-01-13 15:14:51 +00:00
Erik Johnston c050f493dd Add comment 2017-01-13 15:14:41 +00:00
Erik Johnston ab655dca33 Explicitly close the cursor 2017-01-10 15:15:25 +00:00
Erik Johnston 5a32e9273e Don't disable autocommit 2017-01-10 15:11:27 +00:00
Erik Johnston caddadfc5a Change device_inbox stream index to include user
This makes fetching the nost recently changed users much tricker, and
brings it in line with e.g. presence_stream indices.
2017-01-10 15:04:57 +00:00
Erik Johnston dd52d4de4c Limit number of entries to prefill from cache
Some tables, like device_inbox, take a long time to query at startup for
the stream change cache prefills. This is likely because they are slower
growing streams and so are more fragmented on disk. For now, lets pull
fewer entries out to make startup quicker.

In future, we should add a better index to make it even faster.
2017-01-10 14:34:50 +00:00
Matthew Hodgson bd85434cb3 Merge branch 'release-v0.18.6' into develop 2017-01-05 13:58:19 +00:00
Matthew Hodgson 8cfc0165e9 fix annoying typos 2017-01-05 13:39:43 +00:00
Matthew Hodgson b31ed22738 Merge branch 'release-v0.18.6' into develop 2017-01-05 13:03:02 +00:00