92e34615c5
Track where event stream processing have gotten up to
2018-04-11 12:13:40 +01:00
233699c42e
Merge pull request #2760 from Valodim/pypy
...
Synapse on PyPy
2018-04-11 11:20:01 +01:00
d54cfbb7a8
fix typo
2018-04-10 17:38:16 +01:00
d1e56cfcd1
Fix pep8 error on psycopg2cffi hack
2018-04-10 11:35:29 +02:00
89de934981
Use psycopg2cffi module instead of psycopg2 if running on pypy
...
The psycopg2 package isn't available for PyPy. This commit adds a check
if the runtime is PyPy, and if it is uses psycopg2cffi module in favor
of psycopg2. This is almost a drop-in replacement, except for one place
where an additional cast to string is required.
2018-04-10 11:29:52 +02:00
41e0611895
remove errant print
2018-04-09 18:44:20 +01:00
61b439c904
Fix msec to sec, again
2018-04-09 18:43:48 +01:00
87770300d5
Fix msec to sec
2018-04-09 18:38:59 +01:00
64bc2162ef
Fix psycopg2 interpolation
2018-04-09 17:50:36 +01:00
cd087a265d
Don't use redundant inlineCallbacks
2018-04-06 15:43:27 +01:00
87c864b698
join_rule -> join_policy
2018-04-06 15:43:27 +01:00
ae85c7804e
is_joinable -> join_rule
2018-04-06 15:43:27 +01:00
edb45aae38
pep8
2018-04-06 15:43:27 +01:00
b370fe61c0
Implement group join API
2018-04-06 15:43:27 +01:00
e089100c62
Merge pull request #3045 from matrix-org/dbkr/group_joinable
...
Add joinability for groups
2018-04-05 15:57:49 +01:00
68b0ee4e8d
Merge pull request #3041 from matrix-org/r30_stats
...
R30 stats
2018-04-05 15:37:37 +01:00
22284a6f65
Merge pull request #3060 from matrix-org/rav/kill_event_content
...
Remove uses of events.content
2018-04-05 15:02:17 +01:00
917380e89d
NON NULL -> NOT NULL
2018-04-05 14:32:12 +01:00
700e5e7198
Use DEFAULT join_policy of "invite" in db
2018-04-05 14:01:17 +01:00
b214a04ffc
Document set_group_join_policy
2018-04-05 13:29:16 +01:00
0e5f479fc0
Review comments
...
Use iteritems over item to loop over dict
formatting
2018-04-05 12:16:46 +01:00
eb8d8d6f57
Use join_policy API instead of joinable
...
The API is now under
/groups/$group_id/setting/m.join_policy
and expects a JSON blob of the shape
```json
{
"m.join_policy": {
"type": "invite"
}
}
```
where "invite" could alternatively be "open".
2018-04-03 16:16:40 +01:00
8da39ad98f
Merge pull request #3049 from matrix-org/rav/use_staticjson
...
Use static JSONEncoders
2018-04-03 15:18:32 +01:00
2fe3f848b9
Remove uses of events.content
2018-03-29 23:17:12 +01:00
05630758f2
Use static JSONEncoders
...
using json.dumps with custom options requires us to create a new JSONEncoder on
each call. It's more efficient to create one upfront and reuse it.
2018-03-29 23:13:33 +01:00
fcfe7f6ad3
Use simplejson throughout
...
Let's use simplejson rather than json, for consistency.
2018-03-29 22:45:52 +01:00
b4e37c6f50
pep8
2018-03-29 17:27:39 +01:00
9ee44a372d
Remove need for sqlite specific query
2018-03-29 16:45:34 +01:00
dc7c020b33
fix pep8 errors
2018-03-28 17:25:15 +01:00
241e4e8687
remove twisted deferral cruft
2018-03-28 16:25:53 +01:00
929b34963d
OK, smallint it is then
2018-03-28 14:53:55 +01:00
a838444a70
Grr. Copy the definition from is_admin
2018-03-28 14:50:30 +01:00
4262aba17b
bump schema version
2018-03-28 14:40:03 +01:00
86932be2cb
Support multi client R30 for psql
2018-03-28 14:36:53 +01:00
a164270833
Make column definition that works on both dbs
2018-03-28 14:23:00 +01:00
352e1ff9ed
Add schema delta file
2018-03-28 14:07:57 +01:00
79452edeee
Add joinability for groups
...
Adds API to set the 'joinable' flag, and corresponding flag in the
table.
2018-03-28 14:03:37 +01:00
788e69098c
Add user_ips last seen index
2018-03-28 12:03:13 +01:00
0f890f477e
No need to cast in count_daily_users
2018-03-28 11:49:57 +01:00
545001b9e4
Fix search_user_dir multiple sqlite versions do different things
2018-03-28 11:19:45 +01:00
a32d2548d9
query and call for r30 stats
2018-03-28 10:39:13 +01:00
9187e0762f
count_daily_users failed if db was sqlite due to type failure - presumably this prevcented all sqlite homeservers reporting home
2018-03-28 10:02:32 +01:00
f879127aaa
Merge pull request #3029 from matrix-org/erikj/linearize_generate_user_id
...
Linearize calls to _generate_user_id
2018-03-28 10:00:31 +01:00
e6d87c93f3
Merge pull request #3030 from matrix-org/erikj/no_ujson
...
Remove last usage of ujson
2018-03-28 10:00:06 +01:00
800cfd5774
Comment
2018-03-27 13:30:39 +01:00
152c2ac19e
Fix indent
2018-03-27 13:13:46 +01:00
e70287cff3
Comment
2018-03-27 13:13:38 +01:00
3f49e131d9
Add counter metrics for calculating state delta
...
This will allow us to measure how often we calculate state deltas in
event persistence that we would have been able to calculate at the same
time we calculated the state for the event.
2018-03-27 10:57:35 +01:00
fecb45e0c3
Remove last usage of ujson
2018-03-26 13:32:29 +01:00
8d6dc106d1
Don't use _cursor_to_dict in find_next_generated_user_id_localpart
2018-03-26 12:02:44 +01:00
fa72803490
Merge branch 'master' of github.com:matrix-org/synapse into develop
2018-03-19 11:41:01 +00:00
bf49d2dca8
Replace some ujson with simplejson to make it work
2018-03-16 00:55:44 +00:00
a8ce159be4
Replace some ujson with simplejson to make it work
2018-03-16 00:27:09 +00:00
5b631ff41a
Remove wrong comment
2018-03-16 00:07:08 +00:00
926ba76e23
Replace ujson with simplejson
2018-03-15 23:43:31 +00:00
056a6df546
Merge branch 'develop' into matthew/filter_members
2018-03-14 15:38:05 +00:00
ebd0127999
Merge pull request #2988 from matrix-org/erikj/split_profile_store
...
Split up ProfileStore
2018-03-14 09:41:06 +00:00
f51565e023
Merge pull request #2993 from matrix-org/erikj/is_blocked
...
Add is_blocked to worker store
2018-03-14 09:39:18 +00:00
9f77001e27
pep8
2018-03-14 00:07:47 +00:00
d144ed6ffb
fix bug #2926 (loading all state for a given type from the DB if the state_key is None) ( #2990 )
...
Fixes a regression that had crept in where the caching layer upholds requests for loading state which is filtered by type (but not by state_key), but the DB layer itself would interpret a missing state_key as a request to filter by null state_key rather than returning all state_keys.
2018-03-13 22:36:04 +00:00
ccca02846d
make it work
2018-03-13 22:31:41 +00:00
f0f9a0605b
remove comment now #2969 is fixed
2018-03-13 22:12:15 +00:00
12350e3f9a
merge proper fix to bug 2969
2018-03-13 22:11:58 +00:00
afbf4d3dcc
typoe
2018-03-13 19:48:04 +00:00
865377a70d
disable optimisation for searching for state groups
...
when type filter includes wildcards on state_key
2018-03-13 19:46:04 +00:00
a08726fc42
Add is_blocked to worker store
2018-03-13 18:28:44 +00:00
b2aba9e430
build where_clause sanely
2018-03-13 18:13:44 +00:00
52f7e23c72
PR feedbackz
2018-03-13 18:07:55 +00:00
1b1c137771
fix bug #2926
2018-03-13 17:52:52 +00:00
3518d0ea8f
Split up ProfileStore
2018-03-13 17:36:50 +00:00
889a2a853a
Add Measure block for persist_events
...
This seems like a useful thing to measure.
2018-03-13 10:01:42 +00:00
fdedcd1f4d
correctly handle None state_keys
...
and fix include_other_types thinko
2018-03-12 01:39:06 +00:00
97c0496cfa
fix sqlite where clause
2018-03-12 00:27:06 +00:00
8713365265
typos
2018-03-11 20:10:25 +00:00
9b334b3f97
WIP experiment in lazyloading room members
2018-03-11 20:01:41 +00:00
8cb44da4aa
Fix race in sync when joining room
...
The race happens when the user joins a room at the same time as doing a
sync. We fetch the current token and then get the rooms the user is in.
If the join happens after the current token, but before we get the rooms
we end up sending down a partial room entry in the sync.
This is fixed by looking at the stream ordering of the membership
returned by get_rooms_for_user, and handling the case when that stream
ordering is after the current token.
2018-03-07 11:55:31 +00:00
b2932107bb
Merge pull request #2946 from matrix-org/rav/timestamp_to_purge
...
Implement purge_history by timestamp
2018-03-06 11:20:23 +00:00
7aed50a038
Merge pull request #2948 from matrix-org/erikj/kill_as_sync
...
Remove ability for AS users to call /events and /sync
2018-03-06 11:10:09 +00:00
b6c4b851f1
Merge pull request #2947 from matrix-org/erikj/split_directory_store
...
Split Directory store
2018-03-05 18:17:32 +00:00
d4ffe61d4f
Remove ability for AS users to call /events and /sync
...
This functionality has been deprecated for a while as well as being
broken for a while. Instead of fixing it lets just remove it entirely.
See: https://github.com/matrix-org/matrix-doc/issues/1144
2018-03-05 15:44:46 +00:00
69ce365b79
Fix cache invalidation on deletion
2018-03-05 15:29:03 +00:00
2e223163ff
Split Directory store
2018-03-05 15:11:30 +00:00
f8bfcd7e0d
Provide a means to pass a timestamp to purge_history
2018-03-05 14:37:23 +00:00
d032785aa7
Merge pull request #2943 from matrix-org/rav/fix_find_first_stream_ordering_after_ts
...
Test and fix find_first_stream_ordering_after_ts
2018-03-05 12:26:14 +00:00
c818fcab11
Test and fix find_first_stream_ordering_after_ts
...
It seemed to suffer from a bunch of off-by-one errors.
2018-03-05 12:04:02 +00:00
06a14876e5
Add find_first_stream_ordering_after_ts
...
Expose this as a public function which can be called outside a txn
2018-03-05 11:53:39 +00:00
42174946f8
Merge pull request #2934 from matrix-org/erikj/cache_fix
...
Fix bug with delayed cache invalidation stream
2018-03-05 11:33:17 +00:00
efb79820b4
Fix bug with delayed cache invalidation stream
...
We poked the notifier before updated the current token for the cache
invalidation stream. This mean that sometimes the update wouldn't be
sent until the next time a cache was invalidated.
2018-03-02 14:45:15 +00:00
fafa3e7114
Split registration store
2018-03-02 13:48:27 +00:00
1773df0632
Merge pull request #2925 from matrix-org/erikj/split_sig_fed
...
Split out SignatureStore and EventFederationStore
2018-03-01 17:32:58 +00:00
9e08a93a7b
Merge pull request #2927 from matrix-org/erikj/read_marker_caches
...
Improve caching for read_marker API
2018-03-01 17:12:34 +00:00
4b44f05f19
Fewer lies are better
2018-03-01 17:08:17 +00:00
a83c514d1f
Improve caching for read_marker API
...
We add a new storage function to get a paritcular type of room account
data. This allows us to prefill the cache when updating that acount
data.
2018-03-01 17:08:17 +00:00
33bebb63f3
Add some caches to help read marker API
2018-03-01 17:08:17 +00:00
2ad4d5b5bb
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fed
2018-03-01 16:59:39 +00:00
7c371834cc
Stub out broken function only used for cache
2018-03-01 16:44:13 +00:00
64346be26d
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_stream_store
2018-03-01 16:26:42 +00:00
22518e2833
Merge pull request #2923 from matrix-org/erikj/stream_ago_worker
...
Calculate stream_ordering_month_ago correctly on workers
2018-03-01 16:23:54 +00:00
884b26ae41
Remove unused variables
2018-03-01 16:23:48 +00:00
1b2af11650
Document abstract class and method better
2018-03-01 16:20:57 +00:00
872ff95ed4
Default stream_ordering_*_ago to None
2018-03-01 16:00:05 +00:00
22004b524e
Fix comment typo
2018-03-01 15:59:40 +00:00
f793bc3877
Split out stream store
2018-03-01 15:13:08 +00:00
6411f725be
Calculate stream_ordering_month_ago correctly on workers
2018-03-01 14:20:53 +00:00
a9a2d66cdd
Split out SignatureStore and EventFederationStore
2018-03-01 14:17:53 +00:00
0c8ba5dd1c
Split up RoomStore
2018-03-01 14:01:19 +00:00
493e25d554
Move storage functions for push calculations
...
This will allow push actions for an event to be calculated on workers.
2018-02-27 13:58:16 +00:00
3594dbc6dc
Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidation
...
Fix missing invalidations for receipt storage
2018-02-27 11:34:26 +00:00
2311189ee4
Merge pull request #2903 from matrix-org/erikj/split_roommember_store
...
Split out RoomMemberStore
2018-02-27 11:32:10 +00:00
c57607874c
Merge pull request #2901 from matrix-org/erikj/split_as_stores
...
Split AS stores
2018-02-27 10:07:07 +00:00
8956f0147a
Add comment
2018-02-27 10:06:51 +00:00
e5b4a208ce
Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions
...
Batch inserts into event_push_actions_staging
2018-02-26 14:45:40 +00:00
45b5fe9122
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/handle_unpersisted_events_push
2018-02-26 13:49:24 +00:00
d62ce972f8
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_roommember_store
2018-02-23 11:46:24 +00:00
6ae9a3d2a6
Update copyright
2018-02-23 11:44:49 +00:00
a90c60912f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_event_push_actions
2018-02-23 11:26:31 +00:00
50e8657867
Merge pull request #2902 from matrix-org/erikj/split_events_store
...
Split out get_events and co into a worker store
2018-02-23 11:23:52 +00:00
1cf9e071dd
Merge pull request #2899 from matrix-org/erikj/split_pushers
...
Split PusherStore
2018-02-23 11:23:35 +00:00
d0957753bf
Merge pull request #2898 from matrix-org/erikj/split_push_rules_store
...
Split PushRulesStore
2018-02-23 11:23:23 +00:00
70349872c2
Update copyright
2018-02-23 11:14:35 +00:00
eba93b05bf
Split EventsWorkerStore into separate file
2018-02-23 11:01:21 +00:00
bf8a36e080
Update copyright
2018-02-23 10:52:10 +00:00
5d0f665848
Remove redundant clock
2018-02-23 10:49:58 +00:00
3bd760628b
_event_persist_queue shouldn't be in worker store
2018-02-23 10:49:18 +00:00
eb9b5eec81
Update copyright
2018-02-23 10:42:39 +00:00
c2ecfcc3a4
Update copyright
2018-02-23 10:41:34 +00:00
7e6cf89dc2
Update copyright
2018-02-23 10:39:19 +00:00
26d37f7a63
Update copyright
2018-02-23 10:33:55 +00:00
bb73f55fc6
Use absolute imports
2018-02-23 10:31:16 +00:00
faeb369f15
Fix missing invalidations for receipt storage
2018-02-21 15:19:54 +00:00
3dec9c66b3
Split out RoomMemberStore
2018-02-21 12:07:26 +00:00
46244b2759
Split AS stores
2018-02-21 11:49:34 +00:00
27b094f382
Split out get_events and co into a worker store
2018-02-21 11:41:48 +00:00
c96d547f4d
Actually use new param
2018-02-21 11:03:42 +00:00
d15d237b0d
Split out EventPushActionWorkerStore
2018-02-21 11:01:13 +00:00
6f72765371
Split PusherStore
2018-02-21 10:54:21 +00:00
cbaad969f9
Split PushRulesStore
2018-02-21 10:43:31 +00:00
ca9b9d9703
Split AccountDataStore and TagStore
2018-02-21 10:15:04 +00:00
8fbb4d0d19
Raise exception in abstract method
2018-02-20 17:59:23 +00:00
95e4cffd85
Fix comment
2018-02-20 17:58:40 +00:00
e316bbb4c0
Use abstract base class to access stream IDs
2018-02-20 17:43:57 +00:00
f5ac4dc2d4
Split ReceiptsStore
2018-02-20 16:28:28 +00:00
24087bffa9
Ensure all push actions are deleted from staging
2018-02-20 12:34:31 +00:00
ad0ccf15ea
Refactor _set_push_actions_for_event_and_users_txn to use events_and_contexts
2018-02-20 12:34:28 +00:00
6ff8c87484
Batch inserts into event_push_actions_staging
2018-02-20 11:33:07 +00:00
d1a3325f99
(Really) fix tablescan of event_push_actions on purge
...
commit 278d21b5
2018-02-16 14:02:31 +00:00
6af025d3c4
Fix typo of double is_highlight
2018-02-16 11:35:31 +00:00
012e8e142a
Comments
2018-02-16 11:35:01 +00:00
b96278d6fe
Ensure that we delete staging push actions on errors
2018-02-15 15:47:06 +00:00
4810f7effd
Remove context.push_actions
2018-02-15 15:47:06 +00:00
c714c61853
Update event_push_actions table from staging table
2018-02-15 15:47:06 +00:00
acac21248c
Store push actions in staging area
2018-02-15 15:47:04 +00:00
106906a65e
Don't serialize current state over replication
2018-02-15 13:53:18 +00:00
fd1601c596
Fix state group storage bug in workers
...
We needed to move `_count_state_group_hops_txn` to the
StateGroupWorkerStore.
2018-02-15 11:04:32 +00:00
b8d821aa68
Merge pull request #2867 from matrix-org/rav/rework_purge
...
purge_history cleanups
2018-02-15 09:49:07 +00:00
d28ec43e15
Merge pull request #2769 from matrix-org/matthew/hit_the_gin
...
switch back from GIST to GIN indexes
2018-02-14 16:59:03 +00:00
39bf47319f
purge_history: fix sqlite syntax error
...
apparently sqlite insists on indexes being named
2018-02-14 16:42:19 +00:00
ac27f6a35e
purge_history: handle sqlite asshattery
...
apparently creating a temporary table commits the transaction. because that's a
useful thing.
2018-02-14 16:41:12 +00:00
5978dccff0
remove overzealous exception handling
2018-02-14 15:54:09 +00:00
278d21b5e4
purge_history: fix index use
...
event_push_actions doesn't have an index on event_id, so we need to specify
room_id.
2018-02-14 15:44:51 +00:00
5fcbf1e07c
Rework event purge to use a temporary table
...
... which should speed things up by reducing the amount of data being shuffled
across the connection
2018-02-14 11:02:22 +00:00
c0c9327fe0
Merge pull request #2854 from matrix-org/erikj/event_create_worker
...
Create a worker for event creation
2018-02-13 18:07:10 +00:00
d627174da2
Fix log message in purge_history
...
(we don't just remove remote events)
2018-02-13 16:51:21 +00:00
ddb6a79b68
Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin
2018-02-13 16:45:36 +00:00
0b27ae8dc3
move search reindex to schema 47
...
We're up to schema v47 on develop now, so this will have to go in there to have
an effect.
This might cause an error if somebody has already run it in the v46 guise, and
runs it again in the v47 guise, because it will cause a duplicate entry in the
bbackground_updates table. On the other hand, the entry is removed once it is
complete, and it is unlikely that anyone other than matrix.org has run it on
v46. The update itself is harmless to re-run because it deliberately copes with
the index already existing.
2018-02-13 16:44:46 +00:00
4a6d551704
GIN reindex: Fix syntax errors, improve exception handling
2018-02-13 16:44:46 +00:00
bfdf7b9237
Merge pull request #2864 from matrix-org/rav/persist_event_caching
...
Use StateResolutionHandler to resolve state in persist_events
2018-02-13 14:45:57 +00:00
630caf8a70
style nit
2018-02-13 14:29:22 +00:00
a9b712e9dc
Merge branch 'develop' into matthew/gin_work_mem
2018-02-13 12:16:01 +00:00
10b34dbb9a
Merge pull request #2858 from matrix-org/rav/purge_updates
...
delete_local_events for purge_room_history
2018-02-09 14:11:00 +00:00
39a6b35496
purge: move room_depth update to end
...
... to avoid locking the table for too long
2018-02-09 13:07:41 +00:00
74fcbf741b
delete_local_events for purge_history
...
Add a flag which makes the purger delete local events
2018-02-09 13:07:41 +00:00
e571aef06d
purge: Move cache invalidation to more appropriate place
...
it was a bit of a non-sequitur there
2018-02-09 13:07:41 +00:00
61ffaa8137
bump purge logging to info
...
this thing takes ages and the only sign of any progress is the logs, so having
some logs is useful.
2018-02-09 13:07:41 +00:00
671540dccf
rename delete_old_state -> purge_history
...
(beacause it deletes more than state)
2018-02-09 13:07:41 +00:00
24dd73028a
Add replication http endpoint for event sending
2018-02-07 10:32:32 +00:00
3d33eef6fc
Store state groups separately from events ( #2784 )
...
* Split state group persist into seperate storage func
* Add per database engine code for state group id gen
* Move store_state_group to StateReadStore
This allows other workers to use it, and so resolve state.
* Hook up store_state_group
* Fix tests
* Rename _store_mult_state_groups_txn
* Rename StateGroupReadStore
* Remove redundant _have_persisted_state_group_txn
* Update comments
* Comment compute_event_context
* Set start val for state_group_id_seq
... otherwise we try to recreate old state groups
* Update comments
* Don't store state for outliers
* Update comment
* Update docstring as state groups are ints
2018-02-06 14:31:24 +00:00
b31bf0bb51
Merge pull request #2849 from matrix-org/rav/clean_up_state_delta
...
Remove redundant return value from _calculate_state_delta
2018-02-05 17:42:20 +01:00
ebfe64e3d6
Use StateResolutionHandler to resolve state in persist events
...
... and thus benefit (hopefully) from its cache.
2018-02-05 16:23:26 +00:00
225dc3b4cb
Flatten _get_new_state_after_events
...
rejig the if statements to simplify the logic and reduce indentation
2018-02-05 16:23:25 +00:00
9fcbbe8e7d
Check that events being persisted have state_group
2018-02-05 16:23:25 +00:00
447aed42d2
Add event_map param to resolve_state_groups
2018-02-05 16:23:25 +00:00
ee6fb4cf85
Remove redundant return value from _calculate_state_delta
...
we already have the state from _get_new_state_after_events, so returning it
from _calculate_state_delta is just confusing.
2018-02-05 16:23:20 +00:00
3c7b480ba3
Factor out common code for search insert
...
we can reuse the same code as is used for event insert, for doing the
background index population.
2018-02-05 16:12:14 +00:00
c46e75d3d8
Move store_event_search_txn to SearchStore
...
... as a precursor to making event storing and doing the bg update share some
code.
2018-02-05 15:43:22 +00:00
80b8a28100
Factor out common code for search insert
...
we can reuse the same code as is used for event insert, for doing the
background index population.
2018-02-04 00:23:06 +00:00
bd25f9cf36
Clean up work_mem handling
...
Add some comments and improve exception handling when twiddling work_mem for
the search update
2018-02-03 23:05:41 +00:00
4eeae7ad65
Move store_event_search_txn to SearchStore
...
... as a precursor to making event storing and doing the bg update share some
code.
2018-02-03 22:59:45 +00:00
bb9f0f3cdb
Merge branch 'develop' into matthew/gin_work_mem
2018-02-03 22:40:28 +00:00
6b02fc80d1
Reinstate event_search_postgres_gist handler
...
People may have queued updates for this, so we can't just delete it.
2018-02-02 14:32:51 +00:00
78d6ddba86
Merge pull request #2841 from matrix-org/rav/refactor_calc_state_delta
...
factor _get_new_state_after_events out of _calculate_state_delta
2018-02-02 15:27:15 +01:00
6e87b34f7b
Merge branch 'develop' into travis/admin-list-media
2018-02-01 18:05:47 -07:00
77c0629ebc
Merge pull request #2837 from matrix-org/rav/fix_quarantine_media
...
Fix sql error in quarantine_media
2018-02-01 11:45:58 +01:00
e16e45b1b4
pep8
...
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-31 15:30:38 -07:00
e1e4ec9f9d
factor _get_new_state_after_events out of _calculate_state_delta
...
This reduces the scope of a bunch of variables
2018-01-31 21:32:09 +00:00
63c4383927
Documentation and naming
...
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-31 08:07:52 -07:00
773f0eed1e
Fix sql error in quarantine_media
2018-01-30 15:02:51 +00:00
b387ee17b6
Improve exception handling in persist_event
...
1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that
a Failure object will be constructed using the current exception state,
*including* its stack trace - so the stack trace is saved in the Failure,
leading to better exception reports.
2. Set `consumeErrors=True` on the ObservableDeferred, because we know that
there will always be at least one observer - which avoids a spurious "CRITICAL:
unhandled exception in Deferred" error in the logs
2018-01-29 17:05:33 +00:00
46022025ea
Fix SQL for user search
...
fix some syntax errors for user search when search_all_users is enabled
fixes #2801 , hopefully
2018-01-27 17:21:57 +00:00
a94d9b6b82
Appease the linter
...
These are ids anyways, not mxc uris.
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-20 22:49:46 -07:00
5552ed9a7f
Add an admin route to get all the media in a room
...
This is intended to be used by administrators to monitor the media that is passing through their server, if they wish.
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-20 22:37:53 -07:00
36da256cc6
Merge pull request #2805 from matrix-org/rav/log_state_res
...
Log room when doing state resolution
2018-01-17 18:05:04 +00:00
1224612a79
Log room when doing state resolution
...
Mostly because it helps figure out what is prompting the resolution
2018-01-17 17:11:59 +00:00
a87006f9c7
Merge pull request #2783 from matrix-org/erikj/media_last_accessed
...
Keep track of last access time for local media
2018-01-17 16:39:02 +00:00
06db5c4b76
Merge pull request #2803 from matrix-org/matthew/fix-userdir-sql
...
fix SQL when searching all users
2018-01-17 16:27:54 +00:00
2d9ab533f9
fix SQL when searching all users
2018-01-17 15:58:52 +00:00
390093d45e
Split resolve_events into two functions
...
... so that the return type doesn't depend on the arg types
2018-01-17 15:44:31 +00:00
2fb3a28c98
Remove lost comment
2018-01-17 14:59:44 +00:00
05f98a2224
Keep track of last access time for local media
2018-01-17 10:24:43 +00:00
3d12d97415
Track DB scheduling delay per-request
...
For each request, track the amount of time spent waiting for a db
connection. This entails adding it to the LoggingContext and we may as well add
metrics for it while we are passing.
2018-01-16 17:23:32 +00:00
8615f19d20
rework runInteraction in terms of runConnection
...
... so that we can share the code
2018-01-16 17:08:29 +00:00
64ddec1bc0
Fix a logcontext leak in persist_events
...
ObserveableDeferred expects its callbacks to be called without any
logcontexts, whereas it turns out we were calling them with the logcontext of
the request which initiated the persistence loop.
It seems wrong that we are attributing work done in the persistence loop to the
request that happened to initiate it, so let's solve this by dropping the
logcontext for it.
(I'm not sure this actually causes any real problems other than messages in the
debug log, but let's clean it up anyway)
2018-01-16 11:47:36 +00:00
174eacc8ba
oops
2018-01-09 18:14:32 +00:00
a66f489678
fix GIST->GIN switch
2018-01-09 16:55:51 +00:00
e79db0a673
switch back from GIST to GIN indexes
2018-01-09 16:37:48 +00:00
e365ad329f
oops, tweak work_mem when actually storing
2018-01-09 16:30:30 +00:00
19f9227643
avoid 80s GIN inserts by tweaking work_mem
...
see https://github.com/matrix-org/synapse/issues/2753 for details
2018-01-09 16:25:04 +00:00
b30cd5b107
Remove dead code related to default thumbnails
2018-01-09 14:38:33 +00:00
cdc2cb5d11
fix StoreError syntax
2017-12-05 11:09:31 +00:00
74e0cc74ce
fix pep8 and tests
2017-12-04 15:11:38 +00:00
1bd40ca73e
switch to a simpler 'search_all_users' button as per review feedback
2017-12-04 14:58:39 +00:00
f397153dfc
Merge branch 'develop' into matthew/search-all-local-users
2017-11-30 01:51:38 +00:00
f61e107f63
remove null constraint on user_dir.room_id
2017-11-30 01:43:50 +00:00
4b1fceb913
fix alternation operator for FTS4 - how did this ever work!?
2017-11-30 01:34:03 +00:00
a4bb133b68
fix thinkos galore
2017-11-30 01:17:15 +00:00
3241c7aac3
untested WIP but might actually work
2017-11-29 18:27:05 +00:00
47d99a20d5
Add user_directory_include_pattern config param to expand search results to additional users
...
Initial commit; this doesn't work yet - the LIKE filtering seems too aggressive.
It also needs _do_initial_spam to be aware of prepopulating the whole user_directory_search table with all users...
...and it needs a handle_user_signup() or something to be added so that new signups get incrementally added to the table too.
Committing it here as a WIP
2017-11-29 16:46:45 +00:00
2c6d63922a
Remove pushers when deleting access tokens
...
Whenever an access token is invalidated, we should remove the associated
pushers.
2017-11-29 16:44:35 +00:00
5a4da5bf78
Merge pull request #2697 from matrix-org/rav/fix_urlcache_index_error
...
Fix error on sqlite 3.7
2017-11-27 12:25:48 +00:00
63ccaa5873
Avoid retrying forever on IntegrityError
2017-11-27 12:00:07 +00:00
6b48b3e277
fix sql fails
2017-11-22 18:06:24 +00:00
2908f955d1
Check database in has_completed_background_updates
...
so that the right thing happens on workers.
2017-11-22 18:02:15 +00:00
0b4aa2dc21
Merge pull request #2689 from matrix-org/rav/unlock_account_data_upsert
...
Avoid locking account_data tables for upserts
2017-11-21 13:39:14 +00:00
7098b65cb8
Fix error on sqlite 3.7
...
Create the url_cache index on local_media_repository as a background update, so
that we can detect whether we are on sqlite or not and create a partial or
complete index accordingly.
To avoid running the cleanup job before we have built the index, add a bailout
which will defer the cleanup if the bg updates are still running.
Fixes https://github.com/matrix-org/synapse/issues/2572 .
2017-11-21 11:14:17 +00:00
c46139a17e
Avoid locking account_data tables for upserts
2017-11-16 18:08:01 +00:00
06e5bcfc83
Avoid locking for upsert on pushers tables
...
* replace the upsert into deleted_pushers with an insert
* no need to lock for upsert on pusher_throttle
2017-11-16 17:52:23 +00:00
77a1227870
Fix broken ref to IntegrityError
2017-11-16 16:03:38 +00:00
7ab2b69e18
Avoid locking `pushers` table on upsert
...
Now that _simple_upsert will retry on IntegrityError, we don't need to lock the
table.
2017-11-16 15:32:01 +00:00
10aaa1bc15
_simple_upsert: retry on IntegrityError
...
wrap the call to _simple_upsert_txn in a loop so that we retry on an
integrityerror: this means we can avoid locking the table provided there is an
unique index.
2017-11-16 15:30:15 +00:00
cdc9e50a5d
Cleanup in _simple_upsert_txn
...
Bail out early to reduce indentation
2017-11-16 15:29:10 +00:00
f959c01600
Merge pull request #2661 from matrix-org/rav/statereadstore
...
Pull out bits of StateStore to a mixin
2017-11-15 17:23:01 +00:00
1b83c09c03
Merge pull request #2675 from matrix-org/rav/remove_broken_logcontext_funcs
...
Remove preserve_context_over_{fn, deferred}
2017-11-15 11:13:53 +00:00
35a4b63240
Pull out bits of StateStore to a mixin
...
... so that we don't need to secretly gut-wrench it for use in the slaved
stores. I haven't done the other stores yet, but we should. I'm tired of the
workers breaking every time we tweak the stores because I forgot to gut-wrench
the right method.
fixes https://github.com/matrix-org/synapse/issues/2655 .
2017-11-14 11:43:58 +00:00
4dd1bfa8c1
Revert "Revert "move _state_group_cache to statestore""
...
We're going to fix this properly on this branch, so that the _state_group_cache
can end up in StateGroupReadStore.
This reverts commit ab335edb02
2017-11-14 11:43:58 +00:00
7e6fa29cb5
Remove preserve_context_over_{fn, deferred}
...
Both of these functions ae known to leak logcontexts. Replace the remaining
calls to them and kill them off.
2017-11-14 11:22:42 +00:00
6cfee09be9
Make __init__ consitstent across Store heirarchy
...
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
2017-11-13 10:46:07 +00:00
ab335edb02
Revert "move _state_group_cache to statestore"
...
This reverts commit f5cf3638e9
2017-11-13 10:05:33 +00:00
bfbf1e1f1a
Up cache size of get_global_account_data_by_type_for_user
2017-11-13 09:52:11 +00:00
2dce6b15c3
Fix typo
2017-11-09 15:56:16 +00:00
618c7b816a
Merge pull request #2656 from matrix-org/rav/fix_deactivate
...
Fix 'NoneType' not iterable in /deactivate
2017-11-09 15:20:35 +00:00
889102315e
Fix 'NoneType' not iterable in /deactivate
...
make sure we actually return a value from user_delete_access_tokens
2017-11-09 15:15:33 +00:00
e8814410ef
Have an explicit API to update room config
2017-11-08 16:13:27 +00:00
94ff2cda73
Revert "Modify group room association API to allow modification of is_public"
2017-11-08 15:43:34 +00:00
167eb01d83
Merge pull request #2637 from spantaleev/avoid-noop-media-deletes
...
Avoid no-op media deletes
2017-11-08 11:53:27 +00:00
02a9a93bde
Merge pull request #2649 from matrix-org/rav/fix_delta_on_state_res
...
Fix bug in state group storage
2017-11-08 09:22:13 +00:00
228ccf1fe3
Merge pull request #2643 from matrix-org/matthew/user_dir_typos
...
Fix various embarrassing typos around user_directory and add some doc.
2017-11-07 17:31:11 +00:00
f5cf3638e9
move _state_group_cache to statestore
...
this is internal to statestore, so let's keep it there.
2017-11-07 16:43:00 +00:00
38b265cb51
Remember to pick is_admin out of the db
2017-11-07 11:24:04 +00:00
631fa4a1b7
create new indexes before dropping old ones to keep safetynet in place
2017-11-07 10:41:55 +00:00
4ad883398f
s/users_in_pubic_room/users_in_public_rooms/g
2017-11-04 19:39:40 +00:00
d802e8ca6a
s/users_in_pubic_room/users_in_public_rooms/g
2017-11-04 19:38:13 +00:00
a100700630
fix copyright....
2017-11-04 19:35:49 +00:00
b6b075fd49
s/popualte/populate/
2017-11-04 19:35:33 +00:00
805196fbeb
Avoid no-op media deletes
...
If there are no media entries to delete,
avoid creating transactions, prepared statements
and unnecessary log entries.
Signed-off-by: Slavi Pantaleev <slavi@devture.com>
2017-11-04 09:50:15 +02:00
bc8a5c0330
Notify auth providers on logout
...
Provide a hook by which auth providers can be notified of logouts.
2017-11-01 16:51:51 +00:00
275826f234
Merge pull request #2617 from matrix-org/matthew/auto-displayname
...
automatically set default displayname on register
2017-11-01 16:21:16 +00:00
fbbacb284e
Merge pull request #2613 from matrix-org/rav/kill_refresh_tokens
...
Remove the last vestiges of refresh_tokens
2017-11-01 15:57:35 +00:00
9f7a555b4e
switch to setting default displayname in the storage layer
...
to avoid clobbering guest user displaynames on registration
2017-11-01 15:51:30 +00:00
99354b430e
Merge pull request #2612 from matrix-org/luke/groups-room-relationship-is-public
...
Modify group room association API to allow modification of is_public
2017-11-01 11:08:36 +00:00
318a249c8b
Leave `is_public` as required argument of update_room_group_association
2017-11-01 09:36:01 +00:00
356bcafc44
Remove the last vestiges of refresh_tokens
2017-10-31 20:35:58 +00:00
20fe347906
Modify group room association API to allow modification of is_public
...
also includes renamings to make things more consistent.
2017-10-31 17:04:28 +00:00
1650eb5847
DB schema interface for password auth providers
...
Provide an interface by which password auth providers can register db schema
files to be run at startup
2017-10-31 14:01:53 +00:00
4ab8abbc2b
Merge branch 'erikj/attestation_local_fix' of github.com:matrix-org/synapse into develop
2017-10-27 15:07:08 +01:00
5451cc7792
Request is_public from database
2017-10-27 11:27:43 +01:00
6362298fa5
Create groups with is_public = True
2017-10-27 11:04:20 +01:00
82d8c1bacb
Fixup
2017-10-27 10:30:21 +01:00
195abfe7a5
Remove incorrect attestations
2017-10-27 09:58:13 +01:00
007cd48af6
Recreate groups table instead of adding column
...
Adding a column with non-constant default not possible in sqlite3
2017-10-26 17:55:22 +01:00
713e60b9b6
Awful hack to get default true
2017-10-26 17:38:14 +01:00
cfa4e658e0
Bump schema version to 46
2017-10-26 17:23:49 +01:00
9b2feef9eb
Add is_public to groups table to allow for private groups
...
Prevent group API access to non-members for private groups
Also make all the group code paths consistent with `requester_user_id` always being the User ID of the requesting user.
2017-10-26 16:51:32 +01:00
eaaabc6c4f
replace 'except:' with 'except Exception:'
...
what could possibly go wrong
2017-10-23 15:52:32 +01:00
2e9f5ea31a
Fix logcontext handling for persist_events
...
* don't use preserve_context_over_deferred, which is known broken.
* remove a redundant preserve_fn.
* add/improve some comments
2017-10-17 10:59:30 +01:00
2c5972f87f
Implement GET /groups/$groupId/invited_users
2017-10-16 15:31:11 +01:00
818b08d0e4
peeeeeeeeep8888888888888888888888888888
2017-10-11 15:54:00 +01:00
ea18996f54
Fix group stream replication
...
The stream update functions expect the storage function to return a list
of tuples.
2017-10-11 15:44:39 +01:00
c2c47550f9
Fix schema delta versions
2017-10-11 13:23:15 +01:00
535cc49f27
Merge pull request #2466 from matrix-org/erikj/groups_merged
...
Initial Group Implementation
2017-10-11 13:20:07 +01:00
e8496efe84
Fix up comment
2017-10-09 15:17:34 +01:00
11d62f43c9
Invalidate cache
2017-10-03 14:12:28 +01:00
e4ab96021e
Update comments
2017-10-03 14:10:41 +01:00
30848c0fcd
Ignore incoming events for rooms that we have left
...
When synapse receives an event for a room its not in over federation, it
double checks with the remote server to see if it is in fact in the
room. This is done so that if the server has forgotten about the room
(usually as a result of the database being dropped) it can recover from
it.
However, in the presence of state resets in large rooms, this can cause
a lot of work for servers that have legitimately left. As a hacky
solution that supports both cases we drop incoming events for rooms that
we have explicitly left.
This means that we no longer support the case of servers having
forgotten that they've rejoined a room, but that is sufficiently rare
that we're not going to support it for now.
2017-10-03 11:18:21 +01:00
27955056e0
Merge branch 'develop' into erikj/groups_merged
2017-10-02 16:20:41 +01:00
768f00dedb
Up the limits on number of url cache entries to delete at one time
2017-09-28 14:27:27 +01:00
4dc07e93a8
Add old indices
2017-09-28 14:10:33 +01:00
93247a424a
Only pull out local media that were for url cache
2017-09-28 13:48:14 +01:00
7a44c01d89
Fix typo
2017-09-28 12:46:04 +01:00
ae79764fe5
Change expires column to expires_ts
2017-09-28 12:37:53 +01:00
77f1d24de3
More brackets
2017-09-28 12:23:15 +01:00
9ccb4226ba
Delete expired url cache data
2017-09-28 12:18:06 +01:00
17b8e2bd02
Add remove room API
2017-09-26 15:52:41 +01:00
a8e2a3df32
Add unique index to group_rooms table
2017-09-26 15:39:21 +01:00
e1dec2f1a7
Remove user from group summary when the leave the group
2017-09-21 16:09:57 +01:00
197d82dc07
Correctly return next token
2017-09-20 11:12:11 +01:00
069ae2df12
Fix initial sync
2017-09-20 10:52:12 +01:00
e76d1135dd
Invalidate signing key cache when we gat an update
...
This might make the cache slightly more efficient.
2017-09-20 01:32:42 +01:00
4a9b1cf253
Add user profiles to summary from group server
2017-08-25 16:23:58 +01:00
27ebc5c8f2
Add remote profile cache
2017-08-25 11:25:47 +01:00
97c544f91f
Add _simple_update
2017-08-25 11:11:37 +01:00
2800983f3e
Merge pull request #2410 from matrix-org/erikj/groups_publicise
...
Add ability to publicise group membership
2017-08-21 16:51:56 +01:00
8b50fe5330
Use BOOLEAN rather than TEXT type
2017-08-21 16:37:29 +01:00
175a01f56c
Groups: Fix mising json.load in initial sync
2017-08-21 14:45:56 +01:00
ef8e578677
Add bulk group publicised lookup API
2017-08-09 13:36:22 +01:00
b880ff190a
Allow update group publicity
2017-08-08 14:19:41 +01:00
05e21285aa
Store whether the user wants to publicise their membership of a group
2017-08-08 13:01:46 +01:00
966a70f1fa
Update comment
2017-07-24 17:49:39 +01:00
629cdfb124
Use join rather than joined, etc.
2017-07-24 14:54:05 +01:00
ed666d3969
Fix all the typos
2017-07-24 14:05:09 +01:00
b76ef6ccb8
Include users membership in group in summary API
2017-07-24 13:55:39 +01:00
851aeae7c7
Check users/rooms are in group before adding to summary
2017-07-24 13:40:56 +01:00
96917d5552
Merge pull request #2378 from matrix-org/erikj/group_sync_support
...
Add groups to sync stream
2017-07-21 11:05:39 +01:00
0401604222
Merge pull request #2377 from matrix-org/erikj/group_profile_update
...
Add update group profile API
2017-07-20 17:53:39 +01:00
2cc998fed8
Fix replication. And notify
2017-07-20 17:13:18 +01:00
c544188ee3
Add groups to sync stream
2017-07-20 16:36:42 +01:00
8209b5f033
Fix a storage desc
2017-07-20 16:22:22 +01:00
57826d645b
Fix typo
2017-07-20 13:15:22 +01:00
6f443a74cf
Add update group profile API
2017-07-20 09:46:33 +01:00
94ecd871a0
Fix typos
2017-07-18 16:38:54 +01:00
3e703eb04e
Comment
2017-07-18 10:17:25 +01:00
508460f240
Remove sync stuff
2017-07-18 09:55:46 +01:00
4540730111
Remove unused tables
2017-07-18 09:38:15 +01:00
e96ee95a7e
Remove sync stuff
2017-07-18 09:38:08 +01:00
2f9eafdd36
Add local group server support
2017-07-17 12:03:49 +01:00
85fda57208
Add DEFAULT_ROLE_ID
2017-07-14 14:03:54 +01:00
4b203bdba5
Correctly increment orders
2017-07-14 14:02:31 +01:00
3b0470dba5
Remove unused functions
2017-07-13 13:54:01 +01:00
8575e3160f
Comments
2017-07-13 13:52:41 +01:00
26451a09eb
Comments
2017-07-12 14:47:18 +01:00
a62406aaa5
Add group summary APIs
2017-07-12 11:36:15 +01:00
6d586dc05c
Comment
2017-07-12 09:58:37 +01:00
410b4e14a1
Move comment
2017-07-11 15:44:18 +01:00
e52c391cd4
Rename column to attestation_json
2017-07-11 14:25:46 +01:00
0aac30d53b
Comments
2017-07-11 14:23:50 +01:00
83936293eb
Comments
2017-07-11 11:42:25 +01:00
b8ca494ee9
Initial group server implementation
2017-07-10 15:44:15 +01:00
f502b0dea1
Perf: Don't filter events for push
...
We know the users are joined and we can explicitly check for if they are
ignoring the user, so lets do that.
2017-07-07 14:04:40 +01:00
ed9a7f5436
Merge pull request #2309 from matrix-org/erikj/user_ip_repl
...
Fix up user_ip replication commands
2017-07-06 14:33:14 +01:00
b5e8d529e6
Define CACHE_SIZE_FACTOR once
2017-07-04 09:56:44 +01:00
6ff14ddd2e
Make into list
2017-06-29 15:47:37 +01:00
Erik Johnston