7dd13415db
Remove unused from_token param
2018-05-09 10:58:16 +01:00
27cf170558
Refactor recent events func to use pagination func
...
This also removes a cache that is unlikely to ever get hit.
2018-05-09 10:55:55 +01:00
23ec51c94c
Fix up comments and make function private
2018-05-09 09:55:19 +01:00
696f532453
Reuse existing pagination code for context API
2018-05-08 16:20:19 +01:00
3e6d306e94
Parse tokens before calling DB function
2018-05-08 16:18:58 +01:00
274b8c6025
Only fetch required fields from database
2018-05-08 16:15:25 +01:00
06c0d0ed08
Split paginate_room_events storage function
2018-05-08 16:14:26 +01:00
0a3b51c420
Merge pull request #3141 from matrix-org/erikj/fixup_state
...
Refactor event storage to prepare for changes in state calculations
2018-05-03 10:39:20 +01:00
31c7c29d43
Fix up grammar
2018-05-03 10:38:58 +01:00
dd1a832419
remove user agent from data model, will just join on user_ips
2018-05-01 12:13:49 +01:00
d0857702e8
add inidexes based on usage
2018-05-01 12:12:57 +01:00
99a54bf2af
Merge pull request #3129 from matrix-org/matthew/fix_group_dups
...
remove duplicates from groups tables
2018-04-30 11:47:25 +01:00
db75c86e84
Merge branch 'develop' into py3-xrange-1
2018-04-30 01:02:25 +01:00
adaf3ec87f
fix missing import
2018-04-28 22:39:15 +01:00
006e18b6bb
pep8
2018-04-28 22:32:24 +01:00
42c89c8215
make it work with sqlite
2018-04-28 22:27:30 +01:00
d82b6ea9e6
Move more xrange to six
...
plus a bonus next()
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:57:00 +02:00
fc149b4eeb
Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background
2018-04-27 14:31:23 +01:00
2a13af23bc
Use run_in_background in preference to preserve_fn
...
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
2018-04-27 12:55:51 +01:00
9c3da24561
Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptions
...
Improve exception handling for background processes
2018-04-27 11:47:49 +01:00
6e10eed28e
Refactor event storage to not require state
...
This is in preparation for using contexts that may or may not have the
current_state_ids set. This will allow us to avoid unnecessarily pulling
out state for an event on the master process when using workers.
We also add a check to see if the state groups of the old extremities
are the same as the new ones.
2018-04-27 11:38:02 +01:00
605defb9e4
Add missing consumeErrors
...
In general we want defer.gatherResults to consumeErrors, rather than having
exceptions hanging around and getting logged as CRITICAL unhandled errors.
2018-04-27 11:16:28 +01:00
9255a6cb17
Improve exception handling for background processes
...
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.
This is unsatisfactory for a number of reasons:
- logging on garbage collection is best-effort and may happen some time after
the error, if at all
- it can be hard to figure out where the error actually happened.
- it is logged as a scary CRITICAL error which (a) I always forget to grep for
and (b) it's not really CRITICAL if a background process we don't care about
fails.
So this is an attempt to add exception handling to everything we fire off into
the background.
2018-04-27 11:07:40 +01:00
7ec8e798b4
Fix media admin APIs
2018-04-26 11:31:22 +01:00
fb6015d0a6
pep8
2018-04-25 17:56:11 +01:00
617bf40924
Generate user daily stats
2018-04-25 17:37:29 +01:00
22881b3d69
Also fix reindexing of search
2018-04-25 15:32:04 +01:00
ba3166743c
Fix quarantine media admin API
2018-04-25 15:11:18 +01:00
e3a373f002
remove duplicates from groups tables
...
and rename inconsistently named indexes.
Based on https://github.com/matrix-org/synapse/pull/3128 - thanks @vurpo\!
2018-04-25 14:58:43 +01:00
c09a6daf09
Merge pull request #3110 from NotAFile/py3-six-queue
...
Replace Queue with six.moves.queue
2018-04-20 15:35:00 +01:00
bc381d5798
Merge pull request #3117 from matrix-org/rav/refactor_have_events
...
Refactor store.have_events
2018-04-20 10:26:12 +01:00
b1dfbc3c40
Refactor store.have_events
...
It turns out that most of the time we were calling have_events, we were only
using half of the result. Replace have_events with have_seen_events and
get_rejection_reasons, so that we can see what's going on a bit more clearly.
2018-04-20 10:25:56 +01:00
639480e14a
Avoid creating events with huge numbers of prev_events
...
In most cases, we limit the number of prev_events for a given event to 10
events. This fixes a particular code path which created events with huge
numbers of prev_events.
2018-04-16 18:41:37 +01:00
878995e660
Replace Queue with six.moves.queue
...
and a six.range change which I missed the last time
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-16 00:46:21 +02:00
415aeefd89
Format docstring
2018-04-12 12:07:09 +01:00
23a7f9d7f4
Doc we raise on unknown event
2018-04-12 11:20:51 +01:00
4dae4a97ed
Track last processed event received_ts
2018-04-11 14:27:09 +01:00
92e34615c5
Track where event stream processing have gotten up to
2018-04-11 12:13:40 +01:00
233699c42e
Merge pull request #2760 from Valodim/pypy
...
Synapse on PyPy
2018-04-11 11:20:01 +01:00
d54cfbb7a8
fix typo
2018-04-10 17:38:16 +01:00
d1e56cfcd1
Fix pep8 error on psycopg2cffi hack
2018-04-10 11:35:29 +02:00
89de934981
Use psycopg2cffi module instead of psycopg2 if running on pypy
...
The psycopg2 package isn't available for PyPy. This commit adds a check
if the runtime is PyPy, and if it is uses psycopg2cffi module in favor
of psycopg2. This is almost a drop-in replacement, except for one place
where an additional cast to string is required.
2018-04-10 11:29:52 +02:00
41e0611895
remove errant print
2018-04-09 18:44:20 +01:00
61b439c904
Fix msec to sec, again
2018-04-09 18:43:48 +01:00
87770300d5
Fix msec to sec
2018-04-09 18:38:59 +01:00
64bc2162ef
Fix psycopg2 interpolation
2018-04-09 17:50:36 +01:00
cd087a265d
Don't use redundant inlineCallbacks
2018-04-06 15:43:27 +01:00
87c864b698
join_rule -> join_policy
2018-04-06 15:43:27 +01:00
ae85c7804e
is_joinable -> join_rule
2018-04-06 15:43:27 +01:00
edb45aae38
pep8
2018-04-06 15:43:27 +01:00
b370fe61c0
Implement group join API
2018-04-06 15:43:27 +01:00
e089100c62
Merge pull request #3045 from matrix-org/dbkr/group_joinable
...
Add joinability for groups
2018-04-05 15:57:49 +01:00
68b0ee4e8d
Merge pull request #3041 from matrix-org/r30_stats
...
R30 stats
2018-04-05 15:37:37 +01:00
22284a6f65
Merge pull request #3060 from matrix-org/rav/kill_event_content
...
Remove uses of events.content
2018-04-05 15:02:17 +01:00
917380e89d
NON NULL -> NOT NULL
2018-04-05 14:32:12 +01:00
700e5e7198
Use DEFAULT join_policy of "invite" in db
2018-04-05 14:01:17 +01:00
b214a04ffc
Document set_group_join_policy
2018-04-05 13:29:16 +01:00
0e5f479fc0
Review comments
...
Use iteritems over item to loop over dict
formatting
2018-04-05 12:16:46 +01:00
eb8d8d6f57
Use join_policy API instead of joinable
...
The API is now under
/groups/$group_id/setting/m.join_policy
and expects a JSON blob of the shape
```json
{
"m.join_policy": {
"type": "invite"
}
}
```
where "invite" could alternatively be "open".
2018-04-03 16:16:40 +01:00
8da39ad98f
Merge pull request #3049 from matrix-org/rav/use_staticjson
...
Use static JSONEncoders
2018-04-03 15:18:32 +01:00
2fe3f848b9
Remove uses of events.content
2018-03-29 23:17:12 +01:00
05630758f2
Use static JSONEncoders
...
using json.dumps with custom options requires us to create a new JSONEncoder on
each call. It's more efficient to create one upfront and reuse it.
2018-03-29 23:13:33 +01:00
fcfe7f6ad3
Use simplejson throughout
...
Let's use simplejson rather than json, for consistency.
2018-03-29 22:45:52 +01:00
b4e37c6f50
pep8
2018-03-29 17:27:39 +01:00
9ee44a372d
Remove need for sqlite specific query
2018-03-29 16:45:34 +01:00
dc7c020b33
fix pep8 errors
2018-03-28 17:25:15 +01:00
241e4e8687
remove twisted deferral cruft
2018-03-28 16:25:53 +01:00
929b34963d
OK, smallint it is then
2018-03-28 14:53:55 +01:00
a838444a70
Grr. Copy the definition from is_admin
2018-03-28 14:50:30 +01:00
4262aba17b
bump schema version
2018-03-28 14:40:03 +01:00
86932be2cb
Support multi client R30 for psql
2018-03-28 14:36:53 +01:00
a164270833
Make column definition that works on both dbs
2018-03-28 14:23:00 +01:00
352e1ff9ed
Add schema delta file
2018-03-28 14:07:57 +01:00
79452edeee
Add joinability for groups
...
Adds API to set the 'joinable' flag, and corresponding flag in the
table.
2018-03-28 14:03:37 +01:00
788e69098c
Add user_ips last seen index
2018-03-28 12:03:13 +01:00
0f890f477e
No need to cast in count_daily_users
2018-03-28 11:49:57 +01:00
545001b9e4
Fix search_user_dir multiple sqlite versions do different things
2018-03-28 11:19:45 +01:00
a32d2548d9
query and call for r30 stats
2018-03-28 10:39:13 +01:00
9187e0762f
count_daily_users failed if db was sqlite due to type failure - presumably this prevcented all sqlite homeservers reporting home
2018-03-28 10:02:32 +01:00
f879127aaa
Merge pull request #3029 from matrix-org/erikj/linearize_generate_user_id
...
Linearize calls to _generate_user_id
2018-03-28 10:00:31 +01:00
e6d87c93f3
Merge pull request #3030 from matrix-org/erikj/no_ujson
...
Remove last usage of ujson
2018-03-28 10:00:06 +01:00
800cfd5774
Comment
2018-03-27 13:30:39 +01:00
152c2ac19e
Fix indent
2018-03-27 13:13:46 +01:00
e70287cff3
Comment
2018-03-27 13:13:38 +01:00
3f49e131d9
Add counter metrics for calculating state delta
...
This will allow us to measure how often we calculate state deltas in
event persistence that we would have been able to calculate at the same
time we calculated the state for the event.
2018-03-27 10:57:35 +01:00
fecb45e0c3
Remove last usage of ujson
2018-03-26 13:32:29 +01:00
8d6dc106d1
Don't use _cursor_to_dict in find_next_generated_user_id_localpart
2018-03-26 12:02:44 +01:00
fa72803490
Merge branch 'master' of github.com:matrix-org/synapse into develop
2018-03-19 11:41:01 +00:00
bf49d2dca8
Replace some ujson with simplejson to make it work
2018-03-16 00:55:44 +00:00
a8ce159be4
Replace some ujson with simplejson to make it work
2018-03-16 00:27:09 +00:00
5b631ff41a
Remove wrong comment
2018-03-16 00:07:08 +00:00
926ba76e23
Replace ujson with simplejson
2018-03-15 23:43:31 +00:00
056a6df546
Merge branch 'develop' into matthew/filter_members
2018-03-14 15:38:05 +00:00
ebd0127999
Merge pull request #2988 from matrix-org/erikj/split_profile_store
...
Split up ProfileStore
2018-03-14 09:41:06 +00:00
f51565e023
Merge pull request #2993 from matrix-org/erikj/is_blocked
...
Add is_blocked to worker store
2018-03-14 09:39:18 +00:00
9f77001e27
pep8
2018-03-14 00:07:47 +00:00
d144ed6ffb
fix bug #2926 (loading all state for a given type from the DB if the state_key is None) ( #2990 )
...
Fixes a regression that had crept in where the caching layer upholds requests for loading state which is filtered by type (but not by state_key), but the DB layer itself would interpret a missing state_key as a request to filter by null state_key rather than returning all state_keys.
2018-03-13 22:36:04 +00:00
ccca02846d
make it work
2018-03-13 22:31:41 +00:00
f0f9a0605b
remove comment now #2969 is fixed
2018-03-13 22:12:15 +00:00
12350e3f9a
merge proper fix to bug 2969
2018-03-13 22:11:58 +00:00
afbf4d3dcc
typoe
2018-03-13 19:48:04 +00:00
865377a70d
disable optimisation for searching for state groups
...
when type filter includes wildcards on state_key
2018-03-13 19:46:04 +00:00
a08726fc42
Add is_blocked to worker store
2018-03-13 18:28:44 +00:00
b2aba9e430
build where_clause sanely
2018-03-13 18:13:44 +00:00
52f7e23c72
PR feedbackz
2018-03-13 18:07:55 +00:00
1b1c137771
fix bug #2926
2018-03-13 17:52:52 +00:00
3518d0ea8f
Split up ProfileStore
2018-03-13 17:36:50 +00:00
889a2a853a
Add Measure block for persist_events
...
This seems like a useful thing to measure.
2018-03-13 10:01:42 +00:00
fdedcd1f4d
correctly handle None state_keys
...
and fix include_other_types thinko
2018-03-12 01:39:06 +00:00
97c0496cfa
fix sqlite where clause
2018-03-12 00:27:06 +00:00
8713365265
typos
2018-03-11 20:10:25 +00:00
9b334b3f97
WIP experiment in lazyloading room members
2018-03-11 20:01:41 +00:00
8cb44da4aa
Fix race in sync when joining room
...
The race happens when the user joins a room at the same time as doing a
sync. We fetch the current token and then get the rooms the user is in.
If the join happens after the current token, but before we get the rooms
we end up sending down a partial room entry in the sync.
This is fixed by looking at the stream ordering of the membership
returned by get_rooms_for_user, and handling the case when that stream
ordering is after the current token.
2018-03-07 11:55:31 +00:00
b2932107bb
Merge pull request #2946 from matrix-org/rav/timestamp_to_purge
...
Implement purge_history by timestamp
2018-03-06 11:20:23 +00:00
7aed50a038
Merge pull request #2948 from matrix-org/erikj/kill_as_sync
...
Remove ability for AS users to call /events and /sync
2018-03-06 11:10:09 +00:00
b6c4b851f1
Merge pull request #2947 from matrix-org/erikj/split_directory_store
...
Split Directory store
2018-03-05 18:17:32 +00:00
d4ffe61d4f
Remove ability for AS users to call /events and /sync
...
This functionality has been deprecated for a while as well as being
broken for a while. Instead of fixing it lets just remove it entirely.
See: https://github.com/matrix-org/matrix-doc/issues/1144
2018-03-05 15:44:46 +00:00
69ce365b79
Fix cache invalidation on deletion
2018-03-05 15:29:03 +00:00
2e223163ff
Split Directory store
2018-03-05 15:11:30 +00:00
f8bfcd7e0d
Provide a means to pass a timestamp to purge_history
2018-03-05 14:37:23 +00:00
d032785aa7
Merge pull request #2943 from matrix-org/rav/fix_find_first_stream_ordering_after_ts
...
Test and fix find_first_stream_ordering_after_ts
2018-03-05 12:26:14 +00:00
c818fcab11
Test and fix find_first_stream_ordering_after_ts
...
It seemed to suffer from a bunch of off-by-one errors.
2018-03-05 12:04:02 +00:00
06a14876e5
Add find_first_stream_ordering_after_ts
...
Expose this as a public function which can be called outside a txn
2018-03-05 11:53:39 +00:00
42174946f8
Merge pull request #2934 from matrix-org/erikj/cache_fix
...
Fix bug with delayed cache invalidation stream
2018-03-05 11:33:17 +00:00
efb79820b4
Fix bug with delayed cache invalidation stream
...
We poked the notifier before updated the current token for the cache
invalidation stream. This mean that sometimes the update wouldn't be
sent until the next time a cache was invalidated.
2018-03-02 14:45:15 +00:00
fafa3e7114
Split registration store
2018-03-02 13:48:27 +00:00
1773df0632
Merge pull request #2925 from matrix-org/erikj/split_sig_fed
...
Split out SignatureStore and EventFederationStore
2018-03-01 17:32:58 +00:00
9e08a93a7b
Merge pull request #2927 from matrix-org/erikj/read_marker_caches
...
Improve caching for read_marker API
2018-03-01 17:12:34 +00:00
4b44f05f19
Fewer lies are better
2018-03-01 17:08:17 +00:00
a83c514d1f
Improve caching for read_marker API
...
We add a new storage function to get a paritcular type of room account
data. This allows us to prefill the cache when updating that acount
data.
2018-03-01 17:08:17 +00:00
33bebb63f3
Add some caches to help read marker API
2018-03-01 17:08:17 +00:00
2ad4d5b5bb
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fed
2018-03-01 16:59:39 +00:00
7c371834cc
Stub out broken function only used for cache
2018-03-01 16:44:13 +00:00
64346be26d
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_stream_store
2018-03-01 16:26:42 +00:00
22518e2833
Merge pull request #2923 from matrix-org/erikj/stream_ago_worker
...
Calculate stream_ordering_month_ago correctly on workers
2018-03-01 16:23:54 +00:00
884b26ae41
Remove unused variables
2018-03-01 16:23:48 +00:00
1b2af11650
Document abstract class and method better
2018-03-01 16:20:57 +00:00
872ff95ed4
Default stream_ordering_*_ago to None
2018-03-01 16:00:05 +00:00
22004b524e
Fix comment typo
2018-03-01 15:59:40 +00:00
f793bc3877
Split out stream store
2018-03-01 15:13:08 +00:00
6411f725be
Calculate stream_ordering_month_ago correctly on workers
2018-03-01 14:20:53 +00:00
a9a2d66cdd
Split out SignatureStore and EventFederationStore
2018-03-01 14:17:53 +00:00
0c8ba5dd1c
Split up RoomStore
2018-03-01 14:01:19 +00:00
493e25d554
Move storage functions for push calculations
...
This will allow push actions for an event to be calculated on workers.
2018-02-27 13:58:16 +00:00
3594dbc6dc
Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidation
...
Fix missing invalidations for receipt storage
2018-02-27 11:34:26 +00:00
2311189ee4
Merge pull request #2903 from matrix-org/erikj/split_roommember_store
...
Split out RoomMemberStore
2018-02-27 11:32:10 +00:00
c57607874c
Merge pull request #2901 from matrix-org/erikj/split_as_stores
...
Split AS stores
2018-02-27 10:07:07 +00:00
8956f0147a
Add comment
2018-02-27 10:06:51 +00:00
e5b4a208ce
Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions
...
Batch inserts into event_push_actions_staging
2018-02-26 14:45:40 +00:00
45b5fe9122
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/handle_unpersisted_events_push
2018-02-26 13:49:24 +00:00
d62ce972f8
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_roommember_store
2018-02-23 11:46:24 +00:00
6ae9a3d2a6
Update copyright
2018-02-23 11:44:49 +00:00
a90c60912f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_event_push_actions
2018-02-23 11:26:31 +00:00
50e8657867
Merge pull request #2902 from matrix-org/erikj/split_events_store
...
Split out get_events and co into a worker store
2018-02-23 11:23:52 +00:00
1cf9e071dd
Merge pull request #2899 from matrix-org/erikj/split_pushers
...
Split PusherStore
2018-02-23 11:23:35 +00:00
d0957753bf
Merge pull request #2898 from matrix-org/erikj/split_push_rules_store
...
Split PushRulesStore
2018-02-23 11:23:23 +00:00
70349872c2
Update copyright
2018-02-23 11:14:35 +00:00
eba93b05bf
Split EventsWorkerStore into separate file
2018-02-23 11:01:21 +00:00
bf8a36e080
Update copyright
2018-02-23 10:52:10 +00:00
5d0f665848
Remove redundant clock
2018-02-23 10:49:58 +00:00
3bd760628b
_event_persist_queue shouldn't be in worker store
2018-02-23 10:49:18 +00:00
eb9b5eec81
Update copyright
2018-02-23 10:42:39 +00:00
c2ecfcc3a4
Update copyright
2018-02-23 10:41:34 +00:00
7e6cf89dc2
Update copyright
2018-02-23 10:39:19 +00:00
26d37f7a63
Update copyright
2018-02-23 10:33:55 +00:00
bb73f55fc6
Use absolute imports
2018-02-23 10:31:16 +00:00
faeb369f15
Fix missing invalidations for receipt storage
2018-02-21 15:19:54 +00:00
3dec9c66b3
Split out RoomMemberStore
2018-02-21 12:07:26 +00:00
46244b2759
Split AS stores
2018-02-21 11:49:34 +00:00
27b094f382
Split out get_events and co into a worker store
2018-02-21 11:41:48 +00:00
c96d547f4d
Actually use new param
2018-02-21 11:03:42 +00:00
d15d237b0d
Split out EventPushActionWorkerStore
2018-02-21 11:01:13 +00:00
6f72765371
Split PusherStore
2018-02-21 10:54:21 +00:00
cbaad969f9
Split PushRulesStore
2018-02-21 10:43:31 +00:00
ca9b9d9703
Split AccountDataStore and TagStore
2018-02-21 10:15:04 +00:00
8fbb4d0d19
Raise exception in abstract method
2018-02-20 17:59:23 +00:00
95e4cffd85
Fix comment
2018-02-20 17:58:40 +00:00
e316bbb4c0
Use abstract base class to access stream IDs
2018-02-20 17:43:57 +00:00
f5ac4dc2d4
Split ReceiptsStore
2018-02-20 16:28:28 +00:00
24087bffa9
Ensure all push actions are deleted from staging
2018-02-20 12:34:31 +00:00
ad0ccf15ea
Refactor _set_push_actions_for_event_and_users_txn to use events_and_contexts
2018-02-20 12:34:28 +00:00
6ff8c87484
Batch inserts into event_push_actions_staging
2018-02-20 11:33:07 +00:00
d1a3325f99
(Really) fix tablescan of event_push_actions on purge
...
commit 278d21b5
2018-02-16 14:02:31 +00:00
6af025d3c4
Fix typo of double is_highlight
2018-02-16 11:35:31 +00:00
012e8e142a
Comments
2018-02-16 11:35:01 +00:00
b96278d6fe
Ensure that we delete staging push actions on errors
2018-02-15 15:47:06 +00:00
4810f7effd
Remove context.push_actions
2018-02-15 15:47:06 +00:00
c714c61853
Update event_push_actions table from staging table
2018-02-15 15:47:06 +00:00
acac21248c
Store push actions in staging area
2018-02-15 15:47:04 +00:00
106906a65e
Don't serialize current state over replication
2018-02-15 13:53:18 +00:00
fd1601c596
Fix state group storage bug in workers
...
We needed to move `_count_state_group_hops_txn` to the
StateGroupWorkerStore.
2018-02-15 11:04:32 +00:00
b8d821aa68
Merge pull request #2867 from matrix-org/rav/rework_purge
...
purge_history cleanups
2018-02-15 09:49:07 +00:00
d28ec43e15
Merge pull request #2769 from matrix-org/matthew/hit_the_gin
...
switch back from GIST to GIN indexes
2018-02-14 16:59:03 +00:00
39bf47319f
purge_history: fix sqlite syntax error
...
apparently sqlite insists on indexes being named
2018-02-14 16:42:19 +00:00
ac27f6a35e
purge_history: handle sqlite asshattery
...
apparently creating a temporary table commits the transaction. because that's a
useful thing.
2018-02-14 16:41:12 +00:00
5978dccff0
remove overzealous exception handling
2018-02-14 15:54:09 +00:00
278d21b5e4
purge_history: fix index use
...
event_push_actions doesn't have an index on event_id, so we need to specify
room_id.
2018-02-14 15:44:51 +00:00
5fcbf1e07c
Rework event purge to use a temporary table
...
... which should speed things up by reducing the amount of data being shuffled
across the connection
2018-02-14 11:02:22 +00:00
c0c9327fe0
Merge pull request #2854 from matrix-org/erikj/event_create_worker
...
Create a worker for event creation
2018-02-13 18:07:10 +00:00
d627174da2
Fix log message in purge_history
...
(we don't just remove remote events)
2018-02-13 16:51:21 +00:00
ddb6a79b68
Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin
2018-02-13 16:45:36 +00:00
0b27ae8dc3
move search reindex to schema 47
...
We're up to schema v47 on develop now, so this will have to go in there to have
an effect.
This might cause an error if somebody has already run it in the v46 guise, and
runs it again in the v47 guise, because it will cause a duplicate entry in the
bbackground_updates table. On the other hand, the entry is removed once it is
complete, and it is unlikely that anyone other than matrix.org has run it on
v46. The update itself is harmless to re-run because it deliberately copes with
the index already existing.
2018-02-13 16:44:46 +00:00
4a6d551704
GIN reindex: Fix syntax errors, improve exception handling
2018-02-13 16:44:46 +00:00
bfdf7b9237
Merge pull request #2864 from matrix-org/rav/persist_event_caching
...
Use StateResolutionHandler to resolve state in persist_events
2018-02-13 14:45:57 +00:00
630caf8a70
style nit
2018-02-13 14:29:22 +00:00
a9b712e9dc
Merge branch 'develop' into matthew/gin_work_mem
2018-02-13 12:16:01 +00:00
10b34dbb9a
Merge pull request #2858 from matrix-org/rav/purge_updates
...
delete_local_events for purge_room_history
2018-02-09 14:11:00 +00:00
39a6b35496
purge: move room_depth update to end
...
... to avoid locking the table for too long
2018-02-09 13:07:41 +00:00
74fcbf741b
delete_local_events for purge_history
...
Add a flag which makes the purger delete local events
2018-02-09 13:07:41 +00:00
e571aef06d
purge: Move cache invalidation to more appropriate place
...
it was a bit of a non-sequitur there
2018-02-09 13:07:41 +00:00
61ffaa8137
bump purge logging to info
...
this thing takes ages and the only sign of any progress is the logs, so having
some logs is useful.
2018-02-09 13:07:41 +00:00
671540dccf
rename delete_old_state -> purge_history
...
(beacause it deletes more than state)
2018-02-09 13:07:41 +00:00
24dd73028a
Add replication http endpoint for event sending
2018-02-07 10:32:32 +00:00
3d33eef6fc
Store state groups separately from events ( #2784 )
...
* Split state group persist into seperate storage func
* Add per database engine code for state group id gen
* Move store_state_group to StateReadStore
This allows other workers to use it, and so resolve state.
* Hook up store_state_group
* Fix tests
* Rename _store_mult_state_groups_txn
* Rename StateGroupReadStore
* Remove redundant _have_persisted_state_group_txn
* Update comments
* Comment compute_event_context
* Set start val for state_group_id_seq
... otherwise we try to recreate old state groups
* Update comments
* Don't store state for outliers
* Update comment
* Update docstring as state groups are ints
2018-02-06 14:31:24 +00:00
b31bf0bb51
Merge pull request #2849 from matrix-org/rav/clean_up_state_delta
...
Remove redundant return value from _calculate_state_delta
2018-02-05 17:42:20 +01:00
ebfe64e3d6
Use StateResolutionHandler to resolve state in persist events
...
... and thus benefit (hopefully) from its cache.
2018-02-05 16:23:26 +00:00
225dc3b4cb
Flatten _get_new_state_after_events
...
rejig the if statements to simplify the logic and reduce indentation
2018-02-05 16:23:25 +00:00
9fcbbe8e7d
Check that events being persisted have state_group
2018-02-05 16:23:25 +00:00
447aed42d2
Add event_map param to resolve_state_groups
2018-02-05 16:23:25 +00:00
ee6fb4cf85
Remove redundant return value from _calculate_state_delta
...
we already have the state from _get_new_state_after_events, so returning it
from _calculate_state_delta is just confusing.
2018-02-05 16:23:20 +00:00
3c7b480ba3
Factor out common code for search insert
...
we can reuse the same code as is used for event insert, for doing the
background index population.
2018-02-05 16:12:14 +00:00
c46e75d3d8
Move store_event_search_txn to SearchStore
...
... as a precursor to making event storing and doing the bg update share some
code.
2018-02-05 15:43:22 +00:00
80b8a28100
Factor out common code for search insert
...
we can reuse the same code as is used for event insert, for doing the
background index population.
2018-02-04 00:23:06 +00:00
bd25f9cf36
Clean up work_mem handling
...
Add some comments and improve exception handling when twiddling work_mem for
the search update
2018-02-03 23:05:41 +00:00
4eeae7ad65
Move store_event_search_txn to SearchStore
...
... as a precursor to making event storing and doing the bg update share some
code.
2018-02-03 22:59:45 +00:00
bb9f0f3cdb
Merge branch 'develop' into matthew/gin_work_mem
2018-02-03 22:40:28 +00:00
6b02fc80d1
Reinstate event_search_postgres_gist handler
...
People may have queued updates for this, so we can't just delete it.
2018-02-02 14:32:51 +00:00
78d6ddba86
Merge pull request #2841 from matrix-org/rav/refactor_calc_state_delta
...
factor _get_new_state_after_events out of _calculate_state_delta
2018-02-02 15:27:15 +01:00
6e87b34f7b
Merge branch 'develop' into travis/admin-list-media
2018-02-01 18:05:47 -07:00
77c0629ebc
Merge pull request #2837 from matrix-org/rav/fix_quarantine_media
...
Fix sql error in quarantine_media
2018-02-01 11:45:58 +01:00
e16e45b1b4
pep8
...
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-31 15:30:38 -07:00
e1e4ec9f9d
factor _get_new_state_after_events out of _calculate_state_delta
...
This reduces the scope of a bunch of variables
2018-01-31 21:32:09 +00:00
63c4383927
Documentation and naming
...
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-31 08:07:52 -07:00
773f0eed1e
Fix sql error in quarantine_media
2018-01-30 15:02:51 +00:00
b387ee17b6
Improve exception handling in persist_event
...
1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that
a Failure object will be constructed using the current exception state,
*including* its stack trace - so the stack trace is saved in the Failure,
leading to better exception reports.
2. Set `consumeErrors=True` on the ObservableDeferred, because we know that
there will always be at least one observer - which avoids a spurious "CRITICAL:
unhandled exception in Deferred" error in the logs
2018-01-29 17:05:33 +00:00
46022025ea
Fix SQL for user search
...
fix some syntax errors for user search when search_all_users is enabled
fixes #2801 , hopefully
2018-01-27 17:21:57 +00:00
a94d9b6b82
Appease the linter
...
These are ids anyways, not mxc uris.
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-20 22:49:46 -07:00
5552ed9a7f
Add an admin route to get all the media in a room
...
This is intended to be used by administrators to monitor the media that is passing through their server, if they wish.
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-20 22:37:53 -07:00
36da256cc6
Merge pull request #2805 from matrix-org/rav/log_state_res
...
Log room when doing state resolution
2018-01-17 18:05:04 +00:00
1224612a79
Log room when doing state resolution
...
Mostly because it helps figure out what is prompting the resolution
2018-01-17 17:11:59 +00:00
a87006f9c7
Merge pull request #2783 from matrix-org/erikj/media_last_accessed
...
Keep track of last access time for local media
2018-01-17 16:39:02 +00:00
06db5c4b76
Merge pull request #2803 from matrix-org/matthew/fix-userdir-sql
...
fix SQL when searching all users
2018-01-17 16:27:54 +00:00
2d9ab533f9
fix SQL when searching all users
2018-01-17 15:58:52 +00:00
390093d45e
Split resolve_events into two functions
...
... so that the return type doesn't depend on the arg types
2018-01-17 15:44:31 +00:00
2fb3a28c98
Remove lost comment
2018-01-17 14:59:44 +00:00
05f98a2224
Keep track of last access time for local media
2018-01-17 10:24:43 +00:00
3d12d97415
Track DB scheduling delay per-request
...
For each request, track the amount of time spent waiting for a db
connection. This entails adding it to the LoggingContext and we may as well add
metrics for it while we are passing.
2018-01-16 17:23:32 +00:00
8615f19d20
rework runInteraction in terms of runConnection
...
... so that we can share the code
2018-01-16 17:08:29 +00:00
64ddec1bc0
Fix a logcontext leak in persist_events
...
ObserveableDeferred expects its callbacks to be called without any
logcontexts, whereas it turns out we were calling them with the logcontext of
the request which initiated the persistence loop.
It seems wrong that we are attributing work done in the persistence loop to the
request that happened to initiate it, so let's solve this by dropping the
logcontext for it.
(I'm not sure this actually causes any real problems other than messages in the
debug log, but let's clean it up anyway)
2018-01-16 11:47:36 +00:00
174eacc8ba
oops
2018-01-09 18:14:32 +00:00
a66f489678
fix GIST->GIN switch
2018-01-09 16:55:51 +00:00
e79db0a673
switch back from GIST to GIN indexes
2018-01-09 16:37:48 +00:00
e365ad329f
oops, tweak work_mem when actually storing
2018-01-09 16:30:30 +00:00
19f9227643
avoid 80s GIN inserts by tweaking work_mem
...
see https://github.com/matrix-org/synapse/issues/2753 for details
2018-01-09 16:25:04 +00:00
b30cd5b107
Remove dead code related to default thumbnails
2018-01-09 14:38:33 +00:00
cdc2cb5d11
fix StoreError syntax
2017-12-05 11:09:31 +00:00
74e0cc74ce
fix pep8 and tests
2017-12-04 15:11:38 +00:00
1bd40ca73e
switch to a simpler 'search_all_users' button as per review feedback
2017-12-04 14:58:39 +00:00
f397153dfc
Merge branch 'develop' into matthew/search-all-local-users
2017-11-30 01:51:38 +00:00
f61e107f63
remove null constraint on user_dir.room_id
2017-11-30 01:43:50 +00:00
4b1fceb913
fix alternation operator for FTS4 - how did this ever work!?
2017-11-30 01:34:03 +00:00
a4bb133b68
fix thinkos galore
2017-11-30 01:17:15 +00:00
3241c7aac3
untested WIP but might actually work
2017-11-29 18:27:05 +00:00
47d99a20d5
Add user_directory_include_pattern config param to expand search results to additional users
...
Initial commit; this doesn't work yet - the LIKE filtering seems too aggressive.
It also needs _do_initial_spam to be aware of prepopulating the whole user_directory_search table with all users...
...and it needs a handle_user_signup() or something to be added so that new signups get incrementally added to the table too.
Committing it here as a WIP
2017-11-29 16:46:45 +00:00
2c6d63922a
Remove pushers when deleting access tokens
...
Whenever an access token is invalidated, we should remove the associated
pushers.
2017-11-29 16:44:35 +00:00
5a4da5bf78
Merge pull request #2697 from matrix-org/rav/fix_urlcache_index_error
...
Fix error on sqlite 3.7
2017-11-27 12:25:48 +00:00
63ccaa5873
Avoid retrying forever on IntegrityError
2017-11-27 12:00:07 +00:00
6b48b3e277
fix sql fails
2017-11-22 18:06:24 +00:00
2908f955d1
Check database in has_completed_background_updates
...
so that the right thing happens on workers.
2017-11-22 18:02:15 +00:00
0b4aa2dc21
Merge pull request #2689 from matrix-org/rav/unlock_account_data_upsert
...
Avoid locking account_data tables for upserts
2017-11-21 13:39:14 +00:00
7098b65cb8
Fix error on sqlite 3.7
...
Create the url_cache index on local_media_repository as a background update, so
that we can detect whether we are on sqlite or not and create a partial or
complete index accordingly.
To avoid running the cleanup job before we have built the index, add a bailout
which will defer the cleanup if the bg updates are still running.
Fixes https://github.com/matrix-org/synapse/issues/2572 .
2017-11-21 11:14:17 +00:00
c46139a17e
Avoid locking account_data tables for upserts
2017-11-16 18:08:01 +00:00
06e5bcfc83
Avoid locking for upsert on pushers tables
...
* replace the upsert into deleted_pushers with an insert
* no need to lock for upsert on pusher_throttle
2017-11-16 17:52:23 +00:00
77a1227870
Fix broken ref to IntegrityError
2017-11-16 16:03:38 +00:00
7ab2b69e18
Avoid locking `pushers` table on upsert
...
Now that _simple_upsert will retry on IntegrityError, we don't need to lock the
table.
2017-11-16 15:32:01 +00:00
10aaa1bc15
_simple_upsert: retry on IntegrityError
...
wrap the call to _simple_upsert_txn in a loop so that we retry on an
integrityerror: this means we can avoid locking the table provided there is an
unique index.
2017-11-16 15:30:15 +00:00
cdc9e50a5d
Cleanup in _simple_upsert_txn
...
Bail out early to reduce indentation
2017-11-16 15:29:10 +00:00
f959c01600
Merge pull request #2661 from matrix-org/rav/statereadstore
...
Pull out bits of StateStore to a mixin
2017-11-15 17:23:01 +00:00
1b83c09c03
Merge pull request #2675 from matrix-org/rav/remove_broken_logcontext_funcs
...
Remove preserve_context_over_{fn, deferred}
2017-11-15 11:13:53 +00:00
35a4b63240
Pull out bits of StateStore to a mixin
...
... so that we don't need to secretly gut-wrench it for use in the slaved
stores. I haven't done the other stores yet, but we should. I'm tired of the
workers breaking every time we tweak the stores because I forgot to gut-wrench
the right method.
fixes https://github.com/matrix-org/synapse/issues/2655 .
2017-11-14 11:43:58 +00:00
4dd1bfa8c1
Revert "Revert "move _state_group_cache to statestore""
...
We're going to fix this properly on this branch, so that the _state_group_cache
can end up in StateGroupReadStore.
This reverts commit ab335edb02
2017-11-14 11:43:58 +00:00
7e6fa29cb5
Remove preserve_context_over_{fn, deferred}
...
Both of these functions ae known to leak logcontexts. Replace the remaining
calls to them and kill them off.
2017-11-14 11:22:42 +00:00
6cfee09be9
Make __init__ consitstent across Store heirarchy
...
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
2017-11-13 10:46:07 +00:00
ab335edb02
Revert "move _state_group_cache to statestore"
...
This reverts commit f5cf3638e9
2017-11-13 10:05:33 +00:00
bfbf1e1f1a
Up cache size of get_global_account_data_by_type_for_user
2017-11-13 09:52:11 +00:00
2dce6b15c3
Fix typo
2017-11-09 15:56:16 +00:00
618c7b816a
Merge pull request #2656 from matrix-org/rav/fix_deactivate
...
Fix 'NoneType' not iterable in /deactivate
2017-11-09 15:20:35 +00:00
889102315e
Fix 'NoneType' not iterable in /deactivate
...
make sure we actually return a value from user_delete_access_tokens
2017-11-09 15:15:33 +00:00
e8814410ef
Have an explicit API to update room config
2017-11-08 16:13:27 +00:00
94ff2cda73
Revert "Modify group room association API to allow modification of is_public"
2017-11-08 15:43:34 +00:00
167eb01d83
Merge pull request #2637 from spantaleev/avoid-noop-media-deletes
...
Avoid no-op media deletes
2017-11-08 11:53:27 +00:00
02a9a93bde
Merge pull request #2649 from matrix-org/rav/fix_delta_on_state_res
...
Fix bug in state group storage
2017-11-08 09:22:13 +00:00
228ccf1fe3
Merge pull request #2643 from matrix-org/matthew/user_dir_typos
...
Fix various embarrassing typos around user_directory and add some doc.
2017-11-07 17:31:11 +00:00
f5cf3638e9
move _state_group_cache to statestore
...
this is internal to statestore, so let's keep it there.
2017-11-07 16:43:00 +00:00
38b265cb51
Remember to pick is_admin out of the db
2017-11-07 11:24:04 +00:00
631fa4a1b7
create new indexes before dropping old ones to keep safetynet in place
2017-11-07 10:41:55 +00:00
4ad883398f
s/users_in_pubic_room/users_in_public_rooms/g
2017-11-04 19:39:40 +00:00
d802e8ca6a
s/users_in_pubic_room/users_in_public_rooms/g
2017-11-04 19:38:13 +00:00
a100700630
fix copyright....
2017-11-04 19:35:49 +00:00
b6b075fd49
s/popualte/populate/
2017-11-04 19:35:33 +00:00
Erik Johnston