Commit Graph

2593 Commits

Author SHA1 Message Date
Erik Johnston 7ec8e798b4 Fix media admin APIs 2018-04-26 11:31:22 +01:00
Neil Johnson fb6015d0a6 pep8 2018-04-25 17:56:11 +01:00
Neil Johnson 617bf40924 Generate user daily stats 2018-04-25 17:37:29 +01:00
Erik Johnston 22881b3d69 Also fix reindexing of search 2018-04-25 15:32:04 +01:00
Erik Johnston ba3166743c Fix quarantine media admin API 2018-04-25 15:11:18 +01:00
Matthew Hodgson e3a373f002 remove duplicates from groups tables
and rename inconsistently named indexes.
Based on https://github.com/matrix-org/synapse/pull/3128 - thanks @vurpo\!
2018-04-25 14:58:43 +01:00
Richard van der Hoff c09a6daf09
Merge pull request #3110 from NotAFile/py3-six-queue
Replace Queue with six.moves.queue
2018-04-20 15:35:00 +01:00
Richard van der Hoff bc381d5798
Merge pull request #3117 from matrix-org/rav/refactor_have_events
Refactor store.have_events
2018-04-20 10:26:12 +01:00
Richard van der Hoff b1dfbc3c40 Refactor store.have_events
It turns out that most of the time we were calling have_events, we were only
using half of the result. Replace have_events with have_seen_events and
get_rejection_reasons, so that we can see what's going on a bit more clearly.
2018-04-20 10:25:56 +01:00
Richard van der Hoff 639480e14a Avoid creating events with huge numbers of prev_events
In most cases, we limit the number of prev_events for a given event to 10
events. This fixes a particular code path which created events with huge
numbers of prev_events.
2018-04-16 18:41:37 +01:00
Adrian Tschira 878995e660 Replace Queue with six.moves.queue
and a six.range change which I missed the last time

Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-16 00:46:21 +02:00
Erik Johnston 415aeefd89 Format docstring 2018-04-12 12:07:09 +01:00
Erik Johnston 23a7f9d7f4 Doc we raise on unknown event 2018-04-12 11:20:51 +01:00
Erik Johnston 4dae4a97ed Track last processed event received_ts 2018-04-11 14:27:09 +01:00
Erik Johnston 92e34615c5 Track where event stream processing have gotten up to 2018-04-11 12:13:40 +01:00
Richard van der Hoff 233699c42e
Merge pull request #2760 from Valodim/pypy
Synapse on PyPy
2018-04-11 11:20:01 +01:00
Neil Johnson d54cfbb7a8 fix typo 2018-04-10 17:38:16 +01:00
Richard van der Hoff d1e56cfcd1 Fix pep8 error on psycopg2cffi hack 2018-04-10 11:35:29 +02:00
Vincent Breitmoser 89de934981 Use psycopg2cffi module instead of psycopg2 if running on pypy
The psycopg2 package isn't available for PyPy.  This commit adds a check
if the runtime is PyPy, and if it is uses psycopg2cffi module in favor
of psycopg2. This is almost a drop-in replacement, except for one place
where an additional cast to string is required.
2018-04-10 11:29:52 +02:00
Neil Johnson 41e0611895 remove errant print 2018-04-09 18:44:20 +01:00
Neil Johnson 61b439c904 Fix msec to sec, again 2018-04-09 18:43:48 +01:00
Neil Johnson 87770300d5 Fix msec to sec 2018-04-09 18:38:59 +01:00
Neil Johnson 64bc2162ef Fix psycopg2 interpolation 2018-04-09 17:50:36 +01:00
Luke Barnard cd087a265d Don't use redundant inlineCallbacks 2018-04-06 15:43:27 +01:00
Luke Barnard 87c864b698 join_rule -> join_policy 2018-04-06 15:43:27 +01:00
Luke Barnard ae85c7804e is_joinable -> join_rule 2018-04-06 15:43:27 +01:00
David Baker edb45aae38 pep8 2018-04-06 15:43:27 +01:00
David Baker b370fe61c0 Implement group join API 2018-04-06 15:43:27 +01:00
Luke Barnard e089100c62
Merge pull request #3045 from matrix-org/dbkr/group_joinable
Add joinability for groups
2018-04-05 15:57:49 +01:00
Neil Johnson 68b0ee4e8d
Merge pull request #3041 from matrix-org/r30_stats
R30 stats
2018-04-05 15:37:37 +01:00
Richard van der Hoff 22284a6f65
Merge pull request #3060 from matrix-org/rav/kill_event_content
Remove uses of events.content
2018-04-05 15:02:17 +01:00
Luke Barnard 917380e89d NON NULL -> NOT NULL 2018-04-05 14:32:12 +01:00
Luke Barnard 700e5e7198 Use DEFAULT join_policy of "invite" in db 2018-04-05 14:01:17 +01:00
Luke Barnard b214a04ffc Document set_group_join_policy 2018-04-05 13:29:16 +01:00
Neil Johnson 0e5f479fc0 Review comments
Use iteritems over item to loop over dict
formatting
2018-04-05 12:16:46 +01:00
Luke Barnard eb8d8d6f57 Use join_policy API instead of joinable
The API is now under
 /groups/$group_id/setting/m.join_policy

and expects a JSON blob of the shape

```json
{
  "m.join_policy": {
    "type": "invite"
  }
}
```

where "invite" could alternatively be "open".
2018-04-03 16:16:40 +01:00
Richard van der Hoff 8da39ad98f
Merge pull request #3049 from matrix-org/rav/use_staticjson
Use static JSONEncoders
2018-04-03 15:18:32 +01:00
Richard van der Hoff 2fe3f848b9 Remove uses of events.content 2018-03-29 23:17:12 +01:00
Richard van der Hoff 05630758f2 Use static JSONEncoders
using json.dumps with custom options requires us to create a new JSONEncoder on
each call. It's more efficient to create one upfront and reuse it.
2018-03-29 23:13:33 +01:00
Richard van der Hoff fcfe7f6ad3 Use simplejson throughout
Let's use simplejson rather than json, for consistency.
2018-03-29 22:45:52 +01:00
Neil Johnson b4e37c6f50 pep8 2018-03-29 17:27:39 +01:00
Neil Johnson 9ee44a372d Remove need for sqlite specific query 2018-03-29 16:45:34 +01:00
Neil Johnson dc7c020b33 fix pep8 errors 2018-03-28 17:25:15 +01:00
Neil Johnson 241e4e8687 remove twisted deferral cruft 2018-03-28 16:25:53 +01:00
David Baker 929b34963d OK, smallint it is then 2018-03-28 14:53:55 +01:00
David Baker a838444a70 Grr. Copy the definition from is_admin 2018-03-28 14:50:30 +01:00
Neil Johnson 4262aba17b bump schema version 2018-03-28 14:40:03 +01:00
Neil Johnson 86932be2cb Support multi client R30 for psql 2018-03-28 14:36:53 +01:00
David Baker a164270833 Make column definition that works on both dbs 2018-03-28 14:23:00 +01:00
David Baker 352e1ff9ed Add schema delta file 2018-03-28 14:07:57 +01:00
David Baker 79452edeee Add joinability for groups
Adds API to set the 'joinable' flag, and corresponding flag in the
table.
2018-03-28 14:03:37 +01:00
Neil Johnson 788e69098c Add user_ips last seen index 2018-03-28 12:03:13 +01:00
Neil Johnson 0f890f477e No need to cast in count_daily_users 2018-03-28 11:49:57 +01:00
Neil Johnson 545001b9e4 Fix search_user_dir multiple sqlite versions do different things 2018-03-28 11:19:45 +01:00
Neil Johnson a32d2548d9 query and call for r30 stats 2018-03-28 10:39:13 +01:00
Neil Johnson 9187e0762f count_daily_users failed if db was sqlite due to type failure - presumably this prevcented all sqlite homeservers reporting home 2018-03-28 10:02:32 +01:00
Erik Johnston f879127aaa
Merge pull request #3029 from matrix-org/erikj/linearize_generate_user_id
Linearize calls to _generate_user_id
2018-03-28 10:00:31 +01:00
Erik Johnston e6d87c93f3
Merge pull request #3030 from matrix-org/erikj/no_ujson
Remove last usage of ujson
2018-03-28 10:00:06 +01:00
Erik Johnston 800cfd5774 Comment 2018-03-27 13:30:39 +01:00
Erik Johnston 152c2ac19e Fix indent 2018-03-27 13:13:46 +01:00
Erik Johnston e70287cff3 Comment 2018-03-27 13:13:38 +01:00
Erik Johnston 3f49e131d9 Add counter metrics for calculating state delta
This will allow us to measure how often we calculate state deltas in
event persistence that we would have been able to calculate at the same
time we calculated the state for the event.
2018-03-27 10:57:35 +01:00
Erik Johnston fecb45e0c3 Remove last usage of ujson 2018-03-26 13:32:29 +01:00
Erik Johnston 8d6dc106d1 Don't use _cursor_to_dict in find_next_generated_user_id_localpart 2018-03-26 12:02:44 +01:00
Erik Johnston fa72803490 Merge branch 'master' of github.com:matrix-org/synapse into develop 2018-03-19 11:41:01 +00:00
Erik Johnston a8ce159be4 Replace some ujson with simplejson to make it work 2018-03-16 00:27:09 +00:00
Erik Johnston 5b631ff41a Remove wrong comment 2018-03-16 00:07:08 +00:00
Erik Johnston 926ba76e23 Replace ujson with simplejson 2018-03-15 23:43:31 +00:00
Erik Johnston ebd0127999
Merge pull request #2988 from matrix-org/erikj/split_profile_store
Split up ProfileStore
2018-03-14 09:41:06 +00:00
Erik Johnston f51565e023
Merge pull request #2993 from matrix-org/erikj/is_blocked
Add is_blocked to worker store
2018-03-14 09:39:18 +00:00
Matthew Hodgson d144ed6ffb
fix bug #2926 (loading all state for a given type from the DB if the state_key is None) (#2990)
Fixes a regression that had crept in where the caching layer upholds requests for loading state which is filtered by type (but not by state_key), but the DB layer itself would interpret a missing state_key as a request to filter by null state_key rather than returning all state_keys.
2018-03-13 22:36:04 +00:00
Erik Johnston a08726fc42 Add is_blocked to worker store 2018-03-13 18:28:44 +00:00
Erik Johnston 3518d0ea8f Split up ProfileStore 2018-03-13 17:36:50 +00:00
Richard van der Hoff 889a2a853a Add Measure block for persist_events
This seems like a useful thing to measure.
2018-03-13 10:01:42 +00:00
Erik Johnston 8cb44da4aa Fix race in sync when joining room
The race happens when the user joins a room at the same time as doing a
sync. We fetch the current token and then get the rooms the user is in.
If the join happens after the current token, but before we get the rooms
we end up sending down a partial room entry in the sync.

This is fixed by looking at the stream ordering of the membership
returned by get_rooms_for_user, and handling the case when that stream
ordering is after the current token.
2018-03-07 11:55:31 +00:00
Richard van der Hoff b2932107bb
Merge pull request #2946 from matrix-org/rav/timestamp_to_purge
Implement purge_history by timestamp
2018-03-06 11:20:23 +00:00
Erik Johnston 7aed50a038
Merge pull request #2948 from matrix-org/erikj/kill_as_sync
Remove ability for AS users to call /events and /sync
2018-03-06 11:10:09 +00:00
Erik Johnston b6c4b851f1
Merge pull request #2947 from matrix-org/erikj/split_directory_store
Split Directory store
2018-03-05 18:17:32 +00:00
Erik Johnston d4ffe61d4f Remove ability for AS users to call /events and /sync
This functionality has been deprecated for a while as well as being
broken for a while. Instead of fixing it lets just remove it entirely.

See: https://github.com/matrix-org/matrix-doc/issues/1144
2018-03-05 15:44:46 +00:00
Erik Johnston 69ce365b79 Fix cache invalidation on deletion 2018-03-05 15:29:03 +00:00
Erik Johnston 2e223163ff Split Directory store 2018-03-05 15:11:30 +00:00
Richard van der Hoff f8bfcd7e0d Provide a means to pass a timestamp to purge_history 2018-03-05 14:37:23 +00:00
Richard van der Hoff d032785aa7
Merge pull request #2943 from matrix-org/rav/fix_find_first_stream_ordering_after_ts
Test and fix find_first_stream_ordering_after_ts
2018-03-05 12:26:14 +00:00
Richard van der Hoff c818fcab11 Test and fix find_first_stream_ordering_after_ts
It seemed to suffer from a bunch of off-by-one errors.
2018-03-05 12:04:02 +00:00
Richard van der Hoff 06a14876e5 Add find_first_stream_ordering_after_ts
Expose this as a public function which can be called outside a txn
2018-03-05 11:53:39 +00:00
Erik Johnston 42174946f8
Merge pull request #2934 from matrix-org/erikj/cache_fix
Fix bug with delayed cache invalidation stream
2018-03-05 11:33:17 +00:00
Erik Johnston efb79820b4 Fix bug with delayed cache invalidation stream
We poked the notifier before updated the current token for the cache
invalidation stream. This mean that sometimes the update wouldn't be
sent until the next time a cache was invalidated.
2018-03-02 14:45:15 +00:00
Erik Johnston fafa3e7114 Split registration store 2018-03-02 13:48:27 +00:00
Erik Johnston 1773df0632
Merge pull request #2925 from matrix-org/erikj/split_sig_fed
Split out SignatureStore and EventFederationStore
2018-03-01 17:32:58 +00:00
Erik Johnston 9e08a93a7b
Merge pull request #2927 from matrix-org/erikj/read_marker_caches
Improve caching for read_marker API
2018-03-01 17:12:34 +00:00
Erik Johnston 4b44f05f19 Fewer lies are better 2018-03-01 17:08:17 +00:00
Erik Johnston a83c514d1f Improve caching for read_marker API
We add a new storage function to get a paritcular type of room account
data. This allows us to prefill the cache when updating that acount
data.
2018-03-01 17:08:17 +00:00
Erik Johnston 33bebb63f3 Add some caches to help read marker API 2018-03-01 17:08:17 +00:00
Erik Johnston 2ad4d5b5bb Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fed 2018-03-01 16:59:39 +00:00
Erik Johnston 7c371834cc Stub out broken function only used for cache 2018-03-01 16:44:13 +00:00
Erik Johnston 64346be26d Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_stream_store 2018-03-01 16:26:42 +00:00
Erik Johnston 22518e2833
Merge pull request #2923 from matrix-org/erikj/stream_ago_worker
Calculate stream_ordering_month_ago correctly on workers
2018-03-01 16:23:54 +00:00
Erik Johnston 884b26ae41 Remove unused variables 2018-03-01 16:23:48 +00:00
Erik Johnston 1b2af11650 Document abstract class and method better 2018-03-01 16:20:57 +00:00
Erik Johnston 872ff95ed4 Default stream_ordering_*_ago to None 2018-03-01 16:00:05 +00:00
Erik Johnston 22004b524e Fix comment typo 2018-03-01 15:59:40 +00:00
Erik Johnston f793bc3877 Split out stream store 2018-03-01 15:13:08 +00:00
Erik Johnston 6411f725be Calculate stream_ordering_month_ago correctly on workers 2018-03-01 14:20:53 +00:00
Erik Johnston a9a2d66cdd Split out SignatureStore and EventFederationStore 2018-03-01 14:17:53 +00:00
Erik Johnston 0c8ba5dd1c Split up RoomStore 2018-03-01 14:01:19 +00:00
Erik Johnston 493e25d554 Move storage functions for push calculations
This will allow push actions for an event to be calculated on workers.
2018-02-27 13:58:16 +00:00
Erik Johnston 3594dbc6dc
Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidation
Fix missing invalidations for receipt storage
2018-02-27 11:34:26 +00:00
Erik Johnston 2311189ee4
Merge pull request #2903 from matrix-org/erikj/split_roommember_store
Split out RoomMemberStore
2018-02-27 11:32:10 +00:00
Erik Johnston c57607874c
Merge pull request #2901 from matrix-org/erikj/split_as_stores
Split AS stores
2018-02-27 10:07:07 +00:00
Erik Johnston 8956f0147a Add comment 2018-02-27 10:06:51 +00:00
Erik Johnston e5b4a208ce
Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions
Batch inserts into event_push_actions_staging
2018-02-26 14:45:40 +00:00
Erik Johnston 45b5fe9122 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/handle_unpersisted_events_push 2018-02-26 13:49:24 +00:00
Erik Johnston d62ce972f8 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_roommember_store 2018-02-23 11:46:24 +00:00
Erik Johnston 6ae9a3d2a6 Update copyright 2018-02-23 11:44:49 +00:00
Erik Johnston a90c60912f Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_event_push_actions 2018-02-23 11:26:31 +00:00
Erik Johnston 50e8657867
Merge pull request #2902 from matrix-org/erikj/split_events_store
Split out get_events and co into a worker store
2018-02-23 11:23:52 +00:00
Erik Johnston 1cf9e071dd
Merge pull request #2899 from matrix-org/erikj/split_pushers
Split PusherStore
2018-02-23 11:23:35 +00:00
Erik Johnston d0957753bf
Merge pull request #2898 from matrix-org/erikj/split_push_rules_store
Split PushRulesStore
2018-02-23 11:23:23 +00:00
Erik Johnston 70349872c2 Update copyright 2018-02-23 11:14:35 +00:00
Erik Johnston eba93b05bf Split EventsWorkerStore into separate file 2018-02-23 11:01:21 +00:00
Erik Johnston bf8a36e080 Update copyright 2018-02-23 10:52:10 +00:00
Erik Johnston 5d0f665848 Remove redundant clock 2018-02-23 10:49:58 +00:00
Erik Johnston 3bd760628b _event_persist_queue shouldn't be in worker store 2018-02-23 10:49:18 +00:00
Erik Johnston eb9b5eec81 Update copyright 2018-02-23 10:42:39 +00:00
Erik Johnston c2ecfcc3a4 Update copyright 2018-02-23 10:41:34 +00:00
Erik Johnston 7e6cf89dc2 Update copyright 2018-02-23 10:39:19 +00:00
Erik Johnston 26d37f7a63 Update copyright 2018-02-23 10:33:55 +00:00
Erik Johnston bb73f55fc6 Use absolute imports 2018-02-23 10:31:16 +00:00
Erik Johnston faeb369f15 Fix missing invalidations for receipt storage 2018-02-21 15:19:54 +00:00
Erik Johnston 3dec9c66b3 Split out RoomMemberStore 2018-02-21 12:07:26 +00:00
Erik Johnston 46244b2759 Split AS stores 2018-02-21 11:49:34 +00:00
Erik Johnston 27b094f382 Split out get_events and co into a worker store 2018-02-21 11:41:48 +00:00
Erik Johnston c96d547f4d Actually use new param 2018-02-21 11:03:42 +00:00
Erik Johnston d15d237b0d Split out EventPushActionWorkerStore 2018-02-21 11:01:13 +00:00
Erik Johnston 6f72765371 Split PusherStore 2018-02-21 10:54:21 +00:00
Erik Johnston cbaad969f9 Split PushRulesStore 2018-02-21 10:43:31 +00:00
Erik Johnston ca9b9d9703 Split AccountDataStore and TagStore 2018-02-21 10:15:04 +00:00
Erik Johnston 8fbb4d0d19 Raise exception in abstract method 2018-02-20 17:59:23 +00:00
Erik Johnston 95e4cffd85 Fix comment 2018-02-20 17:58:40 +00:00
Erik Johnston e316bbb4c0 Use abstract base class to access stream IDs 2018-02-20 17:43:57 +00:00
Erik Johnston f5ac4dc2d4 Split ReceiptsStore 2018-02-20 16:28:28 +00:00
Erik Johnston 24087bffa9 Ensure all push actions are deleted from staging 2018-02-20 12:34:31 +00:00
Erik Johnston ad0ccf15ea Refactor _set_push_actions_for_event_and_users_txn to use events_and_contexts 2018-02-20 12:34:28 +00:00
Erik Johnston 6ff8c87484 Batch inserts into event_push_actions_staging 2018-02-20 11:33:07 +00:00
Richard van der Hoff d1a3325f99 (Really) fix tablescan of event_push_actions on purge
commit 278d21b5 added new code to avoid the tablescan, but didn't remove the
old :/
2018-02-16 14:02:31 +00:00
Erik Johnston 6af025d3c4 Fix typo of double is_highlight 2018-02-16 11:35:31 +00:00
Erik Johnston 012e8e142a Comments 2018-02-16 11:35:01 +00:00
Erik Johnston b96278d6fe Ensure that we delete staging push actions on errors 2018-02-15 15:47:06 +00:00
Erik Johnston 4810f7effd Remove context.push_actions 2018-02-15 15:47:06 +00:00
Erik Johnston c714c61853 Update event_push_actions table from staging table 2018-02-15 15:47:06 +00:00
Erik Johnston acac21248c Store push actions in staging area 2018-02-15 15:47:04 +00:00
Erik Johnston 106906a65e Don't serialize current state over replication 2018-02-15 13:53:18 +00:00
Erik Johnston fd1601c596 Fix state group storage bug in workers
We needed to move `_count_state_group_hops_txn` to the
StateGroupWorkerStore.
2018-02-15 11:04:32 +00:00
Richard van der Hoff b8d821aa68
Merge pull request #2867 from matrix-org/rav/rework_purge
purge_history cleanups
2018-02-15 09:49:07 +00:00
Richard van der Hoff d28ec43e15
Merge pull request #2769 from matrix-org/matthew/hit_the_gin
switch back from GIST to GIN indexes
2018-02-14 16:59:03 +00:00
Richard van der Hoff 39bf47319f purge_history: fix sqlite syntax error
apparently sqlite insists on indexes being named
2018-02-14 16:42:19 +00:00
Richard van der Hoff ac27f6a35e purge_history: handle sqlite asshattery
apparently creating a temporary table commits the transaction. because that's a
useful thing.
2018-02-14 16:41:12 +00:00
Richard van der Hoff 5978dccff0 remove overzealous exception handling 2018-02-14 15:54:09 +00:00
Richard van der Hoff 278d21b5e4 purge_history: fix index use
event_push_actions doesn't have an index on event_id, so we need to specify
room_id.
2018-02-14 15:44:51 +00:00
Richard van der Hoff 5fcbf1e07c Rework event purge to use a temporary table
... which should speed things up by reducing the amount of data being shuffled
across the connection
2018-02-14 11:02:22 +00:00
Erik Johnston c0c9327fe0
Merge pull request #2854 from matrix-org/erikj/event_create_worker
Create a worker for event creation
2018-02-13 18:07:10 +00:00
Richard van der Hoff d627174da2 Fix log message in purge_history
(we don't just remove remote events)
2018-02-13 16:51:21 +00:00
Richard van der Hoff ddb6a79b68 Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin 2018-02-13 16:45:36 +00:00
Richard van der Hoff 0b27ae8dc3 move search reindex to schema 47
We're up to schema v47 on develop now, so this will have to go in there to have
an effect.

This might cause an error if somebody has already run it in the v46 guise, and
runs it again in the v47 guise, because it will cause a duplicate entry in the
bbackground_updates table. On the other hand, the entry is removed once it is
complete, and it is unlikely that anyone other than matrix.org has run it on
v46. The update itself is harmless to re-run because it deliberately copes with
the index already existing.
2018-02-13 16:44:46 +00:00
Richard van der Hoff 4a6d551704 GIN reindex: Fix syntax errors, improve exception handling 2018-02-13 16:44:46 +00:00
Richard van der Hoff bfdf7b9237
Merge pull request #2864 from matrix-org/rav/persist_event_caching
Use StateResolutionHandler to resolve state in persist_events
2018-02-13 14:45:57 +00:00
Richard van der Hoff 630caf8a70
style nit 2018-02-13 14:29:22 +00:00
Richard van der Hoff a9b712e9dc Merge branch 'develop' into matthew/gin_work_mem 2018-02-13 12:16:01 +00:00
Richard van der Hoff 10b34dbb9a
Merge pull request #2858 from matrix-org/rav/purge_updates
delete_local_events for purge_room_history
2018-02-09 14:11:00 +00:00
Richard van der Hoff 39a6b35496 purge: move room_depth update to end
... to avoid locking the table for too long
2018-02-09 13:07:41 +00:00
Richard van der Hoff 74fcbf741b delete_local_events for purge_history
Add a flag which makes the purger delete local events
2018-02-09 13:07:41 +00:00
Richard van der Hoff e571aef06d purge: Move cache invalidation to more appropriate place
it was a bit of a non-sequitur there
2018-02-09 13:07:41 +00:00
Richard van der Hoff 61ffaa8137 bump purge logging to info
this thing takes ages and the only sign of any progress is the logs, so having
some logs is useful.
2018-02-09 13:07:41 +00:00
Richard van der Hoff 671540dccf rename delete_old_state -> purge_history
(beacause it deletes more than state)
2018-02-09 13:07:41 +00:00
Erik Johnston 24dd73028a Add replication http endpoint for event sending 2018-02-07 10:32:32 +00:00
Erik Johnston 3d33eef6fc
Store state groups separately from events (#2784)
* Split state group persist into seperate storage func

* Add per database engine code for state group id gen

* Move store_state_group to StateReadStore

This allows other workers to use it, and so resolve state.

* Hook up store_state_group

* Fix tests

* Rename _store_mult_state_groups_txn

* Rename StateGroupReadStore

* Remove redundant _have_persisted_state_group_txn

* Update comments

* Comment compute_event_context

* Set start val for state_group_id_seq

... otherwise we try to recreate old state groups

* Update comments

* Don't store state for outliers

* Update comment

* Update docstring as state groups are ints
2018-02-06 14:31:24 +00:00
Richard van der Hoff b31bf0bb51
Merge pull request #2849 from matrix-org/rav/clean_up_state_delta
Remove redundant return value from _calculate_state_delta
2018-02-05 17:42:20 +01:00
Richard van der Hoff ebfe64e3d6 Use StateResolutionHandler to resolve state in persist events
... and thus benefit (hopefully) from its cache.
2018-02-05 16:23:26 +00:00
Richard van der Hoff 225dc3b4cb Flatten _get_new_state_after_events
rejig the if statements to simplify the logic and reduce indentation
2018-02-05 16:23:25 +00:00
Richard van der Hoff 9fcbbe8e7d Check that events being persisted have state_group 2018-02-05 16:23:25 +00:00
Richard van der Hoff 447aed42d2 Add event_map param to resolve_state_groups 2018-02-05 16:23:25 +00:00
Richard van der Hoff ee6fb4cf85 Remove redundant return value from _calculate_state_delta
we already have the state from _get_new_state_after_events, so returning it
from _calculate_state_delta is just confusing.
2018-02-05 16:23:20 +00:00
Richard van der Hoff 3c7b480ba3 Factor out common code for search insert
we can reuse the same code as is used for event insert, for doing the
background index population.
2018-02-05 16:12:14 +00:00
Richard van der Hoff c46e75d3d8 Move store_event_search_txn to SearchStore
... as a precursor to making event storing and doing the bg update share some
code.
2018-02-05 15:43:22 +00:00
Richard van der Hoff 80b8a28100 Factor out common code for search insert
we can reuse the same code as is used for event insert, for doing the
background index population.
2018-02-04 00:23:06 +00:00
Richard van der Hoff bd25f9cf36 Clean up work_mem handling
Add some comments and improve exception handling when twiddling work_mem for
the search update
2018-02-03 23:05:41 +00:00
Richard van der Hoff 4eeae7ad65 Move store_event_search_txn to SearchStore
... as a precursor to making event storing and doing the bg update share some
code.
2018-02-03 22:59:45 +00:00
Richard van der Hoff bb9f0f3cdb Merge branch 'develop' into matthew/gin_work_mem 2018-02-03 22:40:28 +00:00
Richard van der Hoff 6b02fc80d1 Reinstate event_search_postgres_gist handler
People may have queued updates for this, so we can't just delete it.
2018-02-02 14:32:51 +00:00
Richard van der Hoff 78d6ddba86
Merge pull request #2841 from matrix-org/rav/refactor_calc_state_delta
factor _get_new_state_after_events out of _calculate_state_delta
2018-02-02 15:27:15 +01:00
Travis Ralston 6e87b34f7b
Merge branch 'develop' into travis/admin-list-media 2018-02-01 18:05:47 -07:00
Richard van der Hoff 77c0629ebc
Merge pull request #2837 from matrix-org/rav/fix_quarantine_media
Fix sql error in quarantine_media
2018-02-01 11:45:58 +01:00
Travis Ralston e16e45b1b4 pep8
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-31 15:30:38 -07:00
Richard van der Hoff e1e4ec9f9d factor _get_new_state_after_events out of _calculate_state_delta
This reduces the scope of a bunch of variables
2018-01-31 21:32:09 +00:00
Travis Ralston 63c4383927 Documentation and naming
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-31 08:07:52 -07:00
Richard van der Hoff 773f0eed1e Fix sql error in quarantine_media 2018-01-30 15:02:51 +00:00
Richard van der Hoff b387ee17b6 Improve exception handling in persist_event
1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that
a Failure object will be constructed using the current exception state,
*including* its stack trace - so the stack trace is saved in the Failure,
leading to better exception reports.

2. Set `consumeErrors=True` on the ObservableDeferred, because we know that
there will always be at least one observer - which avoids a spurious "CRITICAL:
unhandled exception in Deferred" error in the logs
2018-01-29 17:05:33 +00:00
Richard van der Hoff 46022025ea Fix SQL for user search
fix some syntax errors for user search when search_all_users is enabled

fixes #2801, hopefully
2018-01-27 17:21:57 +00:00
Travis Ralston a94d9b6b82 Appease the linter
These are ids anyways, not mxc uris.

Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-20 22:49:46 -07:00
Travis Ralston 5552ed9a7f Add an admin route to get all the media in a room
This is intended to be used by administrators to monitor the media that is passing through their server, if they wish.

Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-20 22:37:53 -07:00