Commit Graph

115 Commits

Author SHA1 Message Date
Amber Brown 32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Amber Brown 7efd1d87c2 Run black on the rest of the storage module (#4996) 2019-04-03 10:07:29 +01:00
Erik Johnston 5bec8d660d Make starting pushers faster during start up
We start all pushers on start up and immediately start a background
process to fetch push to send. This makes start up incredibly painful
when dealing with many pushers.

Instead, let's do a quick fast DB check to see if there *may* be push to
send and only start the background processes for those pushers. We also
stagger starting up and doing those checks so that we don't try and
handle all pushers at once.
2019-04-02 16:59:13 +01:00
Richard van der Hoff 03751a6420 Fix some looping_call calls which were broken in #3604
It turns out that looping_call does check the deferred returned by its
callback, and (at least in the case of client_ips), we were relying on this,
and I broke it in #3604.

Update run_as_background_process to return the deferred, and make sure we
return it to clock.looping_call.
2018-07-26 11:48:08 +01:00
Richard van der Hoff 371da42ae4 Wrap a number of things that run in the background
This will reduce the number of "Starting db connection from sentinel context"
warnings, and will help with our metrics.
2018-07-25 09:41:12 +01:00
Amber Brown 49af402019 run isort 2018-07-09 16:09:20 +10:00
Amber Brown 6350bf925e
Attempt to be more performant on PyPy (#3462) 2018-06-28 14:49:57 +01:00
Erik Johnston 75dc3ddeab Make push actions rotation configurable 2018-06-22 14:44:37 +01:00
Amber Brown 77ac14b960
Pass around the reactor explicitly (#3385) 2018-06-22 09:37:10 +01:00
Adrian Tschira 933bf2dd35 replace some iteritems with six
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:59:26 +02:00
Erik Johnston 6406b70aeb Use stream rather depth ordering for push actions
This simplifies things as it is, but will also allow us to change the
way we traverse topologically without having to update the way push
actions work.
2018-05-11 15:30:11 +01:00
Richard van der Hoff 9255a6cb17 Improve exception handling for background processes
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.

This is unsatisfactory for a number of reasons:
 - logging on garbage collection is best-effort and may happen some time after
   the error, if at all
 - it can be hard to figure out where the error actually happened.
 - it is logged as a scary CRITICAL error which (a) I always forget to grep for
   and (b) it's not really CRITICAL if a background process we don't care about
   fails.

So this is an attempt to add exception handling to everything we fire off into
the background.
2018-04-27 11:07:40 +01:00
Erik Johnston fa72803490 Merge branch 'master' of github.com:matrix-org/synapse into develop 2018-03-19 11:41:01 +00:00
Erik Johnston 926ba76e23 Replace ujson with simplejson 2018-03-15 23:43:31 +00:00
Richard van der Hoff d032785aa7
Merge pull request #2943 from matrix-org/rav/fix_find_first_stream_ordering_after_ts
Test and fix find_first_stream_ordering_after_ts
2018-03-05 12:26:14 +00:00
Richard van der Hoff c818fcab11 Test and fix find_first_stream_ordering_after_ts
It seemed to suffer from a bunch of off-by-one errors.
2018-03-05 12:04:02 +00:00
Richard van der Hoff 06a14876e5 Add find_first_stream_ordering_after_ts
Expose this as a public function which can be called outside a txn
2018-03-05 11:53:39 +00:00
Erik Johnston efb79820b4 Fix bug with delayed cache invalidation stream
We poked the notifier before updated the current token for the cache
invalidation stream. This mean that sometimes the update wouldn't be
sent until the next time a cache was invalidated.
2018-03-02 14:45:15 +00:00
Erik Johnston 872ff95ed4 Default stream_ordering_*_ago to None 2018-03-01 16:00:05 +00:00
Erik Johnston 22004b524e Fix comment typo 2018-03-01 15:59:40 +00:00
Erik Johnston 6411f725be Calculate stream_ordering_month_ago correctly on workers 2018-03-01 14:20:53 +00:00
Erik Johnston 493e25d554 Move storage functions for push calculations
This will allow push actions for an event to be calculated on workers.
2018-02-27 13:58:16 +00:00
Erik Johnston e5b4a208ce
Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions
Batch inserts into event_push_actions_staging
2018-02-26 14:45:40 +00:00
Erik Johnston 45b5fe9122 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/handle_unpersisted_events_push 2018-02-26 13:49:24 +00:00
Erik Johnston eb9b5eec81 Update copyright 2018-02-23 10:42:39 +00:00
Erik Johnston c96d547f4d Actually use new param 2018-02-21 11:03:42 +00:00
Erik Johnston d15d237b0d Split out EventPushActionWorkerStore 2018-02-21 11:01:13 +00:00
Erik Johnston 24087bffa9 Ensure all push actions are deleted from staging 2018-02-20 12:34:31 +00:00
Erik Johnston ad0ccf15ea Refactor _set_push_actions_for_event_and_users_txn to use events_and_contexts 2018-02-20 12:34:28 +00:00
Erik Johnston 6ff8c87484 Batch inserts into event_push_actions_staging 2018-02-20 11:33:07 +00:00
Erik Johnston 6af025d3c4 Fix typo of double is_highlight 2018-02-16 11:35:31 +00:00
Erik Johnston 012e8e142a Comments 2018-02-16 11:35:01 +00:00
Erik Johnston b96278d6fe Ensure that we delete staging push actions on errors 2018-02-15 15:47:06 +00:00
Erik Johnston c714c61853 Update event_push_actions table from staging table 2018-02-15 15:47:06 +00:00
Erik Johnston acac21248c Store push actions in staging area 2018-02-15 15:47:04 +00:00
Richard van der Hoff 6cfee09be9 Make __init__ consitstent across Store heirarchy
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
2017-11-13 10:46:07 +00:00
Erik Johnston 00957d1aa4 User Cursor.__iter__ instead of fetchall
This prevents unnecessary construction of lists
2017-03-23 17:53:49 +00:00
Erik Johnston b2d20e94fa Remove lock from rotate notifs 2017-02-22 14:24:02 +00:00
Erik Johnston b7442c3e2b Store looping call 2017-02-21 13:59:25 +00:00
Erik Johnston 699be7d1be Fix up notif rotation 2017-02-18 14:42:39 +00:00
Erik Johnston 138e030cfe Comment 2017-02-16 15:03:36 +00:00
Erik Johnston 502ae6c663 Comment 2017-02-16 14:47:11 +00:00
Erik Johnston e6acf0c399 Store the default push actions in a more efficient manner 2017-02-16 14:40:24 +00:00
Erik Johnston ce3c8df6df Less aggressive timers 2017-02-14 13:41:24 +00:00
Erik Johnston 095b45c165 Aggregate event push actions 2017-02-14 13:39:41 +00:00
Erik Johnston fac3c03087 Be more agressive about purging old room event_push_actions 2017-02-01 18:27:24 +00:00
Erik Johnston 1985860c6e Comment 2016-11-23 15:59:59 +00:00
Erik Johnston 2ac516850b More efficient notif count queries 2016-11-23 15:57:04 +00:00
Erik Johnston b17af156c7 Remove where clause 2016-09-12 17:05:54 +01:00
Erik Johnston c94de0ab60 Add WHERE clause support to index creation 2016-09-12 16:55:01 +01:00