Commit Graph

40 Commits

Author SHA1 Message Date
Patrick Cloke 509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
Amber Brown 020add5099
Update black to 19.10b0 (#6304)
* update version of black and also fix the mypy config being overridden
2019-11-01 02:43:24 +11:00
Erik Johnston 96bdd661b8
Remove redundant return 2019-08-13 12:50:36 +01:00
Erik Johnston d02e41dcb2 Handle pusher being deleted during processing.
Instead of throwing a StoreError lets break out of processing loop and
mark the pusher as stopped.
2019-08-01 13:44:12 +01:00
Amber Brown 32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Erik Johnston 2ebeda48b2 Add test 2019-06-07 12:15:33 +01:00
Erik Johnston 5bec8d660d Make starting pushers faster during start up
We start all pushers on start up and immediately start a background
process to fetch push to send. This makes start up incredibly painful
when dealing with many pushers.

Instead, let's do a quick fast DB check to see if there *may* be push to
send and only start the background processes for those pushers. We also
stagger starting up and doing those checks so that we don't try and
handle all pushers at once.
2019-04-02 16:59:13 +01:00
Amber Brown 0dce9e1379
Write some tests for the email pusher (#4095) 2018-10-30 23:55:43 +11:00
Richard van der Hoff e564306e31 sanity-check the is_processing flag
... and rename it, for even more sanity
2018-10-24 09:23:33 +01:00
Richard van der Hoff f749607c91 Make on_started synchronous too
This brings it into line with on_new_notifications and on_new_receipts. It
requires a little bit of hoop-jumping in EmailPusher to load the throttle
params before the first loop.
2018-10-22 16:12:11 +01:00
Richard van der Hoff e7a16c6210 Remove redundant run_as_background_process() from pusherpool
`on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher`
now always return synchronously, so we can remove the `defer.gatherResults` on
their results, and the `run_as_background_process` wrappers can be removed too
because the PusherPool methods will now complete quickly enough.
2018-10-22 16:12:11 +01:00
Richard van der Hoff c7273c11bc Give pushers their own background logcontext
Each pusher has its own loop which runs for as long as it has work to do. This
should run in its own background thread with its own logcontext, as other
similar loops elsewhere in the system do - which means that CPU usage is
consistently attributed to that loop, rather than to whatever request happened
to start the loop.
2018-10-22 16:12:11 +01:00
Amber Brown 49af402019 run isort 2018-07-09 16:09:20 +10:00
Amber Brown 07cad26d65
Remove all global reactor imports & pass it around explicitly (#3424) 2018-06-25 14:08:28 +01:00
Richard van der Hoff 9255a6cb17 Improve exception handling for background processes
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.

This is unsatisfactory for a number of reasons:
 - logging on garbage collection is best-effort and may happen some time after
   the error, if at all
 - it can be hard to figure out where the error actually happened.
 - it is logged as a scary CRITICAL error which (a) I always forget to grep for
   and (b) it's not really CRITICAL if a background process we don't care about
   fails.

So this is an attempt to add exception handling to everything we fire off into
the background.
2018-04-27 11:07:40 +01:00
Richard van der Hoff eaaabc6c4f replace 'except:' with 'except Exception:'
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Erik Johnston 11c2a3655f Only load jinja2 templates once
Instead of every time a new email pusher is created, as loading jinja2
templates is slow.
2017-05-22 17:48:58 +01:00
Erik Johnston 7455ba436a Ensure we pass positive ints to delay function 2017-02-22 12:08:14 +00:00
Erik Johnston f0ca088280 Reduce redundant database work in email pusher
Update the last stream ordering if the
`get_unread_push_actions_for_user_in_range_for_email` returns no new
push actions. This reduces the range that it needs to check next
iteration.
2016-10-18 10:52:47 +01:00
Erik Johnston b260f92936 Ignore AlreadyCalled errors on timer cancel 2016-07-31 16:00:12 +01:00
Mark Haines 0a7d3cd00f Create separate methods for getting messages to push
for the email and http pushers rather than trying to make a single
method that will work with their conflicting requirements.

The http pusher needs to get the messages in ascending stream order, and
doesn't want to miss a message.

The email pusher needs to get the messages in descending timestamp order,
and doesn't mind if it misses messages.
2016-07-28 20:24:24 +01:00
David Baker 812b5de0fe Merge remote-tracking branch 'origin/develop' into dbkr/email_unsubscribe 2016-06-02 15:33:28 +01:00
Erik Johnston 356f13c069 Disable INCLUDE_ALL_UNREAD_NOTIFS 2016-06-02 14:07:38 +01:00
Matthew Hodgson f84b89f0c6 if an email pusher specifies a brand param, use it 2016-06-02 13:29:48 +01:00
David Baker a15ad60849 Email unsubscribing that may in theory, work
Were it not for that fact that you can't use the base handler in the pusher because it pulls in the world. Comitting while I fix that on a different branch.
2016-06-02 11:44:15 +01:00
Matthew Hodgson 989bdc9e56 Tune email notifs to make them quieter:
* After initial 10 minute window, only alert every 24h for room notifs
 * Reset room state after 6h of idleness
 * Synchronise throttles for messages sent in the same notif, so the 24 hourly notifs 'line up'
 * Fix the email subjects to say what triggered the notification
 * Order the rooms in reverse activity order in the email, so the 'reason' room should always come first
2016-05-23 19:24:11 +01:00
Matthew Hodgson cbd2adc95e tune email notifs, fix CSS a bit, and add debugging details 2016-05-16 18:58:38 +01:00
David Baker b084e4d963 Add constant for throttle multiplier 2016-04-29 20:14:55 +01:00
David Baker 35b7b8e4bc Remove unused function 2016-04-29 20:10:34 +01:00
David Baker 6b9b6a9169 Remove unused arg 2016-04-29 20:02:52 +01:00
David Baker 3facde2536 Remove rather pointless get function 2016-04-29 14:36:45 +01:00
David Baker 4364ea1272 Stop processing notifs once we've sent a mail 2016-04-29 14:31:27 +01:00
David Baker cc0874cf71 Put back real delay before mailing 2016-04-28 17:00:40 +01:00
David Baker 8781083960 Better grammar for multiple messages in a room
Say who the messages are from if there's no room name, otherwise it's a bit nonsensical
2016-04-27 15:30:41 +01:00
David Baker c553797c4f No inlineCallbacks necessary on this 2016-04-22 17:27:54 +01:00
David Baker c10ed26c30 Flesh out email templating
Mostly WIP porting the room name calculation logic from the web client so our room names in the email mirror the clients.
2016-04-21 19:19:07 +01:00
David Baker 2ed0adb075 Generate mails from a template 2016-04-20 18:35:29 +01:00
David Baker f63bd4ff47 Send a rather basic email notif
Also pep8 fixes
2016-04-20 13:02:01 +01:00
David Baker e2a01455af Add single instance & logging stuff
Copy the stuff over from http pusher that prevents multiple instances of process running at once and sets up logging and measure blocks.
2016-04-19 14:52:58 +01:00
David Baker 07d765209d First bits of emailpusher
Mostly logic of when to send an email
2016-04-19 14:24:36 +01:00