Commit Graph

270 Commits

Author SHA1 Message Date
Amber Brown a8990fa2ec Merge remote-tracking branch 'origin/develop' into 3218-official-prom 2018-05-22 10:50:26 -05:00
Richard van der Hoff 8810685df9 Stub out ServerNoticesSender on the workers
... and have the sync endpoints call it directly rather than obsure indirection
via PresenceHandler
2018-05-22 11:54:51 +01:00
Richard van der Hoff 9ea219c514 Send users a server notice about consent
When a user first syncs, we will send them a server notice asking them to
consent to the privacy policy if they have not already done so.
2018-05-22 11:54:51 +01:00
Amber Brown df9f72d9e5 replacing portions 2018-05-21 19:47:37 -05:00
Adrian Tschira 933bf2dd35 replace some iteritems with six
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:59:26 +02:00
Richard van der Hoff d43d480d86 Remove unused `update_external_syncs`
This method isn't used anywhere. Burninate it.
2018-05-17 18:22:19 +01:00
Richard van der Hoff 9255a6cb17 Improve exception handling for background processes
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.

This is unsatisfactory for a number of reasons:
 - logging on garbage collection is best-effort and may happen some time after
   the error, if at all
 - it can be hard to figure out where the error actually happened.
 - it is logged as a scary CRITICAL error which (a) I always forget to grep for
   and (b) it's not really CRITICAL if a background process we don't care about
   fails.

So this is an attempt to add exception handling to everything we fire off into
the background.
2018-04-27 11:07:40 +01:00
Erik Johnston 265b993b8a Split replication layer into two 2018-03-13 10:55:47 +00:00
Erik Johnston c3f79c9da5 Split out edu/query registration to a separate class 2018-03-13 10:24:27 +00:00
Matthew Hodgson 34c3d0a386 typo 2017-11-17 01:54:02 +00:00
Richard van der Hoff eaaabc6c4f replace 'except:' with 'except Exception:'
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Erik Johnston ec5c4499f4 Make presence use cached users/hosts in room 2017-05-16 16:01:43 +01:00
Erik Johnston c7ddb5ef7a Reuse get_interested_parties 2017-04-12 10:16:26 +01:00
Erik Johnston 414522aed5 Move get_interested_parties 2017-04-11 15:33:26 +01:00
Erik Johnston 2be8a281d2 Comments 2017-04-11 15:28:24 +01:00
Erik Johnston 6308ac45b0 Move get_interested_remotes back to presence handler 2017-04-11 15:19:26 +01:00
Erik Johnston 29574fd5b3 Reduce federation presence replication traffic
This is mainly done by moving the calculation of where to send presence
updates from the presence handler to the transaction queue, so we only
need to send the presence event (and not the destinations) across the
replication connection. Before we were duplicating by sending the full
state across once per destination.
2017-04-10 16:48:30 +01:00
Erik Johnston 877c029c16 Use iteritems 2017-04-06 15:51:22 +01:00
Erik Johnston 36d2b66f90 Add a timestamp to USER_SYNC command
This timestamp is used to indicate when the user last sync'd
2017-03-31 15:42:22 +01:00
Erik Johnston 9d0170ac6c Fix up presence 2017-03-31 11:36:32 +01:00
Erik Johnston 63fcc42990 Remove user from process_presence when stops syncing 2017-03-30 14:26:08 +01:00
Erik Johnston e9dd8370b0 Add functions to presence to support remote syncs
The TCP replication protocol streams deltas of who has started or
stopped syncing. This is different from the HTTP API which periodically
sends the full list of users who are syncing. This commit adds support
for the new TCP style of sending deltas.
2017-03-30 13:25:14 +01:00
Erik Johnston 7fc1f1e2b6 Cache hosts in room 2017-03-24 11:46:24 +00:00
Erik Johnston 6957bfdca6 Don't recreate so many sets 2017-03-16 11:54:26 +00:00
Erik Johnston 54d2b7e596 Merge pull request #2014 from Half-Shot/hs/fix-appservice-presence
Add fallback to last_active_ts if it beats the last sync time on a presence timeout.
2017-03-15 17:37:15 +00:00
Erik Johnston f83ac78201 Cache set of users whose presence the other user should see 2017-03-15 15:29:19 +00:00
Will Hunt e6032054bf Add a great comment to handle_timeout for active vs sync times. 2017-03-15 15:24:48 +00:00
Will Hunt ebf5a6b14c Add fallback to last_active_ts if it beats the last sync time. 2017-03-15 15:17:16 +00:00
Erik Johnston e892457a03 Comment 2017-03-15 15:01:39 +00:00
Erik Johnston 6c82de5100 Format presence events on the edges instead of reformatting them multiple times 2017-03-15 14:27:34 +00:00
Erik Johnston 9e617cd4c2 Cache get_presence storage 2017-02-13 13:50:03 +00:00
Erik Johnston 54a79c1d37 Make presence.get_new_events a bit faster
We do this by caching the set of users a user shares rooms with.
2017-02-02 13:07:18 +00:00
Erik Johnston fd1c18c088 Use DB cache of joined users for presence 2017-01-30 17:00:24 +00:00
Erik Johnston 59ef517e6b Use new federation_sender DI 2016-11-16 14:47:52 +00:00
Paul "LeoNerd" Evans ca35e54d6b Fix typo "persiting" 2016-09-13 13:26:33 +01:00
Erik Johnston 52b2318777 Clobber EDUs in send queue 2016-09-09 15:59:08 +01:00
Erik Johnston 6c4d582144 Deduplicate presence in _update_states 2016-09-09 14:28:22 +01:00
Erik Johnston 8b93af662d Check the user_id for presence/typing matches origin 2016-09-08 15:07:38 +01:00
Erik Johnston 3c4208a057 Record counts of state changes 2016-09-06 11:31:01 +01:00
Erik Johnston 438ef47637 Short circuit if presence is the same 2016-09-06 10:28:35 +01:00
Erik Johnston 74a3b4a650 Fiddle should_notify to better report stats 2016-09-06 10:23:38 +01:00
Erik Johnston 69054e3d4c Record why we have chosen to notify 2016-09-05 14:12:11 +01:00
Erik Johnston 265d847ffd Fix typo in log line 2016-09-01 14:50:06 +01:00
Erik Johnston c7f665d700 Merge pull request #1055 from matrix-org/erikj/occaisonally_persist
Occaisonally persist unpersisted presence updates
2016-08-30 15:59:00 +01:00
Erik Johnston 097330bae8 Check correct variable 2016-08-30 15:50:20 +01:00
Erik Johnston 21b977ccfe Occaisonally persist unpersisted presence updates 2016-08-30 15:39:50 +01:00
Erik Johnston bc1a8b1f7a Don't notify for online -> online transitions.
Specifically, if currently_active remains true then we should not notify
if only the last active time changes.
2016-08-30 15:05:32 +01:00
Erik Johnston bed10f9880 Use state handler instead of get_users_in_room/get_joined_hosts 2016-08-26 14:54:30 +01:00
Erik Johnston 4e1cebd56f Make synchrotron accept /events 2016-08-12 15:31:44 +01:00
Will Hunt 2510db3e76 Don't change status_msg on /sync 2016-08-10 12:59:59 +01:00
Erik Johnston 216a05b3e3 .values() returns list of sets 2016-06-06 16:00:09 +01:00
Erik Johnston 96dc600579 Fix typos 2016-06-06 15:44:41 +01:00
Erik Johnston 377eb480ca Fire after 30s not 8h 2016-06-06 15:14:21 +01:00
Erik Johnston 4ce84a1acd Change metric style 2016-06-03 13:49:16 +01:00
Erik Johnston a7ff5a1770 Presence metrics. Change def of small delta 2016-06-03 13:40:55 +01:00
Mark Haines 70599ce925 Allow external processes to mark a user as syncing. (#812)
* Add infrastructure to the presence handler to track sync requests in external processes

* Expire stale entries for dead external processes

* Add an http endpoint for making users as syncing

Add some docstrings and comments.

* Fixes
2016-06-02 15:20:15 +01:00
Mark Haines b153f5b150 Merge pull request #787 from matrix-org/markjh/liberate_presence_handler
Move the presence handler out of the Handlers object
2016-05-17 10:09:43 +01:00
Mark Haines 1a3a2002ff Spell "domain" correctly
s/domian/domain/g
2016-05-16 19:17:23 +01:00
Mark Haines e168abbcff Don't inherit PresenceHandler from BaseHandler, remove references to self.hs from presence handler 2016-05-16 19:08:40 +01:00
Mark Haines 3b86ecfa79 Move the presence handler out of the Handlers object 2016-05-16 18:56:37 +01:00
Erik Johnston 08dfa8eee2 Add and use get_domian_from_id 2016-05-09 10:36:03 +01:00
Erik Johnston 56b5e83e36 Reduce database inserts when sending transactions 2016-05-06 11:20:18 +01:00
Mark Haines 7a38612620 Remove another unused function from presence 2016-03-17 11:54:19 +00:00
Mark Haines 673c96ce97 Remove dead code left over from presence changes 2016-03-17 11:03:47 +00:00
Erik Johnston 2a78dac60d Merge pull request #612 from matrix-org/erikj/cache_size
Add environment variable SYNAPSE_CACHE_FACTOR, default it to 0.1
2016-03-02 14:36:10 +00:00
Mark Haines 60a0f81c7a Add a /replication API for extracting the updates that happened on
synapse

This is necessary for replicating the data in synapse to be visible to a
separate service because presence and typing notifications aren't stored
in a database so won't be visible to another process.

This API can be used to either get the raw data by requesting the tables
themselves or to just receive notifications for updates by following the
streams meta-stream.

Returns updates for each table requested a JSON array of arrays with a
row for each row in the table.

Each table is prefixed by a header row with the: name of the table,
current stream_id position for the table, number of rows, number of
columns and the names of the columns.
This is followed by the rows that have been added to the server since
the requester last asked.

The API has a timeout and is hooked up to the notifier so that a slave
can long poll for updates.
2016-03-01 14:49:41 +00:00
Erik Johnston 910fc0f28f Add enviroment variable SYNAPSE_CACHE_FACTOR, default it to 0.1 2016-03-01 12:56:39 +00:00
Erik Johnston b5f77eb12a Check presence token interval is less than 100, rather than the token itself 2016-02-23 15:47:37 +00:00
Erik Johnston e3e0ac6ec7 Merge pull request #602 from matrix-org/erikj/presence
Change the way we figure out presence updates for small deltas
2016-02-23 15:18:41 +00:00
Erik Johnston 13f86c3489 Handle get_all_entities_changed returning None 2016-02-23 15:05:37 +00:00
Erik Johnston 6e0209112b Add comments 2016-02-23 14:57:45 +00:00
Erik Johnston c77dae7a1a Change the way we figure out presence updates for small deltas 2016-02-23 14:54:40 +00:00
Erik Johnston a7b2ce32f7 Merge pull request #600 from matrix-org/erikj/presence
Measure PresenceEventSource.get_new_events
2016-02-23 14:54:26 +00:00
Erik Johnston 02e928cf9b Don't include defer.returnValue in Measure block 2016-02-23 14:52:16 +00:00
Erik Johnston 56a94ccd9e Measure PresenceEventSource.get_new_events 2016-02-23 14:52:16 +00:00
Erik Johnston 138c405974 Pick up currently_active across federation 2016-02-23 10:40:11 +00:00
Erik Johnston 3dbaeef58c Correctly filter states 2016-02-19 12:27:35 +00:00
Erik Johnston 42ac5f0c1a Only send presence updates to remote hosts if user is ours 2016-02-19 12:19:56 +00:00
Erik Johnston 24d9f2c140 Add Measures to presence 2016-02-19 11:50:48 +00:00
Erik Johnston 4a95eb0a12 Add presence metric 2016-02-19 11:32:04 +00:00
Erik Johnston be799453aa Remove spurious extra metrics 2016-02-19 11:29:33 +00:00
Erik Johnston 929cb0ed7d Don't set currently_active for remote presence 2016-02-19 10:58:27 +00:00
Erik Johnston 5f4eca3816 More metrics 2016-02-19 10:21:41 +00:00
Erik Johnston 5614b4dafb Add presence metrics 2016-02-19 09:50:54 +00:00
Erik Johnston 48b652bcbe Remove invalid arg. 2016-02-18 14:57:09 +00:00
Erik Johnston b4796a62ee Add unit test 2016-02-18 11:52:33 +00:00
Erik Johnston fe95f2217c Add stuff pulled from the DB to the cache 2016-02-18 10:26:24 +00:00
Erik Johnston 8351538873 PEP8 2016-02-18 10:12:12 +00:00
Erik Johnston 112283e230 Prefix TS fields with _ts 2016-02-18 10:11:43 +00:00
Erik Johnston b31ec214a5 Remove status_msg when going offline. Don't offline -> online if you send a message 2016-02-18 09:54:08 +00:00
Erik Johnston 114b929f8b Check presence state is a valid one 2016-02-18 09:16:32 +00:00
Erik Johnston ddca9c56fc Move if statement 2016-02-18 09:11:53 +00:00
Erik Johnston 58371fa263 Comment 2016-02-18 09:09:50 +00:00
Erik Johnston c229c87398 Remove spurious comment 2016-02-17 15:48:29 +00:00
Erik Johnston e5999bfb1a Initial cut 2016-02-17 15:40:50 +00:00
Erik Johnston 2c1fbea531 Fix up logcontexts 2016-02-08 14:26:45 +00:00
Daniel Wagner-Hall d83d004ccd Fix flake8 warnings for new flake8 2016-02-02 17:18:50 +00:00
Matthew Hodgson 6c28ac260c copyrights 2016-01-07 04:26:29 +00:00
Mark Haines a9526831a4 Wrap calls to distributor.fire in appropriately named functions so that static analysis can work out want is calling what 2015-12-01 20:53:04 +00:00
Mark Haines 31069ecf6a Rename presence_handler.send_invite to presence_handler.send_presence_invite to distinguish it from normal invites 2015-12-01 15:59:45 +00:00