Commit Graph

154 Commits

Author SHA1 Message Date
Erik Johnston 64e7e11853 Implement cache replication stream 2016-08-15 11:16:45 +01:00
Erik Johnston 5aa024e501 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_reader 2016-07-29 11:24:56 +01:00
Mark Haines 0a7d3cd00f Create separate methods for getting messages to push
for the email and http pushers rather than trying to make a single
method that will work with their conflicting requirements.

The http pusher needs to get the messages in ascending stream order, and
doesn't want to miss a message.

The email pusher needs to get the messages in descending timestamp order,
and doesn't mind if it misses messages.
2016-07-28 20:24:24 +01:00
Erik Johnston ec8b217722 Add destination retry to slave store 2016-07-28 17:35:53 +01:00
Erik Johnston 76b89d0edb Add slace storage functions for public room list 2016-07-28 17:03:40 +01:00
Erik Johnston 0fcbca531f Add get_auth_chain to slave store 2016-07-28 16:36:28 +01:00
Erik Johnston 6ede23ff1b Add more key storage funcs into slave store 2016-07-28 15:41:26 +01:00
Erik Johnston aede7248ab Split out a FederationReader process 2016-07-21 17:37:44 +01:00
David Baker 46b7362304 pep8 2016-06-24 11:44:57 +01:00
David Baker 870c45913e Use similar naming we use in email notifs for push
Fixes https://github.com/vector-im/vector-web/issues/1654
2016-06-24 11:41:11 +01:00
Erik Johnston 05e01f21d7 Remove event fetching from DB threads 2016-06-03 17:22:13 +01:00
Mark Haines 20eccd84d4 Merge pull request #827 from matrix-org/markjh/more_slaved_methods
Add methods to events, account data and receipt slaves
2016-06-03 11:46:21 +01:00
Mark Haines b09f348530 Merge pull request #824 from matrix-org/markjh/slaved_presence_store
Add a slaved store for presence
2016-06-03 11:26:33 +01:00
Mark Haines 81cf449daa Add methods to events, account data and receipt slaves
Adds the methods needed by /sync to the slaved events,
account data and receipt stores.
2016-06-03 11:19:27 +01:00
Mark Haines 3ae915b27e Add a slaved store for presence 2016-06-03 11:05:53 +01:00
Mark Haines f88d747f79 Add a comment explaining why the filter cache doesn't need exipiring 2016-06-03 11:03:10 +01:00
Mark Haines 0eae075723 Add slaved stores for filters, tokens, and push rules 2016-06-03 10:58:03 +01:00
Mark Haines 70599ce925 Allow external processes to mark a user as syncing. (#812)
* Add infrastructure to the presence handler to track sync requests in external processes

* Expire stale entries for dead external processes

* Add an http endpoint for making users as syncing

Add some docstrings and comments.

* Fixes
2016-06-02 15:20:15 +01:00
Mark Haines 0cb441fedd Move typing handler out of the Handlers object 2016-05-17 15:58:46 +01:00
Mark Haines 3b86ecfa79 Move the presence handler out of the Handlers object 2016-05-16 18:56:37 +01:00
Mark Haines 0466454b00 Assert that stream replicated stream positions are ints 2016-05-13 17:33:44 +01:00
Mark Haines 1f71f386f6 Merge branch 'develop' into dbkr/email_notifs_on_pusher 2016-05-13 16:59:56 +01:00
Mark Haines 206eb9fd94 Shift some of the state_group methods into the SlavedEventStore 2016-05-13 16:58:14 +01:00
Mark Haines 3abab26458 Add a slaved datastore for account data 2016-05-13 15:34:06 +01:00
Mark Haines 0e792e7903 Log the stream IDs in an order that makes sense 2016-05-13 11:54:44 +01:00
Mark Haines 3547e66bc6 Make sure we advance our stream position 2016-05-13 11:53:00 +01:00
David Baker 5f46be19a7 Pass through get_events to pusher too 2016-05-10 18:43:40 +02:00
David Baker d46b18a00f Pass through _get_event_txn 2016-05-10 18:27:06 +02:00
Erik Johnston 8940281d1b Don't warn 2016-05-05 15:10:03 +01:00
Erik Johnston 5d8a93a10e Add some log information at returned replication streams 2016-05-05 10:29:21 +01:00
Mark Haines 8d7ad44331 Report per request metrics for all of the things using request_handler 2016-04-28 10:57:49 +01:00
Mark Haines 8a65666454 Fix backfill replication to advance the stream correctly 2016-04-27 15:38:43 +01:00
Mark Haines 78741cf025 Merge pull request #743 from matrix-org/markjh/slave_pushers
Replicate the pushers
2016-04-21 17:21:29 +01:00
Mark Haines 31719ad124 Merge pull request #744 from matrix-org/markjh/replication_remove_pusher
Add a replication endpoint for deleting pushers
2016-04-21 17:10:49 +01:00
Mark Haines cfe1ff4bdb Add a replication endpoint for deleting pushers 2016-04-21 16:33:05 +01:00
Mark Haines d4823efad9 Replicate the pushers 2016-04-21 16:18:00 +01:00
Mark Haines c0d8e0eb63 Replicate push actions 2016-04-21 15:25:58 +01:00
Mark Haines f52dd35ac3 Merge pull request #738 from matrix-org/markjh/slaved_receipts
Add a slaved receipts store
2016-04-19 17:31:59 +01:00
Mark Haines 5bbd424ee0 Add a slaved receipts store 2016-04-19 17:14:08 +01:00
Mark Haines e99365f601 Replicate get_invited_rooms_for_user 2016-04-19 15:22:14 +01:00
Mark Haines ceb599e789 Add tests for redactions 2016-04-07 16:52:07 +01:00
Mark Haines 60ec9793fb Add tests for get_latest_event_ids_in_room and get_current_state 2016-04-07 13:17:56 +01:00
Mark Haines 6bfec56796 Test that room membership is replicated 2016-04-06 16:20:13 +01:00
Mark Haines 75fb9ac1be Add a slaved events store class
Add a test to check that get_room_names_and_aliases does the same
thing on both the master and on the slave data store.
2016-04-06 14:18:35 +01:00
Mark Haines 1d4deff25a Separate generating the replication response...
from doing the http request parsing to make it easier
to write unit tests for replication.
2016-04-05 11:23:57 +01:00
Mark Haines 62e395f0e3 Merge pull request #676 from matrix-org/markjh/replicate_stateIII
Add replication streams for ex outliers and current state resets
2016-03-31 11:20:57 +01:00
Mark Haines 2ec5426035 Use a namedtuple rather than tuple unpacking 2016-03-31 10:33:02 +01:00
Matthew Hodgson 9113316b0e typo 2016-03-30 17:29:42 +01:00
Mark Haines 1fbb094c6f Add replication streams for ex outliers and current state resets 2016-03-30 17:19:56 +01:00
Mark Haines 31a9eceda5 Add a replication stream for state groups 2016-03-30 16:01:58 +01:00
Mark Haines b6e8420aee Add replication stream for pushers 2016-03-15 17:33:10 +00:00
Mark Haines ebcbb23226 s/stream_ordering/event_stream_ordering/ in push 2016-03-04 16:15:23 +00:00
Mark Haines 2223204eba Hook push rules up to the replication API 2016-03-02 17:26:20 +00:00
Mark Haines 60a0f81c7a Add a /replication API for extracting the updates that happened on
synapse

This is necessary for replicating the data in synapse to be visible to a
separate service because presence and typing notifications aren't stored
in a database so won't be visible to another process.

This API can be used to either get the raw data by requesting the tables
themselves or to just receive notifications for updates by following the
streams meta-stream.

Returns updates for each table requested a JSON array of arrays with a
row for each row in the table.

Each table is prefixed by a header row with the: name of the table,
current stream_id position for the table, number of rows, number of
columns and the names of the columns.
This is followed by the rows that have been added to the server since
the requester last asked.

The API has a timeout and is hooked up to the notifier so that a slave
can long poll for updates.
2016-03-01 14:49:41 +00:00