Commit Graph

79 Commits

Author SHA1 Message Date
Andrew Morgan 4548d1f87e
Remove unnecessary parentheses around return statements (#5931)
Python will return a tuple whether there are parentheses around the returned values or not.

I'm just sick of my editor complaining about this all over the place :)
2019-08-30 16:28:26 +01:00
Amber Brown 4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00
Amber Brown 32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Amber Brown 7efd1d87c2 Run black on the rest of the storage module (#4996) 2019-04-03 10:07:29 +01:00
Richard van der Hoff 9bde730ef8
Fix bug where read-receipts lost their timestamps (#4927)
Make sure that they are sent correctly over the replication stream.

Fixes: #4898
2019-03-25 16:38:05 +00:00
Richard van der Hoff 2db49ea476 Add some debug about processing read receipts.
I'm hoping to establish which rooms are having lots of RRs sent for them, and
how old the events are when they are sent.
2019-03-04 18:19:40 +00:00
Erik Johnston aff1dfdf3d Update return value docstring 2018-07-12 09:45:37 +01:00
Erik Johnston 6ccefef07a Use 'is not None' and add comments 2018-07-10 18:12:39 +01:00
Erik Johnston 05f5dabc10 Use stream cache in get_linearized_receipts_for_room
This avoids us from uncessarily hitting the database when there has been
no change for the room
2018-07-10 17:22:42 +01:00
Amber Brown 49af402019 run isort 2018-07-09 16:09:20 +10:00
Amber Brown 6350bf925e
Attempt to be more performant on PyPy (#3462) 2018-06-28 14:49:57 +01:00
Richard van der Hoff 9f797a24a4 Handle RRs which arrive before their events 2018-06-01 14:01:43 +01:00
Richard van der Hoff 857e6fd8b6 Ignore depth when updating read-receipts
Order read receipts by stream ordering instead of depth
2018-06-01 12:18:11 +01:00
Richard van der Hoff 8b1c856d81 Fix error in handling receipts
Fixes an error which has been happening ever since #2158 (v0.21.0-rc1):

> TypeError: argument of type 'ObservableDeferred' is not iterable

fixes #3234
2018-05-18 09:15:35 +01:00
Erik Johnston 6406b70aeb Use stream rather depth ordering for push actions
This simplifies things as it is, but will also allow us to change the
way we traverse topologically without having to update the way push
actions work.
2018-05-11 15:30:11 +01:00
Erik Johnston fa72803490 Merge branch 'master' of github.com:matrix-org/synapse into develop 2018-03-19 11:41:01 +00:00
Erik Johnston 926ba76e23 Replace ujson with simplejson 2018-03-15 23:43:31 +00:00
Erik Johnston faeb369f15 Fix missing invalidations for receipt storage 2018-02-21 15:19:54 +00:00
Erik Johnston 8fbb4d0d19 Raise exception in abstract method 2018-02-20 17:59:23 +00:00
Erik Johnston 95e4cffd85 Fix comment 2018-02-20 17:58:40 +00:00
Erik Johnston e316bbb4c0 Use abstract base class to access stream IDs 2018-02-20 17:43:57 +00:00
Erik Johnston f5ac4dc2d4 Split ReceiptsStore 2018-02-20 16:28:28 +00:00
Richard van der Hoff 6cfee09be9 Make __init__ consitstent across Store heirarchy
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
2017-11-13 10:46:07 +00:00
Erik Johnston 58ebb96cce Fix invalidation of get_users_with_read_receipts_in_room 2017-05-19 14:38:50 +01:00
Erik Johnston 119cb9bbcf Reduce cache size by not storing deferreds
Currently the cache descriptors store deferreds rather than raw values,
this is a simple way of triggering only one database hit and sharing the
result if two callers attempt to get the same value.

However, there are a few caches that simply store a mapping from string
to string (or int). These caches can have a large number of entries,
under the assumption that each entry is small. However, the size of a
deferred (specifically the size of ObservableDeferred) is signigicantly
larger than that of the raw value, 2kb vs 32b.

This PR therefore changes the cache descriptors to store the raw values
rather than the deferreds.

As a side effect cached storage function now either return a deferred or
the actual value, as the cached list decriptor already does. This is
fine as we always end up just yield'ing on the returned value
eventually, which handles that case correctly.
2017-04-25 10:23:11 +01:00
Erik Johnston 00957d1aa4 User Cursor.__iter__ instead of fetchall
This prevents unnecessary construction of lists
2017-03-23 17:53:49 +00:00
Erik Johnston 095b45c165 Aggregate event push actions 2017-02-14 13:39:41 +00:00
Richard van der Hoff bd07a35c29 Fix result of insert_receipt
This should fix the absence of notifications when new receipts arrive.
2016-12-08 12:11:34 +00:00
Erik Johnston 5405351b14 Lower get_linearized_receipts_for_room cache size 2016-08-31 16:19:44 +01:00
Matthew Hodgson 6e80c03d45 Merge branch 'develop' into dbkr/notifications_api 2016-08-20 00:16:18 +01:00
Erik Johnston ba214a5e32 Remove lru option 2016-08-19 14:17:11 +01:00
David Baker 1e4217c90c Explicit join 2016-08-18 17:53:44 +01:00
David Baker b4ecf0b886 Merge remote-tracking branch 'origin/develop' into dbkr/notifications_api 2016-08-11 14:09:13 +01:00
Erik Johnston 696d7c5937 Merge pull request #809 from matrix-org/erikj/cache_receipts_in_room
Add get_users_with_read_receipts_in_room cache
2016-06-03 10:58:24 +01:00
Erik Johnston 43db0d9f6a Add get_users_with_read_receipts_in_room cache 2016-06-01 10:54:32 +01:00
David Baker a24bc5b2dc Add GET /notifications API 2016-05-23 18:33:51 +01:00
David Baker 149fa411e2 Only delete push actions after 30 days 2016-05-20 15:25:12 +01:00
Mark Haines b7381d5338 Allow receipts for events we haven't seen in the db 2016-05-13 15:46:41 +01:00
Erik Johnston e00e8f2166 Merge pull request #769 from matrix-org/erikj/push_actions_delete
Delete old pushers
2016-05-13 14:41:36 +01:00
Erik Johnston 6da7f39d95 Use tree cache for get_linearized_receipts_for_room 2016-05-13 11:41:23 +01:00
Erik Johnston 183f23f10d Delete old pushers 2016-05-03 14:22:33 +01:00
Mark Haines 871357d539 Check that somethign has happend before running the selects 2016-04-27 11:54:13 +01:00
David Baker 9c99ab4572 Merge remote-tracking branch 'origin/develop' into dbkr/pushers_use_event_actions 2016-04-07 16:35:22 +01:00
David Baker 92e3071623 Send badge count pushes.
Also fix bugs with retrying.
2016-04-07 15:39:53 +01:00
Mark Haines 87f2dec8d4 Make the cache objects be per instance rather than being global 2016-04-06 13:08:05 +01:00
Mark Haines e36bfbab38 Use a stream id generator for backfilled ids 2016-04-01 13:29:05 +01:00
Erik Johnston acdfef7b14 Intern all the things 2016-03-23 16:25:54 +00:00
Mark Haines a612ce6659 Merge pull request #489 from matrix-org/markjh/replication
Add a /replication API for extracting the updates that happened on synapse.
2016-03-01 15:08:24 +00:00
Mark Haines 60a0f81c7a Add a /replication API for extracting the updates that happened on
synapse

This is necessary for replicating the data in synapse to be visible to a
separate service because presence and typing notifications aren't stored
in a database so won't be visible to another process.

This API can be used to either get the raw data by requesting the tables
themselves or to just receive notifications for updates by following the
streams meta-stream.

Returns updates for each table requested a JSON array of arrays with a
row for each row in the table.

Each table is prefixed by a header row with the: name of the table,
current stream_id position for the table, number of rows, number of
columns and the names of the columns.
This is followed by the rows that have been added to the server since
the requester last asked.

The API has a timeout and is hooked up to the notifier so that a slave
can long poll for updates.
2016-03-01 14:49:41 +00:00
Mark Haines 54172924c8 Load the current id in the IdGenerator constructor
Rather than loading them lazily. This allows us to remove all
the yield statements and spurious arguments for the get_next
methods.

It also allows us to replace all instances of get_next_txn with
get_next since get_next no longer needs to access the db.
2016-03-01 14:32:56 +00:00