Commit Graph

423 Commits

Author SHA1 Message Date
Erik Johnston 43adf2521c
Refactor presence so we can prune user in room caches (#13313)
See #10826 and #10786 for context as to why we had to disable pruning on
those caches.

Now that `get_users_who_share_room_with_user` is called frequently only
for presence, we just need to make calls to it less frequent and then we
can remove the various levels of caching that is going on.
2022-07-25 09:21:06 +00:00
Nick Mills-Barrett 2ee0b6ef4b
Safe async event cache (#13308)
Fix race conditions in the async cache invalidation logic, by separating
the async & local invalidation calls and ensuring any async call i
executed first.

Signed off by Nick @ Beeper (@Fizzadar).
2022-07-19 11:25:29 +00:00
Erik Johnston f721f1baba
Revert "Make all `process_replication_rows` methods async (#13304)" (#13312)
This reverts commit 5d4028f217.
2022-07-18 14:28:14 +01:00
Nick Mills-Barrett 5d4028f217
Make all `process_replication_rows` methods async (#13304)
More prep work for asyncronous caching, also makes all process_replication_rows methods consistent (presence handler already is so).

Signed off by Nick @ Beeper (@Fizzadar)
2022-07-17 22:19:43 +01:00
Erik Johnston e5716b631c
Don't pull out the full state when calculating push actions (#13078) 2022-07-11 20:08:39 +00:00
Erik Johnston 44de53bb79
Reduce state pulled from DB due to sending typing and receipts over federation (#12964)
Reducing the amount of state we pull from the DB is useful as fetching state is expensive in terms of DB, CPU and memory.
2022-06-06 16:46:11 +01:00
Erik Johnston 888a29f412
Wait for lazy join to complete when getting current state (#12872) 2022-06-01 16:02:53 +01:00
Patrick Cloke a8da046907
Invalidate the get_users_in_room{_with_profile} caches only when necessary. (#11878)
The get_users_in_room and get_users_in_room_with_profiles
are now only invalidated when the  membership of a room changes,
instead of during any state change in the room.
2022-02-02 12:24:07 -05:00
Sean Quah 5305a5e881
Type hint the constructors of the data store classes (#11555) 2021-12-13 17:05:00 +00:00
Sean Quah ffd858aa68
Add type hints to `synapse/storage/databases/main/events_worker.py` (#11411)
Also refactor the stream ID trackers/generators a bit and try to
document them better.
2021-11-26 18:41:31 +00:00
Dan Callahan 52ed9655ed
Remove unnecessary SystemRandom from SQLBaseStore (#9987)
It's not obvious that instances of SQLBaseStore each need their own
instances of random.SystemRandom(); let's just use random directly.

Introduced by 52839886d6

Signed-off-by: Dan Callahan <danc@element.io>
2021-05-14 10:59:10 +01:00
Erik Johnston d0aee697ac
Use get_current_users_in_room from store and not StateHandler (#9910) 2021-05-05 16:49:34 +01:00
Andrew Morgan fe604a022a
Remove various bits of compatibility code for Python <3.6 (#9879)
I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.
2021-04-27 13:13:07 +01:00
Richard van der Hoff 294c675033
Remove `synapse.types.Collection` (#9856)
This is no longer required, since we have dropped support for Python 3.5.
2021-04-22 16:43:50 +01:00
Jonathan de Jong 4b965c862d
Remove redundant "coding: utf-8" lines (#9786)
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Patrick Cloke b7748d3c00
Import HomeServer from the proper module. (#9665) 2021-03-23 07:12:48 -04:00
Patrick Cloke 637282bb50
Add additional type hints to the storage module. (#8980) 2020-12-30 08:09:53 -05:00
Richard van der Hoff 97647b33c2
Replace DeferredCache with LruCache where possible (#8563)
Most of these uses don't need a full-blown DeferredCache; LruCache is lighter and more appropriate.
2020-10-19 12:20:29 +01:00
Patrick Cloke eebf52be06
Be stricter about JSON that is accepted by Synapse (#8106) 2020-08-19 07:26:03 -04:00
Patrick Cloke fbe930dad2
Convert the roommember database to async/await. (#8070) 2020-08-12 12:14:34 -04:00
Erik Johnston a7bdf98d01
Rename database classes to make some sense (#8033) 2020-08-05 21:38:57 +01:00
Patrick Cloke f460da6031
Consistently use `db_to_json` to convert from database values to JSON objects. (#7849) 2020-07-16 11:32:19 -04:00
Richard van der Hoff e6027562e2 remove `builtins.buffer` code from storage code
this is no longer needed on python 3
2020-05-15 19:37:41 +01:00
Erik Johnston d7983b63a6
Support any process writing to cache invalidation stream. (#7436) 2020-05-07 13:51:08 +01:00
Patrick Cloke 509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
Erik Johnston 5d7a6ad223
Allow streaming cache invalidate all to workers. (#6749) 2020-01-22 10:37:00 +00:00
Richard van der Hoff 85db7f73be
Add a background update to clear tombstoned rooms from the directory (#6648)
* Add a background update to clear tombstoned rooms from the directory

* use the ABC metaclass
2020-01-07 14:18:43 +00:00
Erik Johnston 2284eb3a53
Add database config class (#6513)
This encapsulates config for a given database and is the way to get new
connections.
2019-12-18 10:45:12 +00:00
Erik Johnston d537be1ebd Pass Database into the data store 2019-12-06 15:49:44 +00:00
Erik Johnston 9a4fb457cf Change DataStores to accept 'database' param. 2019-12-06 13:30:06 +00:00
Erik Johnston 8863624f78 Comments 2019-12-05 10:46:37 +00:00
Erik Johnston 756d4942f5 Move DB pool and helper functions into dedicated Database class 2019-12-05 10:46:37 +00:00
Erik Johnston ee86abb2d6 Remove underscore from SQLBaseStore functions 2019-12-04 16:23:43 +00:00
Erik Johnston c2f525a525 Don't call SQLBaseStore methods from outside stores 2019-12-04 16:23:43 +00:00
Erik Johnston 9186c105a0 Revert "Move get_user_count_txn out of base store"
This reverts commit 00f0d67566.

Its going to get removed soon, so lets not make merge conflicts.
2019-12-04 15:46:19 +00:00
Erik Johnston 00f0d67566 Move get_user_count_txn out of base store 2019-12-04 15:21:14 +00:00
Erik Johnston 1056d6885a Move cache invalidation to main data store 2019-12-04 15:21:14 +00:00
Erik Johnston 6b2867096b Move event fetch vars to EventWorkStore 2019-12-04 15:21:14 +00:00
Erik Johnston ddd48b6851 Move account validity bg updates to registration store 2019-12-04 15:21:14 +00:00
Richard van der Hoff c01d543584
Make sure that we close cursors before returning from a query (#6408)
There are lots of words in the comment as to why this is a good idea.

Fixes #6403.
2019-11-25 21:03:17 +00:00
Andrew Morgan 3916e1b97a
Clean up newline quote marks around the codebase (#6362) 2019-11-21 12:00:14 +00:00
Andrew Morgan 745a48625d
Fix guest -> real account upgrade with account validity enabled (#6359) 2019-11-14 12:02:05 +00:00
Andrew Morgan 54fef094b3
Remove usage of deprecated logger.warn method from codebase (#6271)
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
2019-10-31 10:23:24 +00:00
Erik Johnston b54b1e759a Fix SQLite take 2 2019-10-10 16:19:40 +01:00
Erik Johnston 3bc687508f Remove add_in_list_sql_clause 2019-10-10 15:35:46 +01:00
Erik Johnston 9d06fb9cb1 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/postgres_any 2019-10-10 14:38:21 +01:00
Erik Johnston b4fbf71187 Add helper funcs to use postgres ANY
This means that we can write queries with `col = ANY(?)`, which helps
postgres.
2019-10-10 13:15:24 +01:00
Erik Johnston 132279a46f Patch inlinecallbacks for log contexts 2019-09-27 15:11:14 +01:00
Richard van der Hoff 49ef8ec399
Fix a cache-invalidation bug for worker-based deployments (#5920)
Some of the caches on worker processes were not being correctly invalidated
when a room's state was changed in a way that did not affect the membership
list of the room.

We need to make sure we send out cache invalidations even when no memberships
are changing.
2019-08-28 10:18:16 +01:00
Amber Brown 4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00