Commit Graph

457 Commits

Author SHA1 Message Date
Erik Johnston 4a33a6dd19 Move background update handling out of store 2019-12-05 11:11:26 +00:00
Erik Johnston 756d4942f5 Move DB pool and helper functions into dedicated Database class 2019-12-05 10:46:37 +00:00
Erik Johnston ee86abb2d6 Remove underscore from SQLBaseStore functions 2019-12-04 16:23:43 +00:00
Amber Brown 0f87b912ab
Implementation of MSC2314 (#6176) 2019-11-28 08:54:07 +11:00
Hubert Chathi 0d27aba900
add etag and count to key backup endpoints (#5858) 2019-11-27 16:14:44 -05:00
Erik Johnston 6a0092d371 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history 2019-11-04 13:29:35 +00:00
Hubert Chathi 6f4bc6d01d Merge branch 'develop' into cross-signing_federation 2019-10-31 22:38:21 -04:00
Erik Johnston cd581338cf Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history 2019-10-31 15:19:26 +00:00
Hubert Chathi bb6cec27a5 rename get_devices_by_remote to get_device_updates_by_remote 2019-10-30 14:57:34 -04:00
Erik Johnston 7c8c97e635 Split purge API into events vs state 2019-10-30 15:23:37 +00:00
Erik Johnston 69f0054ce6 Port to use state storage 2019-10-30 14:46:54 +00:00
Hubert Chathi 7d7eac61be Merge branch 'develop' into cross-signing_federation 2019-10-30 10:17:10 -04:00
Erik Johnston ec6de1cc7d Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_persistence_store 2019-10-30 13:37:04 +00:00
Erik Johnston 326b3dace7 Make ObservableDeferred.observe() always return deferred.
This makes it easier to use in an async/await world.

Also fixes a bug where cache descriptors would occaisonally return a raw
value rather than a deferred.
2019-10-30 11:35:46 +00:00
Hubert Chathi 4697c0de0b remove unneeded imports 2019-10-25 10:47:02 -04:00
Hubert Chathi 4cf3a30a20 switch to using HomeserverTestCase 2019-10-25 10:42:07 -04:00
Hubert Chathi c40d7244f8 Merge branch 'develop' into cross-signing_federation 2019-10-24 22:31:25 -04:00
Hubert Chathi 29a0bc5637 remove some unnecessary lines 2019-10-24 21:43:02 -04:00
Hubert Chathi 848cd388d9 delete keys when deleting backups 2019-10-24 21:21:51 -04:00
Erik Johnston 3ca4c7c516 Use new EventPersistenceStore 2019-10-23 16:15:03 +01:00
Hubert Chathi 3e3f9b684e fix unit test 2019-10-22 22:26:30 -04:00
Erik Johnston c66a06ac6b Move storage classes into a main "data store".
This is in preparation for having multiple data stores that offer
different functionality, e.g. splitting out state or event storage.
2019-10-21 16:05:06 +01:00
Hubert Chathi 06fc66c81e
Merge pull request #6193 from matrix-org/uhoreg/interpret_device_key_in_storage
make storage layer in charge of interpreting the device key data
2019-10-11 15:19:06 -04:00
Neil Johnson a0d0ba7862
Fix MAU reaping where reserved users are specified. (#6168) 2019-10-11 09:38:26 +01:00
Hubert Chathi 4908fb3b30 make storage layer in charge of interpreting the device key data 2019-10-10 19:15:30 -04:00
Erik Johnston bc244627ac Fix postgres unit tests 2019-10-10 15:37:53 +01:00
Erik Johnston b4fe7e19c0
Merge pull request #6146 from matrix-org/erikj/fix_destination_retry_timings
Fix errors storing large retry intervals.
2019-10-02 11:05:22 +01:00
Erik Johnston f44f1d2e83 Fix errors storing large retry intervals.
We have set the max retry interval to a value larger than a postgres or
sqlite int can hold, which caused exceptions when updating the
destinations table.

To fix postgres we need to change the column to a bigint, and for sqlite
we lower the max interval to 2**62 (which is still incredibly long).
2019-10-02 10:36:27 +01:00
Erik Johnston ce7a3e7e27 Fix fetching censored redactions from DB
Fetching a censored redactions caused an exception due to the code
expecting redactions to have a `redact` key, which redacted redactions
don't have.
2019-10-02 10:14:01 +01:00
Neil Johnson 034db2ba21 Fix dummy event insertion consent bug (#6053)
Fixes #5905
2019-09-26 11:47:53 +01:00
Erik Johnston faac453f08 Test that pruning of old user IPs works 2019-09-24 15:53:17 +01:00
Erik Johnston acb62a7cc6 Test background update 2019-09-23 16:59:45 +01:00
Erik Johnston 51d28272e2 Query devices table for last seen info.
This is a) simpler than querying user_ips directly and b) means we can
purge older entries from user_ips without losing the required info.

The storage functions now no longer return the access_token, since it
was unused.
2019-09-23 16:59:45 +01:00
Richard van der Hoff 1e19ce00bf
Add 'failure_ts' column to 'destinations' table (#6016)
Track the time that a server started failing at, for general analysis purposes.
2019-09-17 11:41:54 +01:00
Erik Johnston 470dc621ae
Merge pull request #5934 from matrix-org/erikj/censor_redactions
Censor redactions in DB after a month
2019-09-09 15:29:39 +01:00
Amber Brown 55d5b3af88
Servers-known-about statistic (#5981) 2019-09-07 01:45:51 +10:00
Erik Johnston ad9b64b496 Fix test 2019-09-05 17:17:47 +01:00
Erik Johnston 3ff0422d2d Make redaction retention period configurable 2019-09-05 17:16:45 +01:00
Erik Johnston a4bf72c30c Censor redactions in DB after a month 2019-08-30 11:29:17 +01:00
Half-Shot 0fb5189072 Fix registration test 2019-08-23 09:25:35 +01:00
Richard van der Hoff fb86217553
Merge pull request #5788 from matrix-org/rav/metaredactions
Fix handling of redactions of redactions
2019-08-12 12:25:19 +01:00
Erik Johnston 63eb4a1b62
Merge pull request #5746 from matrix-org/erikj/test_bg_update_currnet_state
Add unit test for current state membership bg update
2019-07-30 10:00:02 +01:00
Richard van der Hoff 4e97eb89e5 Handle loops in redaction events 2019-07-30 00:00:34 +01:00
Richard van der Hoff 1cad8d7b6f
Convert RedactionTestCase to modern test style (#5768) 2019-07-26 07:38:55 +01:00
Erik Johnston 73bbaf2bc6 Add unit test for current state membership bg update 2019-07-23 17:00:22 +01:00
Amber Brown 4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00
Amber Brown 7ad1d76356
Support Prometheus_client 0.4.0+ (#5636) 2019-07-18 23:57:15 +10:00
Richard van der Hoff 5f158ec039
Implement access token expiry (#5660)
Record how long an access token is valid for, and raise a soft-logout once it
expires.
2019-07-12 17:26:02 +01:00
Richard van der Hoff 953dbb7980
Remove access-token support from RegistrationStore.register (#5642)
The 'token' param is no longer used anywhere except the tests, so let's kill
that off too.
2019-07-10 16:26:49 +01:00
Amber Brown 32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Erik Johnston 554609288b Run as background process and fix comments 2019-06-19 11:33:03 +01:00
Erik Johnston b42f90470f Add experimental option to reduce extremities.
Adds new config option `cleanup_extremities_with_dummy_events` which
periodically sends dummy events to rooms with more than 10 extremities.

THIS IS REALLY EXPERIMENTAL.
2019-06-18 15:02:18 +01:00
Erik Johnston 3ed595e327 Prometheus histograms are cumalative 2019-06-14 14:07:32 +01:00
Amber H. Brown a10c8dae85 fix prometheus rendering error 2019-06-14 21:09:33 +10:00
Amber Brown 6312d6cc7c
Expose statistics on extrems to prometheus (#5384) 2019-06-13 22:40:52 +10:00
Andrew Morgan 2d1d7b7e6f Prevent multiple device list updates from breaking a batch send (#5156)
fixes #5153
2019-06-06 23:54:00 +01:00
Neil Johnson a11865016e
Set default room version to v4. (#5379)
Set default room version to v4.
2019-06-06 20:13:47 +01:00
Erik Johnston d7add713a8 Add test 2019-05-29 15:59:06 +01:00
Richard van der Hoff b75537beaf Store key validity time in the storage layer
This is a first step to checking that the key is valid at the required moment.

The idea here is that, rather than passing VerifyKey objects in and out of the
storage layer, we instead pass FetchKeyResult objects, which simply wrap the
VerifyKey and add a valid_until_ts field.
2019-05-23 11:52:22 +01:00
Richard van der Hoff 2e052110ee
Rewrite store_server_verify_key to store several keys at once (#5234)
Storing server keys hammered the database a bit. This replaces the
implementation which stored a single key, with one which can do many updates at
once.
2019-05-23 11:45:39 +01:00
Erik Johnston 4fb44fb5b9 Expose DataStore._get_events as get_events_as_list
This is in preparation for reaction work which requires it.
2019-05-14 13:37:44 +01:00
Amber Brown b36c82576e
Run Black on the tests again (#5170) 2019-05-10 00:12:11 -05:00
Richard van der Hoff 12f9d51e82
Add admin api for sending server_notices (#5121) 2019-05-02 11:59:16 +01:00
Richard van der Hoff 40e576e29c Move admin api impl to its own package
It doesn't really belong under rest/client/v1 any more.
2019-05-01 15:44:30 +01:00
Richard van der Hoff 18b69be00f Rewrite Datastore.get_server_verify_keys
Rewrite this so that it doesn't hammer the database.
2019-04-09 00:00:10 +01:00
Richard van der Hoff 0084309cd2 Rewrite test_keys as a HomeserverTestCase 2019-04-08 23:59:57 +01:00
Neil Johnson e8419554ff
Remove presence lists (#4989)
Remove presence list support as per MSC 1819
2019-04-03 11:11:15 +01:00
Richard van der Hoff 54a87a7b08
Collect room-version variations into one place (#4969)
Collect all the things that make room-versions different to one another into
one place, so that it's easier to define new room versions.
2019-04-01 10:24:38 +01:00
Amber Brown 282c97327f
Migrate the user directory initial population to a background task (#4864) 2019-03-19 04:50:24 +11:00
Amber Brown d306bd1b26 fixup 2019-03-12 22:38:01 +11:00
Amber Brown f6135d06cf
Rewrite userdir to be faster (#4537) 2019-03-07 01:22:53 -08:00
Amber Brown a06614bd2a
UPSERT many functionality (#4644) 2019-02-20 23:03:30 +11:00
Amber Brown 3f189c902e
Fix flake8 (#4519) 2019-01-30 10:53:17 +00:00
Amber Brown 7072fe3084
Fix UPSERTs on SQLite 3.24+ (#4477) 2019-01-28 15:43:32 +00:00
Erik Johnston 9770ed91c2 Fix tests 2019-01-25 10:32:26 +00:00
Erik Johnston be6a7e47fa
Revert "Require event format version to parse or create events" 2019-01-25 10:23:51 +00:00
Erik Johnston efb8ed1d45 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/require_format_version 2019-01-24 18:52:34 +00:00
Amber Brown 58f6c48183
Use native UPSERTs where possible (#4306) 2019-01-24 21:31:54 +11:00
Erik Johnston e79ba9eb34 Fix tests 2019-01-24 09:28:16 +00:00
Amber Brown 7960c26fda
Fix adding new rows instead of updating them if one of the key values is a NULL in upserts. (#4369) 2019-01-09 22:26:25 +11:00
Neil Johnson d2f7c4e6b1
create support user (#4141)
Allow for the creation of a support user.

A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.
2018-12-14 18:20:59 +00:00
Richard van der Hoff 30da50a5b8
Initialise user displayname from SAML2 data (#4272)
When we register a new user from SAML2 data, initialise their displayname
correctly.
2018-12-07 14:44:46 +01:00
Travis Ralston 835779f7fb Add option to track MAU stats (but not limit people) (#3830) 2018-11-15 18:08:27 +00:00
Erik Johnston abaa93c158 Add test to assert set_e2e_device_keys correctly returns False on no-op 2018-11-08 14:06:44 +00:00
Erik Johnston cb53ce9d64
Refactor state group lookup to reduce DB hits (#4011)
Currently when fetching state groups from the data store we make two
hits two the database: once for members and once for non-members (unless
request is filtered to one or the other). This adds needless load to the
datbase, so this PR refactors the lookup to make only a single database
hit.
2018-10-25 17:49:55 +01:00
Neil Johnson ea69a84bbb fix style inconsistencies 2018-10-24 17:18:08 +01:00
Neil Johnson 6105c6101f fix race condiftion in calling initialise_reserved_users 2018-10-23 15:24:58 +01:00
Erik Johnston 9c834b8ee9 Add tests 2018-10-02 16:22:39 +01:00
Amber Brown 6e05fd032c
Fix userconsent on Python 3 (#3938) 2018-10-02 00:11:58 +10:00
Richard van der Hoff ae6ad4cf41
docstrings and unittests for storage.state (#3958)
I spent ages trying to figure out how I was going mad...
2018-09-27 11:22:25 +01:00
Amber Brown 1f3f5fcf52
Fix client IPs being broken on Python 3 (#3908) 2018-09-20 20:14:34 +10:00
Neil Johnson 8decd6233d improve naming 2018-09-12 16:22:15 +01:00
Neil Johnson 0ddf486724 expose number of real reserved users 2018-09-12 11:58:52 +01:00
Amber Brown 52ec6e9dfa
Port tests/ to Python 3 (#3808) 2018-09-07 02:58:18 +10:00
Neil Johnson 84a750e0c3 ensure guests never enter mau list 2018-09-06 17:22:53 +01:00
Neil Johnson 61b05727fa guest users should not be part of mau total 2018-09-05 22:30:36 +01:00
Amber Brown 77055dba92
Fix tests on postgresql (#3740) 2018-09-04 02:21:48 +10:00
Amber Brown 14e4d4f4bf
Port storage/ to Python 3 (#3725) 2018-08-31 00:19:58 +10:00
Richard van der Hoff 414fa36f3e Fix up tests 2018-08-28 17:21:05 +01:00
Erik Johnston cd77270a66 Implement trail users 2018-08-23 19:17:19 +01:00
Erik Johnston 3bf8bab8f9
Merge pull request #3673 from matrix-org/erikj/refactor_state_handler
Refactor state module to support multiple room versions
2018-08-22 10:04:55 +01:00
Matthew Hodgson bb81e78ec6
Split the state_group_cache in two (#3726)
Splits the state_group_cache in two.

One half contains normal state events; the other contains member events.

The idea is that the lazyloading common case of: "I want a subset of member events plus all of the other state" can be accomplished efficiently by splitting the cache into two, and asking for "all events" from the non-members cache, and "just these keys" from the members cache.  This means we can avoid having to make DictionaryCache aware of these sort of complicated queries, whilst letting LL requests benefit from the caching.

Previously we were unable to sensibly use the caching and had to pull all state from the DB irrespective of the filtering, which made things slow.  Hopefully fixes https://github.com/matrix-org/synapse/issues/3720.
2018-08-22 00:56:37 +02:00
Erik Johnston 4d664278af Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler 2018-08-20 14:49:43 +01:00
Neil Johnson bcfeb44afe call reap on start up and fix under reaping bug 2018-08-16 22:55:32 +01:00
Matthew Hodgson 2f78f432c4 speed up /members and add at= and membership params (#3568) 2018-08-15 16:35:22 +01:00
Neil Johnson 1522ed9c07 in case max_mau is less than I think 2018-08-14 16:52:30 +01:00
Neil Johnson 414d54b61a
Merge pull request #3670 from matrix-org/neilj/mau_sync_block
Block ability to read via sync if mau limit exceeded
2018-08-14 15:21:31 +00:00
Amber Brown 99dd975dae
Run tests under PostgreSQL (#3423) 2018-08-13 16:47:46 +10:00
black 8b3d9b6b19 Run black. 2018-08-10 23:54:09 +10:00
Amber Brown b37c472419
Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678) 2018-08-10 23:50:21 +10:00
Neil Johnson 885ea9c602 rename _user_last_seen_monthly_active 2018-08-09 18:02:12 +01:00
Erik Johnston 3e19beb941 Fix tests 2018-08-09 14:58:49 +01:00
Amber Brown 2511f3f8a0
Test fixes for Python 3 (#3647) 2018-08-09 12:22:01 +10:00
Neil Johnson e8eba2b4e3 implement reserved users for mau limits 2018-08-07 17:49:43 +01:00
Neil Johnson e10830e976 wip commit - tests failing 2018-08-03 17:55:50 +01:00
Neil Johnson 5593ff6773 fix (lots of) py3 test failures 2018-08-03 14:59:17 +01:00
Neil Johnson b2aab04d2c fix py3 test failure 2018-08-03 14:12:56 +01:00
Neil Johnson 950807d93a fix caching and tests 2018-08-03 13:49:53 +01:00
Neil Johnson 9180061b49 remove unused count_monthly_users 2018-08-02 15:55:29 +01:00
Neil Johnson c4ffbecb68 fix test, update constructor call 2018-08-02 13:51:05 +01:00
Neil Johnson 00f99f74b1 insertion into monthly_active_users 2018-08-02 13:47:19 +01:00
Neil Johnson 165e067033 Revert "change monthly_active_users table to be a single column"
This reverts commit ec716a35b2.
2018-08-02 10:59:58 +01:00
Neil Johnson ec716a35b2 change monthly_active_users table to be a single column 2018-08-01 17:54:37 +01:00
Neil Johnson d766f26de9 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker 2018-08-01 17:49:41 +01:00
Neil Johnson c480c4c962 fix isort 2018-08-01 14:25:58 +01:00
Neil Johnson 4e6e00152c fix known broken test 2018-08-01 11:48:37 +01:00
Neil Johnson 6ef983ce5c api into monthly_active_users table 2018-07-31 16:36:24 +01:00
Neil Johnson df2235e7fa coding style 2018-07-31 13:16:20 +01:00
Neil Johnson 7d05406a07 fix user_ips counting 2018-07-31 12:03:23 +01:00
Neil Johnson cefac79c10 monthly_active_tests 2018-07-30 22:08:09 +01:00
Matthew Hodgson 0620d27f4d flake8 2018-07-25 17:21:17 +01:00
Matthew Hodgson 0a7ee0ab8b add tests for _get_some_state_from_cache 2018-07-25 16:33:52 +01:00
Matthew Hodgson cd241d6bda incorporate more review 2018-07-24 12:39:40 +01:00
Matthew Hodgson 650daf5628 make test work 2018-07-19 20:49:44 +01:00
Matthew Hodgson 1fa4f7e03e first cut of a UT for testing state store (untested) 2018-07-19 20:19:32 +01:00
Amber Brown 49af402019 run isort 2018-07-09 16:09:20 +10:00
Richard van der Hoff a995fdae39 fix tests 2018-05-29 20:19:29 +01:00
Richard van der Hoff 9ea219c514 Send users a server notice about consent
When a user first syncs, we will send them a server notice asking them to
consent to the privacy policy if they have not already done so.
2018-05-22 11:54:51 +01:00
Erik Johnston 6406b70aeb Use stream rather depth ordering for push actions
This simplifies things as it is, but will also allow us to change the
way we traverse topologically without having to update the way push
actions work.
2018-05-11 15:30:11 +01:00
Richard van der Hoff 46beeb9a30 Fix a couple of logcontext leaks in unit tests
... which were making other, innocent, tests, fail.

Plus remove a spurious unittest.DEBUG which was making the output noisy.
2018-05-02 15:46:22 +01:00
Richard van der Hoff 692a3cc806
Merge pull request #3103 from NotAFile/py3-baseexcepton-message
Use str(e) instead of e.message
2018-04-20 15:34:49 +01:00
Richard van der Hoff 639480e14a Avoid creating events with huge numbers of prev_events
In most cases, we limit the number of prev_events for a given event to 10
events. This fixes a particular code path which created events with huge
numbers of prev_events.
2018-04-16 18:41:37 +01:00
Adrian Tschira 1515560f5c Use str(e) instead of e.message
Doing this I learned e.message was pretty shortlived, added in 2.6,
they realized it was a bad idea and deprecated it in 2.7

Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 20:32:42 +02:00
Erik Johnston cb9f8e527c s/replication_client/federation_client/ 2018-03-13 13:26:52 +00:00
Erik Johnston 6ea27fafad Fix tests 2018-03-13 10:55:47 +00:00
Richard van der Hoff 2c911d75e8 Fix comment typo 2018-03-05 12:24:49 +00:00
Richard van der Hoff c818fcab11 Test and fix find_first_stream_ordering_after_ts
It seemed to suffer from a bunch of off-by-one errors.
2018-03-05 12:04:02 +00:00
Erik Johnston e5b4a208ce
Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions
Batch inserts into event_push_actions_staging
2018-02-26 14:45:40 +00:00
Erik Johnston 25634ed152 Fix test 2018-02-20 12:40:44 +00:00
Erik Johnston e440e28456 Fix unit tests 2018-02-20 11:41:40 +00:00
Erik Johnston 3a061cae26 Fix unit test 2018-02-15 16:31:59 +00:00
Erik Johnston 770b2252ca s/_create_new_client_event/create_new_client_event/ 2018-02-06 16:40:30 +00:00
Erik Johnston 25c0a020f4 Updates tests 2018-02-05 16:01:48 +00:00
Richard van der Hoff d1f3490e75 Add tests for user directory search 2018-01-27 17:21:57 +00:00
Richard van der Hoff 63ef607f1f Fix tests for Store.__init__ update
Fix the test to pass the right number of args to the Store constructors
2017-11-13 10:46:08 +00:00
Richard van der Hoff 02237ce725 Fix tests for refresh_token removal 2017-11-01 10:19:42 +00:00
Richard van der Hoff eaaabc6c4f replace 'except:' with 'except Exception:'
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Richard van der Hoff 9b714abf35 Remove dead class
This isn't used anywhere.
2017-10-17 10:43:36 +01:00
Erik Johnston ed9a7f5436 Merge pull request #2309 from matrix-org/erikj/user_ip_repl
Fix up user_ip replication commands
2017-07-06 14:33:14 +01:00
Erik Johnston 29a4066a4d Update test 2017-07-04 10:21:25 +01:00
Erik Johnston 8c23221666 Fix up 2017-06-27 15:53:45 +01:00
Erik Johnston a0a561ae85 Fix up client ips to read from pending data 2017-06-27 14:46:12 +01:00
Erik Johnston 93e7a38370 Remove unhelpful test 2017-06-15 09:30:54 +01:00
Erik Johnston 3e5a62ecd8 Add more granular event send metrics 2017-05-02 14:23:26 +01:00
Erik Johnston eefd9fee81 Fix up tests 2017-03-30 14:14:46 +01:00
Erik Johnston 00957d1aa4 User Cursor.__iter__ instead of fetchall
This prevents unnecessary construction of lists
2017-03-23 17:53:49 +00:00
Richard van der Hoff 95f21c7a66 Fix caching of remote servers' signature keys
The `@cached` decorator on `KeyStore._get_server_verify_key` was missing
its `num_args` parameter, which meant that it was returning the wrong key for
any server which had more than one recorded key.

By way of a fix, change the default for `num_args` to be *all* arguments. To
implement that, factor out a common base class for `CacheDescriptor` and `CacheListDescriptor`.
2017-03-22 15:11:30 +00:00
Erik Johnston 095b45c165 Aggregate event push actions 2017-02-14 13:39:41 +00:00
Erik Johnston c7a26b7c32 Fix unit tests 2017-01-30 17:11:24 +00:00
Erik Johnston b3e1f2aa7a Fix unit tests 2017-01-26 17:16:24 +00:00
Erik Johnston c974116f19 Implement device key caching over federation 2017-01-26 16:07:24 +00:00
Erik Johnston f85b6ca494 Speed up cache size calculation
Instead of calculating the size of the cache repeatedly, which can take
a long time now that it can use a callback, instead cache the size and
update that on insertion and deletion.

This requires changing the cache descriptors to have two caches, one for
pending deferreds and the other for the actual values. There's no reason
to evict from the pending deferreds as they won't take up any more
memory.
2017-01-17 11:18:13 +00:00
Richard van der Hoff 12f3b9000c fix imports 2016-11-30 17:45:49 +00:00
Richard van der Hoff aa09d6b8f0 Rip out more refresh_token code
We might as well treat all refresh_tokens as invalid. Just return a 403 from
/tokenrefresh, so that we don't have a load of dead, untestable code hanging
around.

Still TODO: removing the table from the schema.
2016-11-30 17:40:18 +00:00
Erik Johnston 524d61bf7e Fix tests 2016-11-21 11:53:02 +00:00
Erik Johnston 35e2cc8b52 Merge pull request #1155 from matrix-org/erikj/pluggable_pwd_auth
Implement pluggable password auth
2016-10-12 11:41:20 +01:00
Patrik Oldsberg 9bfc617791 storage/appservice: make appservice methods only relying on the cache synchronous 2016-10-06 15:24:59 +02:00
Erik Johnston 850b103b36 Implement pluggable password auth
Allows delegating the password auth to an external module. This also
moves the LDAP auth to using this system, allowing it to be removed from
the synapse tree entirely in the future.
2016-10-03 10:36:40 +01:00
Erik Johnston 93b32d4515 Fix unit tests 2016-08-26 15:40:27 +01:00
Erik Johnston 45fd2c8942 Ensure invalidation list does not grow unboundedly 2016-08-19 16:09:16 +01:00
Erik Johnston c0d7d9d642 Rename to on_invalidate 2016-08-19 15:13:58 +01:00
Erik Johnston dc76a3e909 Make cache_context an explicit option 2016-08-19 15:02:38 +01:00
Erik Johnston ba214a5e32 Remove lru option 2016-08-19 14:17:11 +01:00
Erik Johnston 4161ff2fc4 Add concept of cache contexts 2016-08-19 14:17:07 +01:00
Richard van der Hoff 98385888b8 PEP8 2016-08-03 15:42:08 +01:00
Richard van der Hoff 68264d7404 Include device name in /keys/query response
Add an 'unsigned' section which includes the device display name.
2016-08-03 15:42:08 +01:00
Mark Haines 0a7d3cd00f Create separate methods for getting messages to push
for the email and http pushers rather than trying to make a single
method that will work with their conflicting requirements.

The http pusher needs to get the messages in ascending stream order, and
doesn't want to miss a message.

The email pusher needs to get the messages in descending timestamp order,
and doesn't mind if it misses messages.
2016-07-28 20:24:24 +01:00
Richard van der Hoff 8e02494166 Delete refresh tokens when deleting devices 2016-07-26 11:10:37 +01:00
Richard van der Hoff 012b4c1913 Implement updating devices
You can update the displayname of devices now.
2016-07-26 07:35:48 +01:00
Richard van der Hoff 42f4feb2b7 PEP8 2016-07-25 12:25:06 +01:00
Richard van der Hoff 465117d7ca Fix background_update tests
A bit of a cleanup for background_updates, and make sure that the real
background updates have run before we start the unit tests, so that they don't
interfere with the tests.
2016-07-25 12:10:42 +01:00
Richard van der Hoff 40a1c96617 Fix PEP8 errors 2016-07-20 18:06:28 +01:00
Richard van der Hoff bc8f265f0a GET /devices endpoint
implement a GET /devices endpoint which lists all of the user's devices.

It also returns the last IP where we saw that device, so there is some dancing
to fish that out of the user_ips table.
2016-07-20 16:42:32 +01:00
Erik Johnston 1e2a7f18a1 Merge pull request #922 from matrix-org/erikj/file_api2
Feature: Add filter to /messages. Add 'contains_url' to filter.
2016-07-20 10:40:48 +01:00
Richard van der Hoff f863a52cea Add device_id support to /login
Add a 'devices' table to the storage, as well as a 'device_id' column to
refresh_tokens.

Allow the client to pass a device_id, and initial_device_display_name, to
/login. If login is successful, then register the device in the devices table
if it wasn't known already. If no device_id was supplied, make one up.

Associate the device_id with the access token and refresh token, so that we can
get at it again later. Ensure that the device_id is copied from the refresh
token to the access_token when the token is refreshed.
2016-07-18 16:39:44 +01:00
Erik Johnston a98d215204 Add filter param to /messages API 2016-07-14 16:30:56 +01:00
Erik Johnston 84379062f9 Fix AS retries, but with correct ordering 2016-06-07 10:24:50 +01:00
Erik Johnston 310197bab5 Fix AS retries 2016-06-07 09:34:50 +01:00
Erik Johnston 05e01f21d7 Remove event fetching from DB threads 2016-06-03 17:22:13 +01:00
Mark Haines 816df9f267 get_room_members is unused now 2016-05-16 19:51:43 +01:00