Commit Graph

193 Commits

Author SHA1 Message Date
Erik Johnston eefd9fee81 Fix up tests 2017-03-30 14:14:46 +01:00
Erik Johnston 00957d1aa4 User Cursor.__iter__ instead of fetchall
This prevents unnecessary construction of lists
2017-03-23 17:53:49 +00:00
Richard van der Hoff 95f21c7a66 Fix caching of remote servers' signature keys
The `@cached` decorator on `KeyStore._get_server_verify_key` was missing
its `num_args` parameter, which meant that it was returning the wrong key for
any server which had more than one recorded key.

By way of a fix, change the default for `num_args` to be *all* arguments. To
implement that, factor out a common base class for `CacheDescriptor` and `CacheListDescriptor`.
2017-03-22 15:11:30 +00:00
Erik Johnston 095b45c165 Aggregate event push actions 2017-02-14 13:39:41 +00:00
Erik Johnston c7a26b7c32 Fix unit tests 2017-01-30 17:11:24 +00:00
Erik Johnston b3e1f2aa7a Fix unit tests 2017-01-26 17:16:24 +00:00
Erik Johnston c974116f19 Implement device key caching over federation 2017-01-26 16:07:24 +00:00
Erik Johnston f85b6ca494 Speed up cache size calculation
Instead of calculating the size of the cache repeatedly, which can take
a long time now that it can use a callback, instead cache the size and
update that on insertion and deletion.

This requires changing the cache descriptors to have two caches, one for
pending deferreds and the other for the actual values. There's no reason
to evict from the pending deferreds as they won't take up any more
memory.
2017-01-17 11:18:13 +00:00
Richard van der Hoff 12f3b9000c fix imports 2016-11-30 17:45:49 +00:00
Richard van der Hoff aa09d6b8f0 Rip out more refresh_token code
We might as well treat all refresh_tokens as invalid. Just return a 403 from
/tokenrefresh, so that we don't have a load of dead, untestable code hanging
around.

Still TODO: removing the table from the schema.
2016-11-30 17:40:18 +00:00
Erik Johnston 524d61bf7e Fix tests 2016-11-21 11:53:02 +00:00
Erik Johnston 35e2cc8b52 Merge pull request #1155 from matrix-org/erikj/pluggable_pwd_auth
Implement pluggable password auth
2016-10-12 11:41:20 +01:00
Patrik Oldsberg 9bfc617791 storage/appservice: make appservice methods only relying on the cache synchronous 2016-10-06 15:24:59 +02:00
Erik Johnston 850b103b36 Implement pluggable password auth
Allows delegating the password auth to an external module. This also
moves the LDAP auth to using this system, allowing it to be removed from
the synapse tree entirely in the future.
2016-10-03 10:36:40 +01:00
Erik Johnston 93b32d4515 Fix unit tests 2016-08-26 15:40:27 +01:00
Erik Johnston 45fd2c8942 Ensure invalidation list does not grow unboundedly 2016-08-19 16:09:16 +01:00
Erik Johnston c0d7d9d642 Rename to on_invalidate 2016-08-19 15:13:58 +01:00
Erik Johnston dc76a3e909 Make cache_context an explicit option 2016-08-19 15:02:38 +01:00
Erik Johnston ba214a5e32 Remove lru option 2016-08-19 14:17:11 +01:00
Erik Johnston 4161ff2fc4 Add concept of cache contexts 2016-08-19 14:17:07 +01:00
Richard van der Hoff 98385888b8 PEP8 2016-08-03 15:42:08 +01:00
Richard van der Hoff 68264d7404 Include device name in /keys/query response
Add an 'unsigned' section which includes the device display name.
2016-08-03 15:42:08 +01:00
Mark Haines 0a7d3cd00f Create separate methods for getting messages to push
for the email and http pushers rather than trying to make a single
method that will work with their conflicting requirements.

The http pusher needs to get the messages in ascending stream order, and
doesn't want to miss a message.

The email pusher needs to get the messages in descending timestamp order,
and doesn't mind if it misses messages.
2016-07-28 20:24:24 +01:00
Richard van der Hoff 8e02494166 Delete refresh tokens when deleting devices 2016-07-26 11:10:37 +01:00
Richard van der Hoff 012b4c1913 Implement updating devices
You can update the displayname of devices now.
2016-07-26 07:35:48 +01:00
Richard van der Hoff 42f4feb2b7 PEP8 2016-07-25 12:25:06 +01:00
Richard van der Hoff 465117d7ca Fix background_update tests
A bit of a cleanup for background_updates, and make sure that the real
background updates have run before we start the unit tests, so that they don't
interfere with the tests.
2016-07-25 12:10:42 +01:00
Richard van der Hoff 40a1c96617 Fix PEP8 errors 2016-07-20 18:06:28 +01:00
Richard van der Hoff bc8f265f0a GET /devices endpoint
implement a GET /devices endpoint which lists all of the user's devices.

It also returns the last IP where we saw that device, so there is some dancing
to fish that out of the user_ips table.
2016-07-20 16:42:32 +01:00
Erik Johnston 1e2a7f18a1 Merge pull request #922 from matrix-org/erikj/file_api2
Feature: Add filter to /messages. Add 'contains_url' to filter.
2016-07-20 10:40:48 +01:00
Richard van der Hoff f863a52cea Add device_id support to /login
Add a 'devices' table to the storage, as well as a 'device_id' column to
refresh_tokens.

Allow the client to pass a device_id, and initial_device_display_name, to
/login. If login is successful, then register the device in the devices table
if it wasn't known already. If no device_id was supplied, make one up.

Associate the device_id with the access token and refresh token, so that we can
get at it again later. Ensure that the device_id is copied from the refresh
token to the access_token when the token is refreshed.
2016-07-18 16:39:44 +01:00
Erik Johnston a98d215204 Add filter param to /messages API 2016-07-14 16:30:56 +01:00
Erik Johnston 84379062f9 Fix AS retries, but with correct ordering 2016-06-07 10:24:50 +01:00
Erik Johnston 310197bab5 Fix AS retries 2016-06-07 09:34:50 +01:00
Erik Johnston 05e01f21d7 Remove event fetching from DB threads 2016-06-03 17:22:13 +01:00
Mark Haines 816df9f267 get_room_members is unused now 2016-05-16 19:51:43 +01:00
Mark Haines 7e2f971c08 Remove some unused functions (#711)
* Remove some unused functions

* get_room_events_stream is only used in tests

* is_exclusive_room might actually be something we want
2016-04-08 14:01:56 +01:00
Erik Johnston 8aab9d87fa Don't require config to create database 2016-04-06 14:15:45 +01:00
Mark Haines 673c96ce97 Remove dead code left over from presence changes 2016-03-17 11:03:47 +00:00
Daniel Wagner-Hall 95481e7ba7 Merge pull request #571 from matrix-org/daniel/asids
Mark AS users with their AS's ID
2016-03-03 13:56:28 +00:00
Erik Johnston ce2cdced61 Move cache size fiddling to descriptors only. Fix tests 2016-03-01 13:21:46 +00:00
Mark Haines 700487a7c7 Fix flake8 warnings for tests 2016-02-19 15:34:38 +00:00
Erik Johnston 9da9826b85 Remove old tests. 2016-02-18 10:46:16 +00:00
Daniel Wagner-Hall 763360594d Mark AS users with their AS's ID 2016-02-11 17:26:42 +00:00
Erik Johnston f8aae79a72 Simplify get_rooms 2016-02-03 13:24:35 +00:00
Erik Johnston 0487c9441f Fix tests 2016-01-27 17:33:13 +00:00
Daniel Wagner-Hall 2680043bc6 Require ID and as_token be unique for ASs
Defaults ID to as_token if not specified. This will change
when IDs are fully supported.
2016-01-14 14:34:01 +00:00
David Baker c232780081 Merge pull request #456 from matrix-org/store_event_actions
Send unread notification counts
2016-01-08 14:47:15 +00:00
Matthew Hodgson 6c28ac260c copyrights 2016-01-07 04:26:29 +00:00
David Baker 0e48f7f245 fix tests 2016-01-06 16:46:41 +00:00
Oleg Girko 4013216fcc Fix mock import in tests.
For some reason, one test imports Mock class from mock.mock
rather than from mock.
This change fixes this error.

Signed-off-by: Oleg Girko <ol@infoserver.lv>
2015-12-06 20:50:11 +00:00
Erik Johnston e21cef9bb5 Merge pull request #359 from matrix-org/markjh/incremental_indexing
Incremental background updates for db indexes
2015-11-11 17:19:51 +00:00
Daniel Wagner-Hall cf437900e0 Return world_readable and guest_can_join in /publicRooms 2015-11-10 17:10:27 +00:00
Mark Haines 36c58b18a3 Test for background updates 2015-11-10 15:51:40 +00:00
Daniel Wagner-Hall 771ca56c88 Remove more unused parameters 2015-11-02 15:31:57 +00:00
Mark Haines 1ee3d26432 synapse/storage/_base.py:_simple_selectupdate_one was unused 2015-09-23 10:30:03 +01:00
Daniel Wagner-Hall 7213588083 Implement configurable stats reporting
SYN-287

This requires that HS owners either opt in or out of stats reporting.

When --generate-config is passed, --report-stats must be specified
If an already-generated config is used, and doesn't have the
report_stats key, it is requested to be set.
2015-09-22 12:57:40 +01:00
Daniel Wagner-Hall a9d8bd95e7 Stop looking up "admin", which we never read 2015-08-25 16:29:39 +01:00
Daniel Wagner-Hall a0b181bd17 Remove completely unused concepts from codebase
Removes device_id and ClientInfo

device_id is never actually written, and the matrix.org DB has no
non-null entries for it. Right now, it's just cluttering up code.

This doesn't remove the columns from the database, because that's
fiddly.
2015-08-25 16:23:06 +01:00
Daniel Wagner-Hall cecbd636e9 /tokenrefresh POST endpoint
This allows refresh tokens to be exchanged for (access_token,
refresh_token).

It also starts issuing them on login, though no clients currently
interpret them.
2015-08-20 16:21:35 +01:00
Daniel Wagner-Hall 13a6517d89 s/by_token/by_access_token/g
We're about to have two kinds of token, access and refresh
2015-08-20 16:01:29 +01:00
Erik Johnston 2df8dd9b37 Move all the caches into their own package, synapse.util.caches 2015-08-11 18:00:59 +01:00
Erik Johnston b8e386db59 Change Cache to not use *args in its interface 2015-08-07 11:52:21 +01:00
Erik Johnston 7eea3e356f Make @cached cache deferreds rather than the deferreds' values 2015-08-06 13:33:34 +01:00
Erik Johnston d8866d7277 Caches should be bound to instances.
Before, caches were global and so different instances of the stores
would share caches. This caused problems in the unit tests.
2015-06-03 14:45:17 +01:00
Erik Johnston a2c4f3f150 Fix daedlock 2015-05-15 10:54:04 +01:00
Erik Johnston 43c2e8deae Add support for using executemany 2015-05-05 15:13:25 +01:00
Erik Johnston 22d7a59306 Fix tests after commit 9a0579 2015-04-08 16:57:14 +01:00
Erik Johnston 4fe95094d1 Merge branch 'develop' of github.com:matrix-org/synapse into mysql 2015-04-07 18:05:39 +01:00
Erik Johnston 9236136f3a Make work in both Maria and SQLite. Fix tests 2015-04-01 14:12:33 +01:00
Kegan Dougal cf1fa59f4b Use a sender localpart instead of a user ID.
Form the user ID at runtime instead, This gives less room for error in AS
config files since they cannot specify the domain of another HS.
2015-03-31 13:48:03 +01:00
Kegan Dougal c217504949 Edit SQL schema to use string IDs not ints. Use token as ID. Update tests. 2015-03-31 12:07:56 +01:00
Kegan Dougal b59aa74556 Fix tests and missing returns on deferreds. 2015-03-31 11:35:45 +01:00
Kegan Dougal d33ae65efc Remove more reg/unreg methods. Read config not database for cache. 2015-03-31 11:00:00 +01:00
Kegan Dougal 4edcbcee3b Merge branch 'develop' into application-services-txn-reliability
Conflicts:
	synapse/storage/__init__.py
2015-03-26 10:07:59 +00:00
Paul "LeoNerd" Evans 9ba6487b3f Allow a choice of LRU behaviour for Cache() by using LruCache() or OrderedDict() 2015-03-25 19:05:34 +00:00
Paul "LeoNerd" Evans 7ab9f91a60 Unit-test that Cache() key eviction is ordered 2015-03-25 18:50:43 +00:00
Paul "LeoNerd" Evans 0f86312c4c Pull out the cache logic from the @cached wrapper into its own class we can reuse 2015-03-20 18:25:42 +00:00
Erik Johnston 7e282a53a5 Tidy up _simple_... methods 2015-03-20 15:05:10 +00:00
Kegan Dougal b98cd03193 Use event IDs instead of dumping event content in the txns table. 2015-03-09 17:25:20 +00:00
Kegan Dougal 0a60bbf4fa Finish appservice txn storage impl and tests. 2015-03-09 15:53:03 +00:00
Kegan Dougal 1ead1caa18 Implement create_appservice_txn with tests. 2015-03-09 13:54:20 +00:00
Kegan Dougal 1c2dcf762a Partially implement txn store methods with tests. 2015-03-09 13:10:31 +00:00
Kegan Dougal de190e49d5 Add more unit tests for exclusive namespaces. 2015-02-27 11:51:06 +00:00
Kegan Dougal 127efeeb68 Update unit tests to use new format. 2015-02-27 11:10:48 +00:00
Paul "LeoNerd" Evans f53fcbce97 Use cache.pop() instead of a separate membership test + del [] 2015-02-23 18:30:45 +00:00
Paul "LeoNerd" Evans e76d485e29 Allow @cached-wrapped functions to have a prefill method for setting entries 2015-02-23 15:41:54 +00:00
Paul "LeoNerd" Evans ebc3db295b Take named arguments to @cached() decorator, add a 'max_entries' limit 2015-02-19 18:36:02 +00:00
Kegan Dougal cb43fbeeb4 Fix tests which broke when event caching was introduced. 2015-02-11 16:46:01 +00:00
Kegan Dougal f2fdcb7c4b Merge branch 'develop' into application-services 2015-02-11 16:43:26 +00:00
Mark Haines f5a70e0d2e Add a cache for get_event 2015-02-11 15:01:15 +00:00
Mark Haines 896253e085 Factor out some of the common homeserver setup code into a
setup_test_homeserver function in utils.
2015-02-11 11:37:30 +00:00
Kegan Dougal 27091f146a Add hs_token column and generate a different token f.e application service. 2015-02-05 10:08:12 +00:00
Kegan Dougal 89f2e8fbdf Fix bug in store defer. Add more unit tests. 2015-02-04 15:21:03 +00:00
Mark Haines 388581e087 Extract the id token of the token when authing users, include the token and device_id in the internal meta data for the event along with the transaction id when sending events 2015-01-28 16:58:23 +00:00
Mark Haines ada711504e Replace hs.parse_roomalias with RoomAlias.from_string 2015-01-23 13:21:58 +00:00
Mark Haines 1c06c48ce2 Replace hs.parse_roomid with RoomID.from_string 2015-01-23 11:55:12 +00:00
Mark Haines 5759bec43c Replace hs.parse_userid with UserID.from_string 2015-01-23 11:47:15 +00:00
Mark Haines 16bfabb9c5 Fix manifest. Ignore contrib and docs directories when checking manifest against source control. 2015-01-22 14:32:51 +00:00
Mark Haines 3891597eb3 Remove unused functions 2015-01-13 15:57:26 +00:00