Commit Graph

278 Commits

Author SHA1 Message Date
Neil Johnson cefac79c10 monthly_active_tests 2018-07-30 22:08:09 +01:00
Matthew Hodgson 0620d27f4d flake8 2018-07-25 17:21:17 +01:00
Matthew Hodgson 0a7ee0ab8b add tests for _get_some_state_from_cache 2018-07-25 16:33:52 +01:00
Matthew Hodgson cd241d6bda incorporate more review 2018-07-24 12:39:40 +01:00
Matthew Hodgson 650daf5628 make test work 2018-07-19 20:49:44 +01:00
Matthew Hodgson 1fa4f7e03e first cut of a UT for testing state store (untested) 2018-07-19 20:19:32 +01:00
Amber Brown 49af402019 run isort 2018-07-09 16:09:20 +10:00
Richard van der Hoff a995fdae39 fix tests 2018-05-29 20:19:29 +01:00
Richard van der Hoff 9ea219c514 Send users a server notice about consent
When a user first syncs, we will send them a server notice asking them to
consent to the privacy policy if they have not already done so.
2018-05-22 11:54:51 +01:00
Erik Johnston 6406b70aeb Use stream rather depth ordering for push actions
This simplifies things as it is, but will also allow us to change the
way we traverse topologically without having to update the way push
actions work.
2018-05-11 15:30:11 +01:00
Richard van der Hoff 46beeb9a30 Fix a couple of logcontext leaks in unit tests
... which were making other, innocent, tests, fail.

Plus remove a spurious unittest.DEBUG which was making the output noisy.
2018-05-02 15:46:22 +01:00
Richard van der Hoff 692a3cc806
Merge pull request #3103 from NotAFile/py3-baseexcepton-message
Use str(e) instead of e.message
2018-04-20 15:34:49 +01:00
Richard van der Hoff 639480e14a Avoid creating events with huge numbers of prev_events
In most cases, we limit the number of prev_events for a given event to 10
events. This fixes a particular code path which created events with huge
numbers of prev_events.
2018-04-16 18:41:37 +01:00
Adrian Tschira 1515560f5c Use str(e) instead of e.message
Doing this I learned e.message was pretty shortlived, added in 2.6,
they realized it was a bad idea and deprecated it in 2.7

Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 20:32:42 +02:00
Erik Johnston cb9f8e527c s/replication_client/federation_client/ 2018-03-13 13:26:52 +00:00
Erik Johnston 6ea27fafad Fix tests 2018-03-13 10:55:47 +00:00
Richard van der Hoff 2c911d75e8 Fix comment typo 2018-03-05 12:24:49 +00:00
Richard van der Hoff c818fcab11 Test and fix find_first_stream_ordering_after_ts
It seemed to suffer from a bunch of off-by-one errors.
2018-03-05 12:04:02 +00:00
Erik Johnston e5b4a208ce
Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions
Batch inserts into event_push_actions_staging
2018-02-26 14:45:40 +00:00
Erik Johnston 25634ed152 Fix test 2018-02-20 12:40:44 +00:00
Erik Johnston e440e28456 Fix unit tests 2018-02-20 11:41:40 +00:00
Erik Johnston 3a061cae26 Fix unit test 2018-02-15 16:31:59 +00:00
Erik Johnston 770b2252ca s/_create_new_client_event/create_new_client_event/ 2018-02-06 16:40:30 +00:00
Erik Johnston 25c0a020f4 Updates tests 2018-02-05 16:01:48 +00:00
Richard van der Hoff d1f3490e75 Add tests for user directory search 2018-01-27 17:21:57 +00:00
Richard van der Hoff 63ef607f1f Fix tests for Store.__init__ update
Fix the test to pass the right number of args to the Store constructors
2017-11-13 10:46:08 +00:00
Richard van der Hoff 02237ce725 Fix tests for refresh_token removal 2017-11-01 10:19:42 +00:00
Richard van der Hoff eaaabc6c4f replace 'except:' with 'except Exception:'
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Richard van der Hoff 9b714abf35 Remove dead class
This isn't used anywhere.
2017-10-17 10:43:36 +01:00
Erik Johnston ed9a7f5436 Merge pull request #2309 from matrix-org/erikj/user_ip_repl
Fix up user_ip replication commands
2017-07-06 14:33:14 +01:00
Erik Johnston 29a4066a4d Update test 2017-07-04 10:21:25 +01:00
Erik Johnston 8c23221666 Fix up 2017-06-27 15:53:45 +01:00
Erik Johnston a0a561ae85 Fix up client ips to read from pending data 2017-06-27 14:46:12 +01:00
Erik Johnston 93e7a38370 Remove unhelpful test 2017-06-15 09:30:54 +01:00
Erik Johnston 3e5a62ecd8 Add more granular event send metrics 2017-05-02 14:23:26 +01:00
Erik Johnston eefd9fee81 Fix up tests 2017-03-30 14:14:46 +01:00
Erik Johnston 00957d1aa4 User Cursor.__iter__ instead of fetchall
This prevents unnecessary construction of lists
2017-03-23 17:53:49 +00:00
Richard van der Hoff 95f21c7a66 Fix caching of remote servers' signature keys
The `@cached` decorator on `KeyStore._get_server_verify_key` was missing
its `num_args` parameter, which meant that it was returning the wrong key for
any server which had more than one recorded key.

By way of a fix, change the default for `num_args` to be *all* arguments. To
implement that, factor out a common base class for `CacheDescriptor` and `CacheListDescriptor`.
2017-03-22 15:11:30 +00:00
Erik Johnston 095b45c165 Aggregate event push actions 2017-02-14 13:39:41 +00:00
Erik Johnston c7a26b7c32 Fix unit tests 2017-01-30 17:11:24 +00:00
Erik Johnston b3e1f2aa7a Fix unit tests 2017-01-26 17:16:24 +00:00
Erik Johnston c974116f19 Implement device key caching over federation 2017-01-26 16:07:24 +00:00
Erik Johnston f85b6ca494 Speed up cache size calculation
Instead of calculating the size of the cache repeatedly, which can take
a long time now that it can use a callback, instead cache the size and
update that on insertion and deletion.

This requires changing the cache descriptors to have two caches, one for
pending deferreds and the other for the actual values. There's no reason
to evict from the pending deferreds as they won't take up any more
memory.
2017-01-17 11:18:13 +00:00
Richard van der Hoff 12f3b9000c fix imports 2016-11-30 17:45:49 +00:00
Richard van der Hoff aa09d6b8f0 Rip out more refresh_token code
We might as well treat all refresh_tokens as invalid. Just return a 403 from
/tokenrefresh, so that we don't have a load of dead, untestable code hanging
around.

Still TODO: removing the table from the schema.
2016-11-30 17:40:18 +00:00
Erik Johnston 524d61bf7e Fix tests 2016-11-21 11:53:02 +00:00
Erik Johnston 35e2cc8b52 Merge pull request #1155 from matrix-org/erikj/pluggable_pwd_auth
Implement pluggable password auth
2016-10-12 11:41:20 +01:00
Patrik Oldsberg 9bfc617791 storage/appservice: make appservice methods only relying on the cache synchronous 2016-10-06 15:24:59 +02:00
Erik Johnston 850b103b36 Implement pluggable password auth
Allows delegating the password auth to an external module. This also
moves the LDAP auth to using this system, allowing it to be removed from
the synapse tree entirely in the future.
2016-10-03 10:36:40 +01:00
Erik Johnston 93b32d4515 Fix unit tests 2016-08-26 15:40:27 +01:00
Erik Johnston 45fd2c8942 Ensure invalidation list does not grow unboundedly 2016-08-19 16:09:16 +01:00
Erik Johnston c0d7d9d642 Rename to on_invalidate 2016-08-19 15:13:58 +01:00
Erik Johnston dc76a3e909 Make cache_context an explicit option 2016-08-19 15:02:38 +01:00
Erik Johnston ba214a5e32 Remove lru option 2016-08-19 14:17:11 +01:00
Erik Johnston 4161ff2fc4 Add concept of cache contexts 2016-08-19 14:17:07 +01:00
Richard van der Hoff 98385888b8 PEP8 2016-08-03 15:42:08 +01:00
Richard van der Hoff 68264d7404 Include device name in /keys/query response
Add an 'unsigned' section which includes the device display name.
2016-08-03 15:42:08 +01:00
Mark Haines 0a7d3cd00f Create separate methods for getting messages to push
for the email and http pushers rather than trying to make a single
method that will work with their conflicting requirements.

The http pusher needs to get the messages in ascending stream order, and
doesn't want to miss a message.

The email pusher needs to get the messages in descending timestamp order,
and doesn't mind if it misses messages.
2016-07-28 20:24:24 +01:00
Richard van der Hoff 8e02494166 Delete refresh tokens when deleting devices 2016-07-26 11:10:37 +01:00
Richard van der Hoff 012b4c1913 Implement updating devices
You can update the displayname of devices now.
2016-07-26 07:35:48 +01:00
Richard van der Hoff 42f4feb2b7 PEP8 2016-07-25 12:25:06 +01:00
Richard van der Hoff 465117d7ca Fix background_update tests
A bit of a cleanup for background_updates, and make sure that the real
background updates have run before we start the unit tests, so that they don't
interfere with the tests.
2016-07-25 12:10:42 +01:00
Richard van der Hoff 40a1c96617 Fix PEP8 errors 2016-07-20 18:06:28 +01:00
Richard van der Hoff bc8f265f0a GET /devices endpoint
implement a GET /devices endpoint which lists all of the user's devices.

It also returns the last IP where we saw that device, so there is some dancing
to fish that out of the user_ips table.
2016-07-20 16:42:32 +01:00
Erik Johnston 1e2a7f18a1 Merge pull request #922 from matrix-org/erikj/file_api2
Feature: Add filter to /messages. Add 'contains_url' to filter.
2016-07-20 10:40:48 +01:00
Richard van der Hoff f863a52cea Add device_id support to /login
Add a 'devices' table to the storage, as well as a 'device_id' column to
refresh_tokens.

Allow the client to pass a device_id, and initial_device_display_name, to
/login. If login is successful, then register the device in the devices table
if it wasn't known already. If no device_id was supplied, make one up.

Associate the device_id with the access token and refresh token, so that we can
get at it again later. Ensure that the device_id is copied from the refresh
token to the access_token when the token is refreshed.
2016-07-18 16:39:44 +01:00
Erik Johnston a98d215204 Add filter param to /messages API 2016-07-14 16:30:56 +01:00
Erik Johnston 84379062f9 Fix AS retries, but with correct ordering 2016-06-07 10:24:50 +01:00
Erik Johnston 310197bab5 Fix AS retries 2016-06-07 09:34:50 +01:00
Erik Johnston 05e01f21d7 Remove event fetching from DB threads 2016-06-03 17:22:13 +01:00
Mark Haines 816df9f267 get_room_members is unused now 2016-05-16 19:51:43 +01:00
Mark Haines 7e2f971c08 Remove some unused functions (#711)
* Remove some unused functions

* get_room_events_stream is only used in tests

* is_exclusive_room might actually be something we want
2016-04-08 14:01:56 +01:00
Erik Johnston 8aab9d87fa Don't require config to create database 2016-04-06 14:15:45 +01:00
Mark Haines 673c96ce97 Remove dead code left over from presence changes 2016-03-17 11:03:47 +00:00
Daniel Wagner-Hall 95481e7ba7 Merge pull request #571 from matrix-org/daniel/asids
Mark AS users with their AS's ID
2016-03-03 13:56:28 +00:00
Erik Johnston ce2cdced61 Move cache size fiddling to descriptors only. Fix tests 2016-03-01 13:21:46 +00:00
Mark Haines 700487a7c7 Fix flake8 warnings for tests 2016-02-19 15:34:38 +00:00
Erik Johnston 9da9826b85 Remove old tests. 2016-02-18 10:46:16 +00:00
Daniel Wagner-Hall 763360594d Mark AS users with their AS's ID 2016-02-11 17:26:42 +00:00
Erik Johnston f8aae79a72 Simplify get_rooms 2016-02-03 13:24:35 +00:00
Erik Johnston 0487c9441f Fix tests 2016-01-27 17:33:13 +00:00
Daniel Wagner-Hall 2680043bc6 Require ID and as_token be unique for ASs
Defaults ID to as_token if not specified. This will change
when IDs are fully supported.
2016-01-14 14:34:01 +00:00
David Baker c232780081 Merge pull request #456 from matrix-org/store_event_actions
Send unread notification counts
2016-01-08 14:47:15 +00:00
Matthew Hodgson 6c28ac260c copyrights 2016-01-07 04:26:29 +00:00
David Baker 0e48f7f245 fix tests 2016-01-06 16:46:41 +00:00
Oleg Girko 4013216fcc Fix mock import in tests.
For some reason, one test imports Mock class from mock.mock
rather than from mock.
This change fixes this error.

Signed-off-by: Oleg Girko <ol@infoserver.lv>
2015-12-06 20:50:11 +00:00
Erik Johnston e21cef9bb5 Merge pull request #359 from matrix-org/markjh/incremental_indexing
Incremental background updates for db indexes
2015-11-11 17:19:51 +00:00
Daniel Wagner-Hall cf437900e0 Return world_readable and guest_can_join in /publicRooms 2015-11-10 17:10:27 +00:00
Mark Haines 36c58b18a3 Test for background updates 2015-11-10 15:51:40 +00:00
Daniel Wagner-Hall 771ca56c88 Remove more unused parameters 2015-11-02 15:31:57 +00:00
Mark Haines 1ee3d26432 synapse/storage/_base.py:_simple_selectupdate_one was unused 2015-09-23 10:30:03 +01:00
Daniel Wagner-Hall 7213588083 Implement configurable stats reporting
SYN-287

This requires that HS owners either opt in or out of stats reporting.

When --generate-config is passed, --report-stats must be specified
If an already-generated config is used, and doesn't have the
report_stats key, it is requested to be set.
2015-09-22 12:57:40 +01:00
Daniel Wagner-Hall a9d8bd95e7 Stop looking up "admin", which we never read 2015-08-25 16:29:39 +01:00
Daniel Wagner-Hall a0b181bd17 Remove completely unused concepts from codebase
Removes device_id and ClientInfo

device_id is never actually written, and the matrix.org DB has no
non-null entries for it. Right now, it's just cluttering up code.

This doesn't remove the columns from the database, because that's
fiddly.
2015-08-25 16:23:06 +01:00
Daniel Wagner-Hall cecbd636e9 /tokenrefresh POST endpoint
This allows refresh tokens to be exchanged for (access_token,
refresh_token).

It also starts issuing them on login, though no clients currently
interpret them.
2015-08-20 16:21:35 +01:00
Daniel Wagner-Hall 13a6517d89 s/by_token/by_access_token/g
We're about to have two kinds of token, access and refresh
2015-08-20 16:01:29 +01:00
Erik Johnston 2df8dd9b37 Move all the caches into their own package, synapse.util.caches 2015-08-11 18:00:59 +01:00
Erik Johnston b8e386db59 Change Cache to not use *args in its interface 2015-08-07 11:52:21 +01:00
Erik Johnston 7eea3e356f Make @cached cache deferreds rather than the deferreds' values 2015-08-06 13:33:34 +01:00
Erik Johnston d8866d7277 Caches should be bound to instances.
Before, caches were global and so different instances of the stores
would share caches. This caused problems in the unit tests.
2015-06-03 14:45:17 +01:00
Erik Johnston a2c4f3f150 Fix daedlock 2015-05-15 10:54:04 +01:00
Erik Johnston 43c2e8deae Add support for using executemany 2015-05-05 15:13:25 +01:00
Erik Johnston 22d7a59306 Fix tests after commit 9a0579 2015-04-08 16:57:14 +01:00
Erik Johnston 4fe95094d1 Merge branch 'develop' of github.com:matrix-org/synapse into mysql 2015-04-07 18:05:39 +01:00
Erik Johnston 9236136f3a Make work in both Maria and SQLite. Fix tests 2015-04-01 14:12:33 +01:00
Kegan Dougal cf1fa59f4b Use a sender localpart instead of a user ID.
Form the user ID at runtime instead, This gives less room for error in AS
config files since they cannot specify the domain of another HS.
2015-03-31 13:48:03 +01:00
Kegan Dougal c217504949 Edit SQL schema to use string IDs not ints. Use token as ID. Update tests. 2015-03-31 12:07:56 +01:00
Kegan Dougal b59aa74556 Fix tests and missing returns on deferreds. 2015-03-31 11:35:45 +01:00
Kegan Dougal d33ae65efc Remove more reg/unreg methods. Read config not database for cache. 2015-03-31 11:00:00 +01:00
Kegan Dougal 4edcbcee3b Merge branch 'develop' into application-services-txn-reliability
Conflicts:
	synapse/storage/__init__.py
2015-03-26 10:07:59 +00:00
Paul "LeoNerd" Evans 9ba6487b3f Allow a choice of LRU behaviour for Cache() by using LruCache() or OrderedDict() 2015-03-25 19:05:34 +00:00
Paul "LeoNerd" Evans 7ab9f91a60 Unit-test that Cache() key eviction is ordered 2015-03-25 18:50:43 +00:00
Paul "LeoNerd" Evans 0f86312c4c Pull out the cache logic from the @cached wrapper into its own class we can reuse 2015-03-20 18:25:42 +00:00
Erik Johnston 7e282a53a5 Tidy up _simple_... methods 2015-03-20 15:05:10 +00:00
Kegan Dougal b98cd03193 Use event IDs instead of dumping event content in the txns table. 2015-03-09 17:25:20 +00:00
Kegan Dougal 0a60bbf4fa Finish appservice txn storage impl and tests. 2015-03-09 15:53:03 +00:00
Kegan Dougal 1ead1caa18 Implement create_appservice_txn with tests. 2015-03-09 13:54:20 +00:00
Kegan Dougal 1c2dcf762a Partially implement txn store methods with tests. 2015-03-09 13:10:31 +00:00
Kegan Dougal de190e49d5 Add more unit tests for exclusive namespaces. 2015-02-27 11:51:06 +00:00
Kegan Dougal 127efeeb68 Update unit tests to use new format. 2015-02-27 11:10:48 +00:00
Paul "LeoNerd" Evans f53fcbce97 Use cache.pop() instead of a separate membership test + del [] 2015-02-23 18:30:45 +00:00
Paul "LeoNerd" Evans e76d485e29 Allow @cached-wrapped functions to have a prefill method for setting entries 2015-02-23 15:41:54 +00:00
Paul "LeoNerd" Evans ebc3db295b Take named arguments to @cached() decorator, add a 'max_entries' limit 2015-02-19 18:36:02 +00:00
Kegan Dougal cb43fbeeb4 Fix tests which broke when event caching was introduced. 2015-02-11 16:46:01 +00:00
Kegan Dougal f2fdcb7c4b Merge branch 'develop' into application-services 2015-02-11 16:43:26 +00:00
Mark Haines f5a70e0d2e Add a cache for get_event 2015-02-11 15:01:15 +00:00
Mark Haines 896253e085 Factor out some of the common homeserver setup code into a
setup_test_homeserver function in utils.
2015-02-11 11:37:30 +00:00
Kegan Dougal 27091f146a Add hs_token column and generate a different token f.e application service. 2015-02-05 10:08:12 +00:00
Kegan Dougal 89f2e8fbdf Fix bug in store defer. Add more unit tests. 2015-02-04 15:21:03 +00:00
Mark Haines 388581e087 Extract the id token of the token when authing users, include the token and device_id in the internal meta data for the event along with the transaction id when sending events 2015-01-28 16:58:23 +00:00
Mark Haines ada711504e Replace hs.parse_roomalias with RoomAlias.from_string 2015-01-23 13:21:58 +00:00
Mark Haines 1c06c48ce2 Replace hs.parse_roomid with RoomID.from_string 2015-01-23 11:55:12 +00:00
Mark Haines 5759bec43c Replace hs.parse_userid with UserID.from_string 2015-01-23 11:47:15 +00:00
Mark Haines 16bfabb9c5 Fix manifest. Ignore contrib and docs directories when checking manifest against source control. 2015-01-22 14:32:51 +00:00
Mark Haines 3891597eb3 Remove unused functions 2015-01-13 15:57:26 +00:00
Erik Johnston 3c77d13aa5 Kill off synapse.api.events.* 2014-12-16 11:29:05 +00:00
Erik Johnston 9191292b0f Fix prev_content 2014-12-11 15:16:55 +00:00
Erik Johnston e72b16f9a3 Fix redaction tests 2014-12-11 13:38:52 +00:00
Erik Johnston 8cdebce470 Fix redactions. Fix 'age' key 2014-12-11 13:25:19 +00:00
Erik Johnston 4d6af0dde3 Fix some tests 2014-12-10 18:00:57 +00:00
Erik Johnston a46e5ef621 SYN-163: Add an order by rowid to selects.
This should fix the bug where the edges of the graph get returned in a
different order than they were inserted in, and so no get_event no
longer returned the exact same JSON as was inserted. This meant that
signature checks failed.
2014-11-24 10:56:36 +00:00
Erik Johnston 4ebdb19682 Fix SQLBaseStoreTestCase 2014-11-10 15:32:35 +00:00
Erik Johnston 3cd9c02f71 Fix stream test. 2014-11-10 15:29:19 +00:00
Erik Johnston e2cebe26e8 Fix room_member storage test 2014-11-10 15:24:15 +00:00
Erik Johnston c174d19d1e Fix redaction storage test 2014-11-10 15:21:41 +00:00
Erik Johnston 3791b75000 Fix a couple more storage tests 2014-11-05 13:48:36 +00:00
Erik Johnston 2fcce3b3c5 Remove stale tests 2014-11-05 13:43:36 +00:00
Erik Johnston da80ebcc6b Fix redaction storage test 2014-11-05 13:28:57 +00:00
Erik Johnston e06adc6d7e SYN-2: Allow server admins to delete room aliases 2014-09-30 11:31:42 +01:00
Erik Johnston d96cb61f26 Unbreak tests after changing storage API 2014-09-29 15:35:57 +01:00