Commit Graph

844 Commits

Author SHA1 Message Date
Erik Johnston e440e28456 Fix unit tests 2018-02-20 11:41:40 +00:00
Erik Johnston 3a061cae26 Fix unit test 2018-02-15 16:31:59 +00:00
Erik Johnston 4810f7effd Remove context.push_actions 2018-02-15 15:47:06 +00:00
Erik Johnston 617199d73d
Merge pull request #2847 from matrix-org/erikj/separate_event_creation
Split event creation into a separate handler
2018-02-06 17:01:17 +00:00
Erik Johnston 770b2252ca s/_create_new_client_event/create_new_client_event/ 2018-02-06 16:40:30 +00:00
Erik Johnston 3d33eef6fc
Store state groups separately from events (#2784)
* Split state group persist into seperate storage func

* Add per database engine code for state group id gen

* Move store_state_group to StateReadStore

This allows other workers to use it, and so resolve state.

* Hook up store_state_group

* Fix tests

* Rename _store_mult_state_groups_txn

* Rename StateGroupReadStore

* Remove redundant _have_persisted_state_group_txn

* Update comments

* Comment compute_event_context

* Set start val for state_group_id_seq

... otherwise we try to recreate old state groups

* Update comments

* Don't store state for outliers

* Update comment

* Update docstring as state groups are ints
2018-02-06 14:31:24 +00:00
Erik Johnston 25c0a020f4 Updates tests 2018-02-05 16:01:48 +00:00
Richard van der Hoff db91e72ade
Merge pull request #2844 from matrix-org/rav/evicted_metrics
montoring metrics for number of cache evictions
2018-02-05 16:34:36 +01:00
Richard van der Hoff bc496df192 report metrics on number of cache evictions 2018-02-05 15:34:01 +00:00
Erik Johnston a1beca0e25 Fix broken unit test for media storage 2018-02-05 12:44:03 +00:00
Erik Johnston 1f881e0746
Merge pull request #2791 from matrix-org/erikj/media_storage_refactor
Ensure media is in local cache before thumbnailing
2018-02-05 11:28:52 +00:00
Richard van der Hoff 6da4c4d3bd Factor out resolve_state_groups to a separate handler
We extract the storage-independent bits of the state group resolution out to a
separate functiom, and stick it in a new handler, in preparation for its use
from the storage layer.
2018-02-01 12:40:04 +00:00
Richard van der Hoff d1f3490e75 Add tests for user directory search 2018-01-27 17:21:57 +00:00
Richard van der Hoff 88b9c5cbf0 Make it possible to run tests against postgres 2018-01-27 17:15:24 +00:00
Richard van der Hoff d7eacc4f87 Create dbpool as normal in tests
... instead of creating our own special SQLiteMemoryDbPool, whose purpose was a
bit of a mystery.

For some reason this makes one of the tests run slightly slower, so bump the
sleep(). Sorry.
2018-01-27 17:15:15 +00:00
Richard van der Hoff b178eca261 Run on_new_connection for unit tests
Configure the connectionpool used for unit tests to run the `on_new_connection`
function.
2018-01-27 17:06:04 +00:00
Richard van der Hoff 6e9bf67f18 Remove unused/bitrotted MemoryDataStore
This isn't used, and looks thoroughly bitrotted.
2018-01-26 00:35:15 +00:00
Richard van der Hoff 2b91846497 Remove spurious unittest.DEBUG 2018-01-26 00:34:27 +00:00
Richard van der Hoff b2da272b77
Merge pull request #2821 from matrix-org/rav/matthew_test_fixes
Matthew's fixes to the unit tests
2018-01-22 20:43:56 +00:00
Matthew Hodgson ab9f844aaf
Add federation_domain_whitelist option (#2820)
Add federation_domain_whitelist

gives a way to restrict which domains your HS is allowed to federate with.
useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network
2018-01-22 19:11:18 +01:00
Richard van der Hoff 5c431f421c Matthew's fixes to the unit tests
Extracted from https://github.com/matrix-org/synapse/pull/2820
2018-01-22 16:46:16 +00:00
Matthew Hodgson d84f65255e
Merge pull request #2813 from matrix-org/matthew/registrations_require_3pid
add registrations_require_3pid and allow_local_3pids
2018-01-22 13:57:22 +00:00
Matthew Hodgson 81d037dbd8 mock registrations_require_3pid 2018-01-19 00:28:08 +00:00
Erik Johnston ce4f66133e Add unit tests 2018-01-18 16:32:11 +00:00
Erik Johnston b6dc7044a9
Merge pull request #2804 from matrix-org/erikj/file_consumer
Add decent impl of a FileConsumer
2018-01-18 16:31:33 +00:00
Erik Johnston 1432f7ccd5 Move test stuff to tests 2018-01-18 11:57:57 +00:00
Erik Johnston bc67e7d260 Add decent impl of a FileConsumer
Twisted core doesn't have a general purpose one, so we need to write one
ourselves.

Features:
- All writing happens in background thread
- Supports both push and pull producers
- Push producers get paused if the consumer falls behind
2018-01-17 16:43:03 +00:00
Richard van der Hoff 44a498418c Optimise LoggingContext creation and copying
It turns out that the only thing we use the __dict__ of LoggingContext for is
`request`, and given we create lots of LoggingContexts and then copy them every
time we do a db transaction or log line, using the __dict__ seems a bit
redundant. Let's try to optimise things by making the request attribute
explicit.
2018-01-16 15:49:42 +00:00
Richard van der Hoff a767f06e3f
Merge pull request #2765 from matrix-org/rav/fix_room_uts
Fix flaky test_rooms UTs
2018-01-09 14:21:03 +00:00
Richard van der Hoff f8fa5ae4af enable twisted delayedcall debugging in UTs 2018-01-09 12:06:45 +00:00
Richard van der Hoff 374c4d4ced Remove dead code
pointless function is pointless
2018-01-09 12:06:45 +00:00
Richard van der Hoff 142fb0a7d4 Disable user_directory updates for UTs
Fix flakiness in the UTs caused by the user_directory being updated in the
background
2018-01-09 12:06:45 +00:00
Richard van der Hoff 0211464ba2 Fix broken config UTs
https://github.com/matrix-org/synapse/pull/2755 broke log-config generation,
which in turn broke the unit tests.
2018-01-09 11:28:33 +00:00
Matthew Hodgson 33cb7ef0b7
Merge pull request #2723 from matrix-org/matthew/search-all-local-users
Add all local users to the user_directory and optionally search them
2017-12-05 11:09:47 +00:00
Richard van der Hoff d5f9fb06b0 Refactor UI auth implementation
Instead of returning False when auth is incomplete, throw an exception which
can be caught with a wrapper.
2017-12-05 09:40:05 +00:00
Matthew Hodgson 7b86c1fdcd try make tests work a bit more... 2017-12-04 17:10:03 +00:00
Matthew Hodgson 95f8a713dc erik told me to 2017-12-04 16:56:25 +00:00
Luke Barnard ab1dc84779 Add extra space before inline comment 2017-11-16 18:22:40 +00:00
Luke Barnard e836bdf734 Fix tests 2017-11-16 18:14:39 +00:00
Richard van der Hoff 63ef607f1f Fix tests for Store.__init__ update
Fix the test to pass the right number of args to the Store constructors
2017-11-13 10:46:08 +00:00
Ilya Zhuravlev 8a4a0ddea6 Fix appservice tests to account for new behavior of notify_interested_services 2017-11-02 23:19:57 +03:00
Richard van der Hoff 02237ce725 Fix tests for refresh_token removal 2017-11-01 10:19:42 +00:00
Richard van der Hoff 9ded00f221 fix tests 2017-10-31 14:21:13 +00:00
Richard van der Hoff eaaabc6c4f replace 'except:' with 'except Exception:'
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Richard van der Hoff 1135193dfd Validate group ids when parsing
May as well do it whenever we parse a Group ID. We check the sigil and basic
structure here so it makes sense to check the grammar in the same place.
2017-10-21 00:30:39 +01:00
Richard van der Hoff dbdfd8967d Merge pull request #2546 from matrix-org/rav/remove_dead_event_injector
Remove dead class
2017-10-17 13:21:22 +01:00
Richard van der Hoff b8e40d146f Merge pull request #2547 from matrix-org/rav/test_make_deferred_yieldable
Add some tests for make_deferred_yieldable
2017-10-17 13:21:09 +01:00
Richard van der Hoff a6ad8148b9 Fix name of test_logcontext
The file under test is logcontext.py, not log_context.py
2017-10-17 10:53:34 +01:00
Richard van der Hoff 5b5f35ccc0 Add some tests for make_deferred_yieldable 2017-10-17 10:52:31 +01:00
Richard van der Hoff 9b714abf35 Remove dead class
This isn't used anywhere.
2017-10-17 10:43:36 +01:00
David Baker 33122c5a1b Fix test 2017-10-17 10:39:50 +01:00
Richard van der Hoff cc794d60e7 Merge pull request #2532 from matrix-org/rav/fix_linearizer
Fix stackoverflow and logcontexts from linearizer
2017-10-11 17:29:32 +01:00
Richard van der Hoff 4fad8efbfb Fix stackoverflow and logcontexts from linearizer
1. make it not blow out the stack when there are more than 50 things waiting
   for a lock. Fixes https://github.com/matrix-org/synapse/issues/2505.

2. Make it not mess up the log contexts.
2017-10-11 15:05:05 +01:00
David Baker 27955056e0 Merge branch 'develop' into erikj/groups_merged 2017-10-02 16:20:41 +01:00
Richard van der Hoff f65e31d22f Do an AAAA lookup on SRV record targets (#2462)
Support SRV records which point at AAAA records, as well as A records.

Fixes https://github.com/matrix-org/synapse/issues/2405
2017-09-22 20:26:47 +01:00
Richard van der Hoff 72472456d8 Add some more tests for Keyring 2017-09-20 01:32:42 +01:00
Richard van der Hoff 9864efa532 Fix concurrent server_key requests (#2458)
Fix a bug where we could end up firing off multiple requests for server_keys
for the same server at the same time.
2017-09-19 23:25:44 +01:00
Richard van der Hoff aa620d09a0 Add a config option to block all room invites (#2457)
- allows sysadmins the ability to lock down their servers so that people can't
send their users room invites.
2017-09-19 16:08:14 +01:00
Erik Johnston b5feaa5a49 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/groups_merged 2017-09-19 11:07:45 +01:00
Richard van der Hoff 290777b3d9 Clean up and document handling of logcontexts in Keyring (#2452)
I'm still unclear on what the intended behaviour for
`verify_json_objects_for_server` is, but at least I now understand the
behaviour of most of the things it calls...
2017-09-18 18:31:01 +01:00
Erik Johnston bf81f3cf2c Split out profile handler to fix tests 2017-08-25 14:34:56 +01:00
Erik Johnston c544188ee3 Add groups to sync stream 2017-07-20 16:36:42 +01:00
Erik Johnston ed9a7f5436 Merge pull request #2309 from matrix-org/erikj/user_ip_repl
Fix up user_ip replication commands
2017-07-06 14:33:14 +01:00
Erik Johnston 29a4066a4d Update test 2017-07-04 10:21:25 +01:00
Erik Johnston 8c23221666 Fix up 2017-06-27 15:53:45 +01:00
Erik Johnston a0a561ae85 Fix up client ips to read from pending data 2017-06-27 14:46:12 +01:00
Erik Johnston 052c5d19d5 Merge pull request #2281 from matrix-org/erikj/phone_home_stats
Fix phone home stats
2017-06-15 12:46:23 +01:00
Erik Johnston 93e7a38370 Remove unhelpful test 2017-06-15 09:30:54 +01:00
Erik Johnston 23da638360 Fix typing tests 2017-05-26 10:02:04 +01:00
Erik Johnston 11c2a3655f Only load jinja2 templates once
Instead of every time a new email pusher is created, as loading jinja2
templates is slow.
2017-05-22 17:48:58 +01:00
Erik Johnston bbfe4e996c Make get_state_groups_from_groups faster.
Most of the time was spent copying a dict to filter out sentinel values
that indicated that keys did not exist in the dict. The sentinel values
were added to ensure that we cached the non-existence of keys.

By updating DictionaryCache to keep track of which keys were known to
not exist itself we can remove a dictionary copy.
2017-05-17 15:12:15 +01:00
Richard van der Hoff de042b3b88 Do some logging when one-time-keys get claimed
might help us figure out if https://github.com/vector-im/riot-web/issues/3868
has happened.
2017-05-09 19:04:56 +01:00
Richard van der Hoff a7e9d8762d Allow clients to upload one-time-keys with new sigs
When a client retries a key upload, don't give an error if the signature has
changed (but the key is the same).

Fixes https://github.com/vector-im/riot-android/issues/1208, hopefully.
2017-05-09 19:04:56 +01:00
Erik Johnston 3e5a62ecd8 Add more granular event send metrics 2017-05-02 14:23:26 +01:00
Erik Johnston 85a0d6c7ab Remove test of replication resource 2017-04-11 10:59:27 +01:00
Erik Johnston a5c401bd12 Merge pull request #2097 from matrix-org/erikj/repl_tcp_client
Move to using TCP replication
2017-04-05 09:36:21 +01:00
Erik Johnston 3a1f3f8388 Change slave storage to use new replication interface
As the TCP replication uses a slightly different API and streams than
the HTTP replication.

This breaks HTTP replication.
2017-04-03 15:34:19 +01:00
Erik Johnston 9cee0ce7db Merge pull request #2075 from matrix-org/erikj/cache_speed
Speed up cached function access
2017-03-31 10:10:56 +01:00
Erik Johnston 350333a09a Merge pull request #2076 from matrix-org/erikj/as_perf
Make AS's faster
2017-03-31 09:43:10 +01:00
Erik Johnston eefd9fee81 Fix up tests 2017-03-30 14:14:46 +01:00
Richard van der Hoff f9b4bb05e0 Fix the logcontext handling in the cache wrappers (#2077)
The cache wrappers had a habit of leaking the logcontext into the reactor while
the lookup function was running, and then not restoring it correctly when the
lookup function had completed. It's all the fault of
`preserve_context_over_{fn,deferred}` which are basically a bit broken.
2017-03-30 13:22:24 +01:00
Erik Johnston 650f0e69f2 Compile the regex's used in ASes 2017-03-28 13:27:21 +01:00
Erik Johnston 57cfa513f5 Merge pull request #2054 from matrix-org/erikj/user_iter_cursor
Reduce some CPU work on DB threads
2017-03-24 11:40:08 +00:00
Richard van der Hoff 9397edb28b Merge pull request #2050 from matrix-org/rav/federation_backoff
push federation retry limiter down to matrixfederationclient
2017-03-23 22:27:01 +00:00
Richard van der Hoff 06ce7335e9 Merge pull request #2052 from matrix-org/rav/time_bound_deferred
Fix time_bound_deferred to throw the right exception
2017-03-23 22:22:54 +00:00
Erik Johnston 00957d1aa4 User Cursor.__iter__ instead of fetchall
This prevents unnecessary construction of lists
2017-03-23 17:53:49 +00:00
pik 566641a0b5 use jsonschema.FormatChecker for RoomID and UserID strings
* use a valid filter in rest/client/v2_alpha test

Signed-off-by: pik <alexander.maznev@gmail.com>
2017-03-23 11:42:41 -03:00
pik acafcf1c5b Add valid filter tests, flake8, fix typo
Signed-off-by: pik <alexander.maznev@gmail.com>
2017-03-23 11:42:10 -03:00
pik e56c79c114 check_valid_filter using JSONSchema
* add invalid filter tests

Signed-off-by: pik <alexander.maznev@gmail.com>
2017-03-23 11:42:07 -03:00
Richard van der Hoff 0bfea9a2be fix tests 2017-03-23 13:20:08 +00:00
Richard van der Hoff b88a323ffb Fix time_bound_deferred to throw the right exception
Due to a failure to instantiate DeferredTimedOutError, time_bound_deferred
would throw a CancelledError when the deferred timed out, which was rather
confusing.
2017-03-23 12:07:11 +00:00
Richard van der Hoff 4bd597d9fc push federation retry limiter down to matrixfederationclient
rather than having to instrument everywhere we make a federation call,
make the MatrixFederationHttpClient manage the retry limiter.
2017-03-23 09:28:46 +00:00
Richard van der Hoff 95f21c7a66 Fix caching of remote servers' signature keys
The `@cached` decorator on `KeyStore._get_server_verify_key` was missing
its `num_args` parameter, which meant that it was returning the wrong key for
any server which had more than one recorded key.

By way of a fix, change the default for `num_args` to be *all* arguments. To
implement that, factor out a common base class for `CacheDescriptor` and `CacheListDescriptor`.
2017-03-22 15:11:30 +00:00
Erik Johnston 37a187bfab Merge pull request #2033 from matrix-org/erikj/repl_speed
Don't send the full event json over replication
2017-03-21 13:11:15 +00:00
Erik Johnston 737f283a07 Fix unit test 2017-03-20 14:03:58 +00:00
Richard van der Hoff f40c2db05a Stop preserve_fn leaking context into the reactor
Fix a bug in ``logcontext.preserve_fn`` which made it leak context into the
reactor, and add a test for it.

Also, get rid of ``logcontext.reset_context_after_deferred``, which tried to do
the same thing but had its own, different, set of bugs.
2017-03-18 00:07:43 +00:00
Will Hunt c144292373 Modify test_user_sync so it doesn't look at last_active_ts over last_user_sync_ts 2017-03-15 15:38:57 +00:00
Erik Johnston 095b45c165 Aggregate event push actions 2017-02-14 13:39:41 +00:00
Erik Johnston 51adaac953 Fix email push in pusher worker
This was broken when device list updates were implemented, as Mailer
could no longer instantiate an AuthHandler due to a dependency on
federation sending.
2017-02-02 10:53:36 +00:00
Erik Johnston 692daf6f54 Remote membership tests for replication
This is because it now relies of the caches stream, which only works on
postgres. We are trying to test with sqlite.
2017-01-31 16:10:16 +00:00