feb496056e
preserve_fn some deferred-returning things
...
In `Notifier._on_new_room_event`, `preserve_fn` around its subroutines which
return deferreds, so that it is safe to call it with an active logcontext.
2017-04-03 15:41:17 +01:00
e2eebf1696
Fix fixme in preserve_fn
...
`preserve_fn` is no longer used as a decorator anywhere, so we can safely fix a
fixme therein.
2017-04-03 15:38:02 +01:00
36c28bc467
Update all the workers and master to use TCP replication
2017-04-03 15:35:52 +01:00
3a1f3f8388
Change slave storage to use new replication interface
...
As the TCP replication uses a slightly different API and streams than
the HTTP replication.
This breaks HTTP replication.
2017-04-03 15:34:19 +01:00
52bfa604e1
Add basic replication client handler and factory
2017-04-03 15:34:13 +01:00
0a6a966e2b
Always advance stream tokens
2017-04-03 15:22:56 +01:00
773e1c6d68
Remove spurious @preserve_fn decorators
...
Remove `@preserve_fn` decorators on `on_new_room_event`,
`_notify_pending_new_room_events`, `_on_new_room_event`, `on_new_event`, and
`on_new_replication_data` - none of these functions return a deferred, and the
decorator does nothing unless the wrapped function returns a deferred, so the
decorator was a no-op.
2017-04-03 15:14:11 +01:00
0d1c85e643
Merge branch 'release-v0.20.0' of github.com:matrix-org/synapse into develop
2017-04-03 14:58:14 +01:00
1df7c28661
Use callbacks to notify tcp replication rather than deferreds
2017-03-31 15:42:51 +01:00
36d2b66f90
Add a timestamp to USER_SYNC command
...
This timestamp is used to indicate when the user last sync'd
2017-03-31 15:42:22 +01:00
8a240e4f9c
Merge pull request #2078 from APwhitehat/assertuserfriendly
...
add user friendly report of assertion error in synctl.py
2017-03-31 14:41:49 +01:00
142b6b4abf
Merge pull request #2011 from matrix-org/matthew/turn_allow_guests
...
add setting (on by default) to support TURN for guests
2017-03-31 14:37:09 +01:00
2a06b44be2
Merge pull request #1986 from matrix-org/matthew/enable_guest_3p
...
enable guest access for the 3pl/3pid APIs
2017-03-31 14:36:03 +01:00
9ee397b440
switch to allow_guest=True for authing 3Ps as per PR feedback
2017-03-31 13:54:26 +01:00
9d0170ac6c
Fix up presence
2017-03-31 11:36:32 +01:00
bfcf016714
Fix up docs
2017-03-31 11:19:24 +01:00
9cee0ce7db
Merge pull request #2075 from matrix-org/erikj/cache_speed
...
Speed up cached function access
2017-03-31 10:10:56 +01:00
350333a09a
Merge pull request #2076 from matrix-org/erikj/as_perf
...
Make AS's faster
2017-03-31 09:43:10 +01:00
4d17add8de
Remove unused instance variable
2017-03-31 09:38:27 +01:00
27b1b4a2c9
Speed up copy_and_replace
2017-03-30 17:50:31 +01:00
5b5b171f3e
Docs
2017-03-30 17:05:53 +01:00
b282fe7170
Revert log context change
2017-03-30 17:03:59 +01:00
9ff4e0e91b
Bump version and changelog
2017-03-30 16:37:40 +01:00
0834d1a70c
Merge pull request #2079 from matrix-org/erikj/push_regex_cache
...
Cache glob to regex at a higher level for push
2017-03-30 16:28:18 +01:00
63fcc42990
Remove user from process_presence when stops syncing
2017-03-30 14:26:08 +01:00
6194a64ae9
Doc new instance variables
2017-03-30 14:19:10 +01:00
014fee93b3
Manually calculate cache key as getcallargs is expensive
...
This is because getcallargs recomputes the getargspec, amongst other
things, which we don't need to do as its already been done
2017-03-30 14:14:46 +01:00
86780a8bc3
Don't convert to deferreds when not necessary
2017-03-30 14:14:36 +01:00
3ba2859e0c
Add tcp replication listener type and hook it up
2017-03-30 13:31:10 +01:00
e9dd8370b0
Add functions to presence to support remote syncs
...
The TCP replication protocol streams deltas of who has started or
stopped syncing. This is different from the HTTP API which periodically
sends the full list of users who are syncing. This commit adds support
for the new TCP style of sending deltas.
2017-03-30 13:25:14 +01:00
4d7fc7f977
Add server side resource for tcp replication
2017-03-30 13:24:45 +01:00
f9b4bb05e0
Fix the logcontext handling in the cache wrappers ( #2077 )
...
The cache wrappers had a habit of leaking the logcontext into the reactor while
the lookup function was running, and then not restoring it correctly when the
lookup function had completed. It's all the fault of
`preserve_context_over_{fn,deferred}` which are basically a bit broken.
2017-03-30 13:22:24 +01:00
7450693435
Initial TCP protocol implementation
...
This defines the low level TCP replication protocol
2017-03-30 12:54:46 +01:00
8da6f0be48
Define the various streams we will replicate
2017-03-30 12:54:46 +01:00
11880103b1
Make federation send queue take the current position
2017-03-30 12:54:36 +01:00
7984708a55
Add a simple hook to wait for replication traffic
2017-03-30 11:57:52 +01:00
24d35ab47b
Add new storage functions for new replication
...
The new replication protocol will keep all the streams separate, rather
than muxing multiple streams into one.
2017-03-30 11:48:35 +01:00
30348c924c
Use txn.fetchall() so we can reuse txn
2017-03-30 10:30:05 +01:00
6cdca71079
synctl.py: wait for synapse to stop before restarting ( #2020 )
2017-03-29 19:20:13 +01:00
305d16d612
add user friendly report of assertion error in synctl.py
...
Signed-off-by: Anant Prakash <anantprakashjsr@gmail.com>
2017-03-29 20:41:39 +05:30
a3810136fe
Cache glob to regex at a higher level for push
2017-03-29 15:53:14 +01:00
3ce8d59176
Increase cache size for _get_state_group_for_event
2017-03-29 14:31:46 +01:00
b9c2ae6788
Merge pull request #1849 from matrix-org/erikj/state_typo
...
Fix bug where current_state_events renamed to current_state_ids
2017-03-29 11:53:42 +01:00
85be3dde81
Bail early if remote wouldn't be retried ( #2064 )
...
* Bail early if remote wouldn't be retried
* Don't always return true
* Just use get_retry_limiter
* Spelling
2017-03-29 11:48:27 +01:00
2f8b580b64
Merge pull request #2053 from matrix-org/erikj/e2e_one_time_upsert
...
Don't user upsert to persist new one time keys
2017-03-29 11:44:23 +01:00
c5b0bdd542
Merge pull request #2067 from matrix-org/erikj/notify_on_fed
...
Notify on new federation traffic
2017-03-29 11:41:37 +01:00
e4df0e189d
Decrank last commit
2017-03-29 11:02:35 +01:00
4ad613f6be
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/e2e_one_time_upsert
2017-03-29 10:57:19 +01:00
ac6bc55512
Correctly look up key
2017-03-29 10:56:26 +01:00
69efd77749
Add comment
2017-03-29 09:50:05 +01:00
51b156d48a
Cache whether an AS is interested based on members
2017-03-28 13:27:21 +01:00
30f5ffdca2
Remove param and cast at call site
2017-03-28 13:27:21 +01:00
650f0e69f2
Compile the regex's used in ASes
2017-03-28 13:27:21 +01:00
d28db583da
Merge pull request #2063 from matrix-org/erikj/device_list_batch
...
Batch sending of device list pokes
2017-03-28 11:35:41 +01:00
58a35366be
The algorithm is part of the key id
2017-03-28 11:34:37 +01:00
bac9bf1b12
Typo
2017-03-27 18:02:17 +01:00
d82c42837f
Short circuit if all new events have same state group
2017-03-27 18:00:47 +01:00
35b4aa04be
Notify on new federation traffic
2017-03-27 14:07:47 +01:00
2a28b79e04
Batch sending of device list pokes
2017-03-24 14:44:49 +00:00
281553afe6
Merge pull request #2062 from matrix-org/erikj/presence_replication
...
Use presence replication stream to invalidate cache
2017-03-24 13:57:45 +00:00
987f4945b4
Actually call invalidate
2017-03-24 13:28:20 +00:00
31d56c3fb5
Merge pull request #2061 from matrix-org/erikj/add_transaction_store
...
Add slave transaction store to workers who send federation requests
2017-03-24 13:24:27 +00:00
09f79aaad0
Use presence replication stream to invalidate cache
...
Instead of using the cache invalidation replication stream to invalidate
the _get_presence_cache, we can instead rely on the presence replication
stream. This reduces the amount of replication traffic considerably.
2017-03-24 13:21:08 +00:00
23e0ff840a
Merge pull request #2060 from matrix-org/erikj/cache_hosts_in_room
...
Cache hosts in room
2017-03-24 13:07:22 +00:00
48e7697911
Add slave transaction store
2017-03-24 13:05:30 +00:00
f136c89d5e
Merge pull request #2058 from matrix-org/rav/logcontext_leaks_2
...
try not to drop context after federation requests
2017-03-24 12:47:26 +00:00
01fc847f7f
Merge pull request #2057 from matrix-org/rav/missing_yield_2
...
Add another missing yield on check_device_registered
2017-03-24 12:46:43 +00:00
7fc1f1e2b6
Cache hosts in room
2017-03-24 11:46:24 +00:00
57cfa513f5
Merge pull request #2054 from matrix-org/erikj/user_iter_cursor
...
Reduce some CPU work on DB threads
2017-03-24 11:40:08 +00:00
d58b1ffe94
Replace some calls to cursor_to_dict
...
cursor_to_dict can be surprisinglh expensive for large result sets, so lets
only call it when we need to.
2017-03-24 11:07:02 +00:00
e71940aa64
Use iter(items|values)
2017-03-24 10:57:02 +00:00
e36950dec5
Merge pull request #2055 from matrix-org/dbkr/fix_add_msisdn_requestToken
...
Fix token request for addition of phone numbers
2017-03-24 10:32:57 +00:00
f902e89d4b
Merge pull request #2056 from matrix-org/dbkr/fix_invite_reject
...
Fix rejection of invites not reaching sync
2017-03-24 10:32:51 +00:00
a380f041c2
try not to drop context after federation requests
...
preserve_context_over_fn uses a ContextPreservingDeferred, which only restores
context for the duration of its callbacks, which isn't really correct, and
means that subsequent operations in the same request can end up without their
logcontexts.
2017-03-23 22:36:21 +00:00
9397edb28b
Merge pull request #2050 from matrix-org/rav/federation_backoff
...
push federation retry limiter down to matrixfederationclient
2017-03-23 22:27:01 +00:00
06ce7335e9
Merge pull request #2052 from matrix-org/rav/time_bound_deferred
...
Fix time_bound_deferred to throw the right exception
2017-03-23 22:22:54 +00:00
13c8749ac9
Add another missing yield on check_device_registered
2017-03-23 22:18:53 +00:00
e1f1784f99
Fix rejection of invites not reaching sync
...
Always allow the user to see their own leave events, otherwise
they won't see the event if they reject an invite for a room whose
history visibility is set such that they cannot see events before
joining.
2017-03-23 18:50:31 +00:00
86e865d7d2
Oops, remove unintentional change
2017-03-23 18:48:30 +00:00
a2dfab12c5
Fix token request for addition of phone numbers
2017-03-23 18:46:17 +00:00
00957d1aa4
User Cursor.__iter__ instead of fetchall
...
This prevents unnecessary construction of lists
2017-03-23 17:53:49 +00:00
250ce11ab9
Add jsonschema to python_dependencies.py
...
Signed-off-by: pik <alexander.maznev@gmail.com>
2017-03-23 11:42:47 -03:00
566641a0b5
use jsonschema.FormatChecker for RoomID and UserID strings
...
* use a valid filter in rest/client/v2_alpha test
Signed-off-by: pik <alexander.maznev@gmail.com>
2017-03-23 11:42:41 -03:00
acafcf1c5b
Add valid filter tests, flake8, fix typo
...
Signed-off-by: pik <alexander.maznev@gmail.com>
2017-03-23 11:42:10 -03:00
e56c79c114
check_valid_filter using JSONSchema
...
* add invalid filter tests
Signed-off-by: pik <alexander.maznev@gmail.com>
2017-03-23 11:42:07 -03:00
6ebe2d23b1
Raise a more helpful exception
2017-03-23 13:48:30 +00:00
e64655c25d
Don't user upsert to persist new one time keys
...
Instead we no-op duplicate one time key uploads, an error if the key_id
already exists but encodes a different key.
2017-03-23 13:17:00 +00:00
5a16cb4bf0
Ignore backoff history for invites, aliases, and roomdirs
...
Add a param to the federation client which lets us ignore historical backoff
data for federation queries, and set it for a handful of operations.
2017-03-23 12:23:22 +00:00
b88a323ffb
Fix time_bound_deferred to throw the right exception
...
Due to a failure to instantiate DeferredTimedOutError, time_bound_deferred
would throw a CancelledError when the deferred timed out, which was rather
confusing.
2017-03-23 12:07:11 +00:00
55366814a6
Merge pull request #2048 from matrix-org/rav/missing_yield
...
Add a missing yield in device key upload
2017-03-23 09:29:44 +00:00
4bd597d9fc
push federation retry limiter down to matrixfederationclient
...
rather than having to instrument everywhere we make a federation call,
make the MatrixFederationHttpClient manage the retry limiter.
2017-03-23 09:28:46 +00:00
ad8a26e361
MatrixFederationHttpClient: clean up
...
rename _create_request to _request, and push ascii-encoding of `destination`
and `path` down into it
2017-03-23 00:27:04 +00:00
19b9366d73
Fix a couple of logcontext leaks
...
Use preserve_fn to correctly manage the logcontexts around things we don't want
to yield on.
2017-03-23 00:17:46 +00:00
e08f81d96a
Add a missing yield in device key upload
...
(this would only very very rarely actually be a useful thing, so the main
problem was the logcontext leak...)
2017-03-23 00:16:43 +00:00
35e0cfb54d
Merge pull request #2044 from matrix-org/rav/crypto_docs
...
fix up some key verif docstrings
2017-03-22 17:12:55 +00:00
7b67848042
Merge pull request #2042 from matrix-org/rav/fix_key_caching
...
Fix caching of remote servers' signature keys
2017-03-22 17:11:55 +00:00
95f21c7a66
Fix caching of remote servers' signature keys
...
The `@cached` decorator on `KeyStore._get_server_verify_key` was missing
its `num_args` parameter, which meant that it was returning the wrong key for
any server which had more than one recorded key.
By way of a fix, change the default for `num_args` to be *all* arguments. To
implement that, factor out a common base class for `CacheDescriptor` and `CacheListDescriptor`.
2017-03-22 15:11:30 +00:00
d101488c5f
Merge branch 'master' into develop
2017-03-21 22:48:21 +01:00
64778693be
fix up some key verif docstrings
2017-03-21 13:27:50 +00:00
37a187bfab
Merge pull request #2033 from matrix-org/erikj/repl_speed
...
Don't send the full event json over replication
2017-03-21 13:11:15 +00:00
633dcc316c
Bump changelog and version
2017-03-20 16:37:14 +00:00
c36d15d2de
Add some debug to help diagnose weird federation issue
2017-03-20 15:36:14 +00:00
aac6d1fc9b
PEP8
2017-03-20 13:47:56 +00:00
bd08ee7a46
Merge pull request #2026 from matrix-org/rav/logcontext_docs
...
Logcontext docs
2017-03-20 12:05:21 +00:00
eddce9d74a
Merge pull request #2027 from matrix-org/rav/logcontext_leaks
...
A few fixes to logcontext things
2017-03-20 11:53:36 +00:00
f40c2db05a
Stop preserve_fn leaking context into the reactor
...
Fix a bug in ``logcontext.preserve_fn`` which made it leak context into the
reactor, and add a test for it.
Also, get rid of ``logcontext.reset_context_after_deferred``, which tried to do
the same thing but had its own, different, set of bugs.
2017-03-18 00:07:43 +00:00
067b00d49d
Run the reactor with the sentinel logcontext
...
This fixes a class of 'Unexpected logcontext' messages, which were happening
because the logcontext was somewhat arbitrarily swapping between the sentinel
and the `run` logcontext.
2017-03-18 00:07:43 +00:00
994d7ae7c5
Remove broken use of clock.call_later
...
background_updates was using `call_later` in a way that leaked the logcontext
into the reactor.
We could have rewritten it to do it properly, but given that we weren't using
the fancier facilities provided by `call_later`, we might as well just use
`async.sleep`, which does the logcontext stuff properly.
2017-03-18 00:01:37 +00:00
d2d146a314
Logcontext docs
2017-03-17 23:59:28 +00:00
61f471f779
Don't send the full event json over replication
2017-03-17 15:50:01 +00:00
0c01f829ae
Avoid resetting state on rejected events
...
When we get a rejected event, give it the same state_group as its prev_event,
rather than no state_group at all.
This should fix https://github.com/matrix-org/synapse/issues/1935 .
2017-03-17 15:06:08 +00:00
5068fb16a5
Refactoring and cleanups
...
A few non-functional changes:
* A bunch of docstrings to document types
* Split `EventsStore._persist_events_txn` up a bit. Hopefully it's a bit more
readable.
* Rephrase `EventFederationStore._update_min_depth_for_room_txn` to avoid
mind-bending conditional.
* Rephrase rejected/outlier conditional in `_update_outliers_txn` to avoid
mind-bending conditional.
2017-03-17 15:06:07 +00:00
2abe85d50e
Merge pull request #2016 from matrix-org/rav/queue_pdus_during_join
...
Queue up federation PDUs while a room join is in progress
2017-03-17 11:32:44 +00:00
da146657c9
Comments
2017-03-16 13:04:07 +00:00
a158c36a8a
Comment
2017-03-16 11:57:45 +00:00
6957bfdca6
Don't recreate so many sets
2017-03-16 11:54:26 +00:00
2ccf3b241c
Implement no op for room stream in sync
2017-03-16 11:06:41 +00:00
9ce53a3861
Queue up federation PDUs while a room join is in progress
...
This just takes the existing `room_queues` logic and moves it out to
`on_receive_pdu` instead of `_process_received_pdu`, which ensures that we
don't start trying to fetch prev_events and whathaveyou until the join has
completed.
2017-03-15 18:01:11 +00:00
54d2b7e596
Merge pull request #2014 from Half-Shot/hs/fix-appservice-presence
...
Add fallback to last_active_ts if it beats the last sync time on a presence timeout.
2017-03-15 17:37:15 +00:00
a8f96c63aa
Comment
2017-03-15 16:01:01 +00:00
f83ac78201
Cache set of users whose presence the other user should see
2017-03-15 15:29:19 +00:00
e6032054bf
Add a great comment to handle_timeout for active vs sync times.
2017-03-15 15:24:48 +00:00
ebf5a6b14c
Add fallback to last_active_ts if it beats the last sync time.
2017-03-15 15:17:16 +00:00
e892457a03
Comment
2017-03-15 15:01:39 +00:00
a297155a97
Remove unused import
2017-03-15 14:49:25 +00:00
6c82de5100
Format presence events on the edges instead of reformatting them multiple times
2017-03-15 14:27:34 +00:00
0ad44acb5a
Merge pull request #1997 from matrix-org/dbkr/cas_partialdownload
...
Handle PartialDownloadError in CAS login
2017-03-15 13:52:34 +00:00
0970e0307e
typo
2017-03-15 12:40:42 +00:00
5aa42d4292
set default for turn_allow_guests correctly
2017-03-15 12:40:13 +00:00
e0ff66251f
add setting (on by default) to support TURN for guests
2017-03-15 12:22:18 +00:00
29ed09e80a
Fix assertion to stop transaction queue getting wedged
...
... and update some docstrings to correctly reflect the types being used.
get_new_device_msgs_for_remote can return a long under some circumstances,
which was being stored in last_device_list_stream_id_by_dest, and was then
upsetting things on the next loop.
2017-03-15 12:16:55 +00:00
872e75a3d5
Add some metrics on notifier
2017-03-15 10:56:51 +00:00
7827251daf
Merge pull request #1994 from matrix-org/dbkr/msisdn_signin_2
...
Phone number registration / login support v2
2017-03-15 09:59:54 +00:00
b5d1c68beb
Implement reset_context_after_deferred
...
to correctly reset the context when we fire off a deferred we aren't going to
wait for.
2017-03-15 02:21:07 +00:00
f2ed64eaaf
Merge pull request #1992 from matrix-org/rav/fix_media_loop
...
Fix routing loop when fetching remote media
2017-03-14 23:40:35 +00:00
bad72b0b8e
Merge pull request #2002 from matrix-org/erikj/dont_sync_by_default
...
Reduce number of spurious sync result generations.
2017-03-14 16:58:45 +00:00
fd2eef49c8
Reduce spurious calls to generate sync
2017-03-14 16:33:45 +00:00
1d09586599
Address review comments
...
- don't blindly proxy all HTTPRequestExceptions
- log unexpected exceptions at error
- avoid `isinstance`
- improve docs on `from_http_response_exception`
2017-03-14 14:15:37 +00:00
7f237800e9
re-refactor exception heirarchy
...
Give CodeMessageException back its `msg` attribute, and use that to hold the
HTTP status message for HttpResponseException.
2017-03-14 14:15:37 +00:00
1ece06273e
Handle PartialDownloadError in CAS login
2017-03-14 13:37:36 +00:00
bb256ac96f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/public_list_speed
2017-03-14 11:35:05 +00:00
cc7a294e2e
Fix current_state_events table to not lie
...
If we try and persist two state events that have the same ancestor we
calculate the wrong current state when persisting those events.
2017-03-14 10:57:43 +00:00
7b6ed9871e
Use extend instead of +=
2017-03-14 10:49:55 +00:00
d79a687d85
Oops, remove print
2017-03-14 10:40:20 +00:00
f29d85d9e4
Merge pull request #1993 from matrix-org/luke/delete-devices
...
Implement delete_devices API
2017-03-14 10:02:56 +00:00
bbeeb97f75
Implement _simple_delete_many_txn, use it to delete devices
...
(But this doesn't implement the same for deleting access tokens or e2e keys.
Also respond to code review.
2017-03-13 17:53:23 +00:00
0a9945220e
Fix registration for broken clients
...
Only offer msisdn flows if the x_show_msisdn option is given.
2017-03-13 17:29:38 +00:00
73a5f06652
Support registration / login with phone number
...
Changes from https://github.com/matrix-org/synapse/pull/1971
2017-03-13 17:27:51 +00:00
c077c3277b
Flake
2017-03-13 16:45:38 +00:00
31f3ca1b2b
Merge pull request #1990 from matrix-org/rav/log_config_comments
...
Add helpful texts to logger config options
2017-03-13 16:42:12 +00:00
Richard van der Hoff