8521ae13e3
Merge pull request #3654 from matrix-org/rav/room_versions
...
Support for room versioning
2018-08-08 17:10:53 +01:00
d2f3ef98ac
Merge branch 'develop' into neilj/disable_hs
2018-08-08 15:55:47 +00:00
2b5baebeba
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/disable_hs
2018-08-08 13:47:07 +01:00
be59910b93
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/reserved_users
2018-08-08 13:45:21 +01:00
990fe9fc23
Merge pull request #3633 from matrix-org/neilj/mau_tracker
...
API for monthly_active_users table
2018-08-08 12:44:15 +00:00
312ae74746
typos
2018-08-08 13:33:16 +01:00
360ba89c50
Don't fail requests to unbind 3pids for non supporting ID servers
...
Older identity servers may not support the unbind 3pid request, so we
shouldn't fail the requests if we received one of 400/404/501. The
request still fails if we receive e.g. 500 responses, allowing clients
to retry requests on transient identity server errors that otherwise do
support the API.
Fixes #3661
2018-08-08 12:06:18 +01:00
bebe325e6c
Rename POST param to METHOD
2018-08-08 10:36:18 +01:00
5011417632
Fixup logging and docstrings
2018-08-08 10:29:58 +01:00
bab94da79c
fix metric name
2018-08-07 22:11:45 +01:00
53bca4690b
more metrics for the federation and appservice senders
2018-08-07 19:09:48 +01:00
ef3589063a
prevent total number of reserved users being too large
2018-08-07 17:59:27 +01:00
e8eba2b4e3
implement reserved users for mau limits
2018-08-07 17:49:43 +01:00
3523f5432a
Don't expose default_room_version as config opt
2018-08-07 12:51:57 +01:00
848431be1d
fix for rc1
2018-08-07 21:13:22 +10:00
3b9662339b
version
2018-08-07 21:10:52 +10:00
cd9765805e
Allow ratelimiting on workers
2018-08-07 10:50:28 +01:00
495cb100d1
Allow profile changes to happen on workers
2018-08-07 10:50:26 +01:00
ca9bc1f4fe
Fix occasional glitches in the synapse_event_persisted_position metric
...
Every so often this metric glitched to a negative number. I'm assuming it was
due to backfilled events.
2018-08-07 10:00:03 +01:00
a74b25faaa
WIP building out mau reserved users
2018-08-06 23:25:25 +01:00
7daa8a78c5
load mau limit threepids
2018-08-06 22:55:05 +01:00
89834d9a29
Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs
2018-08-06 21:56:56 +01:00
e54794f5b6
Fix postgres compatibility bug
2018-08-06 21:55:54 +01:00
7bcf126b18
Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs
2018-08-06 21:39:44 +01:00
1911c037cb
update comments to reflect new sig
2018-08-06 18:01:46 +01:00
33bd07d062
Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs
2018-08-06 17:52:28 +01:00
16d78be315
make use of _simple_select_one_onecol, improved comments
2018-08-06 17:51:15 +01:00
19a17068f1
Check m.room.create for sane room_versions
2018-08-06 16:11:24 +01:00
96a9a29645
Pull in necessary stores in federation_reader
2018-08-06 15:23:57 +01:00
62ace05c45
Move necessary storage functions to worker classes
2018-08-06 15:23:38 +01:00
1e2bed9656
Import all functions from TransactionStore
2018-08-06 15:23:38 +01:00
a3f5bf79a0
Add EDU/query handling over replication
2018-08-06 15:23:31 +01:00
e26dbd82ef
Add replication APIs for persisting federation events
2018-08-06 15:02:28 +01:00
051a99c400
Fix isort
2018-08-06 14:29:31 +01:00
f900d50824
include known room versions in outgoing make_joins
2018-08-06 13:45:37 +01:00
42c6823827
disable HS from config
2018-08-04 22:07:04 +01:00
e40a510fbf
py3 fix
2018-08-03 23:19:13 +01:00
886be75ad1
bug fixes
2018-08-03 22:29:03 +01:00
16d9701892
Return M_NOT_FOUND when a profile could not be found. ( #3596 )
2018-08-03 19:08:05 +01:00
e10830e976
wip commit - tests failing
2018-08-03 17:55:50 +01:00
3777fa26aa
sanity check response from make_join
2018-08-03 16:08:32 +01:00
0d63d93ca8
Enforce compatibility when processing make_join requests
...
Reject make_join requests from servers which do not support the room version.
Also include the room version in the response.
2018-08-03 16:08:32 +01:00
0ca459ea33
Basic support for room versioning
...
This is the first tranche of support for room versioning. It includes:
* setting the default room version in the config file
* new room_version param on the createRoom API
* storing the version of newly-created rooms in the m.room.create event
* fishing the version of existing rooms out of the m.room.create event
2018-08-03 16:08:32 +01:00
15c1ae45e5
Docstrings for BaseFederationServlet
...
... to save me reverse-engineering this stuff again.
2018-08-03 16:08:32 +01:00
5593ff6773
fix (lots of) py3 test failures
2018-08-03 14:59:17 +01:00
950807d93a
fix caching and tests
2018-08-03 13:49:53 +01:00
897c51d274
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker
2018-08-03 13:40:47 +01:00
cb298ff623
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_repl_servlet
2018-08-03 09:25:15 +01:00
c0affa7b4f
update generate_monthly_active_users, and reap_monthly_active_users
2018-08-02 23:03:01 +01:00
5e2f7b8084
typo
2018-08-02 22:47:48 +01:00
4ecb4bdac9
wip attempt at caching
2018-08-02 22:41:05 +01:00
1fa98495d0
Merge pull request #3639 from matrix-org/rav/refactor_error_handling
...
Clean up handling of errors from outbound requests
2018-08-02 17:38:24 +01:00
bdae8f2e68
Merge pull request #3638 from matrix-org/rav/refactor_federation_client_exception_handling
...
Factor out exception handling in federation_client
2018-08-02 17:37:46 +01:00
74b1d46ad9
do mau checks based on monthly_active_users table
2018-08-02 16:57:35 +01:00
9180061b49
remove unused count_monthly_users
2018-08-02 15:55:29 +01:00
704c3e6239
Merge branch 'master' into develop
2018-08-02 15:43:30 +01:00
43ecfe0b10
Synapse 0.33.1 (2018-08-02)
...
===========================
SECURITY FIXES
--------------
- Fix a potential issue where servers could request events for rooms they have not joined. (`#3641 <https://github.com/matrix-org/synapse/issues/3641 >`_)
- Fix a potential issue where users could see events in private rooms before they joined. (`#3642 <https://github.com/matrix-org/synapse/issues/3642 >`_)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAABAgAGBQJbYxcCAAoJEIofk9V1tejVg8YIAJU1xoZ2zSnordJczpvVZbzx
Rsstk/wvE2dSGJL22gfSLzQNCbQvF4aOul1LAGJfbEQWc46BcNpyuWUxjTd3FaF7
SfbWdkDh1w3ROSuiPA8j4CnH6EwP0w2itIBhFG7JpVEXjRgCyc1iMUl/oQESv82g
UQyr6t/G68mE8xHm0eqvNfCjVAMSZDAnOzYZa7vfZJBqlZiGHB6Z1nsvsS3nZqQd
BQQi/pQTXxsIL2egSvuycykiZtEZcm+QDRFi6hIatv5LRFOsQUqKVTB/D7XOoSbP
RPOzcaSjv4mZ9a4NBzFMVVzzTRbr2EnipPd2ePrnJoOnl2eNRO4Won8zl4h+OQc=
=U/ir
-----END PGP SIGNATURE-----
Merge tag 'v0.33.1'
Synapse 0.33.1 (2018-08-02)
===========================
SECURITY FIXES
--------------
- Fix a potential issue where servers could request events for rooms they have not joined. (`#3641 <https://github.com/matrix-org/synapse/issues/3641 >`_)
- Fix a potential issue where users could see events in private rooms before they joined. (`#3642 <https://github.com/matrix-org/synapse/issues/3642 >`_)
2018-08-02 15:40:44 +01:00
14a4e7d5a4
Prepare 0.33.1
2018-08-02 15:31:04 +01:00
0bf5ec0db7
Check room visibility for /event/ requests
...
Make sure that the user has permission to view the requeseted event for
/event/{eventId} and /room/{roomId}/event/{eventId} requests.
Also check that the event is in the given room for
/room/{roomId}/event/{eventId}, for sanity.
2018-08-02 15:03:27 +01:00
14fa9d4d92
Avoid extra db lookups
...
Since we're about to look up the events themselves anyway, we can skip the
extra db queries here.
2018-08-02 13:55:51 +01:00
0a65450d04
Validation for events/rooms in fed requests
...
When we get a federation request which refers to an event id, make sure that
said event is in the room the caller claims it is in.
(patch supplied by @turt2live)
2018-08-02 13:48:40 +01:00
00f99f74b1
insertion into monthly_active_users
2018-08-02 13:47:19 +01:00
4a6725d9d1
Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/mau_tracker
2018-08-02 11:04:18 +01:00
165e067033
Revert "change monthly_active_users table to be a single column"
...
This reverts commit ec716a35b2
2018-08-02 10:59:58 +01:00
40c1c59cf4
Merge pull request #3621 from matrix-org/erikj/split_fed_store
...
Split out DB writes in federation handler
2018-08-02 10:41:42 +01:00
08281fe6b7
self.db_conn unused
2018-08-01 23:26:24 +01:00
c21d82bab3
normalise reaping query
2018-08-01 23:24:38 +01:00
ec716a35b2
change monthly_active_users table to be a single column
2018-08-01 17:54:37 +01:00
d766f26de9
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker
2018-08-01 17:49:41 +01:00
085435e13a
Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits
...
Initial impl of capping MAU
2018-08-01 15:58:45 +00:00
b8d7d3996b
Merge pull request #3620 from fuzzmz/return-404-room-not-found
...
return 404 if room not found
2018-08-01 16:34:32 +01:00
b7f203a566
count_monthly_users is now async
2018-08-01 16:17:42 +01:00
7ff44d9215
improve clarity
2018-08-01 16:17:00 +01:00
01e93f48ed
Kill off MatrixCodeMessageException
...
This code brings the SimpleHttpClient into line with the
MatrixFederationHttpClient by having it raise HttpResponseExceptions when a
request fails (rather than trying to parse for matrix errors and maybe raising
MatrixCodeMessageException).
Then, whenever we were checking for MatrixCodeMessageException and turning them
into SynapseErrors, we now need to check for HttpResponseExceptions and call
to_synapse_error.
2018-08-01 16:02:46 +01:00
018d75a148
Refactor code for turning HttpResponseException into SynapseError
...
This commit replaces SynapseError.from_http_response_exception with
HttpResponseException.to_synapse_error.
The new method actually returns a ProxiedRequestError, which allows us to pass
through additional metadata from the API call.
2018-08-01 16:02:46 +01:00
fa7dc889f1
Be more careful which errors we send back over the C-S API
...
We really shouldn't be sending all CodeMessageExceptions back over the C-S API;
it will include things like 401s which we shouldn't proxy.
That means that we need to explicitly turn a few HttpResponseExceptions into
SynapseErrors in the federation layer.
The effect of the latter is that the matrix errcode will get passed through
correctly to calling clients, which might help with some of the random
M_UNKNOWN errors when trying to join rooms.
2018-08-01 16:02:38 +01:00
c82ccd3027
Factor out exception handling in federation_client
...
Factor out the error handling from make_membership_event, send_join, and
send_leave, so that it can be shared.
2018-08-01 16:01:04 +01:00
da7785147d
Python 3: Convert some unicode/bytes uses ( #3569 )
2018-08-02 00:54:06 +10:00
303f1c851f
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits
2018-08-01 13:42:50 +01:00
a6d7b74915
update docs
2018-08-01 13:39:14 +01:00
4b256b9271
_persist_auth_tree no longer returns anything
2018-08-01 13:39:07 +01:00
4e5ac901dd
clean up
2018-08-01 12:03:57 +01:00
f9f5559971
fix comment
2018-08-01 12:03:42 +01:00
0aba3d361a
count_monthly_users() async
2018-08-01 11:47:58 +01:00
2c54f1c225
remove need to plot limit_usage_by_mau
2018-08-01 11:46:59 +01:00
f49147d14f
Merge pull request #3634 from matrix-org/rav/wtf_is_a_replication_layer
...
rename replication_layer to federation_client
2018-08-01 10:29:29 +01:00
cab782c17e
Merge pull request #3384 from matrix-org/rav/rewrite_cachedlist_decorator
...
Rewrite cache list decorator
2018-08-01 10:28:56 +01:00
6023cdd227
remove errant print
2018-08-01 10:27:17 +01:00
7931393495
make count_monthly_users async synapse/handlers/auth.py
2018-08-01 10:21:56 +01:00
c507fa15ce
only need to loop if mau limiting is enabled
2018-08-01 10:20:42 +01:00
37be52ac34
limt -> limit
2018-07-31 16:29:09 -06:00
70af98e361
return NotFoundError if room not found
...
Per the Client-Server API[0] we should return
`M_NOT_FOUND` if the room isn't found instead
of generic SynapseError.
This ensures that /directory/list API returns
404 for room not found instead of 400.
[0]: https://matrix.org/docs/spec/client_server/unstable.html#get-matrix-client-r0-directory-list-room-roomid
Signed-off-by: Serban Constantin <serban.constantin@gmail.com>
2018-07-31 21:47:23 +03:00
5e2ee64660
Merge pull request #3628 from turt2live/travis/goodby-pdu-failures
...
Remove pdu_failures from transactions
2018-07-31 12:13:09 -06:00
6ef983ce5c
api into monthly_active_users table
2018-07-31 16:36:24 +01:00
bdbdceeafa
rename replication_layer to federation_client
...
I have HAD ENOUGH of trying to remember wtf a replication layer is in terms of
classes.
2018-07-31 15:44:05 +01:00
443da003bc
Use new helper base class for membership requests
2018-07-31 14:32:23 +01:00
729b672823
Use new helper base class for ReplicationSendEventRestServlet
2018-07-31 14:32:23 +01:00
d81602b75a
Add helper base class for generating new replication endpoints
...
This will hopefully reduce the boiler plate required to implement new
internal HTTP requests.
2018-07-31 14:32:20 +01:00
5de936caa1
Merge pull request #3612 from matrix-org/rav/store_heirarchy
...
Make EventStore inherit from EventFederationStore
2018-07-31 13:44:04 +01:00
df2235e7fa
coding style
2018-07-31 13:16:20 +01:00
0bc9b9e397
reinstate explicit include of EventsWorkerStore
2018-07-31 13:11:04 +01:00
e908b86832
Remove pdu_failures from transactions
...
The field is never read from, and all the opportunities given to populate it are not utilized. It should be very safe to remove this.
2018-07-30 16:28:47 -06:00
254e8267e2
Only import secrets when available
...
secrets got introduced in python 3.6 so this class is not available
in 3.5 and before.
This now checks for the current running version and only tries using
secrets if the version is 3.6 or above
Signed-Off-By: Matthias Kesler <krombel@krombel.de>
2018-07-30 23:59:02 +02:00
21276ff846
remove errant logging
2018-07-30 22:42:12 +01:00
fef7e58ac6
actually close conn
2018-07-30 22:29:44 +01:00
9b13817e06
factor out metrics from __init__ to app/homeserver
2018-07-30 22:07:07 +01:00
251e6c1210
limit register and sign in on number of monthly users
2018-07-30 15:55:57 +01:00
143f1a2532
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store
2018-07-30 09:56:18 +01:00
2903e65aff
fix isort
2018-07-29 19:47:08 +02:00
a8cbce0ced
fix invalidation
2018-07-27 16:17:17 +01:00
e9b2d047f6
make /context lazyload & filter aware ( #3567 )
...
make /context lazyload & filter aware.
2018-07-27 15:12:50 +01:00
f102c05856
Rewrite cache list decorator
...
Because it was complicated and annoyed me. I suspect this will be more
efficient too.
2018-07-27 13:47:04 +01:00
8e3f75b39a
fix accidental removal of hs
2018-07-27 12:17:31 +02:00
b0b5566f36
Merge pull request #3391 from t3chguy/t3chguy/default_inviter_display_name_3pid
...
if inviter_display_name == ""||None then default to inviter MXID
2018-07-27 10:08:39 +01:00
7041cd872b
Merge branch 'develop' into send_sni_for_federation_requests
2018-07-27 09:17:11 +01:00
d42455cdc9
Merge pull request #3616 from matrix-org/travis/event_id_send_leave
...
Update the send_leave path to be an event_id
2018-07-26 23:25:17 +01:00
65c8dee900
Merge pull request #3614 from matrix-org/rav/stop_populating_event_content
...
Stop populating events.content
2018-07-26 22:54:08 +01:00
a75231b507
Deduplicate redundant lazy-loaded members ( #3331 )
...
* attempt at deduplicating lazy-loaded members
as per the proposal; we can deduplicate redundant lazy-loaded members
which are sent in the same sync sequence. we do this heuristically
rather than requiring the client to somehow tell us which members it
has chosen to cache, by instead caching the last N members sent to
a client, and not sending them again. For now we hardcode N to 100.
Each cache for a given (user,device) tuple is in turn cached for up to
X minutes (to avoid the caches building up). For now we hardcode X to 30.
* add include_redundant_members filter option & make it work
* remove stale todo
* add tests for _get_some_state_from_cache
* incorporate review
2018-07-26 22:51:30 +01:00
9e68b1bd2d
Merge pull request #3613 from matrix-org/rav/stop_using_event_edges_room_id
...
Remove some redundant joins on event_edges.room_id
2018-07-26 22:31:01 +01:00
7d32f0d745
Update the send_leave path to be an event_id
...
It's still not used, however the parameter is an event ID not a transaction ID.
2018-07-26 14:41:59 -06:00
ef9d51b081
Merge pull request #3610 from matrix-org/rav/fix_looping_calls
...
Fix some looping_call calls which were broken in #3604
2018-07-26 14:55:57 +01:00
51d7df1915
Create the column nullable
...
There's no real point in ever making the column non-nullable, and doing so
breaks the sytests.
2018-07-26 14:54:04 +01:00
5c1d301fd9
Stop populating events.content
...
This field is no longer read from, so we should stop populating it. Once we're
happy that this doesn't break everything, and a rollback is unlikely, we can
think about dropping the column.
2018-07-26 14:43:02 +01:00
bd4b25f4d0
Remove some redundant joins on event_edges.room_id
...
We've long passed the point where it's possible to have the same event_id in
different tables, so these join conditions are redundant: we can just join on
event_id.
event_edges is of non-trivial size, and the room_id column is wasteful, so
let's stop reading from it. In future, we can stop writing to it, and then drop
it.
2018-07-26 13:19:08 +01:00
1b4d73fa52
comment on event_edges
2018-07-26 12:53:51 +01:00
21e878ebb6
Make EventStore inherit from EventFederationStore
...
(since it uses methods therein)
Turns out that we had a bunch of things which were incorrectly importing
EventWorkerStore from events.py rather than events_worker.py, which broke once
I removed the import into events.py.
2018-07-26 12:48:51 +01:00
03751a6420
Fix some looping_call calls which were broken in #3604
...
It turns out that looping_call does check the deferred returned by its
callback, and (at least in the case of client_ips), we were relying on this,
and I broke it in #3604 .
Update run_as_background_process to return the deferred, and make sure we
return it to clock.looping_call.
2018-07-26 11:48:08 +01:00
1bcd0490c2
Merge pull request #2970 from matrix-org/matthew/filter_members
...
Implement the lazy_load_members room state filter parameter
2018-07-26 00:03:01 +01:00
bc7944e6d2
switch missing_types to be a bool
2018-07-25 23:36:31 +01:00
d8e65ed7e1
Fix a minor documentation typo in on_make_leave
2018-07-25 15:44:41 -06:00
2565804030
Merge branch 'develop' into matthew/filter_members
2018-07-25 17:27:49 +01:00
7d9fb88617
incorporate more review.
2018-07-25 16:33:50 +01:00
78a691d005
Split out DB writes in federation handler
...
This will allow us to easily add an internal replication API to proxy
these reqeusts to master, so that we can move federation APIs to
workers.
2018-07-25 16:22:56 +01:00
3849f7f69f
Merge pull request #3603 from matrix-org/erikj/handle_outliers
...
Correctly handle outliers during persist events
2018-07-25 13:24:04 +01:00
cee1ae1b72
Merge pull request #3606 from matrix-org/rav/logcontext_fixes_once_more
...
Fix another logcontext leak in _persist_events
2018-07-25 11:56:00 +01:00
32b30e15f2
Merge pull request #3607 from matrix-org/rav/fix_persist_events_integrity_error
...
Fix occasional 'tuple index out of range' error
2018-07-25 11:55:45 +01:00
4081bd1560
Merge pull request #3605 from matrix-org/rav/fix_update_remote_profile_cache
...
Fix updating of cached remote profiles
2018-07-25 11:54:46 +01:00
7780a7b47c
Actually fix it by adding continue
2018-07-25 11:13:20 +01:00
1be94440d3
Fix occasional 'tuple index out of range' error
...
This fixes a bug in _delete_existing_rows_txn which was introduced in #3435
(though it's been on matrix-org-hotfixes for *years*). This code is only called
when there is some sort of conflict the first time we try to persist an event,
so it only happens rarely. Still, the exceptions are annoying.
2018-07-25 11:05:58 +01:00
07defd5fe6
Fix another logcontext leak in _persist_events
...
We need to run the errback in the sentinel context to avoid losing our own
context.
Also: add logging to runInteraction to help identify where "Starting db
connection from sentinel context" warnings are coming from
2018-07-25 10:53:23 +01:00
55acd6856c
Fix updating of cached remote profiles
...
_update_remote_profile_cache was missing its `defer.inlineCallbacks`, so when
it was called, would just return a generator object, without actually running
any of the method body.
2018-07-25 10:34:48 +01:00
f59be4eb0e
Fix unit tests
...
on_notifier_poke no longer runs synchonously, so we have to do a different hack
to make sure that the replication data has been sent. Let's actually listen for
its arrival.
2018-07-25 10:30:36 +01:00
a297ff2b16
Fix typo in conditional
2018-07-25 09:48:01 +01:00
371da42ae4
Wrap a number of things that run in the background
...
This will reduce the number of "Starting db connection from sentinel context"
warnings, and will help with our metrics.
2018-07-25 09:41:12 +01:00
ec56121b0d
Correctly handle outliers during persist events
...
We incorrectly asserted that all contexts must have a non None state
group without consider outliers. This would usually be fine as the
assertion would never be hit, as there is a shortcut during persistence
if the forward extremities don't change.
However, if the outlier is being persisted with non-outlier events, the
function would be called and the assertion would be hit.
Fixes #3601
2018-07-25 09:35:02 +01:00
cb5c37a57c
handle the edge case for _get_some_state_from_cache where types is []
2018-07-24 20:34:45 +01:00
87951d3891
Merge branch 'develop' of github.com:matrix-org/synapse into t3chguy/default_inviter_display_name_3pid
2018-07-24 17:17:46 +01:00
8b8c4f34a3
Replace usage of get_current_toke with StreamToken.START
...
This allows us to handle /context/ requests on the client_reader worker
without having to pull in all the various stream handlers (e.g.
precence, typing, pushers etc). The only thing the token gets used for
is pagination, and that ignores everything but the room portion of the
token.
2018-07-24 16:49:17 +01:00
3188973857
Pull out did_forget to worker store
2018-07-24 16:49:14 +01:00
60a1d147a7
Merge pull request #3595 from matrix-org/erikj/use_deltas
...
Use deltas to calculate current state deltas
2018-07-24 15:41:18 +01:00
709c309b0e
Expand on docstring comment about return value
2018-07-24 15:12:50 +01:00
8f65ab98d2
Remove unnecessary iteritems
2018-07-24 15:08:01 +01:00
ed0dd68731
Fixup comment (and indent)
2018-07-24 14:31:38 +01:00
811ac73a42
Don't fetch state from the database unless needed
2018-07-24 14:18:23 +01:00
a79410e7b8
Have _get_new_state_after_events return delta
...
If we have a delta from the existing to new current state, then we can
reuse that rather than manually working it out by fetching both lots of
state.
2018-07-24 14:14:17 +01:00
81946db9cf
Merge pull request #3587 from matrix-org/rav/better_exception_logging
...
Improve logging for exceptions when handling PDUs
2018-07-24 14:12:12 +01:00
a321f78991
Merge pull request #3586 from matrix-org/rav/optimise_resolve_state_groups
...
Fixes and optimisations for resolve_state_groups
2018-07-24 14:11:45 +01:00
93b0722c50
Merge pull request #3583 from matrix-org/rav/remove_who_forgot_in_room
...
Remove redundant checks on room forgottenness
2018-07-24 14:11:11 +01:00
454f59b7ad
Merge branch 'develop' into matthew/filter_members
2018-07-24 14:03:37 +01:00
1a01a5b964
clarify comment on p_ids
2018-07-24 14:03:15 +01:00
223341205e
Don't require to_delete to have event_ids
2018-07-24 14:02:40 +01:00
e22700c3dd
consider non-filter_type types as wildcards, thus missing from the state-group-cache
2018-07-24 13:59:07 +01:00
eb1d911ab7
rather than adding ll_ids, remove them from p_ids
2018-07-24 13:40:49 +01:00
d19fba3655
Merge branch 'develop' into matthew/filter_members
2018-07-24 12:39:54 +01:00
cd241d6bda
incorporate more review
2018-07-24 12:39:40 +01:00
30bfed5aa5
Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room
2018-07-24 11:46:09 +01:00
97acd385a3
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/speed_up_calculate_state_delta
2018-07-24 11:32:13 +01:00
0fa73e4a63
Remove unnecessary if
2018-07-24 11:19:23 +01:00
ff5426f6b8
Speed up _calculate_state_delta
2018-07-24 10:55:11 +01:00
a678145010
Merge branch 'develop' into rav/logcontext_fixes
2018-07-24 10:43:30 +01:00
d436ad332c
Merge pull request #3555 from matrix-org/erikj/client_apis_move
...
Make client_reader support some more read only APIs
2018-07-24 10:42:28 +01:00
536bc63a4e
Merge branch 'develop' into erikj/client_apis_move
2018-07-24 09:57:05 +01:00
cf2d15c6a9
another couple of logcontext leaks
2018-07-24 00:57:48 +01:00
8dff6e0322
Logcontext fixes
...
Fix some random logcontext leaks.
2018-07-24 00:37:17 +01:00
69fb5dbdab
fix idiocy
2018-07-24 00:04:44 +01:00
1938cffaea
Add some measure blocks to persist_events
...
... to help us figure out where 40% of CPU is going
2018-07-23 23:48:19 +01:00
efcdacad7d
handle case where types is [] on postgres correctly
2018-07-23 22:41:05 +01:00
ce0c18dec5
Improve logging for exceptions handling PDUs
...
when we get an exception handling a federation PDU, log the whole stacktrace.
2018-07-23 22:13:19 +01:00
c1f80effbe
Handle delta_ids being None in _update_context_for_auth_events
...
it's easier to create the new state group as a delta from the existing one.
(There's an outside chance this will help with
https://github.com/matrix-org/synapse/issues/3364 )
2018-07-23 22:06:50 +01:00
adfe29ec0b
Merge branch 'develop' into matthew/filter_members
2018-07-23 19:21:37 +01:00
254fb430d1
incorporate review
2018-07-23 19:21:20 +01:00
5c705f70c9
Fixes and optimisations for resolve_state_groups
...
First of all, fix the logic which looks for identical input state groups so
that we actually use them. This turned out to be most easily done by factoring
the relevant code out to a separate function so that we could do an early
return.
Secondly, avoid building the whole `conflicted_state` dict (which was only ever
used as a boolean flag).
Thirdly, replace the construction of the `state` dict (which mapped from keys
to events that set them), with an optimistic construction of the resolution
result assuming there will be no conflicts. This should be no slower than
building the old `state` dict, and:
- in the conflicted case, we'll short-cut it, saving part of the work
- in the unconflicted case, it saves rebuilding the resolution from the
`state` dict.
Finally, do a couple of s/values/itervalues/.
2018-07-23 19:10:11 +01:00
8b9f164fff
Comments
2018-07-23 17:43:01 +01:00
50c60e5fad
Only get cached state from context in persist_event
...
We don't want to bother pulling out the current state from the DB since
until we know we have to. Checking the context for state is just an
optimisation.
2018-07-23 17:21:40 +01:00
4f5cc8e4e7
Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room
2018-07-23 17:15:12 +01:00
dae6dc1e77
Remove redundant checks on room forgottenness
...
Fixes #3550
2018-07-23 17:13:34 +01:00
0faa3223cd
Fix missing attributes on workers.
...
This was missed during the transition from attribute to getter for
getting state from context.
2018-07-23 16:28:00 +01:00
999bcf9d01
Fix EventContext when using workers
...
We were:
1. Not correctly setting all attributes
2. Using defer.inlineCallbacks in a non-generator
2018-07-23 15:24:21 +01:00
4797ed000e
Update docstrings to make sense
2018-07-23 15:05:56 +01:00
0b0b24cb82
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/client_apis_move
2018-07-23 13:21:15 +01:00
027bc01a1b
Add support for updating state
2018-07-23 13:17:25 +01:00
e42510ba63
Use new getters
2018-07-23 13:17:22 +01:00
440b8845b5
Make EventContext lazy load state
2018-07-23 12:56:56 +01:00
842cdece42
pep8
2018-07-23 12:31:35 +01:00
acbfdc3442
Refcator EventContext to accept state during init
2018-07-23 12:20:55 +01:00
9b34f3ea3a
Merge branch 'develop' into matthew/sync_deleted_devices
2018-07-23 10:03:28 +01:00
c1bf2b587e
add trailing comma
2018-07-23 09:56:23 +01:00
3132b89f12
Make the rest of the .iterwhatever go away ( #3562 )
2018-07-21 15:47:18 +10:00
5c88bb722f
Move PaginationHandler to its own file
2018-07-20 15:32:23 +01:00
0ecf68aedc
Move check_in_room_or_world_readable to Auth
2018-07-20 15:30:59 +01:00
3d6df84658
Test and fix support for cancellation in Linearizer
2018-07-20 13:59:55 +01:00
4f67623674
Merge pull request #3571 from matrix-org/rav/limiter_fixes
...
A set of improvements to the Limiter
2018-07-20 13:53:03 +01:00
e1a237eaab
Admin API for creating new users ( #3415 )
2018-07-20 22:41:13 +10:00
7c712f95bb
Combine Limiter and Linearizer
...
Linearizer was effectively a Limiter with max_count=1, so rather than
maintaining two sets of code, let's combine them.
2018-07-20 13:11:43 +01:00
8462c26485
Improvements to the Limiter
...
* give them names, to improve logging
* use a deque rather than a list for efficiency
2018-07-20 12:50:27 +01:00
d7275eecf3
Add a sleep to the Limiter to fix stack overflows.
...
Fixes #3570
2018-07-20 12:37:12 +01:00
2f558300cc
fix thinkos; unbreak tests
2018-07-19 19:22:27 +01:00
bcaec2915a
incorporate review
2018-07-19 19:03:50 +01:00
924eb34d94
add a filtered_types param to limit filtering to specific types
2018-07-19 18:32:02 +01:00
5f3658baf5
Merge pull request #3377 from Valodim/note-affinity
...
document that the affinity package is required for the cpu_affinity setting
2018-07-19 14:35:06 +01:00
9a8acdc720
Merge branch 'master' into develop
2018-07-19 21:16:44 +10:00
38f53399a2
0.33 final
2018-07-19 21:11:40 +10:00
95ccb6e2ec
Don't spew errors because we can't save metrics ( #3563 )
2018-07-19 20:58:18 +10:00
be3adfc331
merge develop pydoc for _get_state_for_groups
2018-07-19 11:26:04 +01:00
9e40834f74
yes, we do need to invalidate the device_id_exists_cache when deleting a remote device
2018-07-19 11:15:10 +01:00
f1a15ea206
revert 00bc979
...
... we've fixed the things that caused the warnings, so we should reinstate the
warning.
2018-07-19 11:14:20 +01:00
1ffb7bec20
Merge remote-tracking branch 'origin/release-v0.33.0' into develop
2018-07-19 11:12:33 +01:00
c754e006f4
Merge pull request #3556 from matrix-org/rav/background_processes
...
Run things as background processes
2018-07-19 11:04:18 +01:00
a97c845271
Move v1-only APIs into their own module & isolate deprecated ones ( #3460 )
2018-07-19 20:03:33 +10:00
c0685f67c0
spell out that include_deleted_devices requires include_all_devices
2018-07-19 10:59:02 +01:00
00bc979137
Disable logcontext warning
...
Temporary workaround to #3518 while we release 0.33.0.
2018-07-19 10:51:15 +01:00
6f62a6ef21
Merge pull request #3554 from matrix-org/erikj/response_metrics_code
...
Add response code to response timer metrics
2018-07-19 10:25:18 +01:00
8c69b735e3
Make Distributor run its processes as a background process
...
This is more involved than it might otherwise be, because the current
implementation just drops its logcontexts and runs everything in the sentinel
context.
It turns out that we aren't actually using a bunch of the functionality here
(notably suppress_failures and the fact that Distributor.fire returns a
deferred), so the easiest way to fix this is actually by simplifying a bunch of
code.
2018-07-18 20:55:05 +01:00
667fba68f3
Run things as background processes
...
This fixes #3518 , and ensures that we get useful logs and metrics for lots of
things that happen in the background.
(There are certainly more things that happen in the background; these are just
the common ones I've found running a single-process synapse locally).
2018-07-18 20:55:05 +01:00
9b596177ae
Add some room read only APIs to client_reader
2018-07-18 15:33:03 +01:00
bacdf0cbf9
Move RoomContextHandler out of Handlers
...
This is in preparation for moving GET /context/ to a worker
2018-07-18 15:33:03 +01:00
8cb8df55e9
Split MessageHandler into read only and writers
...
This will let us call the read only parts from workers, and so be able
to move some APIs off of master, e.g. the `/state` API.
2018-07-18 15:33:03 +01:00
65d6a0e477
Merge pull request #3553 from matrix-org/rav/background_process_tracking
...
Resource tracking for background processes
2018-07-18 14:27:50 +01:00
5bd0a47fcd
pep8
2018-07-18 14:19:00 +01:00
e45a46b6e4
Add response code to response timer metrics
2018-07-18 13:59:36 +01:00
dab00faa83
Merge pull request #3367 from matrix-org/rav/drop_re_signing_hacks
...
Remove event re-signing hacks
2018-07-18 12:46:27 +01:00
c91a44572e
Merge pull request #3514 from matrix-org/dbkr/turn_dont_add_defaults
...
Comment dummy TURN parameters in default config
2018-07-18 11:56:18 +01:00
6e3fc657b4
Resource tracking for background processes
...
This introduces a mechanism for tracking resource usage by background
processes, along with an example of how it will be used.
This will help address #3518 , but more importantly will give us better insights
into things which are happening but not being shown up by the request metrics.
We *could* do this with Measure blocks, but:
- I think having them pulled out as a completely separate metric class will
make it easier to distinguish top-level processes from those which are
nested.
- I want to be able to report on in-flight background processes, and I don't
think we want to do this for *all* Measure blocks.
2018-07-18 10:50:33 +01:00
5f3d02f6eb
bump to 0.33.0rc1
2018-07-18 12:52:56 +10:00
79eb339c66
add a comment
2018-07-17 14:53:34 +01:00
d897be6a98
Fix visibility of events from erased users over federation
2018-07-17 14:02:07 +01:00
9c04b4abf9
Merge pull request #3541 from matrix-org/rav/optimize_filter_events_for_server
...
Refactor and optimze filter_events_for_server
2018-07-17 14:01:39 +01:00
94440ae994
fix imports
2018-07-17 11:51:26 +01:00
bc006b3c9d
Refactor REST API tests to use explicit reactors ( #3351 )
2018-07-17 20:43:18 +10:00
2172a3d8cb
add a comment
2018-07-17 11:13:57 +01:00
b2aa05a8d6
Use efficient .intersection
2018-07-17 11:07:04 +01:00
547b1355d3
Fix perf regression in PR #3530
...
The get_entities_changed function was changed to return all changed
entities since the given stream position, rather than only those changed
from a given list of entities. This resulted in the function incorrectly
returning large numbers of entities that, for example, caused large
increases in database usage.
2018-07-17 10:27:51 +01:00
3fe0938b76
Merge pull request #3530 from matrix-org/erikj/stream_cache
...
Don't return unknown entities in get_entities_changed
2018-07-17 13:44:46 +10:00
09e29fb58b
Attempt to make _filter_events_for_server more efficient
2018-07-16 14:06:09 +01:00
78a9ddcf9a
rerun isort with latest version
2018-07-16 14:23:25 +02:00
ea69d35651
Move filter_events_for_server out of FederationHandler
...
for easier unit testing.
2018-07-16 13:06:24 +01:00
4a27000548
check isort by travis
2018-07-16 13:57:33 +02:00
505530f36a
Merge remote-tracking branch 'upstream/develop' into send_sni_for_federation_requests
...
# Conflicts:
# synapse/crypto/context_factory.py
2018-07-14 20:24:46 +02:00
8a4f05fefb
Fix develop because I broke it :( ( #3535 )
2018-07-14 09:51:00 +10:00
8532953c04
Merge pull request #3534 from krombel/use_parse_and_asserts_from_servlet
...
Use parse and asserts from http.servlet
2018-07-14 09:09:19 +10:00
Richard van der Hoff