Commit Graph

9197 Commits

Author SHA1 Message Date
Matthew Hodgson ca0b052307 fix factoring out of _should_replace_room_key 2018-08-12 19:14:31 -04:00
Matthew Hodgson cac0253799 rename room_key_version table correctly, and fix opt args 2018-08-12 19:14:31 -04:00
Matthew Hodgson 0abb205b47 blindly incorporate PR review - needs testing & fixing 2018-08-12 19:14:31 -04:00
Matthew Hodgson 69e51c7ba4 make /room_keys/version work 2018-08-12 19:14:31 -04:00
Matthew Hodgson 8ae64b270f implement /room_keys/version too (untested) 2018-08-12 19:14:31 -04:00
Matthew Hodgson cf1e2000f6 document the API 2018-08-12 19:13:09 -04:00
Matthew Hodgson 6b8c07abc2 make it work and fix pep8 2018-08-12 19:13:09 -04:00
Matthew Hodgson 0bc4627a73 interim WIP checkin; doesn't build yet 2018-08-12 18:23:10 -04:00
Matthew Hodgson 53ace904b2 total WIP skeleton for /room_keys API 2018-08-12 18:23:10 -04:00
Neil Johnson 31fa743567 fix sqlite/postgres incompatibility in reap_monthly_active_users 2018-08-11 22:38:34 +01:00
Neil Johnson 6c6aba76e1 implementation of server notices to alert on hitting resource limits 2018-08-10 15:12:59 +01:00
Richard van der Hoff c31793a784 Merge branch 'rav/fix_linearizer_cancellation' into develop 2018-08-10 14:57:27 +01:00
Richard van der Hoff c08f9d95b2 log *after* reloading log config
... because logging *before* reloading means the log message gets lost in the old MemoryLogger
2018-08-10 14:56:48 +01:00
Amber Brown b37c472419
Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678) 2018-08-10 23:50:21 +10:00
Andrej Shadura c75b71a397
Use recaptcha_ajax.js directly from Google
The script recaptcha_ajax.js contains minified non-free code which
we probably cannot redistribute. Since it talks to Google servers
anyway, it is better to just download it from Google directly instead
of shipping it.

This fixes #1932.
2018-08-10 13:36:14 +02:00
Richard van der Hoff 3c0213a217
Merge pull request #3439 from vojeroen/send_sni_for_federation_requests
send SNI for federation requests
2018-08-10 12:23:54 +01:00
Richard van der Hoff 638d35ef08 Fix linearizer cancellation on twisted < 18.7
Turns out that cancellation of inlineDeferreds didn't really work properly
until Twisted 18.7. This commit refactors Linearizer.queue to avoid
inlineCallbacks.
2018-08-10 10:59:09 +01:00
Neil Johnson 01021c812f wip at implementing MSC 7075 2018-08-09 22:16:00 +01:00
Jeroen 2e9c73e8ca more generic conversion of str/bytes to unicode 2018-08-09 21:31:26 +02:00
Jeroen 64899341dc include private functions from twisted 2018-08-09 21:04:22 +02:00
Neil Johnson 885ea9c602 rename _user_last_seen_monthly_active 2018-08-09 18:02:12 +01:00
Neil Johnson c1f9dec92a fix errant parenthesis 2018-08-09 17:43:26 +01:00
Neil Johnson 09cf130898 only block on sync where user is not part of the mau cohort 2018-08-09 17:39:12 +01:00
Erik Johnston bb99b1f550 Add fast path in state res for zero prev events 2018-08-09 14:58:49 +01:00
Erik Johnston ce6db0e547 Choose state algorithm based on room version 2018-08-09 14:58:47 +01:00
Erik Johnston 152c0aa58e Add constants for room versions 2018-08-09 14:55:47 +01:00
Erik Johnston 119451dcd1 Refactor state module
We split out the actual state resolution algorithm to prepare for having
multiple versions.
2018-08-09 14:55:47 +01:00
Neil Johnson 69ce057ea6 block sync if auth checks fail 2018-08-09 12:26:27 +01:00
Neil Johnson 3dce9050cf Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block 2018-08-09 11:42:02 +01:00
Neil Johnson 0ad98e38d0
Merge pull request #3655 from matrix-org/neilj/disable_hs
Flag to disable HS without disabling federation
2018-08-09 10:41:43 +00:00
Neil Johnson a5ef110749 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block 2018-08-09 11:40:37 +01:00
Erik Johnston a6c813761a Docstrings 2018-08-09 10:41:08 +01:00
Erik Johnston b179537f2a Move clean_room_for_join to master 2018-08-09 10:37:38 +01:00
Erik Johnston 72d1902bbe Fixup doc comments 2018-08-09 10:23:49 +01:00
Amber Brown 984376745b Merge branch 'master' into develop 2018-08-09 19:23:47 +10:00
Amber Brown dafe90a6be 0.33.2 2018-08-09 19:20:41 +10:00
Erik Johnston 484a0ebdfc Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles 2018-08-09 10:16:29 +01:00
Erik Johnston 5785b93711 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation 2018-08-09 10:16:16 +01:00
Erik Johnston bf7598f582 Log when we 3pid/unbind request fails 2018-08-09 10:09:56 +01:00
Erik Johnston 2bdafaf3c1
Merge pull request #3632 from matrix-org/erikj/refactor_repl_servlet
Add helper base class for generating new replication endpoints
2018-08-09 10:06:23 +01:00
Erik Johnston 62564797f5 Fixup wording and remove dead code 2018-08-09 09:56:10 +01:00
Richard van der Hoff bb89c84614
Merge pull request #3664 from matrix-org/rav/federation_metrics
more metrics for the federation and appservice senders
2018-08-08 23:16:58 +01:00
Jeroen d5c0ce4cad updated docstring for ServerContextFactory 2018-08-08 19:25:01 +02:00
Neil Johnson e92fb00f32 sync auth blocking 2018-08-08 17:54:49 +01:00
Neil Johnson 839a317c96
fix pep8 too many lines 2018-08-08 17:39:04 +01:00
Neil Johnson 5298d79fb5
Merge branch 'develop' into neilj/disable_hs 2018-08-08 16:13:03 +00:00
Richard van der Hoff 8521ae13e3
Merge pull request #3654 from matrix-org/rav/room_versions
Support for room versioning
2018-08-08 17:10:53 +01:00
Neil Johnson d2f3ef98ac
Merge branch 'develop' into neilj/disable_hs 2018-08-08 15:55:47 +00:00
Neil Johnson 2b5baebeba Merge branch 'develop' of github.com:matrix-org/synapse into neilj/disable_hs 2018-08-08 13:47:07 +01:00
Neil Johnson be59910b93 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/reserved_users 2018-08-08 13:45:21 +01:00
Neil Johnson 990fe9fc23
Merge pull request #3633 from matrix-org/neilj/mau_tracker
API for monthly_active_users table
2018-08-08 12:44:15 +00:00
Neil Johnson 312ae74746 typos 2018-08-08 13:33:16 +01:00
Erik Johnston 360ba89c50 Don't fail requests to unbind 3pids for non supporting ID servers
Older identity servers may not support the unbind 3pid request, so we
shouldn't fail the requests if we received one of 400/404/501. The
request still fails if we receive e.g. 500 responses, allowing clients
to retry requests on transient identity server errors that otherwise do
support the API.

Fixes #3661
2018-08-08 12:06:18 +01:00
Erik Johnston bebe325e6c Rename POST param to METHOD 2018-08-08 10:36:18 +01:00
Erik Johnston 5011417632 Fixup logging and docstrings 2018-08-08 10:29:58 +01:00
Richard van der Hoff bab94da79c fix metric name 2018-08-07 22:11:45 +01:00
Richard van der Hoff 53bca4690b more metrics for the federation and appservice senders 2018-08-07 19:09:48 +01:00
Neil Johnson ef3589063a prevent total number of reserved users being too large 2018-08-07 17:59:27 +01:00
Neil Johnson e8eba2b4e3 implement reserved users for mau limits 2018-08-07 17:49:43 +01:00
Richard van der Hoff 3523f5432a Don't expose default_room_version as config opt 2018-08-07 12:51:57 +01:00
Amber Brown 848431be1d fix for rc1 2018-08-07 21:13:22 +10:00
Amber Brown 3b9662339b version 2018-08-07 21:10:52 +10:00
Erik Johnston cd9765805e Allow ratelimiting on workers 2018-08-07 10:50:28 +01:00
Erik Johnston 495cb100d1 Allow profile changes to happen on workers 2018-08-07 10:50:26 +01:00
Richard van der Hoff ca9bc1f4fe Fix occasional glitches in the synapse_event_persisted_position metric
Every so often this metric glitched to a negative number. I'm assuming it was
due to backfilled events.
2018-08-07 10:00:03 +01:00
Neil Johnson a74b25faaa WIP building out mau reserved users 2018-08-06 23:25:25 +01:00
Neil Johnson 7daa8a78c5 load mau limit threepids 2018-08-06 22:55:05 +01:00
Neil Johnson 89834d9a29 Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs 2018-08-06 21:56:56 +01:00
Neil Johnson e54794f5b6 Fix postgres compatibility bug 2018-08-06 21:55:54 +01:00
Neil Johnson 7bcf126b18 Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs 2018-08-06 21:39:44 +01:00
Neil Johnson 1911c037cb update comments to reflect new sig 2018-08-06 18:01:46 +01:00
Neil Johnson 33bd07d062 Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs 2018-08-06 17:52:28 +01:00
Neil Johnson 16d78be315 make use of _simple_select_one_onecol, improved comments 2018-08-06 17:51:15 +01:00
Richard van der Hoff 19a17068f1 Check m.room.create for sane room_versions 2018-08-06 16:11:24 +01:00
Erik Johnston 96a9a29645 Pull in necessary stores in federation_reader 2018-08-06 15:23:57 +01:00
Erik Johnston 62ace05c45 Move necessary storage functions to worker classes 2018-08-06 15:23:38 +01:00
Erik Johnston 1e2bed9656 Import all functions from TransactionStore 2018-08-06 15:23:38 +01:00
Erik Johnston a3f5bf79a0 Add EDU/query handling over replication 2018-08-06 15:23:31 +01:00
Erik Johnston e26dbd82ef Add replication APIs for persisting federation events 2018-08-06 15:02:28 +01:00
Erik Johnston 051a99c400 Fix isort 2018-08-06 14:29:31 +01:00
Richard van der Hoff f900d50824 include known room versions in outgoing make_joins 2018-08-06 13:45:37 +01:00
Neil Johnson 42c6823827 disable HS from config 2018-08-04 22:07:04 +01:00
Neil Johnson e40a510fbf py3 fix 2018-08-03 23:19:13 +01:00
Neil Johnson 886be75ad1 bug fixes 2018-08-03 22:29:03 +01:00
Will Hunt 16d9701892 Return M_NOT_FOUND when a profile could not be found. (#3596) 2018-08-03 19:08:05 +01:00
Neil Johnson e10830e976 wip commit - tests failing 2018-08-03 17:55:50 +01:00
Richard van der Hoff 3777fa26aa sanity check response from make_join 2018-08-03 16:08:32 +01:00
Richard van der Hoff 0d63d93ca8 Enforce compatibility when processing make_join requests
Reject make_join requests from servers which do not support the room version.

Also include the room version in the response.
2018-08-03 16:08:32 +01:00
Richard van der Hoff 0ca459ea33 Basic support for room versioning
This is the first tranche of support for room versioning. It includes:
 * setting the default room version in the config file
 * new room_version param on the createRoom API
 * storing the version of newly-created rooms in the m.room.create event
 * fishing the version of existing rooms out of the m.room.create event
2018-08-03 16:08:32 +01:00
Richard van der Hoff 15c1ae45e5 Docstrings for BaseFederationServlet
... to save me reverse-engineering this stuff again.
2018-08-03 16:08:32 +01:00
Neil Johnson 5593ff6773 fix (lots of) py3 test failures 2018-08-03 14:59:17 +01:00
Neil Johnson 950807d93a fix caching and tests 2018-08-03 13:49:53 +01:00
Neil Johnson 897c51d274 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker 2018-08-03 13:40:47 +01:00
Erik Johnston cb298ff623 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_repl_servlet 2018-08-03 09:25:15 +01:00
Neil Johnson c0affa7b4f update generate_monthly_active_users, and reap_monthly_active_users 2018-08-02 23:03:01 +01:00
Neil Johnson 5e2f7b8084 typo 2018-08-02 22:47:48 +01:00
Neil Johnson 4ecb4bdac9 wip attempt at caching 2018-08-02 22:41:05 +01:00
Richard van der Hoff 1fa98495d0
Merge pull request #3639 from matrix-org/rav/refactor_error_handling
Clean up handling of errors from outbound requests
2018-08-02 17:38:24 +01:00
Richard van der Hoff bdae8f2e68
Merge pull request #3638 from matrix-org/rav/refactor_federation_client_exception_handling
Factor out exception handling in federation_client
2018-08-02 17:37:46 +01:00
Neil Johnson 74b1d46ad9 do mau checks based on monthly_active_users table 2018-08-02 16:57:35 +01:00
Neil Johnson 9180061b49 remove unused count_monthly_users 2018-08-02 15:55:29 +01:00
Richard van der Hoff 704c3e6239 Merge branch 'master' into develop 2018-08-02 15:43:30 +01:00
Richard van der Hoff 43ecfe0b10 Synapse 0.33.1 (2018-08-02)
===========================
 
 SECURITY FIXES
 --------------
 
 - Fix a potential issue where servers could request events for rooms they have not joined. (`#3641 <https://github.com/matrix-org/synapse/issues/3641>`_)
 - Fix a potential issue where users could see events in private rooms before they joined. (`#3642 <https://github.com/matrix-org/synapse/issues/3642>`_)
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1
 
 iQEcBAABAgAGBQJbYxcCAAoJEIofk9V1tejVg8YIAJU1xoZ2zSnordJczpvVZbzx
 Rsstk/wvE2dSGJL22gfSLzQNCbQvF4aOul1LAGJfbEQWc46BcNpyuWUxjTd3FaF7
 SfbWdkDh1w3ROSuiPA8j4CnH6EwP0w2itIBhFG7JpVEXjRgCyc1iMUl/oQESv82g
 UQyr6t/G68mE8xHm0eqvNfCjVAMSZDAnOzYZa7vfZJBqlZiGHB6Z1nsvsS3nZqQd
 BQQi/pQTXxsIL2egSvuycykiZtEZcm+QDRFi6hIatv5LRFOsQUqKVTB/D7XOoSbP
 RPOzcaSjv4mZ9a4NBzFMVVzzTRbr2EnipPd2ePrnJoOnl2eNRO4Won8zl4h+OQc=
 =U/ir
 -----END PGP SIGNATURE-----

Merge tag 'v0.33.1'

Synapse 0.33.1 (2018-08-02)
===========================

SECURITY FIXES
--------------

- Fix a potential issue where servers could request events for rooms they have not joined. (`#3641 <https://github.com/matrix-org/synapse/issues/3641>`_)
- Fix a potential issue where users could see events in private rooms before they joined. (`#3642 <https://github.com/matrix-org/synapse/issues/3642>`_)
2018-08-02 15:40:44 +01:00
Richard van der Hoff 14a4e7d5a4 Prepare 0.33.1 2018-08-02 15:31:04 +01:00
Richard van der Hoff 0bf5ec0db7 Check room visibility for /event/ requests
Make sure that the user has permission to view the requeseted event for
/event/{eventId} and /room/{roomId}/event/{eventId} requests.

Also check that the event is in the given room for
/room/{roomId}/event/{eventId}, for sanity.
2018-08-02 15:03:27 +01:00
Richard van der Hoff 14fa9d4d92 Avoid extra db lookups
Since we're about to look up the events themselves anyway, we can skip the
extra db queries here.
2018-08-02 13:55:51 +01:00
Richard van der Hoff 0a65450d04 Validation for events/rooms in fed requests
When we get a federation request which refers to an event id, make sure that
said event is in the room the caller claims it is in.

(patch supplied by @turt2live)
2018-08-02 13:48:40 +01:00
Neil Johnson 00f99f74b1 insertion into monthly_active_users 2018-08-02 13:47:19 +01:00
Neil Johnson 4a6725d9d1 Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/mau_tracker 2018-08-02 11:04:18 +01:00
Neil Johnson 165e067033 Revert "change monthly_active_users table to be a single column"
This reverts commit ec716a35b2.
2018-08-02 10:59:58 +01:00
Erik Johnston 40c1c59cf4
Merge pull request #3621 from matrix-org/erikj/split_fed_store
Split out DB writes in federation handler
2018-08-02 10:41:42 +01:00
Neil Johnson 08281fe6b7 self.db_conn unused 2018-08-01 23:26:24 +01:00
Neil Johnson c21d82bab3 normalise reaping query 2018-08-01 23:24:38 +01:00
Neil Johnson ec716a35b2 change monthly_active_users table to be a single column 2018-08-01 17:54:37 +01:00
Neil Johnson d766f26de9 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker 2018-08-01 17:49:41 +01:00
Neil Johnson 085435e13a
Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits
Initial impl of capping MAU
2018-08-01 15:58:45 +00:00
Richard van der Hoff b8d7d3996b
Merge pull request #3620 from fuzzmz/return-404-room-not-found
return 404 if room not found
2018-08-01 16:34:32 +01:00
Neil Johnson b7f203a566 count_monthly_users is now async 2018-08-01 16:17:42 +01:00
Neil Johnson 7ff44d9215 improve clarity 2018-08-01 16:17:00 +01:00
Richard van der Hoff 01e93f48ed Kill off MatrixCodeMessageException
This code brings the SimpleHttpClient into line with the
MatrixFederationHttpClient by having it raise HttpResponseExceptions when a
request fails (rather than trying to parse for matrix errors and maybe raising
MatrixCodeMessageException).

Then, whenever we were checking for MatrixCodeMessageException and turning them
into SynapseErrors, we now need to check for HttpResponseExceptions and call
to_synapse_error.
2018-08-01 16:02:46 +01:00
Richard van der Hoff 018d75a148 Refactor code for turning HttpResponseException into SynapseError
This commit replaces SynapseError.from_http_response_exception with
HttpResponseException.to_synapse_error.

The new method actually returns a ProxiedRequestError, which allows us to pass
through additional metadata from the API call.
2018-08-01 16:02:46 +01:00
Richard van der Hoff fa7dc889f1 Be more careful which errors we send back over the C-S API
We really shouldn't be sending all CodeMessageExceptions back over the C-S API;
it will include things like 401s which we shouldn't proxy.

That means that we need to explicitly turn a few HttpResponseExceptions into
SynapseErrors in the federation layer.

The effect of the latter is that the matrix errcode will get passed through
correctly to calling clients, which might help with some of the random
M_UNKNOWN errors when trying to join rooms.
2018-08-01 16:02:38 +01:00
Richard van der Hoff c82ccd3027 Factor out exception handling in federation_client
Factor out the error handling from make_membership_event, send_join, and
send_leave, so that it can be shared.
2018-08-01 16:01:04 +01:00
Amber Brown da7785147d
Python 3: Convert some unicode/bytes uses (#3569) 2018-08-02 00:54:06 +10:00
Neil Johnson 303f1c851f Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits 2018-08-01 13:42:50 +01:00
Erik Johnston a6d7b74915 update docs 2018-08-01 13:39:14 +01:00
Erik Johnston 4b256b9271 _persist_auth_tree no longer returns anything 2018-08-01 13:39:07 +01:00
Neil Johnson 4e5ac901dd clean up 2018-08-01 12:03:57 +01:00
Neil Johnson f9f5559971 fix comment 2018-08-01 12:03:42 +01:00
Neil Johnson 0aba3d361a count_monthly_users() async 2018-08-01 11:47:58 +01:00
Neil Johnson 2c54f1c225 remove need to plot limit_usage_by_mau 2018-08-01 11:46:59 +01:00
Richard van der Hoff f49147d14f
Merge pull request #3634 from matrix-org/rav/wtf_is_a_replication_layer
rename replication_layer to federation_client
2018-08-01 10:29:29 +01:00
Richard van der Hoff cab782c17e
Merge pull request #3384 from matrix-org/rav/rewrite_cachedlist_decorator
Rewrite cache list decorator
2018-08-01 10:28:56 +01:00
Neil Johnson 6023cdd227 remove errant print 2018-08-01 10:27:17 +01:00
Neil Johnson 7931393495 make count_monthly_users async synapse/handlers/auth.py 2018-08-01 10:21:56 +01:00
Neil Johnson c507fa15ce only need to loop if mau limiting is enabled 2018-08-01 10:20:42 +01:00
Travis Ralston 37be52ac34
limt -> limit 2018-07-31 16:29:09 -06:00
Serban Constantin 70af98e361
return NotFoundError if room not found
Per the Client-Server API[0] we should return
`M_NOT_FOUND` if the room isn't found instead
of generic SynapseError.

This ensures that /directory/list API returns
404 for room not found instead of 400.

[0]: https://matrix.org/docs/spec/client_server/unstable.html#get-matrix-client-r0-directory-list-room-roomid

Signed-off-by: Serban Constantin <serban.constantin@gmail.com>
2018-07-31 21:47:23 +03:00
Travis Ralston 5e2ee64660
Merge pull request #3628 from turt2live/travis/goodby-pdu-failures
Remove pdu_failures from transactions
2018-07-31 12:13:09 -06:00
Neil Johnson 6ef983ce5c api into monthly_active_users table 2018-07-31 16:36:24 +01:00
Richard van der Hoff bdbdceeafa rename replication_layer to federation_client
I have HAD ENOUGH of trying to remember wtf a replication layer is in terms of
classes.
2018-07-31 15:44:05 +01:00
Erik Johnston 443da003bc Use new helper base class for membership requests 2018-07-31 14:32:23 +01:00
Erik Johnston 729b672823 Use new helper base class for ReplicationSendEventRestServlet 2018-07-31 14:32:23 +01:00
Erik Johnston d81602b75a Add helper base class for generating new replication endpoints
This will hopefully reduce the boiler plate required to implement new
internal HTTP requests.
2018-07-31 14:32:20 +01:00
Richard van der Hoff 5de936caa1
Merge pull request #3612 from matrix-org/rav/store_heirarchy
Make EventStore inherit from EventFederationStore
2018-07-31 13:44:04 +01:00
Neil Johnson df2235e7fa coding style 2018-07-31 13:16:20 +01:00
Richard van der Hoff 0bc9b9e397 reinstate explicit include of EventsWorkerStore 2018-07-31 13:11:04 +01:00
Travis Ralston e908b86832 Remove pdu_failures from transactions
The field is never read from, and all the opportunities given to populate it are not utilized. It should be very safe to remove this.
2018-07-30 16:28:47 -06:00
Krombel 254e8267e2 Only import secrets when available
secrets got introduced in python 3.6 so this class is not available
in 3.5 and before.

This now checks for the current running version and only tries using
secrets if the version is 3.6 or above

Signed-Off-By: Matthias Kesler <krombel@krombel.de>
2018-07-30 23:59:02 +02:00
Neil Johnson 21276ff846 remove errant logging 2018-07-30 22:42:12 +01:00
Neil Johnson fef7e58ac6 actually close conn 2018-07-30 22:29:44 +01:00
Neil Johnson 9b13817e06 factor out metrics from __init__ to app/homeserver 2018-07-30 22:07:07 +01:00
Neil Johnson 251e6c1210 limit register and sign in on number of monthly users 2018-07-30 15:55:57 +01:00
Erik Johnston 143f1a2532 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store 2018-07-30 09:56:18 +01:00
Jeroen 2903e65aff fix isort 2018-07-29 19:47:08 +02:00
Richard van der Hoff a8cbce0ced fix invalidation 2018-07-27 16:17:17 +01:00
Matthew Hodgson e9b2d047f6
make /context lazyload & filter aware (#3567)
make /context lazyload & filter aware.
2018-07-27 15:12:50 +01:00
Richard van der Hoff f102c05856 Rewrite cache list decorator
Because it was complicated and annoyed me. I suspect this will be more
efficient too.
2018-07-27 13:47:04 +01:00
Jeroen 8e3f75b39a fix accidental removal of hs 2018-07-27 12:17:31 +02:00
Richard van der Hoff b0b5566f36
Merge pull request #3391 from t3chguy/t3chguy/default_inviter_display_name_3pid
if inviter_display_name == ""||None then default to inviter MXID
2018-07-27 10:08:39 +01:00
Richard van der Hoff 7041cd872b
Merge branch 'develop' into send_sni_for_federation_requests 2018-07-27 09:17:11 +01:00
Richard van der Hoff d42455cdc9
Merge pull request #3616 from matrix-org/travis/event_id_send_leave
Update the send_leave path to be an event_id
2018-07-26 23:25:17 +01:00
Richard van der Hoff 65c8dee900
Merge pull request #3614 from matrix-org/rav/stop_populating_event_content
Stop populating events.content
2018-07-26 22:54:08 +01:00
Matthew Hodgson a75231b507
Deduplicate redundant lazy-loaded members (#3331)
* attempt at deduplicating lazy-loaded members

as per the proposal; we can deduplicate redundant lazy-loaded members
which are sent in the same sync sequence. we do this heuristically
rather than requiring the client to somehow tell us which members it
has chosen to cache, by instead caching the last N members sent to
a client, and not sending them again.  For now we hardcode N to 100.
Each cache for a given (user,device) tuple is in turn cached for up to
X minutes (to avoid the caches building up).  For now we hardcode X to 30.

* add include_redundant_members filter option & make it work

* remove stale todo

* add tests for _get_some_state_from_cache

* incorporate review
2018-07-26 22:51:30 +01:00
Richard van der Hoff 9e68b1bd2d
Merge pull request #3613 from matrix-org/rav/stop_using_event_edges_room_id
Remove some redundant joins on event_edges.room_id
2018-07-26 22:31:01 +01:00
Travis Ralston 7d32f0d745
Update the send_leave path to be an event_id
It's still not used, however the parameter is an event ID not a transaction ID.
2018-07-26 14:41:59 -06:00
Richard van der Hoff ef9d51b081
Merge pull request #3610 from matrix-org/rav/fix_looping_calls
Fix some looping_call calls which were broken in #3604
2018-07-26 14:55:57 +01:00
Richard van der Hoff 51d7df1915 Create the column nullable
There's no real point in ever making the column non-nullable, and doing so
breaks the sytests.
2018-07-26 14:54:04 +01:00
Richard van der Hoff 5c1d301fd9 Stop populating events.content
This field is no longer read from, so we should stop populating it. Once we're
happy that this doesn't break everything, and a rollback is unlikely, we can
think about dropping the column.
2018-07-26 14:43:02 +01:00
Richard van der Hoff bd4b25f4d0 Remove some redundant joins on event_edges.room_id
We've long passed the point where it's possible to have the same event_id in
different tables, so these join conditions are redundant: we can just join on
event_id.

event_edges is of non-trivial size, and the room_id column is wasteful, so
let's stop reading from it. In future, we can stop writing to it, and then drop
it.
2018-07-26 13:19:08 +01:00
Richard van der Hoff 1b4d73fa52 comment on event_edges 2018-07-26 12:53:51 +01:00
Richard van der Hoff 21e878ebb6 Make EventStore inherit from EventFederationStore
(since it uses methods therein)

Turns out that we had a bunch of things which were incorrectly importing
EventWorkerStore from events.py rather than events_worker.py, which broke once
I removed the import into events.py.
2018-07-26 12:48:51 +01:00
Richard van der Hoff 03751a6420 Fix some looping_call calls which were broken in #3604
It turns out that looping_call does check the deferred returned by its
callback, and (at least in the case of client_ips), we were relying on this,
and I broke it in #3604.

Update run_as_background_process to return the deferred, and make sure we
return it to clock.looping_call.
2018-07-26 11:48:08 +01:00
Matthew Hodgson 1bcd0490c2
Merge pull request #2970 from matrix-org/matthew/filter_members
Implement the lazy_load_members room state filter parameter
2018-07-26 00:03:01 +01:00
Matthew Hodgson bc7944e6d2 switch missing_types to be a bool 2018-07-25 23:36:31 +01:00
Travis Ralston d8e65ed7e1
Fix a minor documentation typo in on_make_leave 2018-07-25 15:44:41 -06:00
Matthew Hodgson 2565804030 Merge branch 'develop' into matthew/filter_members 2018-07-25 17:27:49 +01:00
Matthew Hodgson 7d9fb88617 incorporate more review. 2018-07-25 16:33:50 +01:00
Erik Johnston 78a691d005 Split out DB writes in federation handler
This will allow us to easily add an internal replication API to proxy
these reqeusts to master, so that we can move federation APIs to
workers.
2018-07-25 16:22:56 +01:00
Erik Johnston 3849f7f69f
Merge pull request #3603 from matrix-org/erikj/handle_outliers
Correctly handle outliers during persist events
2018-07-25 13:24:04 +01:00
Richard van der Hoff cee1ae1b72
Merge pull request #3606 from matrix-org/rav/logcontext_fixes_once_more
Fix another logcontext leak in _persist_events
2018-07-25 11:56:00 +01:00
Richard van der Hoff 32b30e15f2
Merge pull request #3607 from matrix-org/rav/fix_persist_events_integrity_error
Fix occasional 'tuple index out of range' error
2018-07-25 11:55:45 +01:00
Richard van der Hoff 4081bd1560
Merge pull request #3605 from matrix-org/rav/fix_update_remote_profile_cache
Fix updating of cached remote profiles
2018-07-25 11:54:46 +01:00
Erik Johnston 7780a7b47c Actually fix it by adding continue 2018-07-25 11:13:20 +01:00
Richard van der Hoff 1be94440d3 Fix occasional 'tuple index out of range' error
This fixes a bug in _delete_existing_rows_txn which was introduced in #3435
(though it's been on matrix-org-hotfixes for *years*). This code is only called
when there is some sort of conflict the first time we try to persist an event,
so it only happens rarely. Still, the exceptions are annoying.
2018-07-25 11:05:58 +01:00
Richard van der Hoff 07defd5fe6 Fix another logcontext leak in _persist_events
We need to run the errback in the sentinel context to avoid losing our own
context.

Also: add logging to runInteraction to help identify where "Starting db
connection from sentinel context" warnings are coming from
2018-07-25 10:53:23 +01:00
Richard van der Hoff 55acd6856c Fix updating of cached remote profiles
_update_remote_profile_cache was missing its `defer.inlineCallbacks`, so when
it was called, would just return a generator object, without actually running
any of the method body.
2018-07-25 10:34:48 +01:00
Richard van der Hoff f59be4eb0e Fix unit tests
on_notifier_poke no longer runs synchonously, so we have to do a different hack
to make sure that the replication data has been sent. Let's actually listen for
its arrival.
2018-07-25 10:30:36 +01:00
Erik Johnston a297ff2b16 Fix typo in conditional 2018-07-25 09:48:01 +01:00
Richard van der Hoff 371da42ae4 Wrap a number of things that run in the background
This will reduce the number of "Starting db connection from sentinel context"
warnings, and will help with our metrics.
2018-07-25 09:41:12 +01:00
Erik Johnston ec56121b0d Correctly handle outliers during persist events
We incorrectly asserted that all contexts must have a non None state
group without consider outliers. This would usually be fine as the
assertion would never be hit, as there is a shortcut during persistence
if the forward extremities don't change.

However, if the outlier is being persisted with non-outlier events, the
function would be called and the assertion would be hit.

Fixes #3601
2018-07-25 09:35:02 +01:00
Matthew Hodgson cb5c37a57c handle the edge case for _get_some_state_from_cache where types is [] 2018-07-24 20:34:45 +01:00
Michael Telatynski 87951d3891
Merge branch 'develop' of github.com:matrix-org/synapse into t3chguy/default_inviter_display_name_3pid 2018-07-24 17:17:46 +01:00
Erik Johnston 8b8c4f34a3 Replace usage of get_current_toke with StreamToken.START
This allows us to handle /context/ requests on the client_reader worker
without having to pull in all the various stream handlers (e.g.
precence, typing, pushers etc). The only thing the token gets used for
is pagination, and that ignores everything but the room portion of the
token.
2018-07-24 16:49:17 +01:00
Erik Johnston 3188973857 Pull out did_forget to worker store 2018-07-24 16:49:14 +01:00
Erik Johnston 60a1d147a7
Merge pull request #3595 from matrix-org/erikj/use_deltas
Use deltas to calculate current state deltas
2018-07-24 15:41:18 +01:00
Erik Johnston 709c309b0e Expand on docstring comment about return value 2018-07-24 15:12:50 +01:00
Erik Johnston 8f65ab98d2 Remove unnecessary iteritems 2018-07-24 15:08:01 +01:00
Erik Johnston ed0dd68731 Fixup comment (and indent) 2018-07-24 14:31:38 +01:00
Erik Johnston 811ac73a42 Don't fetch state from the database unless needed 2018-07-24 14:18:23 +01:00
Erik Johnston a79410e7b8 Have _get_new_state_after_events return delta
If we have a delta from the existing to new current state, then we can
reuse that rather than manually working it out by fetching both lots of
state.
2018-07-24 14:14:17 +01:00
Richard van der Hoff 81946db9cf
Merge pull request #3587 from matrix-org/rav/better_exception_logging
Improve logging for exceptions when handling PDUs
2018-07-24 14:12:12 +01:00
Richard van der Hoff a321f78991
Merge pull request #3586 from matrix-org/rav/optimise_resolve_state_groups
Fixes and optimisations for resolve_state_groups
2018-07-24 14:11:45 +01:00
Richard van der Hoff 93b0722c50
Merge pull request #3583 from matrix-org/rav/remove_who_forgot_in_room
Remove redundant checks on room forgottenness
2018-07-24 14:11:11 +01:00
Matthew Hodgson 454f59b7ad Merge branch 'develop' into matthew/filter_members 2018-07-24 14:03:37 +01:00
Matthew Hodgson 1a01a5b964 clarify comment on p_ids 2018-07-24 14:03:15 +01:00
Erik Johnston 223341205e Don't require to_delete to have event_ids 2018-07-24 14:02:40 +01:00
Matthew Hodgson e22700c3dd consider non-filter_type types as wildcards, thus missing from the state-group-cache 2018-07-24 13:59:07 +01:00
Matthew Hodgson eb1d911ab7 rather than adding ll_ids, remove them from p_ids 2018-07-24 13:40:49 +01:00
Matthew Hodgson d19fba3655 Merge branch 'develop' into matthew/filter_members 2018-07-24 12:39:54 +01:00
Matthew Hodgson cd241d6bda incorporate more review 2018-07-24 12:39:40 +01:00
Richard van der Hoff 30bfed5aa5 Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room 2018-07-24 11:46:09 +01:00
Erik Johnston 97acd385a3 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/speed_up_calculate_state_delta 2018-07-24 11:32:13 +01:00
Erik Johnston 0fa73e4a63 Remove unnecessary if 2018-07-24 11:19:23 +01:00
Erik Johnston ff5426f6b8 Speed up _calculate_state_delta 2018-07-24 10:55:11 +01:00
Richard van der Hoff a678145010
Merge branch 'develop' into rav/logcontext_fixes 2018-07-24 10:43:30 +01:00
Erik Johnston d436ad332c
Merge pull request #3555 from matrix-org/erikj/client_apis_move
Make client_reader support some more read only APIs
2018-07-24 10:42:28 +01:00
Erik Johnston 536bc63a4e
Merge branch 'develop' into erikj/client_apis_move 2018-07-24 09:57:05 +01:00
Richard van der Hoff cf2d15c6a9 another couple of logcontext leaks 2018-07-24 00:57:48 +01:00
Richard van der Hoff 8dff6e0322 Logcontext fixes
Fix some random logcontext leaks.
2018-07-24 00:37:17 +01:00
Richard van der Hoff 69fb5dbdab fix idiocy 2018-07-24 00:04:44 +01:00
Richard van der Hoff 1938cffaea Add some measure blocks to persist_events
... to help us figure out where 40% of CPU is going
2018-07-23 23:48:19 +01:00
Matthew Hodgson efcdacad7d handle case where types is [] on postgres correctly 2018-07-23 22:41:05 +01:00
Richard van der Hoff ce0c18dec5 Improve logging for exceptions handling PDUs
when we get an exception handling a federation PDU, log the whole stacktrace.
2018-07-23 22:13:19 +01:00
Richard van der Hoff c1f80effbe Handle delta_ids being None in _update_context_for_auth_events
it's easier to create the new state group as a delta from the existing one.

(There's an outside chance this will help with
https://github.com/matrix-org/synapse/issues/3364)
2018-07-23 22:06:50 +01:00
Matthew Hodgson adfe29ec0b Merge branch 'develop' into matthew/filter_members 2018-07-23 19:21:37 +01:00
Matthew Hodgson 254fb430d1 incorporate review 2018-07-23 19:21:20 +01:00
Richard van der Hoff 5c705f70c9 Fixes and optimisations for resolve_state_groups
First of all, fix the logic which looks for identical input state groups so
that we actually use them. This turned out to be most easily done by factoring
the relevant code out to a separate function so that we could do an early
return.

Secondly, avoid building the whole `conflicted_state` dict (which was only ever
used as a boolean flag).

Thirdly, replace the construction of the `state` dict (which mapped from keys
to events that set them), with an optimistic construction of the resolution
result assuming there will be no conflicts. This should be no slower than
building the old `state` dict, and:
  - in the conflicted case, we'll short-cut it, saving part of the work
  - in the unconflicted case, it saves rebuilding the resolution from the
    `state` dict.

Finally, do a couple of s/values/itervalues/.
2018-07-23 19:10:11 +01:00
Erik Johnston 8b9f164fff Comments 2018-07-23 17:43:01 +01:00
Erik Johnston 50c60e5fad Only get cached state from context in persist_event
We don't want to bother pulling out the current state from the DB since
until we know we have to. Checking the context for state is just an
optimisation.
2018-07-23 17:21:40 +01:00
Richard van der Hoff 4f5cc8e4e7 Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room 2018-07-23 17:15:12 +01:00
Richard van der Hoff dae6dc1e77 Remove redundant checks on room forgottenness
Fixes #3550
2018-07-23 17:13:34 +01:00
Erik Johnston 0faa3223cd Fix missing attributes on workers.
This was missed during the transition from attribute to getter for
getting state from context.
2018-07-23 16:28:00 +01:00
Erik Johnston 999bcf9d01 Fix EventContext when using workers
We were:
  1. Not correctly setting all attributes
  2. Using defer.inlineCallbacks in a non-generator
2018-07-23 15:24:21 +01:00
Erik Johnston 4797ed000e Update docstrings to make sense 2018-07-23 15:05:56 +01:00
Erik Johnston 0b0b24cb82 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/client_apis_move 2018-07-23 13:21:15 +01:00
Erik Johnston 027bc01a1b Add support for updating state 2018-07-23 13:17:25 +01:00
Erik Johnston e42510ba63 Use new getters 2018-07-23 13:17:22 +01:00
Erik Johnston 440b8845b5 Make EventContext lazy load state 2018-07-23 12:56:56 +01:00
Erik Johnston 842cdece42 pep8 2018-07-23 12:31:35 +01:00
Erik Johnston acbfdc3442 Refcator EventContext to accept state during init 2018-07-23 12:20:55 +01:00
Matthew Hodgson 9b34f3ea3a
Merge branch 'develop' into matthew/sync_deleted_devices 2018-07-23 10:03:28 +01:00
Matthew Hodgson c1bf2b587e
add trailing comma 2018-07-23 09:56:23 +01:00
Amber Brown 3132b89f12
Make the rest of the .iterwhatever go away (#3562) 2018-07-21 15:47:18 +10:00
Erik Johnston 5c88bb722f Move PaginationHandler to its own file 2018-07-20 15:32:23 +01:00
Erik Johnston 0ecf68aedc Move check_in_room_or_world_readable to Auth 2018-07-20 15:30:59 +01:00
Richard van der Hoff 3d6df84658 Test and fix support for cancellation in Linearizer 2018-07-20 13:59:55 +01:00
Richard van der Hoff 4f67623674
Merge pull request #3571 from matrix-org/rav/limiter_fixes
A set of improvements to the Limiter
2018-07-20 13:53:03 +01:00
Amber Brown e1a237eaab
Admin API for creating new users (#3415) 2018-07-20 22:41:13 +10:00
Richard van der Hoff 7c712f95bb Combine Limiter and Linearizer
Linearizer was effectively a Limiter with max_count=1, so rather than
maintaining two sets of code, let's combine them.
2018-07-20 13:11:43 +01:00
Richard van der Hoff 8462c26485 Improvements to the Limiter
* give them names, to improve logging
* use a deque rather than a list for efficiency
2018-07-20 12:50:27 +01:00
Richard van der Hoff d7275eecf3 Add a sleep to the Limiter to fix stack overflows.
Fixes #3570
2018-07-20 12:37:12 +01:00
Matthew Hodgson 2f558300cc fix thinkos; unbreak tests 2018-07-19 19:22:27 +01:00
Matthew Hodgson bcaec2915a incorporate review 2018-07-19 19:03:50 +01:00
Matthew Hodgson 924eb34d94 add a filtered_types param to limit filtering to specific types 2018-07-19 18:32:02 +01:00
Richard van der Hoff 5f3658baf5
Merge pull request #3377 from Valodim/note-affinity
document that the affinity package is required for the cpu_affinity setting
2018-07-19 14:35:06 +01:00
Amber Brown 9a8acdc720 Merge branch 'master' into develop 2018-07-19 21:16:44 +10:00
Amber Brown 38f53399a2 0.33 final 2018-07-19 21:11:40 +10:00
Amber Brown 95ccb6e2ec
Don't spew errors because we can't save metrics (#3563) 2018-07-19 20:58:18 +10:00
Matthew Hodgson be3adfc331 merge develop pydoc for _get_state_for_groups 2018-07-19 11:26:04 +01:00
Matthew Hodgson 9e40834f74 yes, we do need to invalidate the device_id_exists_cache when deleting a remote device 2018-07-19 11:15:10 +01:00
Richard van der Hoff f1a15ea206 revert 00bc979
... we've fixed the things that caused the warnings, so we should reinstate the
warning.
2018-07-19 11:14:20 +01:00
Richard van der Hoff 1ffb7bec20 Merge remote-tracking branch 'origin/release-v0.33.0' into develop 2018-07-19 11:12:33 +01:00
Richard van der Hoff c754e006f4
Merge pull request #3556 from matrix-org/rav/background_processes
Run things as background processes
2018-07-19 11:04:18 +01:00
Amber Brown a97c845271
Move v1-only APIs into their own module & isolate deprecated ones (#3460) 2018-07-19 20:03:33 +10:00
Matthew Hodgson c0685f67c0 spell out that include_deleted_devices requires include_all_devices 2018-07-19 10:59:02 +01:00
Richard van der Hoff 00bc979137 Disable logcontext warning
Temporary workaround to #3518 while we release 0.33.0.
2018-07-19 10:51:15 +01:00
Erik Johnston 6f62a6ef21
Merge pull request #3554 from matrix-org/erikj/response_metrics_code
Add response code to response timer metrics
2018-07-19 10:25:18 +01:00
Richard van der Hoff 8c69b735e3 Make Distributor run its processes as a background process
This is more involved than it might otherwise be, because the current
implementation just drops its logcontexts and runs everything in the sentinel
context.

It turns out that we aren't actually using a bunch of the functionality here
(notably suppress_failures and the fact that Distributor.fire returns a
deferred), so the easiest way to fix this is actually by simplifying a bunch of
code.
2018-07-18 20:55:05 +01:00
Richard van der Hoff 667fba68f3 Run things as background processes
This fixes #3518, and ensures that we get useful logs and metrics for lots of
things that happen in the background.

(There are certainly more things that happen in the background; these are just
the common ones I've found running a single-process synapse locally).
2018-07-18 20:55:05 +01:00
Erik Johnston 9b596177ae Add some room read only APIs to client_reader 2018-07-18 15:33:03 +01:00
Erik Johnston bacdf0cbf9 Move RoomContextHandler out of Handlers
This is in preparation for moving GET /context/ to a worker
2018-07-18 15:33:03 +01:00
Erik Johnston 8cb8df55e9 Split MessageHandler into read only and writers
This will let us call the read only parts from workers, and so be able
to move some APIs off of master, e.g. the `/state` API.
2018-07-18 15:33:03 +01:00
Richard van der Hoff 65d6a0e477
Merge pull request #3553 from matrix-org/rav/background_process_tracking
Resource tracking for background processes
2018-07-18 14:27:50 +01:00
Erik Johnston 5bd0a47fcd pep8 2018-07-18 14:19:00 +01:00
Erik Johnston e45a46b6e4 Add response code to response timer metrics 2018-07-18 13:59:36 +01:00
Richard van der Hoff dab00faa83
Merge pull request #3367 from matrix-org/rav/drop_re_signing_hacks
Remove event re-signing hacks
2018-07-18 12:46:27 +01:00
David Baker c91a44572e
Merge pull request #3514 from matrix-org/dbkr/turn_dont_add_defaults
Comment dummy TURN parameters in default config
2018-07-18 11:56:18 +01:00
Richard van der Hoff 6e3fc657b4 Resource tracking for background processes
This introduces a mechanism for tracking resource usage by background
processes, along with an example of how it will be used.

This will help address #3518, but more importantly will give us better insights
into things which are happening but not being shown up by the request metrics.

We *could* do this with Measure blocks, but:
 - I think having them pulled out as a completely separate metric class will
   make it easier to distinguish top-level processes from those which are
   nested.

 - I want to be able to report on in-flight background processes, and I don't
   think we want to do this for *all* Measure blocks.
2018-07-18 10:50:33 +01:00
Amber Brown 5f3d02f6eb bump to 0.33.0rc1 2018-07-18 12:52:56 +10:00
Richard van der Hoff 79eb339c66 add a comment 2018-07-17 14:53:34 +01:00
Richard van der Hoff d897be6a98 Fix visibility of events from erased users over federation 2018-07-17 14:02:07 +01:00
Richard van der Hoff 9c04b4abf9
Merge pull request #3541 from matrix-org/rav/optimize_filter_events_for_server
Refactor and optimze filter_events_for_server
2018-07-17 14:01:39 +01:00
Richard van der Hoff 94440ae994 fix imports 2018-07-17 11:51:26 +01:00
Amber Brown bc006b3c9d
Refactor REST API tests to use explicit reactors (#3351) 2018-07-17 20:43:18 +10:00
Richard van der Hoff 2172a3d8cb add a comment 2018-07-17 11:13:57 +01:00
Erik Johnston b2aa05a8d6 Use efficient .intersection 2018-07-17 11:07:04 +01:00
Erik Johnston 547b1355d3 Fix perf regression in PR #3530
The get_entities_changed function was changed to return all changed
entities since the given stream position, rather than only those changed
from a given list of entities. This resulted in the function incorrectly
returning large numbers of entities that, for example, caused large
increases in database usage.
2018-07-17 10:27:51 +01:00
Amber Brown 3fe0938b76
Merge pull request #3530 from matrix-org/erikj/stream_cache
Don't return unknown entities in get_entities_changed
2018-07-17 13:44:46 +10:00
Richard van der Hoff 09e29fb58b Attempt to make _filter_events_for_server more efficient 2018-07-16 14:06:09 +01:00
Krombel 78a9ddcf9a rerun isort with latest version 2018-07-16 14:23:25 +02:00
Richard van der Hoff ea69d35651 Move filter_events_for_server out of FederationHandler
for easier unit testing.
2018-07-16 13:06:24 +01:00
Krombel 4a27000548 check isort by travis 2018-07-16 13:57:33 +02:00
Jeroen 505530f36a Merge remote-tracking branch 'upstream/develop' into send_sni_for_federation_requests
# Conflicts:
#	synapse/crypto/context_factory.py
2018-07-14 20:24:46 +02:00
Amber Brown 8a4f05fefb
Fix develop because I broke it :( (#3535) 2018-07-14 09:51:00 +10:00
Amber Brown 8532953c04
Merge pull request #3534 from krombel/use_parse_and_asserts_from_servlet
Use parse and asserts from http.servlet
2018-07-14 09:09:19 +10:00
Amber Brown a2374b2c7f
fix sytests 2018-07-14 07:52:58 +10:00
Amber Brown 33b60c01b5
Make auth & transactions more testable (#3499) 2018-07-14 07:34:49 +10:00
Krombel 516f960ad8 add changelog 2018-07-13 22:19:19 +02:00
Krombel 3366b9c534 rename assert_params_in_request to assert_params_in_dict
the method "assert_params_in_request" does handle dicts and not
requests. A request body has to be parsed to json before this method
can be used
2018-07-13 21:53:01 +02:00
Krombel 32fd6910d0 Use parse_{int,str} and assert from http.servlet
parse_integer and parse_string can take a request and raise errors
in case we have wrong or missing params.
This PR tries to use them more to deduplicate some code and make it
better readable
2018-07-13 21:40:14 +02:00
Richard van der Hoff 2aba1f549c
Merge pull request #3533 from matrix-org/rav/fix_federation_ratelimite_queue
Make FederationRateLimiter queue requests properly
2018-07-13 16:59:18 +01:00
Richard van der Hoff 33b40d0a25 Make FederationRateLimiter queue requests properly
popitem removes the *most recent* item by default [1]. We want the oldest.

Fixes #3524

[1]: https://docs.python.org/2/library/collections.html#collections.OrderedDict.popitem
2018-07-13 16:19:40 +01:00
Erik Johnston 77b692e65d Don't return unknown entities in get_entities_changed
The stream cache keeps track of all entities that have changed since
a particular stream position, so get_entities_changed does not need to
return unknown entites when given a larger stream position.

This makes it consistent with the behaviour of has_entity_changed.
2018-07-13 15:26:10 +01:00
Matthew Hodgson ba22b6a456 typo 2018-07-13 12:03:39 +01:00
Richard van der Hoff 6dff49b8a9
Merge pull request #3521 from matrix-org/rav/optimise_stream_change_cache
Reduce set building in get_entities_changed
2018-07-12 12:08:49 +01:00
Matthew Hodgson 12ec58301f shift to using an explicit deleted flag on m.device_list_update EDUs
and generally make it work.
2018-07-12 11:39:43 +01:00
Richard van der Hoff fa5c2bc082 Reduce set building in get_entities_changed
This line shows up as about 5% of cpu time on a synchrotron:

    not_known_entities = set(entities) - set(self._entity_to_key)

Presumably the problem here is that _entity_to_key can be largeish, and
building a set for its keys every time this function is called is slow.

Here we rewrite the logic to avoid building so many sets.
2018-07-12 11:37:44 +01:00
Richard van der Hoff 482d17b58b Merge branch 'develop' into rav/enforce_report_api 2018-07-12 09:56:28 +01:00
Erik Johnston 0456e05977
Merge pull request #3505 from matrix-org/erikj/receipts_cahce
Use stream cache in get_linearized_receipts_for_room
2018-07-12 09:46:29 +01:00
Erik Johnston aff1dfdf3d Update return value docstring 2018-07-12 09:45:37 +01:00
Matthew Hodgson 5797f5542b WIP to announce deleted devices over federation
Previously we queued up the poke correctly when the device was deleted,
but then the actual EDU wouldn't get sent, as the device was no longer known.
Instead, we now send EDUs for deleted devices too if there's a poke for them.
2018-07-12 01:32:39 +01:00
David Baker 36f4fd3e1e Comment dummy TURN parameters in default config
This default config is parsed and used a base before the actual
config is overlaid, so with these values not commented out, the
code to detect when no turn params were set and refuse to generate
credentials was never firing because the dummy default was always set.
2018-07-11 15:49:29 +01:00
Erik Johnston 6ccefef07a Use 'is not None' and add comments 2018-07-10 18:12:39 +01:00
Matthew Hodgson ea752bdd99 s/becuase/because/g 2018-07-10 17:58:18 +01:00
Erik Johnston 05f5dabc10 Use stream cache in get_linearized_receipts_for_room
This avoids us from uncessarily hitting the database when there has been
no change for the room
2018-07-10 17:22:42 +01:00
Richard van der Hoff c3c29aa196
Attempt to include db threads in cpu usage stats (#3496)
Let's try to include time spent in the DB threads in the per-request/block cpu
usage metrics.
2018-07-10 16:12:36 +01:00
Richard van der Hoff 55370331da
Refactor logcontext resource usage tracking (#3501)
Factor out the resource usage tracking out to a separate object, which can be
passed around and copied independently of the logcontext itself.
2018-07-10 13:56:07 +01:00
Matthew Hodgson 16b10666e7 another typo 2018-07-10 12:28:42 +01:00
Matthew Hodgson 4ea391a6ae typo (i think) 2018-07-10 12:08:09 +01:00
Richard van der Hoff e31e5dee38 Add CPU metrics for _fetch_event_list
add a Measure block on _fetch_event_list, in the hope that we can better
measure CPU usage here.
2018-07-09 18:15:54 +01:00
Jeroen b5e157d895 Merge branch 'develop' into send_sni_for_federation_requests
# Conflicts:
#	synapse/http/endpoint.py
2018-07-09 08:51:11 +02:00
Amber Brown 49af402019 run isort 2018-07-09 16:09:20 +10:00
Amber Brown 3060bcc8e9 version 2018-07-07 10:48:06 +10:00
Amber Brown e845fd41c2
Correct attrs package name in requirements (#3492) 2018-07-07 10:46:59 +10:00
Richard van der Hoff 1cfc2c4790 Prepare 0.32.1 release 2018-07-06 16:50:52 +01:00
Richard van der Hoff 1464a0578a Add explicit dependency on netaddr
the dependencies file, causing failures on upgrade (and presumably for new
installs).
2018-07-06 16:27:17 +01:00
Neil Johnson 277c561766 0.32.0 version bump, update changelog 2018-07-06 15:07:29 +01:00
Amber Brown d196fe42a9 bump version to 0.32.0rc1 2018-07-05 20:22:35 +10:00
Richard van der Hoff 3cf3e08a97 Implementation of server_acls
... as described at
https://docs.google.com/document/d/1EttUVzjc2DWe2ciw4XPtNpUpIl9lWXGEsy2ewDS7rtw.
2018-07-04 19:06:20 +01:00
Richard van der Hoff 546bc9e28b More server_name validation
We need to do a bit more validation when we get a server name, but don't want
to be re-doing it all over the shop, so factor out a separate
parse_and_validate_server_name, and do the extra validation.

Also, use it to verify the server name in the config file.
2018-07-04 18:59:51 +01:00
Erik Johnston 13f7adf84b
Merge pull request #3473 from matrix-org/erikj/thread_cache
Invalidate cache on correct thread
2018-07-04 10:11:38 +01:00
Erik Johnston 40252d13d1
Merge pull request #3474 from matrix-org/erikj/py3_auth
Fix up auth check
2018-07-04 09:41:33 +01:00
Richard van der Hoff a4ab491371
Merge branch 'develop' into rav/drop_re_signing_hacks 2018-07-04 07:13:38 +01:00
Richard van der Hoff 508196e08a
Reject invalid server names (#3480)
Make sure that server_names used in auth headers are sane, and reject them with
a sensible error code, before they disappear off into the depths of the system.
2018-07-03 14:36:14 +01:00
Erik Johnston 2c33b55738 Avoid relying on int vs None comparison
Python 3 doesn't support comparing None to ints
2018-07-02 11:40:32 +01:00
Erik Johnston cbf82dddf1 Ensure that we define sender_domain 2018-07-02 11:37:57 +01:00
Erik Johnston 3905c693c5 Invalidate cache on correct thread 2018-07-02 11:36:44 +01:00
Matthew Hodgson fc4f8f33be replace invalid utf8 with \ufffd 2018-07-02 11:33:02 +01:00
Matthew Hodgson 1c867f5391 a fix which doesn't NPE everywhere 2018-07-01 11:56:33 +01:00
Matthew Hodgson f131bf8d3e don't mix unicode strings with utf8-in-byte-strings
otherwise we explode with:

```
Traceback (most recent call last):
  File /usr/lib/python2.7/logging/handlers.py, line 78, in emit
    logging.FileHandler.emit(self, record)
  File /usr/lib/python2.7/logging/__init__.py, line 950, in emit
    StreamHandler.emit(self, record)
  File /usr/lib/python2.7/logging/__init__.py, line 887, in emit
    self.handleError(record)
  File /usr/lib/python2.7/logging/__init__.py, line 810, in handleError
    None, sys.stderr)
  File /usr/lib/python2.7/traceback.py, line 124, in print_exception
    _print(file, 'Traceback (most recent call last):')
  File /usr/lib/python2.7/traceback.py, line 13, in _print
    file.write(str+terminator)
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_io.py, line 170, in write
    self.log.emit(self.level, format=u{log_io}, log_io=line)
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 144, in emit
    self.observer(event)
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 136, in __call__
    errorLogger = self._errorLoggerForObserver(brokenObserver)
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 156, in _errorLoggerForObserver
    if obs is not observer
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 81, in __init__
    self.log = Logger(observer=self)
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 64, in __init__
    namespace = self._namespaceFromCallingContext()
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 42, in _namespaceFromCallingContext
    return currentframe(2).f_globals[__name__]
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/compat.py, line 93, in currentframe
    for x in range(n + 1):
RuntimeError: maximum recursion depth exceeded while calling a Python object
Logged from file site.py, line 129
  File /usr/lib/python2.7/logging/__init__.py, line 859, in emit
    msg = self.format(record)
  File /usr/lib/python2.7/logging/__init__.py, line 732, in format
    return fmt.format(record)
  File /usr/lib/python2.7/logging/__init__.py, line 471, in format
    record.message = record.getMessage()
  File /usr/lib/python2.7/logging/__init__.py, line 335, in getMessage
    msg = msg % self.args
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 4: ordinal not in range(128)
Logged from file site.py, line 129
```

...where the logger apparently recurses whilst trying to log the error, hitting the
maximum recursion depth and killing everything badly.
2018-07-01 05:08:58 +01:00
Erik Johnston e3b4043800
Merge pull request #3456 from matrix-org/hawkowl/federation-prevevent-checking
Check the state of prev_events a bit more thoroughly when coming over federation
2018-06-29 13:55:02 +01:00
Matthew Hodgson e72234f6bd fix tests 2018-06-28 20:56:07 +01:00
Matthew Hodgson f4f1cda928 add ip_range_whitelist parameter to limit where ASes can connect from 2018-06-28 20:32:00 +01:00
Amber Brown 6350bf925e
Attempt to be more performant on PyPy (#3462) 2018-06-28 14:49:57 +01:00
Amber Brown 72d2143ea8
Revert "Revert "Try to not use as much CPU in the StreamChangeCache"" (#3454) 2018-06-28 11:04:18 +01:00
Amber Brown 99800de63d try and clean up 2018-06-27 11:40:27 +01:00
Amber Brown f03a5d1a17 pep8 2018-06-27 11:38:14 +01:00
Amber Brown a7ecf34b70 cleanups 2018-06-27 11:36:03 +01:00
Amber Brown 77078d6c8e handle federation not telling us about prev_events 2018-06-27 11:27:32 +01:00
Jeroen 95341a8f6f take idna implementation from twisted 2018-06-26 21:15:14 +02:00
Jeroen b7f34ee348 allow self-signed certificates 2018-06-26 20:41:05 +02:00
Matthew Hodgson 8057489b26
Revert "Try to not use as much CPU in the StreamChangeCache" 2018-06-26 18:09:01 +01:00
Matthew Hodgson d91efb06cf
Merge pull request #3451 from matrix-org/hawkowl/sorteddict-api
Try to not use as much CPU in the cache
2018-06-26 17:49:55 +01:00
Amber Brown 1202508067 fixes 2018-06-26 17:29:01 +01:00
Amber Brown bd3d329c88 fixes 2018-06-26 17:28:12 +01:00
Amber Brown abfe4b2957 try and make loading items from the cache faster 2018-06-26 17:25:34 +01:00
David Baker 028490afd4 Fix error on deleting users pending deactivation
Use simple_delete instead of simple_delete_one as commented
2018-06-26 10:52:52 +01:00
Matthew Hodgson c7f6b420ae
Merge pull request #3448 from matrix-org/matthew/gdpr-deactivate-admin-api
add GDPR erase param to deactivate API
2018-06-26 10:43:14 +01:00
Matthew Hodgson 9570aa82eb update doc for deactivate API 2018-06-26 10:42:50 +01:00
Matthew Hodgson 1e788db430 add GDPR erase param to deactivate API 2018-06-26 10:26:54 +01:00
Amber Brown 1d62c4a127
Merge pull request #3438 from turt2live/travis/dont-print-access-tokens-in-logs
Stop including access tokens in warnings in the log
2018-06-26 09:55:55 +01:00
Erik Johnston d72fb9a448
Merge pull request #3442 from matrix-org/matthew/allow-unconsented-parts
allow non-consented users to still part rooms (to let us autopart them)
2018-06-25 20:14:18 +01:00
Erik Johnston df48f7ef37 Actually fix it 2018-06-25 20:03:41 +01:00
Erik Johnston a0e8a53c6d Comment 2018-06-25 19:57:38 +01:00
Erik Johnston 7bdc5c8fa3 Fix bug with assuming wrong type 2018-06-25 19:56:02 +01:00
Erik Johnston ea7a9c0483 Add fast path to _filter_events_for_server
Most rooms have a trivial history visibility like "shared" or
"world_readable", especially large rooms, so lets not bother getting the
full membership of those rooms in that case.
2018-06-25 19:49:13 +01:00
Matthew Hodgson 0269367f18 allow non-consented users to still part rooms (to let us autopart them) 2018-06-25 17:56:10 +01:00
Matthew Hodgson 784189b1f4 typos 2018-06-25 17:37:16 +01:00
Matthew Hodgson 6eb861b67f typo 2018-06-25 17:37:16 +01:00
Erik Johnston 947fea67cb Need to pass reactor to endpoint fac 2018-06-25 15:22:57 +01:00
Erik Johnston 33fdcfa957
Merge pull request #3441 from matrix-org/erikj/redo_erasure
Fix user erasure and re-enable
2018-06-25 14:37:01 +01:00
Erik Johnston eb50c44eaf Add UserErasureWorkerStore to workers 2018-06-25 14:22:24 +01:00
Amber Brown 07cad26d65
Remove all global reactor imports & pass it around explicitly (#3424) 2018-06-25 14:08:28 +01:00
Erik Johnston 244484bf3c Revert "Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility""
This reverts commit 1d009013b3.
2018-06-25 13:42:55 +01:00
Jeroen 07b4f88de9 formatting changes for pep8 2018-06-25 12:31:16 +02:00
Jeroen 3d605853c8 send SNI for federation requests 2018-06-24 22:38:43 +02:00
Travis Ralston ec1e799e17 Don't print invalid access tokens in the logs
Tokens shouldn't be appearing the logs, valid or invalid. 

Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-06-24 12:17:01 -06:00
Richard van der Hoff 1d009013b3 Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility"
This reverts commit ce0d911156, reversing
changes made to b4a5d767a9.
2018-06-22 16:35:10 +01:00
Richard van der Hoff 516f884176
Merge pull request #3435 from matrix-org/rav/fix_event_push_actions_tablescan
Fix event_push_actions tablescan when reinserting events
2018-06-22 15:57:22 +01:00
Mark Haines 9850f66abe Deleting from event_push_actions needs to use an index 2018-06-22 15:54:48 +01:00
Erik Johnston 28ddc6cfbe Also log number of events for serach context 2018-06-22 15:42:11 +01:00
Erik Johnston 4b4cec3989 Add some logging to search queries 2018-06-22 15:42:11 +01:00
Richard van der Hoff 200e11c5bf
Merge pull request #3432 from matrix-org/rav/joined_hosts_cache_non_iterable
Make _get_joined_hosts_cache cache non-iterable
2018-06-22 15:18:51 +01:00
Erik Johnston f8272813a9 Make _get_joined_hosts_cache cache non-iterable 2018-06-22 15:12:26 +01:00
Richard van der Hoff 1d7ad11747
Merge pull request #3430 from matrix-org/rav/configurable_push_action_rotation
Make push actions rotation configurable
2018-06-22 15:07:31 +01:00
Erik Johnston ce0d911156
Merge pull request #3431 from matrix-org/rav/erasure_visibility
Support hiding events from deleted users
2018-06-22 15:06:44 +01:00
Erik Johnston b4a5d767a9
Merge pull request #3428 from matrix-org/erikj/persisted_pdu
Simplify get_persisted_pdu
2018-06-22 14:47:43 +01:00
Erik Johnston f79abda87f
Merge pull request #3427 from matrix-org/erikj/remove_filters
remove dead filter_events_for_clients
2018-06-22 14:45:42 +01:00
Erik Johnston 75dc3ddeab Make push actions rotation configurable 2018-06-22 14:44:37 +01:00
Richard van der Hoff 43e02c409d Disable partial state group caching for wildcard lookups
When _get_state_for_groups is given a wildcard filter, just do a complete
lookup. Hopefully this will give us the best of both worlds by not filling up
the ram if we only need one or two keys, but also making the cache still work
for the federation reader usecase.
2018-06-22 11:52:07 +01:00
Richard van der Hoff 240f192523
Merge pull request #3382 from matrix-org/rav/optimise_state_groups
Optimise state_group_cache update
2018-06-22 11:20:20 +01:00
Richard van der Hoff 70e6501913
Merge pull request #3419 from matrix-org/rav/events_per_request
Log number of events fetched from DB
2018-06-22 11:17:56 +01:00
Richard van der Hoff 0495fe0035 Indirect evt_count updates via method call
so that we can stub it for the sentinel and not have a billion failing UTs
2018-06-22 10:42:28 +01:00
Amber Brown 9a685d60ae
Merge pull request #3418 from matrix-org/rav/fix_metric_desc
Fix description of "python_gc_time" metric
2018-06-22 09:43:26 +01:00
Amber Brown 77ac14b960
Pass around the reactor explicitly (#3385) 2018-06-22 09:37:10 +01:00
Richard van der Hoff cbbfaa4be8 Fix description of "python_gc_time" metric 2018-06-21 10:02:42 +01:00
Amber Brown c2eff937ac
Populate synapse_federation_client_sent_pdu_destinations:count again (#3386) 2018-06-21 09:39:58 +01:00
Amber Brown 99b77aa829
Fix tcp protocol metrics naming (#3410) 2018-06-21 09:39:27 +01:00
Richard van der Hoff b088aafcae Log number of events fetched from DB
When we finish processing a request, log the number of events we fetched from
the database to handle it.

[I'm trying to figure out which requests are responsible for large amounts of
event cache churn. It may turn out to be more helpful to add counts to the
prometheus per-request/block metrics, but that is an extension to this code
anyway.]
2018-06-21 06:15:03 +01:00
Richard van der Hoff aff3d76920
Merge pull request #3416 from matrix-org/rav/restart_indicator
Write a clear restart indicator in logs
2018-06-20 18:14:08 +01:00
Richard van der Hoff 02bfc581f8
Merge pull request #3399 from costacruise/master
Add error code to room creation error
2018-06-20 17:26:25 +01:00
Richard van der Hoff 245d53d32a Write a clear restart indicator in logs
I'm fed up with never being able to find the point a server restarted in the
logs.
2018-06-20 15:33:14 +01:00
Amber Brown f6c4d74f96
Fix inflight requests metric (incorrect name & traceback) (#3413) 2018-06-20 11:18:57 +01:00
Matthew Hodgson ccfdaf68be spell gauge correctly 2018-06-16 07:10:34 +01:00
Richard van der Hoff 9a793f861c Merge branch 'master' into develop 2018-06-14 16:36:01 +01:00
Richard van der Hoff 53969e1960 SECURITY UPDATE: Prevent unauthorised users from setting state events in a room
when there is no `m.room.power_levels` event in force in the room. (PR #3397)
 
 Discussion around the Matrix Spec change proposal for this change can be
 followed at https://github.com/matrix-org/matrix-doc/issues/1304.
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1
 
 iQEcBAABAgAGBQJbIop9AAoJEIofk9V1tejV9lsIAJVH0l5dXROmy1KH/zt16AUA
 CXa6Vv4Vyo6hKad/fZ81OZVRr5ChK/TvbIJVn/SA/muCfdoIFdxhT8eo/pXzO2UW
 zReuLsDhAg+gSvpNus37oWj2FVsAE1HYDZ60lfaapAdZnkFit68d5DQZjO6nZHHA
 YUXcU3GUwj0ZYuUzFzYKMLu6uNNasNkN8h6SS2lF7Bm4JaKDW+mFMfCyJwdIVSEh
 BGhHoVpXdxFysD9s6Mwxqrz3KKg1Jtp7idDkk0x2S2Eh+gxyiDQQokv0oQ3+0+HG
 sgy5Iz2t2CkpS02/j+LOvAZljTmnD0bXu3srGR+25StsoDFP038Am3bfQwtD190=
 =9jsT
 -----END PGP SIGNATURE-----

Merge tag 'v0.31.2'

SECURITY UPDATE: Prevent unauthorised users from setting state events in a room
when there is no `m.room.power_levels` event in force in the room. (PR #3397)

Discussion around the Matrix Spec change proposal for this change can be
followed at https://github.com/matrix-org/matrix-doc/issues/1304.
2018-06-14 16:35:33 +01:00
Richard van der Hoff 7e1c616452 v0.31.2 2018-06-14 16:24:32 +01:00
Richard van der Hoff 61ab08a197
Merge pull request #3397 from matrix-org/rav/adjust_auth_rules
Adjust event auth rules when there is no PL event
2018-06-14 16:09:13 +01:00
Michael Wagner 19cd3120ec Add error code to room creation error
This error code is mentioned in the documentation at https://matrix.org/docs/api/client-server/#!/Room32creation/createRoom
2018-06-14 14:08:40 +02:00
Richard van der Hoff 5c9afd6f80 Make default state_default 50
Make it so that, before there is a power-levels event in the room, you need a
power level of at least 50 to send state.

Partially addresses https://github.com/matrix-org/matrix-doc/issues/1192
2018-06-14 12:38:09 +01:00
Richard van der Hoff 52423607bd Clarify interface for event_auth
stop pretending that it returns a boolean, which just almost gave me a heart
attack.
2018-06-14 12:26:17 +01:00
Amber Brown f116f32ace
add a last seen metric (#3396) 2018-06-14 20:26:59 +10:00
Richard van der Hoff 557b686eac Refactor get_send_level to take a power_levels event
it makes it easier for me to reason about
2018-06-14 11:26:27 +01:00
Amber Brown a61738b316
Remove run_on_reactor (#3395) 2018-06-14 18:27:37 +10:00
Amber Brown 0fde1896cd
Merge pull request #3389 from turt2live/travis/name_metrics
Use the correct flag (enable_metrics) when warning about an incorrect metrics setup
2018-06-13 23:50:10 +10:00
Michael Telatynski 94700e55fa
if inviter_display_name == ""||None then default to inviter MXID
to prevent email invite from "None"
2018-06-13 10:31:01 +01:00
Travis Ralston 45768d1640 Use the RegistryProxy for appservices too
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-06-12 12:55:48 -06:00
Travis Ralston 12285a1a76 The flag is named enable_metrics, not collect_metrics
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-06-12 12:51:31 -06:00
Richard van der Hoff b6faef2ad7 Filter out erased messages
Redact any messges sent by erased users.
2018-06-12 09:53:18 +01:00
Richard van der Hoff f1023ebf4b mark accounts as erased when requested 2018-06-12 09:53:18 +01:00
Richard van der Hoff 3ff8a619f5 UserErasureStore
to store which users have been erased
2018-06-12 09:52:22 +01:00
Richard van der Hoff 9fc5b74b24 simplify get_persisted_pdu
it doesn't make much sense to use get_persisted_pdu on the receive path: just
get the event straight from the store.
2018-06-12 09:51:31 +01:00
Richard van der Hoff bd348f0af6 remove dead filter_events_for_clients
This is only used by filter_events_for_client, so we can simplify the whole
thing by just doing one user at a time, and removing a dead storage function to
boot.
2018-06-12 09:51:31 +01:00
Richard van der Hoff eb32b2ca20 Optimise state_group_cache update
(1) matrix-org-hotfixes has removed the intern calls; let's do the same here.
(2) remove redundant iteritems() so we can used an optimised db update.
2018-06-11 22:56:11 +01:00
David Baker 187a546bff
Merge pull request #3276 from matrix-org/dbkr/unbind
Remove email addresses / phone numbers from ID servers when they're removed from synapse
2018-06-11 16:02:00 +01:00
Matthew Hodgson d6cc369205 fix idiotic typo in state res 2018-06-11 14:43:55 +01:00
Matthew Hodgson c96d882a02 Merge branch 'develop' into matthew/filter_members 2018-06-10 12:26:14 +03:00
Vincent Breitmoser b800834351 add note that the affinity package is required for the cpu_affinity setting 2018-06-09 22:50:29 +02:00
Neil Johnson ed5a0780a4 Merge branch 'master' into develop 2018-06-08 15:47:11 +01:00
Neil Johnson 1032393dfb Changes in synapse v0.31.1 (2018-06-08)
=======================================
 
 v0.31.1 fixes a security bug in the ``get_missing_events`` federation API
 where event visibility rules were not applied correctly.
 
 We are not aware of it being actively exploited but please upgrade asap.
 
 Bug Fixes:
 
 * Fix event filtering in get_missing_events handler (PR #3371)
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEETQ1YthIGLQRddG54CTxDAAxPS/QFAlsalP8ACgkQCTxDAAxP
 S/RADw/+NeDu0LjVpS5Uc4ElHgRBuFSm6l2i4z8rZBBlKSYnuq0Em4WMvLloi/JF
 iAvTOYE7OjmF+gNvmsdH1N7hc1lKdQ2gAlpvaQR/5Qz9NtOVmM3WPZxS7n5jZHvD
 hVSxeO9+GQOwK7rJorqrrsnWHQt0OkLHV6WThFdrgZb1JjWCUDTvw+Hei2uMX2aq
 y2mkMG4TLStHwMvL2qw0h+hFtXywXI796qJR73ZxbEn24YD+kOXeEVkIFi2LT0Pj
 cgkg7WWT32JD43/ioumLupZuhCmpRyxn4fi5gIKpXe5kiLsxOdApzNQSwmoJ+WA8
 7zlrWY+0QDN4pbA5ESLitWSWAT50Ul//uM4nwmM4xEBPHdljXvKyHPsaSCeKLvT9
 RT8pc41TQAqSshlXF8zgIAtStnF3oGel3EBBl1mmM9Un1ULnBFwWDZhlIm+ZZGhJ
 MWoAWNG7j8AQuy0BTUAUr76x7t+/cdSqDuyVl1GO1tbDh0DUWoHZGXCUKrAXnn2T
 SbiFigwOLvEADbvkW7L9Je9CVOi2V5Pg/32X9O8YMEiSz+j5PQEiGefyVh/I/QvV
 Ha/atRpZF2OZ+XUOO5DLZMP/XCXpVgvHuskzfU6LvvVQCXgExuJhRD1PrRGeBaWr
 zjJW+rmY+VeredqX7QkTB3XOGLMLGJfx2FeSMb+j0w3a7/iFj+Q=
 =J80S
 -----END PGP SIGNATURE-----

Merge tag 'v0.31.1'

Changes in synapse v0.31.1 (2018-06-08)
=======================================

v0.31.1 fixes a security bug in the ``get_missing_events`` federation API
where event visibility rules were not applied correctly.

We are not aware of it being actively exploited but please upgrade asap.

Bug Fixes:

* Fix event filtering in get_missing_events handler (PR #3371)
2018-06-08 15:46:18 +01:00
Neil Johnson 0eb4722932 changelog a bump version 2018-06-08 15:21:46 +01:00
Richard van der Hoff c6b1441c52 Fix event filtering in get_missing_events handler 2018-06-08 14:15:31 +01:00
David Baker 0e505b1913
Merge pull request #3372 from matrix-org/rav/better_verification_logging
Try to log more helpful info when a sig verification fails
2018-06-08 13:32:02 +01:00
Richard van der Hoff e82db24a0e Try to log more helpful info when a sig verification fails
Firstly, don't swallow the reason for the failure

Secondly, don't assume all exceptions are verification failures

Thirdly, log a bit of info about the key being used if debug is enabled
2018-06-08 12:13:08 +01:00
Richard van der Hoff 0834b49c6a Fix event filtering in get_missing_events handler 2018-06-08 11:34:46 +01:00
Richard van der Hoff 8503dd0047 Remove event re-signing hacks
These "temporary fixes" have been here three and a half years, and I can't find
any events in the matrix.org database where the calculated signature differs
from what's in the db. It's time for them to go away.
2018-06-07 16:08:29 +01:00
Will Hunt 13d211edc1
Merge pull request #3344 from Half-Shot/hs/as-metrics
Add metrics to track appservice transactions
2018-06-07 11:37:12 +01:00
Richard van der Hoff 1152f495a0
Merge pull request #3363 from matrix-org/rav/fix_purge
Fix event-purge-by-ts admin API
2018-06-07 11:34:46 +01:00
Richard van der Hoff e2acf536d4
Merge pull request #3355 from matrix-org/rav/fix_federation_backfill
Fix federation backfill from sqlite servers
2018-06-07 10:18:01 +01:00
Amber Brown 0160f6601f
Merge pull request #3356 from matrix-org/rav/add_missing_attr_dep
Make sure that attr is installed
2018-06-07 16:33:30 +10:00
Richard van der Hoff f4caf3f83d fix log 2018-06-07 00:26:38 +01:00
Richard van der Hoff 0546715c18 Fix event-purge-by-ts admin API
This got completely broken in 0.30.

Fixes #3300.
2018-06-07 00:15:49 +01:00
Richard van der Hoff 57e3f923d2 Add missing dependency on attr
We've rcently added a dep on `attr`. I don't know why the CI didn't pick this
up, but we should make it explicit anyway.
2018-06-06 17:12:41 +01:00
Richard van der Hoff d3a8c9c55e Fix sql error in _get_state_groups_from_groups
If this was called with a `(type, None)` entry in types (which is supposed to
return all state of type `type`), it would explode with a sql error.
2018-06-06 14:19:01 +01:00
Neil Johnson fef6c2cdcc Merge branch 'master' into develop 2018-06-06 12:28:15 +01:00
Neil Johnson 752b7b32ed Changes in synapse v0.31.0 (2018-06-06)
======================================
 
 Most notable change from v0.30.0 is to switch to python prometheus library to improve system
 stats reporting. WARNING this changes a number of prometheus metrics in a
 backwards-incompatible manner. For more details, see
 `docs/metrics-howto.rst <docs/metrics-howto.rst#removal-of-deprecated-metrics--time-based-counters-becoming-histograms-in-0310>`_.
 
 Bug Fixes:
 
 * Fix metric documentation tables (PR #3341)
 * Fix LaterGuage error handling (694968f)
 * Fix replication metrics (b7e7fd2)
 
 Changes in synapse v0.31.0-rc1 (2018-06-04)
 ==========================================
 
 Features:
 
 * Switch to the Python Prometheus library (PR #3256, #3274)
 * Let users leave the server notice room after joining (PR #3287)
 
 Changes:
 
 * daily user type phone home stats (PR #3264)
 * Use iter* methods for _filter_events_for_server (PR #3267)
 * Docs on consent bits (PR #3268)
 * Remove users from user directory on deactivate (PR #3277)
 * Avoid sending consent notice to guest users (PR #3288)
 * disable CPUMetrics if no /proc/self/stat (PR #3299)
 * Add local and loopback IPv6 addresses to url_preview_ip_range_blacklist (PR #3312) Thanks to @thegcat!
 * Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (PR #3307)
 * Add private IPv6 addresses to example config for url preview blacklist (PR #3317) Thanks to @thegcat!
 * Reduce stuck read-receipts: ignore depth when updating (PR #3318)
 * Put python's logs into Trial when running unit tests (PR #3319)
 
 Changes, python 3 migration:
 
 * Replace some more comparisons with six (PR #3243) Thanks to @NotAFile!
 * replace some iteritems with six (PR #3244) Thanks to @NotAFile!
 * Add batch_iter to utils (PR #3245) Thanks to @NotAFile!
 * use repr, not str (PR #3246) Thanks to @NotAFile!
 * Misc Python3 fixes (PR #3247) Thanks to @NotAFile!
 * Py3 storage/_base.py (PR #3278) Thanks to @NotAFile!
 * more six iteritems (PR #3279) Thanks to @NotAFile!
 * More Misc. py3 fixes (PR #3280) Thanks to @NotAFile!
 * remaining isintance fixes (PR #3281) Thanks to @NotAFile!
 * py3-ize state.py (PR #3283) Thanks to @NotAFile!
 * extend tox testing for py3 to avoid regressions (PR #3302) Thanks to @krombel!
 * use memoryview in py3 (PR #3303) Thanks to @NotAFile!
 
 Bugs:
 
 * Fix federation backfill bugs (PR #3261)
 * federation: fix LaterGauge usage (PR #3328) Thanks to @intelfx!
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEETQ1YthIGLQRddG54CTxDAAxPS/QFAlsXxJAACgkQCTxDAAxP
 S/SJTg//Wtr+Qop9LJh2/leAYXpyqW6P7Ftak0w3aJ3KL3+tYg32yYNoRADCqbp3
 LkrHu8MwbZagHjRUyEWNfDk4jbfq5fwh0JVGmYuUKhG9aF0HYyytKkbW79YzuhdQ
 dfHj9x0xSBOUvgt/husloZSDy0VHC6uyQSAFgFDyHS2y7RPAiGstqLGByv0ciZOk
 pO7TdjkUQcx4Ps7Wgip31NuHy3GY2int6f540pUXoZHLXs7RkfqS2cpF9Z/sTXJ4
 xDLiY7uYNsTcCblwqaiijY5c90xwRB2vLs5CJdKFgyB6PNgg/2wHJqP/WHHEj+F8
 BoSm3Ts7NXQf23pP9CXICe7vXX3J+ruOnC7FOSRobr6KGjn6DUrIZxo1ZepTwpp9
 DIq+1eOFKKjwLQM3Jdi8WBCP63LhYXrTZxreke3jpwdcD7oIO9v6/e9J1gU5xHWa
 Izg+YnWn1JLfq/X8T7YTZddUXGGPkH5i6LZKKkyY8u7LkJ4WR7syuAceUzkOOIAq
 UWO0uEV7IiLnZzZGTtNIlEqtuklmVQTm6bvAgTPabai2JQyngFpH5M/5mPpVSiLV
 QRLwaM56c+5GGZJWef8vxdGeYn+8rFI/UUniJ7358kLJF2IHsxlZu8J0ZZO2HWI2
 ze5Kz0AWRzXLhWzq62Qb2dsiGySrZ7hng1tDxLak2IiusY+9SjM=
 =Mz9U
 -----END PGP SIGNATURE-----

Merge tag 'v0.31.0'

Changes in synapse v0.31.0 (2018-06-06)
======================================

Most notable change from v0.30.0 is to switch to python prometheus library to improve system
stats reporting. WARNING this changes a number of prometheus metrics in a
backwards-incompatible manner. For more details, see
`docs/metrics-howto.rst <docs/metrics-howto.rst#removal-of-deprecated-metrics--time-based-counters-becoming-histograms-in-0310>`_.

Bug Fixes:

* Fix metric documentation tables (PR #3341)
* Fix LaterGuage error handling (694968f)
* Fix replication metrics (b7e7fd2)

Changes in synapse v0.31.0-rc1 (2018-06-04)
==========================================

Features:

* Switch to the Python Prometheus library (PR #3256, #3274)
* Let users leave the server notice room after joining (PR #3287)

Changes:

* daily user type phone home stats (PR #3264)
* Use iter* methods for _filter_events_for_server (PR #3267)
* Docs on consent bits (PR #3268)
* Remove users from user directory on deactivate (PR #3277)
* Avoid sending consent notice to guest users (PR #3288)
* disable CPUMetrics if no /proc/self/stat (PR #3299)
* Add local and loopback IPv6 addresses to url_preview_ip_range_blacklist (PR #3312) Thanks to @thegcat!
* Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (PR #3307)
* Add private IPv6 addresses to example config for url preview blacklist (PR #3317) Thanks to @thegcat!
* Reduce stuck read-receipts: ignore depth when updating (PR #3318)
* Put python's logs into Trial when running unit tests (PR #3319)

Changes, python 3 migration:

* Replace some more comparisons with six (PR #3243) Thanks to @NotAFile!
* replace some iteritems with six (PR #3244) Thanks to @NotAFile!
* Add batch_iter to utils (PR #3245) Thanks to @NotAFile!
* use repr, not str (PR #3246) Thanks to @NotAFile!
* Misc Python3 fixes (PR #3247) Thanks to @NotAFile!
* Py3 storage/_base.py (PR #3278) Thanks to @NotAFile!
* more six iteritems (PR #3279) Thanks to @NotAFile!
* More Misc. py3 fixes (PR #3280) Thanks to @NotAFile!
* remaining isintance fixes (PR #3281) Thanks to @NotAFile!
* py3-ize state.py (PR #3283) Thanks to @NotAFile!
* extend tox testing for py3 to avoid regressions (PR #3302) Thanks to @krombel!
* use memoryview in py3 (PR #3303) Thanks to @NotAFile!

Bugs:

* Fix federation backfill bugs (PR #3261)
* federation: fix LaterGauge usage (PR #3328) Thanks to @intelfx!
2018-06-06 12:27:33 +01:00
Neil Johnson 61134debdc bump version and changelog 2018-06-06 11:26:21 +01:00
Richard van der Hoff ad459a106c
Merge pull request #3349 from t3chguy/redact_as_request_token
Redact AS tokens in log (fixes to #3327)
2018-06-06 10:58:07 +01:00
Michael Telatynski 592c162516
also redact __str__ of ApplicationService used for logging 2018-06-06 10:35:29 +01:00
Michael Telatynski 330432031b
redact_uri in two missed log paths 2018-06-06 10:25:48 +01:00
David Baker bf54c1cf6c pep8 2018-06-06 10:15:33 +01:00
David Baker 3e4bc4488c More doc fixes 2018-06-06 09:44:10 +01:00
Richard van der Hoff 592ee217a3 Merge commit 'b7e7fd2' into release-v0.31.0 2018-06-06 07:02:02 +01:00
Ivan Shapovalov c88d50aa8f federation/send_queue.py: fix usage of sortedcontainers.SortedDict 2018-06-06 02:45:18 +03:00
Richard van der Hoff e316407b5d
Merge pull request #3327 from t3chguy/redact_as_request_token
Strip `access_token` from outgoing requests
2018-06-05 19:08:46 +01:00
Michael Telatynski e6cbf47773
factor out uri redaction into a method on http 2018-06-05 18:31:40 +01:00
David Baker 607bd27c83 fix pep8 2018-06-05 18:10:35 +01:00
David Baker d62162bbec doc fixes 2018-06-05 18:09:13 +01:00
Richard van der Hoff 522bd3c8a3 Merge remote-tracking branch 'origin/master' into develop 2018-06-05 17:42:49 +01:00
Will Hunt d6e3c2c79b Let's try labels instead of label, that might work 2018-06-05 17:30:45 +01:00
Amber Brown f7869f8f8b
Port to sortedcontainers (with tests!) (#3332) 2018-06-06 00:13:57 +10:00
Will Hunt 604cff1a06 Add metrics to track appservice transactions 2018-06-05 13:21:30 +01:00
Richard van der Hoff f29b41fde9
Merge pull request #3324 from matrix-org/rav/remove_dead_method
Remove was_forgotten_at
2018-06-04 18:18:08 +01:00
Richard van der Hoff 28b0490dfd
Merge pull request #3334 from matrix-org/rav/cache_factor_override
Cache factor override system for specific caches
2018-06-04 17:19:33 +01:00
Richard van der Hoff b7e7fd2d0e Fix replication metrics
fix bug introduced in #3256
2018-06-04 16:23:05 +01:00
Neil Johnson 244ab974e7 bump version and changelog 2018-06-04 16:09:58 +01:00
Richard van der Hoff 694968fa81 Hopefully, fix LaterGuage error handling 2018-06-04 15:59:14 +01:00
Erik Johnston 042eedfa2b Add hacky cache factor override system 2018-06-04 15:39:28 +01:00
David Baker c5930d513a Docstring 2018-06-04 12:05:58 +01:00
David Baker 6a29e815fc Fix comment 2018-06-04 12:01:23 +01:00
David Baker e44150a6de Missing yield 2018-06-04 12:01:13 +01:00
David Baker f731e42baf docstring 2018-06-04 12:00:51 +01:00
Matthew Hodgson 28f09fcdd5 Merge branch 'develop' into matthew/filter_members 2018-06-04 00:09:17 +03:00
Matthew Hodgson 5f6122fe10 more comments 2018-06-04 00:08:52 +03:00
Ivan Shapovalov 7d9d75e4e8 federation/send_queue.py: fix usage of LaterGauge
Fixes a startup crash due to commit df9f72d9e5
"replacing portions".
2018-06-03 14:16:17 +03:00
Michael Telatynski 09503126df
Strip `access_token` from outgoing requests using existing regex 2018-06-02 23:25:13 +01:00
Richard van der Hoff c1f4118bb6 Remove was_forgotten_at
This is unused. IT MUST DIE!!!1
̧̪͈̱̹̳͖͙H̵̰̤̰͕̖e̛ ͚͉̗̼̞w̶̩̥͉̮h̩̺̪̩͘ͅọ͎͉̟ ̜̩͔̦̘ͅW̪̫̩̣̲͔̳a͏͔̳͖i͖͜t͓̤̠͓͙s̘̰̩̥̙̝ͅ ̲̠̬̥Be̡̙̫̦h̰̩i̛̫͙͔̭̤̗̲n̳͞d̸ ͎̻͘T̛͇̝̲̹̠̗ͅh̫̦̝ͅe̩̫͟ ͓͖̼W͕̳͎͚̙̥ą̙l̘͚̺͔͞ͅl̳͍̙̤̤̮̳.̢
̟̺̜̙͉Z̤̲̙̙͎̥̝A͎̣͔̙͘L̥̻̗̳̻̳̳͢G͉̖̯͓̞̩̦O̹̹̺!̙͈͎̞̬ *
2018-06-01 18:21:49 +01:00
Richard van der Hoff a9e97dcd65
Merge pull request #3317 from thegcat/feature/3312-add_ipv6_to_blacklist_example_config
Add private IPv6 addresses to example config for url preview blacklist
2018-06-01 14:45:14 +01:00
Neil Johnson 71477f3317
Merge pull request #3264 from matrix-org/neil/sign-up-stats
daily user type phone home stats
2018-06-01 13:42:01 +00:00
Richard van der Hoff 9f797a24a4 Handle RRs which arrive before their events 2018-06-01 14:01:43 +01:00
Richard van der Hoff 857e6fd8b6 Ignore depth when updating read-receipts
Order read receipts by stream ordering instead of depth
2018-06-01 12:18:11 +01:00
Felix Schäfer 4ef76f3ac4 Add private IPv6 addresses to preview blacklist #3312
The added addresses are expected to be local or loopback addresses and
shouldn't be spidered for previews.

Signed-off-by: Felix Schäfer <felix@thegcat.net>
2018-06-01 12:18:35 +02:00
Neil Johnson 4986b084f8 remove unnecessary INSERT 2018-06-01 10:50:40 +01:00
Richard van der Hoff 7e15410f02 Enforce the specified API for report_event
as per
https://matrix.org/docs/spec/client_server/unstable.html#post-matrix-client-r0-rooms-roomid-report-eventid
2018-05-31 18:17:11 +01:00
Amber Brown febe0ec8fd
Run Prometheus on a different port, optionally. (#3274) 2018-05-31 19:04:50 +10:00
Amber Brown c936a52a9e
Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307) 2018-05-31 19:03:47 +10:00
Richard van der Hoff 219c2a322b
remove trailing whitespace 2018-05-30 19:42:19 +01:00
Richard van der Hoff 2e4be8bfd9
fix english and wrap comment 2018-05-30 19:24:12 +01:00
Amber Brown 872cf43516
Merge pull request #3303 from NotAFile/py3-memoryview
use memoryview in py3
2018-05-30 12:51:42 +10:00
Amber Brown debff7ae09
Merge pull request #3281 from NotAFile/py3-six-isinstance
remaining isintance fixes
2018-05-30 12:44:46 +10:00
Richard van der Hoff 34b85df7f5 Update some comments and docstrings in SyncHandler 2018-05-29 22:31:18 +01:00
Richard van der Hoff 4a9cbdbc15 Exempt AS-registered users from doing gdpr 2018-05-29 19:54:32 +01:00
Neil Johnson ab0ef31dc7 create users index on creation_ts 2018-05-29 17:51:08 +01:00
Neil Johnson 558f3d376a create index in background 2018-05-29 17:47:55 +01:00
Neil Johnson c379acd4fd bump version 2018-05-29 17:47:28 +01:00
Adrian Tschira 4b9d0cde97 add remaining memoryview changes 2018-05-29 17:42:43 +02:00
Adrian Tschira 7873cde526 pep8 2018-05-29 17:35:55 +02:00
Adrian Tschira 1afafb3497 use memoryview in py3
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-29 17:14:34 +02:00
Matthew Hodgson 9bbb9f5556 add lazy_load_members to the filter json schema 2018-05-29 04:26:10 +01:00
Matthew Hodgson 8df7bad839 pep8 2018-05-29 02:49:36 +01:00
Matthew Hodgson ff1bc0a279 pep8 2018-05-29 02:32:15 +01:00
Matthew Hodgson 0a240ad36e disable CPUMetrics if no /proc/self/stat
fixes build on macOS again
2018-05-29 02:23:30 +01:00
Matthew Hodgson b69ff33d9e disable CPUMetrics if no /proc/self/stat
fixes build on macOS again
2018-05-29 02:22:27 +01:00
Matthew Hodgson a6c8f7c875 add pydoc 2018-05-29 01:09:55 +01:00
Matthew Hodgson 7a6df013cc merge develop 2018-05-29 00:25:22 +01:00
Amber Brown 57ad76fa4a fix up tests 2018-05-28 19:51:53 +10:00
Amber Brown 3ef5cd74a6 update to more consistently use seconds in any metrics or logging 2018-05-28 19:39:27 +10:00
Amber Brown 5c40ce3777 invalid syntax :( 2018-05-28 19:16:09 +10:00
Amber Brown 357c74a50f add comment about why unreg 2018-05-28 19:14:41 +10:00
Amber Brown a2eb5db4a0 update metrics to be in seconds 2018-05-28 19:10:27 +10:00
Amber Brown 754826a830 Merge remote-tracking branch 'origin/develop' into 3218-official-prom 2018-05-28 18:57:23 +10:00
Ruben Barkow 08ea5fe635
add link to thorough instruction how to configure consent 2018-05-25 23:19:55 +02:00
Richard van der Hoff 66bdae986f Fix default for send_server_notice_to_guests
bool("False") == True...
2018-05-25 11:42:05 +01:00
Richard van der Hoff ba1b163590 Avoid sending consent notice to guest users
we think it makes sense not to send the notices to guest users.
2018-05-25 11:36:43 +01:00
Richard van der Hoff 41921ac01b
Merge pull request #3287 from matrix-org/rav/allow_leaving_server_notices_room
Let users leave the server notice room after joining
2018-05-25 11:15:45 +01:00
Richard van der Hoff 757ed27258 Let users leave the server notice room after joining
They still can't reject invites, but we let them leave it.
2018-05-25 11:07:21 +01:00
Adrian Tschira 4ee4450d66 fix recursion error 2018-05-24 21:44:10 +02:00
Amber Brown 9c36c150e7
Merge pull request #3283 from NotAFile/py3-state
py3-ize state.py
2018-05-24 14:23:33 -05:00
Amber Brown cc1349c06a
Merge pull request #3279 from NotAFile/py3-more-iteritems
more six iteritems
2018-05-24 14:23:13 -05:00
Adrian Tschira 0e61705661 py3-ize state.py 2018-05-24 20:59:00 +02:00
Adrian Tschira dd068ca979 remaining isintance fixes
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-24 20:55:08 +02:00
Adrian Tschira 17a70cf6e9 Misc. py3 fixes
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-24 20:20:33 +02:00
Adrian Tschira 6c16a4ec1b more iteritems 2018-05-24 20:19:06 +02:00
Amber Brown 7ea07c7305
Merge pull request #3278 from NotAFile/py3-storage-base
Py3 storage/_base.py
2018-05-24 13:08:09 -05:00
Amber Brown 1f69693347
Merge pull request #3244 from NotAFile/py3-six-4
replace some iteritems with six
2018-05-24 13:04:07 -05:00
Amber Brown c4fb15a06c
Merge pull request #3246 from NotAFile/py3-repr-string
use repr, not str
2018-05-24 13:00:20 -05:00
Amber Brown 36501068d8
Merge pull request #3247 from NotAFile/py3-misc
Misc Python3 fixes
2018-05-24 12:58:37 -05:00
Amber Brown 2aff6eab6d
Merge pull request #3245 from NotAFile/batch-iter
Add batch_iter to utils
2018-05-24 12:54:12 -05:00
Adrian Tschira 095292304f Py3 storage/_base.py
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-24 18:24:12 +02:00
David Baker 77a23e2e05 Merge remote-tracking branch 'origin/develop' into dbkr/unbind 2018-05-24 16:20:53 +01:00
David Baker ecc4b88bd1
Merge pull request #3277 from matrix-org/dbkr/remove_from_user_dir
Remove users from user directory on deactivate
2018-05-24 16:12:12 +01:00
Erik Johnston 46345187cc
Merge pull request #3243 from NotAFile/py3-six-3
Replace some more comparisons with six
2018-05-24 16:08:57 +01:00
Neil Johnson 037c6db85d Merge branch 'master' into develop 2018-05-24 16:03:44 +01:00
David Baker 7a1af504d7 Remove users from user directory on deactivate 2018-05-24 15:59:58 +01:00
Erik Johnston f72d5a44d5
Merge pull request #3261 from matrix-org/erikj/pagination_fixes
Fix federation backfill bugs
2018-05-24 14:52:03 +01:00
Erik Johnston 68399fc4de
Merge pull request #3267 from matrix-org/erikj/iter_filter
Use iter* methods for _filter_events_for_server
2018-05-24 14:44:57 +01:00
Neil Johnson 91d95a1d8e bump version 2018-05-24 14:05:07 +01:00
David Baker 9700d15611 pep8 2018-05-24 11:23:15 +01:00
David Baker a21a41bad7 comment 2018-05-24 11:19:59 +01:00
David Baker b3bff53178 Unbind 3pids when they're deleted too 2018-05-24 11:08:05 +01:00
Richard van der Hoff 6abcb5d22d
Merge pull request #3273 from matrix-org/rav/server_notices_avatar_url
Allow overriding the server_notices user's avatar
2018-05-24 10:31:43 +01:00
Amber Brown 389dac2c15 pepeightttt 2018-05-23 13:08:59 -05:00
Amber Brown 472a5ec4e2 add back CPU metrics 2018-05-23 13:03:56 -05:00
Amber Brown e987079037 fixes 2018-05-23 13:03:51 -05:00
Richard van der Hoff 9bf4b2bda3 Allow overriding the server_notices user's avatar
probably should have done this in the first place, like @turt2live suggested.
2018-05-23 17:43:30 +01:00
Richard van der Hoff a0b3946fe2
Merge branch 'release-v0.30.0' into rav/localpart_in_consent_uri 2018-05-23 16:06:03 +01:00
Richard van der Hoff 2f7008d4eb
Merge pull request #3271 from matrix-org/rav/consent_uri_in_messages
Support for putting %(consent_uri)s in messages
2018-05-23 16:04:30 +01:00
Richard van der Hoff dda40fb55d fix typo 2018-05-23 15:30:26 +01:00
Richard van der Hoff 3ff6f50eac Use the localpart in the consent uri
... because it's shorter.
2018-05-23 15:28:23 +01:00
Richard van der Hoff 82191b08f6 Support for putting %(consent_uri)s in messages
Make it possible to put the URI in the error message and the server notice that
get sent by the server
2018-05-23 15:24:31 +01:00
David Baker 2c7866d664 Hit the 3pid unbind endpoint on deactivation 2018-05-23 14:38:56 +01:00
Richard van der Hoff 321f02d263 Block attempts to send server notices to remote users 2018-05-23 14:30:47 +01:00
Neil Johnson 3601a240aa bump version and changelog 2018-05-23 12:03:23 +01:00
Erik Johnston 5aaa3189d5 s/values/itervalues/ 2018-05-23 10:13:05 +01:00
Erik Johnston 0a4bca4134 Use iter* methods for _filter_events_for_server 2018-05-23 10:04:23 +01:00
Amber Brown b6063631c3 more cleanup 2018-05-22 17:36:20 -05:00
Amber Brown 53cc2cde1f cleanup 2018-05-22 17:32:57 -05:00
Amber Brown 071206304d cleanup pep8 errors 2018-05-22 16:54:22 -05:00
Amber Brown 85ba83eb51 fixes 2018-05-22 16:28:23 -05:00
Amber Brown 228f1f584e fix the test failures 2018-05-22 15:02:38 -05:00
Erik Johnston e85b5a0ff7 Use iter* methods 2018-05-22 19:02:48 +01:00
Erik Johnston 586b66b197 Fix that states is a dict of dicts 2018-05-22 19:02:36 +01:00
Erik Johnston a17e901f4d Remove unused string formatting param 2018-05-22 18:24:32 +01:00
Erik Johnston 5494c1d71e Don't support limitless pagination
The pagination storage function supported not specifiying a limit on the
number of events returned. This was triggered when using the search or
context API with a limit of zero, which the storage function took to
mean not being limited.
2018-05-22 18:15:21 +01:00
Neil Johnson d8cb7225d2 daily user type phone home stats 2018-05-22 18:09:09 +01:00
hera ad2823ee27 fix synchrotron 2018-05-22 17:47:42 +01:00
Richard van der Hoff 08bfc48abf custom error code for not leaving server notices room 2018-05-22 17:27:27 +01:00
David Baker 0a078026ea comment typo 2018-05-22 17:14:06 +01:00
Amber Brown 8f5a688d42 cleanups, self-registration 2018-05-22 10:56:03 -05:00
Amber Brown a8990fa2ec Merge remote-tracking branch 'origin/develop' into 3218-official-prom 2018-05-22 10:50:26 -05:00
Erik Johnston cb2a2ad791 get_domains_from_state returns list of tuples 2018-05-22 16:23:39 +01:00
Richard van der Hoff 08a14b32ae
Merge pull request #3262 from matrix-org/rav/has_already_consented
Add a 'has_consented' template var to consent forms
2018-05-22 15:35:10 +01:00
Richard van der Hoff 7b36d06a69 Add a 'has_consented' template var to consent forms
fixes #3260
2018-05-22 14:58:34 +01:00
Richard van der Hoff 669400e22f Enable auto-escaping for the consent templates
... to reduce the risk of somebody introducing an html injection attack...
2018-05-22 14:58:34 +01:00
Richard van der Hoff b5b2d5d64b Fix dependency on jinja2
Delay the import of ConsentResource, so that we can get away without jinja2 if
people don't have the consent resource enabled.

Fixes #3259
2018-05-22 14:03:45 +01:00
Richard van der Hoff a5e2941aad Reject attempts to send event before privacy consent is given
Returns an M_CONSENT_NOT_GIVEN error (cf
https://github.com/matrix-org/matrix-doc/issues/1252) if consent is not yet
given.
2018-05-22 12:00:47 +01:00
Richard van der Hoff 8aeb529262
Merge pull request #3236 from matrix-org/rav/consent_notice
Send users a server notice about consent
2018-05-22 11:56:09 +01:00
Richard van der Hoff 8810685df9 Stub out ServerNoticesSender on the workers
... and have the sync endpoints call it directly rather than obsure indirection
via PresenceHandler
2018-05-22 11:54:51 +01:00
Richard van der Hoff d5dca9a04f Move consent config parsing into ConsentConfig
turns out we need to reuse this, so it's better in the config class.
2018-05-22 11:54:51 +01:00
Richard van der Hoff 9ea219c514 Send users a server notice about consent
When a user first syncs, we will send them a server notice asking them to
consent to the privacy policy if they have not already done so.
2018-05-22 11:54:51 +01:00
Richard van der Hoff d14d7b8fdc Rename 'version' param on user consent config
we're going to use it for the version we require too.
2018-05-22 11:54:51 +01:00
Erik Johnston 7948ecf234 Comment 2018-05-22 11:39:43 +01:00
Erik Johnston 020377a550 Fix logcontext resource usage tracking 2018-05-22 11:16:07 +01:00
Erik Johnston 13a8dfba0d
Merge pull request #3252 from matrix-org/erikj/in_flight_requests
Add in flight request metrics
2018-05-22 09:54:30 +01:00
Erik Johnston c435b0b441 Don't store context 2018-05-22 09:34:26 +01:00
Erik Johnston fb2806b186 Move in_flight_requests_count to be a callback metric 2018-05-22 09:31:53 +01:00
Amber Brown fcc525b0b7 rest of the changes 2018-05-21 19:48:57 -05:00
Amber Brown df9f72d9e5 replacing portions 2018-05-21 19:47:37 -05:00
Amber Brown c60e0d5e02 don't need the resource portion 2018-05-21 17:03:20 -05:00
Amber Brown 02c1d29133 look at the Prometheus metrics instead 2018-05-21 17:02:20 -05:00
Amber Brown f258deffcb remove old metrics libs 2018-05-21 17:01:15 -05:00
Richard van der Hoff 413482f578
Merge pull request #3255 from matrix-org/rav/fix_transactions
Stop the transaction cache caching failures
2018-05-21 17:58:08 +01:00
Richard van der Hoff 6e1cb54a05 Fix logcontext leak in HttpTransactionCache
ONE DAY I WILL PURGE THE WORLD OF THIS EVIL
2018-05-21 16:58:20 +01:00
Richard van der Hoff 6d6e7288fe Stop the transaction cache caching failures
The transaction cache has some code which tries to stop it caching failures,
but if the callback function failed straight away, then things would happen
backwards and we'd end up with the failure stuck in the cache.
2018-05-21 16:49:59 +01:00
Erik Johnston dfa70adc33 Add in flight request metrics
This tracks CPU and DB usage while requests are in flight, rather than
when we write the response.
2018-05-21 16:23:06 +01:00
Adrian Tschira 933bf2dd35 replace some iteritems with six
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:59:26 +02:00
Adrian Tschira d9fe2b2d9d Replace some more comparisons with six
plus a bonus b"" string I missed last time

Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:56:31 +02:00
Adrian Tschira 45b55e23d3 Add batch_iter to utils
There's a frequent idiom I noticed where an iterable is split up into a
number of chunks/batches. Unfortunately that method does not work with
iterators like dict.keys() in python3. This implementation works with
iterators.

Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:48:30 +02:00
Adrian Tschira dcc235b47d use stand-in value if maxint is not available
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:35:44 +02:00
Adrian Tschira 73cbdef5f7 fix py3 intern and remove unnecessary py3 encode
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:35:31 +02:00
Adrian Tschira aafb0f6b0d py3-ize url preview 2018-05-19 17:35:20 +02:00
Adrian Tschira b932b4ea25 use repr, not str
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:28:42 +02:00
Neil Johnson 644aac5f73 Tighter filtering for user_daily_visits 2018-05-18 17:10:35 +01:00
Neil Johnson ef466b3a13 fix psql compatability bug 2018-05-18 15:51:21 +01:00
Neil Johnson 2725223f08 Merge branch 'master' into develop 2018-05-18 14:07:50 +01:00
Richard van der Hoff 6d9dc67139
Merge pull request #3232 from matrix-org/rav/server_notices_room
Infrastructure for a server notices room
2018-05-18 11:28:04 +01:00
Richard van der Hoff ed3125b0a1
Merge pull request #3235 from matrix-org/rav/fix_receipts_deferred
Fix error in handling receipts
2018-05-18 11:23:11 +01:00
Richard van der Hoff 67af392712
Merge pull request #3233 from matrix-org/rav/remove_dead_code
Remove unused `update_external_syncs`
2018-05-18 11:22:43 +01:00
Richard van der Hoff 011e1f4010 Better docstrings 2018-05-18 11:22:12 +01:00
Richard van der Hoff 26305788fe Make sure we reject attempts to invite the notices user 2018-05-18 11:18:39 +01:00
Richard van der Hoff d10707c810 Replace inline docstrings with "Attributes" in class docstring 2018-05-18 11:00:55 +01:00
Erik Johnston fa30ac38cc
Merge pull request #3221 from matrix-org/erikj/purge_token
Make purge_history operate on tokens
2018-05-18 10:35:23 +01:00
Richard van der Hoff 8b1c856d81 Fix error in handling receipts
Fixes an error which has been happening ever since #2158 (v0.21.0-rc1):

> TypeError: argument of type 'ObservableDeferred' is not iterable

fixes #3234
2018-05-18 09:15:35 +01:00
Neil Johnson 6958459b50 bump version, change log 2018-05-17 21:35:07 +01:00
Richard van der Hoff 88d3405332 fix missing yield for server_notices_room 2018-05-17 18:33:45 +01:00
Richard van der Hoff d43d480d86 Remove unused `update_external_syncs`
This method isn't used anywhere. Burninate it.
2018-05-17 18:22:19 +01:00
Richard van der Hoff fed62e21ad Infrastructure for a server notices room
Server Notices use a special room which the user can't dismiss. They are
created on demand when some other bit of the code calls send_notice.

(This doesn't actually do much yet becuse we don't call send_notice anywhere)
2018-05-17 17:58:25 +01:00
Richard van der Hoff f8a1e76d64
Merge pull request #3225 from matrix-org/rav/move_creation_handler
Move RoomCreationHandler out of synapse.handlers.Handlers
2018-05-17 17:56:42 +01:00
Erik Johnston f7906203f6
Merge pull request #3212 from matrix-org/erikj/epa_stream
Use stream rather depth ordering for push actions
2018-05-17 12:01:21 +01:00
Richard van der Hoff c46367d0d7 Move RoomCreationHandler out of synapse.handlers.Handlers
Handlers is deprecated nowadays, so let's move this out before I add a new
dependency on it.

Also fix the docstrings on create_room.
2018-05-17 09:08:42 +01:00
Neil Johnson 85b8acdeb4 Merge branch 'master' into develop 2018-05-16 16:48:10 +01:00
Neil Johnson 3c099219e0 bump version and changelog for 0.29.0 2018-05-16 15:44:59 +01:00
Erik Johnston 680530cc7f Clarify comment 2018-05-16 11:47:29 +01:00
Erik Johnston 43e6e82c4d Comments 2018-05-16 11:13:31 +01:00
Neil Johnson dc8930ea9e
Merge pull request #3163 from matrix-org/cohort_analytics
user visit data
2018-05-16 10:09:24 +00:00
Erik Johnston c945af8799 Move and rename variable 2018-05-16 10:52:06 +01:00
Neil Johnson be11a02c4f remove empty line 2018-05-16 10:45:40 +01:00
Neil Johnson a2204cc9cc remove unused method recurring_user_daily_visit_stats 2018-05-16 09:47:20 +01:00
Neil Johnson 31c2502ca8 style and further contraining query 2018-05-16 09:46:43 +01:00
Richard van der Hoff 8030a825c8
Merge pull request #3213 from matrix-org/rav/consent_handler
ConsentResource to gather policy consent from users
2018-05-16 07:19:18 +01:00
Neil Johnson c92a8aa578 pep8 2018-05-15 17:31:11 +01:00
Neil Johnson 05ac15ae82 Limit query load of generate_user_daily_visits
The aim is to keep track of when it was last called and only query from that point in time
2018-05-15 17:01:33 +01:00
Erik Johnston 5f27ed75ad Make purge_history operate on tokens
As we're soon going to change how topological_ordering works
2018-05-15 16:23:50 +01:00
Erik Johnston 37dbee6490 Use events_to_purge table rather than token 2018-05-15 16:23:47 +01:00
Richard van der Hoff 47815edcfa ConsentResource to gather policy consent from users
Hopefully there are enough comments and docs in this that it makes sense on its
own.
2018-05-15 15:11:59 +01:00
Neil Johnson aea80a0118 v0.29.0-rc1: bump version and change log 2018-05-14 15:50:57 +01:00
Neil Johnson f077e97914 instead of inserting user daily visit data at the end of the day, instead insert incrementally through the day 2018-05-14 13:50:58 +01:00
David Baker 8cbbfd16fb
Merge pull request #3201 from matrix-org/dbkr/leave_rooms_on_deactivate
Part user from rooms on account deactivate
2018-05-14 11:31:48 +01:00
Neil Johnson 977765bde2 Merge branch 'develop' of https://github.com/matrix-org/synapse into cohort_analytics 2018-05-14 09:31:42 +01:00
Richard van der Hoff c25d7ba12e
Merge pull request #3208 from matrix-org/rav/more_refactor_request_handler
Set Server header in SynapseRequest
2018-05-11 16:07:47 +01:00
Erik Johnston 6406b70aeb Use stream rather depth ordering for push actions
This simplifies things as it is, but will also allow us to change the
way we traverse topologically without having to update the way push
actions work.
2018-05-11 15:30:11 +01:00
Richard van der Hoff 23e2dfe940
Merge pull request #3209 from damir-manapov/master
transaction_id, destination defined twice
2018-05-11 00:35:13 +01:00
Damir Manapov db18d854cd transaction_id, destination twice 2018-05-10 22:13:31 +03:00
Richard van der Hoff 318711e139 Set Server header in SynapseRequest
(instead of everywhere that writes a response. Or rather, the subset of places
which write responses where we haven't forgotten it).

This also means that we don't have to have the mysterious version_string
attribute in anything with a request handler.

Unfortunately it does mean that we have to pass the version string wherever we
instantiate a SynapseSite, which has been c&ped 150 times, but that is code
that ought to be cleaned up anyway really.
2018-05-10 18:50:27 +01:00
David Baker 6b49628e3b Catch failure to part user from room 2018-05-10 12:23:53 +01:00
David Baker 217bc53c98 Many docstrings 2018-05-10 12:20:40 +01:00
Richard van der Hoff 645cb4bf06 Remove redundant request_handler decorator
This is needless complexity; we might as well use the wrapper directly.

Also rename wrap_request_handler->wrap_json_request_handler.
2018-05-10 12:19:53 +01:00
Richard van der Hoff 09f570b935 Factor wrap_request_handler_with_logging out of wrap_request_handler
... so that it can be used on non-JSON endpoints
2018-05-10 12:19:52 +01:00
Richard van der Hoff 9589a1925e Remove include_metrics param
The metrics are now available via the request, so this is redundant and can go
away at last.
2018-05-10 12:19:52 +01:00
Richard van der Hoff 49e5a613f1 Move outgoing_responses_counter handling to RequestMetrics
it's much neater there.
2018-05-10 12:19:52 +01:00
Richard van der Hoff b8700dd7d0 Bump requests_counter in wrapped_request_handler
less magic
2018-05-10 12:19:52 +01:00
Richard van der Hoff c6f730282c Move RequestMetrics handling into SynapseRequest.processing()
It fits quite nicely here, and opens the path to getting rid of the
"include_metrics" mess.
2018-05-10 12:19:51 +01:00
Richard van der Hoff 09b29f9c4a Make RequestMetrics take a raw time rather than a clock
... which is going to make it easier to move around.
2018-05-10 12:18:52 +01:00
David Baker 4d298506dd Oops, don't call function passed to run_in_background 2018-05-10 11:57:13 +01:00
Richard van der Hoff 8460e48d06 Move request_id management into SynapseRequest 2018-05-10 11:48:17 +01:00
Richard van der Hoff 18e144fe08 Move RequestsMetrics to its own file
This is useful in its own right, because server.py is full of stuff; but more
importantly, I want to do some refactoring that will cause a circular reference
as it is.
2018-05-09 19:55:03 +01:00
Erik Johnston bfe1f73855
Merge pull request #3199 from matrix-org/erikj/pagination_sync
Refactor sync APIs to reuse pagination API
2018-05-09 16:16:56 +01:00
Erik Johnston 5adb75bcba
Merge pull request #3198 from matrix-org/erikj/fixup_return_pagination
Refactor get_recent_events_for_room return type
2018-05-09 16:07:14 +01:00
Erik Johnston d26bec8a43 Add comment to sync as to why code path is split 2018-05-09 15:56:07 +01:00
Erik Johnston fcf55f2255 Fix returned token is no longer a tuple 2018-05-09 15:43:00 +01:00
Erik Johnston 7ce98804ff Fix up comment 2018-05-09 15:42:39 +01:00
Erik Johnston cddf91c8b9 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_membership_change 2018-05-09 15:32:07 +01:00
Erik Johnston 9896dab8f6 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fixup_return_pagination 2018-05-09 15:31:33 +01:00
Erik Johnston 1e5280b7d0
Merge pull request #3196 from matrix-org/erikj/pagination_return
Refactor pagination DB API to return concrete type
2018-05-09 15:27:17 +01:00
Erik Johnston 75552d2148 Update comments 2018-05-09 15:15:38 +01:00
David Baker 294e9a0c9b Prefix internal functions 2018-05-09 15:10:37 +01:00
David Baker 46df23f581 Add the schema file 2018-05-09 15:07:54 +01:00
David Baker 52281e4c54 Indent fail 2018-05-09 15:06:16 +01:00
David Baker 7e8726b8fb Part deactivated users in the background
One room at a time so we don't take out the whole server with leave
events, and restart at server restart.
2018-05-09 14:54:28 +01:00
Erik Johnston c0e08dc45b Remove unused code path from member change DB func
The function is never called without a from_key, so we can remove all
the handling for that scenario.
2018-05-09 14:31:32 +01:00
Erik Johnston 0461ef01b7
Merge pull request #3195 from matrix-org/erikj/pagination_refactor
Refactor recent events func to use pagination func
2018-05-09 14:12:24 +01:00
Erik Johnston e2accd7f1d Refactor sync APIs to reuse pagination API
The sync API often returns events in a topological rather than stream
ordering, e.g. when the user joined the room or on initial sync. When
this happens we can reuse existing pagination storage functions.
2018-05-09 13:43:39 +01:00
Erik Johnston e5ab9cd24b Don't unnecessarily require token to be stream token
This allows calling the `get_recent_event_ids_for_room` function in more
situations.
2018-05-09 11:58:35 +01:00
Erik Johnston c4af4c24ca Refactor get_recent_events_for_room return type
There is no reason to return a tuple of tokens when the last token is
always the token passed as an argument. Changing it makes it consistent
with other storage APIs
2018-05-09 11:55:34 +01:00
Erik Johnston 05e0a2462c Refactor pagination DB API to return concrete type
This makes it easier to document what is being returned by the storage
functions and what some functions expect as arguments.
2018-05-09 11:34:24 +01:00
Erik Johnston 7dd13415db Remove unused from_token param 2018-05-09 10:58:16 +01:00
Erik Johnston 27cf170558 Refactor recent events func to use pagination func
This also removes a cache that is unlikely to ever get hit.
2018-05-09 10:55:55 +01:00
Erik Johnston 1aeb5e28a9
Merge pull request #3193 from matrix-org/erikj/pagination_refactor
Refactor /context to reuse pagination storage functions
2018-05-09 10:22:38 +01:00
Erik Johnston 23ec51c94c Fix up comments and make function private 2018-05-09 09:55:19 +01:00
Erik Johnston 696f532453 Reuse existing pagination code for context API 2018-05-08 16:20:19 +01:00
Erik Johnston 3e6d306e94 Parse tokens before calling DB function 2018-05-08 16:18:58 +01:00
Erik Johnston 274b8c6025 Only fetch required fields from database 2018-05-08 16:15:25 +01:00
Erik Johnston 06c0d0ed08 Split paginate_room_events storage function 2018-05-08 16:14:26 +01:00
David Baker bf98fa0864 Part user from rooms on account deactivate
This implements this very crudely: this probably isn't viable
because parting a user from all their rooms could take a long time,
and if the HS gets restarted in that time the process will be
aborted.
2018-05-08 15:58:35 +01:00
Richard van der Hoff 678e649b78
Merge pull request #3190 from mujx/notif-token-fix
notifications: Convert next_token to string according to the spec
2018-05-08 13:54:47 +01:00
Konstantinos Sideris 88868b2839 notifications: Convert next_token to string according to the spec
Currently the parameter is serialized as an integer.

Signed-off-by: Konstantinos Sideris <sideris.konstantin@gmail.com>
2018-05-05 12:55:02 +03:00
Erik Johnston 6d8ec3462d Note that label values can be anything 2018-05-03 16:25:05 +01:00
Erik Johnston 95b6912045 Fix metrics that have integer value labels 2018-05-03 15:51:04 +01:00
Richard van der Hoff 093d8c415a Merge remote-tracking branch 'origin/develop' into rav/warn_on_logcontext_fail 2018-05-03 14:59:29 +01:00
Richard van der Hoff 0ba609dc6f
Merge pull request #3183 from matrix-org/rav/moar_logcontext_leaks
Fix logcontext leaks in rate limiter
2018-05-03 14:58:13 +01:00
Richard van der Hoff 2117f84323
Merge pull request #3182 from Half-Shot/hs/fix-twisted-shutdown
Fix 'Unhandled Error' logs with Twisted 18.4
2018-05-03 12:40:11 +01:00
Richard van der Hoff a7fe62f0cb Fix logcontext leaks in rate limiter 2018-05-03 12:31:59 +01:00
Will Hunt 2e7a94c36b Don't abortConnection() if the transport connection has already closed. 2018-05-03 12:31:47 +01:00
Richard van der Hoff a2aaa9cb3c
Merge pull request #3178 from matrix-org/rav/fix_request_timeouts
fix http request timeout code
2018-05-03 11:33:26 +01:00
Erik Johnston 0a3b51c420
Merge pull request #3141 from matrix-org/erikj/fixup_state
Refactor event storage to prepare for changes in state calculations
2018-05-03 10:39:20 +01:00
Erik Johnston 31c7c29d43 Fix up grammar 2018-05-03 10:38:58 +01:00
Richard van der Hoff 902673e356
Merge pull request #3161 from NotAFile/remove-v1auth
Make Client-Server API return 403 for invalid token
2018-05-03 10:10:57 +01:00
Erik Johnston 53a5fdf312
Merge pull request #3175 from matrix-org/erikj/escape_metric_values
Escape label values in prometheus metrics
2018-05-03 10:01:04 +01:00
Richard van der Hoff 1dfd650348 add missing param to cancelled_to_request_timed_out_error
This gets two arguments, not one.
2018-05-02 22:42:36 +01:00
Erik Johnston a41117c63b Make _escape_character take MatchObject 2018-05-02 17:27:27 +01:00
Erik Johnston 32015e1109 Escape label values in prometheus metrics 2018-05-02 16:52:42 +01:00
Richard van der Hoff 3a42aed9a1
Merge pull request #3170 from matrix-org/rav/more_logcontext_leaks
Fix a class of logcontext leaks
2018-05-02 16:45:51 +01:00
Richard van der Hoff 415c6b672e Merge branch 'develop' into rav/more_logcontext_leaks 2018-05-02 16:16:01 +01:00
Richard van der Hoff be31adb036 Fix logcontext leak in media repo
Make FileResponder.write_to_consumer uphold the logcontext contract
2018-05-02 16:14:50 +01:00
Richard van der Hoff f22e7cda2c Fix a class of logcontext leaks
So, it turns out that if you have a first `Deferred` `D1`, you can add a
callback which returns another `Deferred` `D2`, and `D2` must then complete
before any further callbacks on `D1` will execute (and later callbacks on `D1`
get the *result* of `D2` rather than `D2` itself).

So, `D1` might have `called=True` (as in, it has started running its
callbacks), but any new callbacks added to `D1` won't get run until `D2`
completes - so if you `yield D1` in an `inlineCallbacks` function, your `yield`
will 'block'.

In conclusion: some of our assumptions in `logcontext` were invalid. We need to
make sure that we don't optimise out the logcontext juggling when this
situation happens. Fortunately, it is easy to detect by checking `D1.paused`.
2018-05-02 11:58:00 +01:00
Richard van der Hoff a8d8bf92e0
Merge pull request #3168 from matrix-org/rav/fix_logformatter
Fix incorrect reference to StringIO
2018-05-02 10:03:36 +01:00
Richard van der Hoff e482f8cd85 Fix incorrect reference to StringIO
This was introduced in 4f2f5171
2018-05-02 09:12:26 +01:00
Neil Johnson 2414178ed6 Merge branch 'master' into develop 2018-05-01 18:53:56 +01:00
Neil Johnson 8570bb84cc
Update __init__.py
bump version
2018-05-01 18:22:53 +01:00
Richard van der Hoff ca7211104e Merge branch 'release-v0.28.1' into develop 2018-05-01 18:16:57 +01:00
Richard van der Hoff d5eee5d601 Merge commit '33f469b' into release-v0.28.1 2018-05-01 18:14:18 +01:00
Richard van der Hoff d858f3bd4e Miscellaneous fixes to python_dependencies
* add some doc about wtf this thing does
* pin Twisted to < 18.4
* add explicit dep on six (fixes #3089)
2018-05-01 18:13:54 +01:00
Richard van der Hoff 33f469ba19 Apply some limits to depth to counter abuse
* When creating a new event, cap its depth to 2^63 - 1
* When receiving events, reject any without a sensible depth

As per https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI
2018-05-01 17:54:19 +01:00
Neil Johnson dd1a832419 remove user agent from data model, will just join on user_ips 2018-05-01 12:13:49 +01:00
Neil Johnson d0857702e8 add inidexes based on usage 2018-05-01 12:12:57 +01:00
Neil Johnson 5917562b60 10 mins seems more reasonable that every minute 2018-05-01 12:12:22 +01:00
Adrian Tschira 6495dbb326 Burminate v1auth
This closes #2602

v1auth was created to account for the differences in status code between
the v1 and v2_alpha revisions of the protocol (401 vs 403 for invalid
tokens). However since those protocols were merged, this makes the r0
version/endpoint internally inconsistent, and violates the
specification for the r0 endpoint.

This might break clients that rely on this inconsistency with the
specification. This is said to affect the legacy angular reference
client. However, I feel that restoring parity with the spec is more
important. Either way, it is critical to inform developers about this
change, in case they rely on the illegal behaviour.

Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-30 22:20:43 +02:00
Will Hunt 2ad3fc36e6 Fixes #3135 - Replace _OpenSSLECCurve with crypto.get_elliptic_curve (#3157)
fixes #3135

Signed-off-by: Will Hunt will@half-shot.uk
2018-04-30 16:21:11 +01:00
Krombel 576b71dd3d add guard for None on purge_history api 2018-04-30 14:29:48 +02:00
Matthew Hodgson 99a54bf2af
Merge pull request #3129 from matrix-org/matthew/fix_group_dups
remove duplicates from groups tables
2018-04-30 11:47:25 +01:00
Richard van der Hoff 63ae5cbf34
Merge pull request #3143 from matrix-org/rav/remove_redundant_preserve_fn
Remove redundant call to preserve_fn
2018-04-30 10:23:59 +01:00
Richard van der Hoff fdb6849b81
Merge pull request #3144 from matrix-org/rav/run_in_background_exception_handling
Trap exceptions thrown within run_in_background
2018-04-30 10:23:02 +01:00
Richard van der Hoff 01e8a52825
Merge pull request #3102 from NotAFile/py3-attributeerror
Make event properties raise AttributeError instead
2018-04-30 09:22:09 +01:00
Adrian Tschira 0c9db26260 add comment explaining attributeerror 2018-04-30 09:49:10 +02:00
Richard van der Hoff 950a32eb47
Merge pull request #3152 from NotAFile/py3-local-imports
make imports local
2018-04-30 01:28:13 +01:00
Richard van der Hoff bc2017a594
Merge pull request #3153 from NotAFile/py3-httplib
move httplib import to six
2018-04-30 01:26:42 +01:00
Richard van der Hoff 683149c1f9
Merge pull request #3151 from NotAFile/py3-xrange-1
Move more xrange to six
2018-04-30 01:20:06 +01:00
Richard van der Hoff 3b0e431c82
Merge pull request #3150 from NotAFile/py3-listcomp-yield
Don't yield in list comprehensions
2018-04-30 01:11:41 +01:00
Richard van der Hoff db75c86e84
Merge branch 'develop' into py3-xrange-1 2018-04-30 01:02:25 +01:00
Richard van der Hoff 2fd96727b1
Merge pull request #3085 from NotAFile/py3-config-text-mode
Open config file in non-bytes mode
2018-04-30 01:00:23 +01:00
Richard van der Hoff b8ee12b978
Merge pull request #3084 from NotAFile/py3-certs-byte-mode
Open certificate files as bytes
2018-04-30 01:00:05 +01:00
Richard van der Hoff 049b0b5af2
Merge pull request #3154 from NotAFile/py3-stringio
Replace stringIO imports with six
2018-04-30 00:59:04 +01:00
Richard van der Hoff af3cc50511 Remove redundant call to preserve_fn
submit_event_for_as doesn't return a deferred anyway, so this is pointless.
2018-04-30 00:48:36 +01:00
Richard van der Hoff dbf6f28d64
Merge pull request #3155 from NotAFile/py3-bytes-1
more bytes strings
2018-04-30 00:38:21 +01:00
Richard van der Hoff aab2e4da60
Merge pull request #3140 from matrix-org/rav/use_run_in_background
Use run_in_background in preference to preserve_fn
2018-04-30 00:34:28 +01:00
Richard van der Hoff 1315d374cc
Merge pull request #3156 from NotAFile/py3-hmac-bytes
Construct HMAC as bytes on py3
2018-04-30 00:33:20 +01:00
Richard van der Hoff 9e2601f830
Merge pull request #3108 from NotAFile/py3-six-urlparse
Use six.moves.urlparse
2018-04-30 00:33:05 +01:00
Adrian Tschira 122593265b Construct HMAC as bytes on py3
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-29 00:19:41 +02:00
Adrian Tschira e9143b6593 more bytes strings
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-29 00:13:57 +02:00
Matthew Hodgson adaf3ec87f fix missing import 2018-04-28 22:39:15 +01:00
Matthew Hodgson 006e18b6bb pep8 2018-04-28 22:32:24 +01:00
Matthew Hodgson 42c89c8215 make it work with sqlite 2018-04-28 22:27:30 +01:00
Adrian Tschira d82b6ea9e6 Move more xrange to six
plus a bonus next()

Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:57:00 +02:00
Adrian Tschira 4f2f5171b7 replace stringIO imports 2018-04-28 13:46:23 +02:00
Adrian Tschira 94f4d7f49e move httplib import to six 2018-04-28 13:43:34 +02:00
Adrian Tschira 57b58e2174 make imports local
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:41:41 +02:00
Adrian Tschira cdb4647a80 Don't yield in list comprehensions
I've tried to grep for more of this with no success.

Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:36:30 +02:00
Adrian Tschira a376d8f761 open log_config in text mode too
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:34:13 +02:00
Richard van der Hoff 453adf00b6 pep8; remove spurious import 2018-04-27 14:32:08 +01:00