51b17ec566
flake8
2018-08-16 17:32:22 +01:00
3c1080b6e4
refactor for readability, and reuse caching for setting tags
2018-08-16 17:02:04 +01:00
eff3ae3b9a
add room tagging
2018-08-16 15:48:34 +01:00
7d5b1a60a3
Fix inbound federation on reader worker
...
Inbound federation requires calculating push, which in turn relies on
having access to account data.
2018-08-16 15:32:20 +01:00
bfb6c58624
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking
2018-08-16 15:10:16 +01:00
a675f9c556
check for room state before deciding on action
2018-08-16 14:53:35 +01:00
c151b32b1d
Add GET media/v1/config ( #3184 )
2018-08-16 14:23:38 +01:00
762a758fea
lazyload aware /messages ( #3589 )
2018-08-16 14:22:47 +01:00
25d2b5d55f
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking
2018-08-16 11:37:17 +01:00
df1e4f259f
WIP impl commiting to get feedback
2018-08-16 11:10:53 +01:00
c055c91655
fix case where empty string state check is evaulated as False
2018-08-16 11:10:19 +01:00
3f543dc021
initial cut at a room summary API ( #3574 )
2018-08-16 09:46:50 +01:00
8cfad2e686
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking
2018-08-15 17:20:38 +01:00
81d727efa9
Merge pull request #3689 from matrix-org/neilj/fix_off_by_1+maus
...
Fix Mau off by one errors
2018-08-15 16:19:41 +00:00
2f78f432c4
speed up /members and add at= and membership params ( #3568 )
2018-08-15 16:35:22 +01:00
fc5d937550
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking
2018-08-15 16:31:40 +01:00
86a00e05e1
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus
2018-08-15 16:27:08 +01:00
eabc5f8271
wip cut at sending resource server notices
2018-08-15 15:04:52 +01:00
c24fc9797b
add new event types
2018-08-15 15:04:30 +01:00
afcd655ab6
Use a producer to stream back responses
...
The problem with dumping all of the json response into the Request object at
once is that doing so starts the timeout for the next request to be received:
so if it takes longer than 60s to stream back the response to the client, the
client never gets it.
The correct solution is to use a Producer; then the timeout is only started
once all of the content is sent over the TCP connection.
2018-08-15 15:04:16 +01:00
dc56c47dc0
Merge pull request #3653 from matrix-org/erikj/split_federation
...
Move more federation APIs to workers
2018-08-15 14:59:02 +01:00
4601129c44
Merge pull request #3687 from matrix-org/neilj/admin_email
...
support admin_email config and pass through into blocking errors,
2018-08-15 13:52:25 +00:00
ef184caf30
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation
2018-08-15 14:25:46 +01:00
488ffe6fdb
Use federation handler function rather than duplicate
...
This involves renaming _persist_events to be a public function.
2018-08-15 14:17:18 +01:00
773db62a22
Rename slave TransactionStore to SlaveTransactionStore
2018-08-15 14:17:06 +01:00
eaaa2248ff
Refactor request logging code
...
This commit moves a bunch of the logic for deciding when to log the receipt and
completion of HTTP requests into SynapseRequest, rather than in the request
handling wrappers.
Advantages of this are:
* we get logs for *all* requests (including OPTIONS and HEADs), rather than
just those that end up hitting handlers we've remembered to decorate
correctly.
* when a request handler wires up a Producer (as the media stuff does
currently, and as other things will do soon), we log at the point that all
of the traffic has been sent to the client.
2018-08-15 13:47:52 +01:00
87a824bad1
clean up AuthError
2018-08-15 11:58:03 +01:00
c4eb97518f
Merge branch 'neilj/update_limits_error_codes' of github.com:matrix-org/synapse into neilj/admin_email
2018-08-15 11:53:01 +01:00
55afba0fc5
update admin email to uri
2018-08-15 11:41:18 +01:00
1c5e690a6b
Merge pull request #3690 from matrix-org/neilj/change_prometheus_mau_metric_name
...
combine mau metrics into one group
2018-08-15 09:53:59 +00:00
fef2e65d12
Merge pull request #3667 from matrix-org/erikj/fixup_unbind
...
Don't fail requests to unbind 3pids for non supporting ID servers
2018-08-15 10:32:12 +01:00
b8429c7c81
update error codes for resource limiting
2018-08-15 10:19:48 +01:00
ab035bdeac
replace admin_email with admin_uri for greater flexibility
2018-08-15 10:16:41 +01:00
19b433e3f4
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/admin_email
2018-08-14 17:44:46 +01:00
b586b8b986
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus
2018-08-14 17:43:22 +01:00
70e48cbbb1
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/change_prometheus_mau_metric_name
2018-08-14 17:40:59 +01:00
9ecbaf8ba8
adding missing yield
2018-08-14 16:55:28 +01:00
e5962f845c
pep8
2018-08-14 16:36:14 +01:00
e7d091fb86
combine mau metrics into one group
2018-08-14 16:26:55 +01:00
414d54b61a
Merge pull request #3670 from matrix-org/neilj/mau_sync_block
...
Block ability to read via sync if mau limit exceeded
2018-08-14 15:21:31 +00:00
2545993ce4
make comments clearer
2018-08-14 15:48:12 +01:00
8f9a7eb58d
support admin_email config and pass through into blocking errors, return AuthError in all cases
2018-08-14 15:11:54 +01:00
c74c71128d
remove blank line
2018-08-14 15:06:24 +01:00
ed4bc3d2fc
fix off by 1s on mau
2018-08-14 15:04:48 +01:00
bd92c8eaa7
Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus
2018-08-14 14:56:45 +01:00
99ebaed8e6
Update register.py
...
remove comments
2018-08-14 14:55:55 +01:00
9b5bf3d858
Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus
2018-08-14 14:51:38 +01:00
e25d87d97b
Merge branch 'neilj/mau_sync_block' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus
2018-08-14 14:32:18 +01:00
e2c9fe0a6a
backout ability to pass in event type to server notices
2018-08-14 13:32:56 +01:00
591bf87c6a
Merge remote-tracking branch 'origin/develop' into neilj/fix_reap_users_in_postgres
2018-08-14 20:56:23 +10:00
9b75c78b4d
support server notice state events for resource limits
2018-08-14 11:20:41 +01:00
63417c31e9
fix typo
2018-08-13 22:36:52 +01:00
f4b49152e2
support admin_email config and pass through into blocking errors, return AuthError in all cases
2018-08-13 21:09:47 +01:00
50bcaf1a27
Merge pull request #3677 from andrewshadura/master
...
Use recaptcha_ajax.js directly from Google
2018-08-13 18:52:56 +01:00
34f51babc5
Revert "Use recaptcha_ajax.js directly from Google"
2018-08-13 18:48:03 +01:00
ce7de9ae6b
Revert "support admin_email config and pass through into blocking errors, return AuthError in all cases"
...
This reverts commit 0d43f991a1
2018-08-13 18:06:18 +01:00
0d43f991a1
support admin_email config and pass through into blocking errors, return AuthError in all cases
2018-08-13 18:00:23 +01:00
99dd975dae
Run tests under PostgreSQL ( #3423 )
2018-08-13 16:47:46 +10:00
31fa743567
fix sqlite/postgres incompatibility in reap_monthly_active_users
2018-08-11 22:38:34 +01:00
6c6aba76e1
implementation of server notices to alert on hitting resource limits
2018-08-10 15:12:59 +01:00
c31793a784
Merge branch 'rav/fix_linearizer_cancellation' into develop
2018-08-10 14:57:27 +01:00
c08f9d95b2
log *after* reloading log config
...
... because logging *before* reloading means the log message gets lost in the old MemoryLogger
2018-08-10 14:56:48 +01:00
b37c472419
Rename async to async_helpers because `async` is a keyword on Python 3.7 ( #3678 )
2018-08-10 23:50:21 +10:00
c75b71a397
Use recaptcha_ajax.js directly from Google
...
The script recaptcha_ajax.js contains minified non-free code which
we probably cannot redistribute. Since it talks to Google servers
anyway, it is better to just download it from Google directly instead
of shipping it.
This fixes #1932 .
2018-08-10 13:36:14 +02:00
3c0213a217
Merge pull request #3439 from vojeroen/send_sni_for_federation_requests
...
send SNI for federation requests
2018-08-10 12:23:54 +01:00
638d35ef08
Fix linearizer cancellation on twisted < 18.7
...
Turns out that cancellation of inlineDeferreds didn't really work properly
until Twisted 18.7. This commit refactors Linearizer.queue to avoid
inlineCallbacks.
2018-08-10 10:59:09 +01:00
01021c812f
wip at implementing MSC 7075
2018-08-09 22:16:00 +01:00
2e9c73e8ca
more generic conversion of str/bytes to unicode
2018-08-09 21:31:26 +02:00
64899341dc
include private functions from twisted
2018-08-09 21:04:22 +02:00
885ea9c602
rename _user_last_seen_monthly_active
2018-08-09 18:02:12 +01:00
c1f9dec92a
fix errant parenthesis
2018-08-09 17:43:26 +01:00
09cf130898
only block on sync where user is not part of the mau cohort
2018-08-09 17:39:12 +01:00
bb99b1f550
Add fast path in state res for zero prev events
2018-08-09 14:58:49 +01:00
ce6db0e547
Choose state algorithm based on room version
2018-08-09 14:58:47 +01:00
152c0aa58e
Add constants for room versions
2018-08-09 14:55:47 +01:00
119451dcd1
Refactor state module
...
We split out the actual state resolution algorithm to prepare for having
multiple versions.
2018-08-09 14:55:47 +01:00
69ce057ea6
block sync if auth checks fail
2018-08-09 12:26:27 +01:00
3dce9050cf
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block
2018-08-09 11:42:02 +01:00
0ad98e38d0
Merge pull request #3655 from matrix-org/neilj/disable_hs
...
Flag to disable HS without disabling federation
2018-08-09 10:41:43 +00:00
a5ef110749
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block
2018-08-09 11:40:37 +01:00
a6c813761a
Docstrings
2018-08-09 10:41:08 +01:00
b179537f2a
Move clean_room_for_join to master
2018-08-09 10:37:38 +01:00
72d1902bbe
Fixup doc comments
2018-08-09 10:23:49 +01:00
984376745b
Merge branch 'master' into develop
2018-08-09 19:23:47 +10:00
dafe90a6be
0.33.2
2018-08-09 19:20:41 +10:00
484a0ebdfc
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles
2018-08-09 10:16:29 +01:00
5785b93711
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation
2018-08-09 10:16:16 +01:00
bf7598f582
Log when we 3pid/unbind request fails
2018-08-09 10:09:56 +01:00
2bdafaf3c1
Merge pull request #3632 from matrix-org/erikj/refactor_repl_servlet
...
Add helper base class for generating new replication endpoints
2018-08-09 10:06:23 +01:00
62564797f5
Fixup wording and remove dead code
2018-08-09 09:56:10 +01:00
bb89c84614
Merge pull request #3664 from matrix-org/rav/federation_metrics
...
more metrics for the federation and appservice senders
2018-08-08 23:16:58 +01:00
d5c0ce4cad
updated docstring for ServerContextFactory
2018-08-08 19:25:01 +02:00
e92fb00f32
sync auth blocking
2018-08-08 17:54:49 +01:00
839a317c96
fix pep8 too many lines
2018-08-08 17:39:04 +01:00
5298d79fb5
Merge branch 'develop' into neilj/disable_hs
2018-08-08 16:13:03 +00:00
8521ae13e3
Merge pull request #3654 from matrix-org/rav/room_versions
...
Support for room versioning
2018-08-08 17:10:53 +01:00
d2f3ef98ac
Merge branch 'develop' into neilj/disable_hs
2018-08-08 15:55:47 +00:00
2b5baebeba
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/disable_hs
2018-08-08 13:47:07 +01:00
be59910b93
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/reserved_users
2018-08-08 13:45:21 +01:00
990fe9fc23
Merge pull request #3633 from matrix-org/neilj/mau_tracker
...
API for monthly_active_users table
2018-08-08 12:44:15 +00:00
312ae74746
typos
2018-08-08 13:33:16 +01:00
360ba89c50
Don't fail requests to unbind 3pids for non supporting ID servers
...
Older identity servers may not support the unbind 3pid request, so we
shouldn't fail the requests if we received one of 400/404/501. The
request still fails if we receive e.g. 500 responses, allowing clients
to retry requests on transient identity server errors that otherwise do
support the API.
Fixes #3661
2018-08-08 12:06:18 +01:00
bebe325e6c
Rename POST param to METHOD
2018-08-08 10:36:18 +01:00
5011417632
Fixup logging and docstrings
2018-08-08 10:29:58 +01:00
bab94da79c
fix metric name
2018-08-07 22:11:45 +01:00
53bca4690b
more metrics for the federation and appservice senders
2018-08-07 19:09:48 +01:00
ef3589063a
prevent total number of reserved users being too large
2018-08-07 17:59:27 +01:00
e8eba2b4e3
implement reserved users for mau limits
2018-08-07 17:49:43 +01:00
3523f5432a
Don't expose default_room_version as config opt
2018-08-07 12:51:57 +01:00
848431be1d
fix for rc1
2018-08-07 21:13:22 +10:00
3b9662339b
version
2018-08-07 21:10:52 +10:00
cd9765805e
Allow ratelimiting on workers
2018-08-07 10:50:28 +01:00
495cb100d1
Allow profile changes to happen on workers
2018-08-07 10:50:26 +01:00
ca9bc1f4fe
Fix occasional glitches in the synapse_event_persisted_position metric
...
Every so often this metric glitched to a negative number. I'm assuming it was
due to backfilled events.
2018-08-07 10:00:03 +01:00
a74b25faaa
WIP building out mau reserved users
2018-08-06 23:25:25 +01:00
7daa8a78c5
load mau limit threepids
2018-08-06 22:55:05 +01:00
89834d9a29
Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs
2018-08-06 21:56:56 +01:00
e54794f5b6
Fix postgres compatibility bug
2018-08-06 21:55:54 +01:00
7bcf126b18
Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs
2018-08-06 21:39:44 +01:00
1911c037cb
update comments to reflect new sig
2018-08-06 18:01:46 +01:00
33bd07d062
Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs
2018-08-06 17:52:28 +01:00
16d78be315
make use of _simple_select_one_onecol, improved comments
2018-08-06 17:51:15 +01:00
19a17068f1
Check m.room.create for sane room_versions
2018-08-06 16:11:24 +01:00
96a9a29645
Pull in necessary stores in federation_reader
2018-08-06 15:23:57 +01:00
62ace05c45
Move necessary storage functions to worker classes
2018-08-06 15:23:38 +01:00
1e2bed9656
Import all functions from TransactionStore
2018-08-06 15:23:38 +01:00
a3f5bf79a0
Add EDU/query handling over replication
2018-08-06 15:23:31 +01:00
e26dbd82ef
Add replication APIs for persisting federation events
2018-08-06 15:02:28 +01:00
051a99c400
Fix isort
2018-08-06 14:29:31 +01:00
f900d50824
include known room versions in outgoing make_joins
2018-08-06 13:45:37 +01:00
42c6823827
disable HS from config
2018-08-04 22:07:04 +01:00
e40a510fbf
py3 fix
2018-08-03 23:19:13 +01:00
886be75ad1
bug fixes
2018-08-03 22:29:03 +01:00
16d9701892
Return M_NOT_FOUND when a profile could not be found. ( #3596 )
2018-08-03 19:08:05 +01:00
e10830e976
wip commit - tests failing
2018-08-03 17:55:50 +01:00
3777fa26aa
sanity check response from make_join
2018-08-03 16:08:32 +01:00
0d63d93ca8
Enforce compatibility when processing make_join requests
...
Reject make_join requests from servers which do not support the room version.
Also include the room version in the response.
2018-08-03 16:08:32 +01:00
0ca459ea33
Basic support for room versioning
...
This is the first tranche of support for room versioning. It includes:
* setting the default room version in the config file
* new room_version param on the createRoom API
* storing the version of newly-created rooms in the m.room.create event
* fishing the version of existing rooms out of the m.room.create event
2018-08-03 16:08:32 +01:00
15c1ae45e5
Docstrings for BaseFederationServlet
...
... to save me reverse-engineering this stuff again.
2018-08-03 16:08:32 +01:00
5593ff6773
fix (lots of) py3 test failures
2018-08-03 14:59:17 +01:00
950807d93a
fix caching and tests
2018-08-03 13:49:53 +01:00
897c51d274
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker
2018-08-03 13:40:47 +01:00
cb298ff623
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_repl_servlet
2018-08-03 09:25:15 +01:00
c0affa7b4f
update generate_monthly_active_users, and reap_monthly_active_users
2018-08-02 23:03:01 +01:00
5e2f7b8084
typo
2018-08-02 22:47:48 +01:00
4ecb4bdac9
wip attempt at caching
2018-08-02 22:41:05 +01:00
1fa98495d0
Merge pull request #3639 from matrix-org/rav/refactor_error_handling
...
Clean up handling of errors from outbound requests
2018-08-02 17:38:24 +01:00
bdae8f2e68
Merge pull request #3638 from matrix-org/rav/refactor_federation_client_exception_handling
...
Factor out exception handling in federation_client
2018-08-02 17:37:46 +01:00
74b1d46ad9
do mau checks based on monthly_active_users table
2018-08-02 16:57:35 +01:00
9180061b49
remove unused count_monthly_users
2018-08-02 15:55:29 +01:00
704c3e6239
Merge branch 'master' into develop
2018-08-02 15:43:30 +01:00
43ecfe0b10
Synapse 0.33.1 (2018-08-02)
...
===========================
SECURITY FIXES
--------------
- Fix a potential issue where servers could request events for rooms they have not joined. (`#3641 <https://github.com/matrix-org/synapse/issues/3641 >`_)
- Fix a potential issue where users could see events in private rooms before they joined. (`#3642 <https://github.com/matrix-org/synapse/issues/3642 >`_)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAABAgAGBQJbYxcCAAoJEIofk9V1tejVg8YIAJU1xoZ2zSnordJczpvVZbzx
Rsstk/wvE2dSGJL22gfSLzQNCbQvF4aOul1LAGJfbEQWc46BcNpyuWUxjTd3FaF7
SfbWdkDh1w3ROSuiPA8j4CnH6EwP0w2itIBhFG7JpVEXjRgCyc1iMUl/oQESv82g
UQyr6t/G68mE8xHm0eqvNfCjVAMSZDAnOzYZa7vfZJBqlZiGHB6Z1nsvsS3nZqQd
BQQi/pQTXxsIL2egSvuycykiZtEZcm+QDRFi6hIatv5LRFOsQUqKVTB/D7XOoSbP
RPOzcaSjv4mZ9a4NBzFMVVzzTRbr2EnipPd2ePrnJoOnl2eNRO4Won8zl4h+OQc=
=U/ir
-----END PGP SIGNATURE-----
Merge tag 'v0.33.1'
Synapse 0.33.1 (2018-08-02)
===========================
SECURITY FIXES
--------------
- Fix a potential issue where servers could request events for rooms they have not joined. (`#3641 <https://github.com/matrix-org/synapse/issues/3641 >`_)
- Fix a potential issue where users could see events in private rooms before they joined. (`#3642 <https://github.com/matrix-org/synapse/issues/3642 >`_)
2018-08-02 15:40:44 +01:00
14a4e7d5a4
Prepare 0.33.1
2018-08-02 15:31:04 +01:00
0bf5ec0db7
Check room visibility for /event/ requests
...
Make sure that the user has permission to view the requeseted event for
/event/{eventId} and /room/{roomId}/event/{eventId} requests.
Also check that the event is in the given room for
/room/{roomId}/event/{eventId}, for sanity.
2018-08-02 15:03:27 +01:00
14fa9d4d92
Avoid extra db lookups
...
Since we're about to look up the events themselves anyway, we can skip the
extra db queries here.
2018-08-02 13:55:51 +01:00
0a65450d04
Validation for events/rooms in fed requests
...
When we get a federation request which refers to an event id, make sure that
said event is in the room the caller claims it is in.
(patch supplied by @turt2live)
2018-08-02 13:48:40 +01:00
00f99f74b1
insertion into monthly_active_users
2018-08-02 13:47:19 +01:00
4a6725d9d1
Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/mau_tracker
2018-08-02 11:04:18 +01:00
165e067033
Revert "change monthly_active_users table to be a single column"
...
This reverts commit ec716a35b2
2018-08-02 10:59:58 +01:00
40c1c59cf4
Merge pull request #3621 from matrix-org/erikj/split_fed_store
...
Split out DB writes in federation handler
2018-08-02 10:41:42 +01:00
08281fe6b7
self.db_conn unused
2018-08-01 23:26:24 +01:00
c21d82bab3
normalise reaping query
2018-08-01 23:24:38 +01:00
ec716a35b2
change monthly_active_users table to be a single column
2018-08-01 17:54:37 +01:00
d766f26de9
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker
2018-08-01 17:49:41 +01:00
085435e13a
Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits
...
Initial impl of capping MAU
2018-08-01 15:58:45 +00:00
b8d7d3996b
Merge pull request #3620 from fuzzmz/return-404-room-not-found
...
return 404 if room not found
2018-08-01 16:34:32 +01:00
b7f203a566
count_monthly_users is now async
2018-08-01 16:17:42 +01:00
7ff44d9215
improve clarity
2018-08-01 16:17:00 +01:00
01e93f48ed
Kill off MatrixCodeMessageException
...
This code brings the SimpleHttpClient into line with the
MatrixFederationHttpClient by having it raise HttpResponseExceptions when a
request fails (rather than trying to parse for matrix errors and maybe raising
MatrixCodeMessageException).
Then, whenever we were checking for MatrixCodeMessageException and turning them
into SynapseErrors, we now need to check for HttpResponseExceptions and call
to_synapse_error.
2018-08-01 16:02:46 +01:00
018d75a148
Refactor code for turning HttpResponseException into SynapseError
...
This commit replaces SynapseError.from_http_response_exception with
HttpResponseException.to_synapse_error.
The new method actually returns a ProxiedRequestError, which allows us to pass
through additional metadata from the API call.
2018-08-01 16:02:46 +01:00
fa7dc889f1
Be more careful which errors we send back over the C-S API
...
We really shouldn't be sending all CodeMessageExceptions back over the C-S API;
it will include things like 401s which we shouldn't proxy.
That means that we need to explicitly turn a few HttpResponseExceptions into
SynapseErrors in the federation layer.
The effect of the latter is that the matrix errcode will get passed through
correctly to calling clients, which might help with some of the random
M_UNKNOWN errors when trying to join rooms.
2018-08-01 16:02:38 +01:00
c82ccd3027
Factor out exception handling in federation_client
...
Factor out the error handling from make_membership_event, send_join, and
send_leave, so that it can be shared.
2018-08-01 16:01:04 +01:00
da7785147d
Python 3: Convert some unicode/bytes uses ( #3569 )
2018-08-02 00:54:06 +10:00
303f1c851f
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits
2018-08-01 13:42:50 +01:00
a6d7b74915
update docs
2018-08-01 13:39:14 +01:00
4b256b9271
_persist_auth_tree no longer returns anything
2018-08-01 13:39:07 +01:00
4e5ac901dd
clean up
2018-08-01 12:03:57 +01:00
f9f5559971
fix comment
2018-08-01 12:03:42 +01:00
0aba3d361a
count_monthly_users() async
2018-08-01 11:47:58 +01:00
2c54f1c225
remove need to plot limit_usage_by_mau
2018-08-01 11:46:59 +01:00
f49147d14f
Merge pull request #3634 from matrix-org/rav/wtf_is_a_replication_layer
...
rename replication_layer to federation_client
2018-08-01 10:29:29 +01:00
cab782c17e
Merge pull request #3384 from matrix-org/rav/rewrite_cachedlist_decorator
...
Rewrite cache list decorator
2018-08-01 10:28:56 +01:00
6023cdd227
remove errant print
2018-08-01 10:27:17 +01:00
7931393495
make count_monthly_users async synapse/handlers/auth.py
2018-08-01 10:21:56 +01:00
c507fa15ce
only need to loop if mau limiting is enabled
2018-08-01 10:20:42 +01:00
37be52ac34
limt -> limit
2018-07-31 16:29:09 -06:00
70af98e361
return NotFoundError if room not found
...
Per the Client-Server API[0] we should return
`M_NOT_FOUND` if the room isn't found instead
of generic SynapseError.
This ensures that /directory/list API returns
404 for room not found instead of 400.
[0]: https://matrix.org/docs/spec/client_server/unstable.html#get-matrix-client-r0-directory-list-room-roomid
Signed-off-by: Serban Constantin <serban.constantin@gmail.com>
2018-07-31 21:47:23 +03:00
5e2ee64660
Merge pull request #3628 from turt2live/travis/goodby-pdu-failures
...
Remove pdu_failures from transactions
2018-07-31 12:13:09 -06:00
6ef983ce5c
api into monthly_active_users table
2018-07-31 16:36:24 +01:00
bdbdceeafa
rename replication_layer to federation_client
...
I have HAD ENOUGH of trying to remember wtf a replication layer is in terms of
classes.
2018-07-31 15:44:05 +01:00
443da003bc
Use new helper base class for membership requests
2018-07-31 14:32:23 +01:00
729b672823
Use new helper base class for ReplicationSendEventRestServlet
2018-07-31 14:32:23 +01:00
d81602b75a
Add helper base class for generating new replication endpoints
...
This will hopefully reduce the boiler plate required to implement new
internal HTTP requests.
2018-07-31 14:32:20 +01:00
5de936caa1
Merge pull request #3612 from matrix-org/rav/store_heirarchy
...
Make EventStore inherit from EventFederationStore
2018-07-31 13:44:04 +01:00
df2235e7fa
coding style
2018-07-31 13:16:20 +01:00
0bc9b9e397
reinstate explicit include of EventsWorkerStore
2018-07-31 13:11:04 +01:00
e908b86832
Remove pdu_failures from transactions
...
The field is never read from, and all the opportunities given to populate it are not utilized. It should be very safe to remove this.
2018-07-30 16:28:47 -06:00
254e8267e2
Only import secrets when available
...
secrets got introduced in python 3.6 so this class is not available
in 3.5 and before.
This now checks for the current running version and only tries using
secrets if the version is 3.6 or above
Signed-Off-By: Matthias Kesler <krombel@krombel.de>
2018-07-30 23:59:02 +02:00
21276ff846
remove errant logging
2018-07-30 22:42:12 +01:00
fef7e58ac6
actually close conn
2018-07-30 22:29:44 +01:00
9b13817e06
factor out metrics from __init__ to app/homeserver
2018-07-30 22:07:07 +01:00
251e6c1210
limit register and sign in on number of monthly users
2018-07-30 15:55:57 +01:00
143f1a2532
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store
2018-07-30 09:56:18 +01:00
2903e65aff
fix isort
2018-07-29 19:47:08 +02:00
a8cbce0ced
fix invalidation
2018-07-27 16:17:17 +01:00
e9b2d047f6
make /context lazyload & filter aware ( #3567 )
...
make /context lazyload & filter aware.
2018-07-27 15:12:50 +01:00
f102c05856
Rewrite cache list decorator
...
Because it was complicated and annoyed me. I suspect this will be more
efficient too.
2018-07-27 13:47:04 +01:00
8e3f75b39a
fix accidental removal of hs
2018-07-27 12:17:31 +02:00
b0b5566f36
Merge pull request #3391 from t3chguy/t3chguy/default_inviter_display_name_3pid
...
if inviter_display_name == ""||None then default to inviter MXID
2018-07-27 10:08:39 +01:00
7041cd872b
Merge branch 'develop' into send_sni_for_federation_requests
2018-07-27 09:17:11 +01:00
d42455cdc9
Merge pull request #3616 from matrix-org/travis/event_id_send_leave
...
Update the send_leave path to be an event_id
2018-07-26 23:25:17 +01:00
65c8dee900
Merge pull request #3614 from matrix-org/rav/stop_populating_event_content
...
Stop populating events.content
2018-07-26 22:54:08 +01:00
a75231b507
Deduplicate redundant lazy-loaded members ( #3331 )
...
* attempt at deduplicating lazy-loaded members
as per the proposal; we can deduplicate redundant lazy-loaded members
which are sent in the same sync sequence. we do this heuristically
rather than requiring the client to somehow tell us which members it
has chosen to cache, by instead caching the last N members sent to
a client, and not sending them again. For now we hardcode N to 100.
Each cache for a given (user,device) tuple is in turn cached for up to
X minutes (to avoid the caches building up). For now we hardcode X to 30.
* add include_redundant_members filter option & make it work
* remove stale todo
* add tests for _get_some_state_from_cache
* incorporate review
2018-07-26 22:51:30 +01:00
9e68b1bd2d
Merge pull request #3613 from matrix-org/rav/stop_using_event_edges_room_id
...
Remove some redundant joins on event_edges.room_id
2018-07-26 22:31:01 +01:00
7d32f0d745
Update the send_leave path to be an event_id
...
It's still not used, however the parameter is an event ID not a transaction ID.
2018-07-26 14:41:59 -06:00
ef9d51b081
Merge pull request #3610 from matrix-org/rav/fix_looping_calls
...
Fix some looping_call calls which were broken in #3604
2018-07-26 14:55:57 +01:00
51d7df1915
Create the column nullable
...
There's no real point in ever making the column non-nullable, and doing so
breaks the sytests.
2018-07-26 14:54:04 +01:00
5c1d301fd9
Stop populating events.content
...
This field is no longer read from, so we should stop populating it. Once we're
happy that this doesn't break everything, and a rollback is unlikely, we can
think about dropping the column.
2018-07-26 14:43:02 +01:00
bd4b25f4d0
Remove some redundant joins on event_edges.room_id
...
We've long passed the point where it's possible to have the same event_id in
different tables, so these join conditions are redundant: we can just join on
event_id.
event_edges is of non-trivial size, and the room_id column is wasteful, so
let's stop reading from it. In future, we can stop writing to it, and then drop
it.
2018-07-26 13:19:08 +01:00
1b4d73fa52
comment on event_edges
2018-07-26 12:53:51 +01:00
21e878ebb6
Make EventStore inherit from EventFederationStore
...
(since it uses methods therein)
Turns out that we had a bunch of things which were incorrectly importing
EventWorkerStore from events.py rather than events_worker.py, which broke once
I removed the import into events.py.
2018-07-26 12:48:51 +01:00
03751a6420
Fix some looping_call calls which were broken in #3604
...
It turns out that looping_call does check the deferred returned by its
callback, and (at least in the case of client_ips), we were relying on this,
and I broke it in #3604 .
Update run_as_background_process to return the deferred, and make sure we
return it to clock.looping_call.
2018-07-26 11:48:08 +01:00
1bcd0490c2
Merge pull request #2970 from matrix-org/matthew/filter_members
...
Implement the lazy_load_members room state filter parameter
2018-07-26 00:03:01 +01:00
bc7944e6d2
switch missing_types to be a bool
2018-07-25 23:36:31 +01:00
d8e65ed7e1
Fix a minor documentation typo in on_make_leave
2018-07-25 15:44:41 -06:00
2565804030
Merge branch 'develop' into matthew/filter_members
2018-07-25 17:27:49 +01:00
7d9fb88617
incorporate more review.
2018-07-25 16:33:50 +01:00
78a691d005
Split out DB writes in federation handler
...
This will allow us to easily add an internal replication API to proxy
these reqeusts to master, so that we can move federation APIs to
workers.
2018-07-25 16:22:56 +01:00
3849f7f69f
Merge pull request #3603 from matrix-org/erikj/handle_outliers
...
Correctly handle outliers during persist events
2018-07-25 13:24:04 +01:00
cee1ae1b72
Merge pull request #3606 from matrix-org/rav/logcontext_fixes_once_more
...
Fix another logcontext leak in _persist_events
2018-07-25 11:56:00 +01:00
32b30e15f2
Merge pull request #3607 from matrix-org/rav/fix_persist_events_integrity_error
...
Fix occasional 'tuple index out of range' error
2018-07-25 11:55:45 +01:00
4081bd1560
Merge pull request #3605 from matrix-org/rav/fix_update_remote_profile_cache
...
Fix updating of cached remote profiles
2018-07-25 11:54:46 +01:00
7780a7b47c
Actually fix it by adding continue
2018-07-25 11:13:20 +01:00
1be94440d3
Fix occasional 'tuple index out of range' error
...
This fixes a bug in _delete_existing_rows_txn which was introduced in #3435
(though it's been on matrix-org-hotfixes for *years*). This code is only called
when there is some sort of conflict the first time we try to persist an event,
so it only happens rarely. Still, the exceptions are annoying.
2018-07-25 11:05:58 +01:00
07defd5fe6
Fix another logcontext leak in _persist_events
...
We need to run the errback in the sentinel context to avoid losing our own
context.
Also: add logging to runInteraction to help identify where "Starting db
connection from sentinel context" warnings are coming from
2018-07-25 10:53:23 +01:00
55acd6856c
Fix updating of cached remote profiles
...
_update_remote_profile_cache was missing its `defer.inlineCallbacks`, so when
it was called, would just return a generator object, without actually running
any of the method body.
2018-07-25 10:34:48 +01:00
f59be4eb0e
Fix unit tests
...
on_notifier_poke no longer runs synchonously, so we have to do a different hack
to make sure that the replication data has been sent. Let's actually listen for
its arrival.
2018-07-25 10:30:36 +01:00
a297ff2b16
Fix typo in conditional
2018-07-25 09:48:01 +01:00
371da42ae4
Wrap a number of things that run in the background
...
This will reduce the number of "Starting db connection from sentinel context"
warnings, and will help with our metrics.
2018-07-25 09:41:12 +01:00
ec56121b0d
Correctly handle outliers during persist events
...
We incorrectly asserted that all contexts must have a non None state
group without consider outliers. This would usually be fine as the
assertion would never be hit, as there is a shortcut during persistence
if the forward extremities don't change.
However, if the outlier is being persisted with non-outlier events, the
function would be called and the assertion would be hit.
Fixes #3601
2018-07-25 09:35:02 +01:00
cb5c37a57c
handle the edge case for _get_some_state_from_cache where types is []
2018-07-24 20:34:45 +01:00
87951d3891
Merge branch 'develop' of github.com:matrix-org/synapse into t3chguy/default_inviter_display_name_3pid
2018-07-24 17:17:46 +01:00
8b8c4f34a3
Replace usage of get_current_toke with StreamToken.START
...
This allows us to handle /context/ requests on the client_reader worker
without having to pull in all the various stream handlers (e.g.
precence, typing, pushers etc). The only thing the token gets used for
is pagination, and that ignores everything but the room portion of the
token.
2018-07-24 16:49:17 +01:00
3188973857
Pull out did_forget to worker store
2018-07-24 16:49:14 +01:00
60a1d147a7
Merge pull request #3595 from matrix-org/erikj/use_deltas
...
Use deltas to calculate current state deltas
2018-07-24 15:41:18 +01:00
709c309b0e
Expand on docstring comment about return value
2018-07-24 15:12:50 +01:00
8f65ab98d2
Remove unnecessary iteritems
2018-07-24 15:08:01 +01:00
ed0dd68731
Fixup comment (and indent)
2018-07-24 14:31:38 +01:00
811ac73a42
Don't fetch state from the database unless needed
2018-07-24 14:18:23 +01:00
a79410e7b8
Have _get_new_state_after_events return delta
...
If we have a delta from the existing to new current state, then we can
reuse that rather than manually working it out by fetching both lots of
state.
2018-07-24 14:14:17 +01:00
Neil Johnson