393a811a41
Fix join ratelimiter breaking profile updates and idempotency ( #8153 )
2020-08-24 18:06:04 +01:00
2df82ae451
Do not apply ratelimiting on joins to appservices ( #8139 )
...
Add new method ratelimiter.can_requester_do_action and ensure that appservices are exempt from being ratelimited.
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Erik Johnston <erik@matrix.org>
2020-08-24 14:53:53 +01:00
3f49f74610
Don't fail /submit_token requests on incorrect session ID if request_token_inhibit_3pid_errors is turned on ( #7991 )
...
* Don't raise session_id errors on submit_token if request_token_inhibit_3pid_errors is set
* Changelog
* Also wait some time before responding to /requestToken
* Incorporate review
* Update synapse/storage/databases/main/registration.py
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Incorporate review
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2020-08-24 11:33:55 +01:00
cbbf9126cb
Do not apply ratelimiting on joins to appservices ( #8139 )
...
Add new method ratelimiter.can_requester_do_action and ensure that appservices are exempt from being ratelimited.
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Erik Johnston <erik@matrix.org>
2020-08-21 15:07:56 +01:00
3f91638da6
Allow denying or shadow banning registrations via the spam checker ( #8034 )
2020-08-20 15:42:58 -04:00
e259d63f73
Stop shadow-banned users from sending invites. ( #8095 )
2020-08-20 15:07:42 -04:00
f594e434c3
Switch the JSON byte producer from a pull to a push producer. ( #8116 )
2020-08-19 08:07:57 -04:00
d294f0e7e1
Remove the unused inlineCallbacks code-paths in the caching code ( #8119 )
2020-08-19 07:09:07 -04:00
76d21d14a0
Separate `get_current_token` into two. ( #8113 )
...
The function is used for two purposes: 1) for subscribers of streams to
get a token they can use to get further updates with, and 2) for
replication to track position of the writers of the stream.
For streams with a single writer the two scenarios produce the same
result, however the situation becomes complicated for streams with
multiple writers. The current `MultiWriterIdGenerator` does not
correctly handle the first case (which is not an issue as its only used
for the `caches` stream which nothing subscribes to outside of
replication).
2020-08-19 10:39:31 +01:00
f40645e60b
Convert events worker database to async/await. ( #8071 )
2020-08-18 16:20:49 -04:00
5cf7c12995
Remove : from allowed client_secret chars ( #8101 )
...
Closes: https://github.com/matrix-org/synapse/issues/6766
Equivalent Sydent PR: https://github.com/matrix-org/sydent/pull/309
I believe it's now time to remove the extra allowed `:` from `client_secret` parameters.
2020-08-18 14:14:27 +01:00
2f4d60a5ba
Iteratively encode JSON responses to avoid blocking the reactor. ( #8013 )
2020-08-18 08:49:59 -04:00
050e20e7ca
Convert some of the general database methods to async ( #8100 )
2020-08-17 12:18:01 -04:00
e04e465b4d
Use the default templates when a custom template file cannot be found ( #8037 )
...
Fixes https://github.com/matrix-org/synapse/issues/6583
2020-08-17 17:05:00 +01:00
ad6190c925
Convert stream database to async/await. ( #8074 )
2020-08-17 07:24:46 -04:00
ac77cdb64e
Add a shadow-banned flag to users. ( #8092 )
2020-08-14 12:37:59 -04:00
dd8f28bd3f
Fix unawaited coroutine error in tests. ( #8072 )
2020-08-13 07:11:39 -04:00
fbe930dad2
Convert the roommember database to async/await. ( #8070 )
2020-08-12 12:14:34 -04:00
5ecc8b5825
Convert devices database to async/await. ( #8069 )
2020-08-12 10:51:42 -04:00
5dd73d029e
Add type hints to handlers.message and events.builder ( #8067 )
2020-08-12 15:05:50 +01:00
d68e10f308
Convert account data, device inbox, and censor events databases to async/await ( #8063 )
2020-08-12 09:29:06 -04:00
a3a59bab7b
Convert appservice, group server, profile and more databases to async ( #8066 )
2020-08-12 09:28:48 -04:00
04faa0bfa9
Convert tags and metrics databases to async/await ( #8062 )
2020-08-11 17:21:20 -04:00
a0acdfa9e9
Converts event_federation and registration databases to async/await ( #8061 )
2020-08-11 17:21:13 -04:00
fcbab08cbd
Add an assertion on prev_events in create_new_client_event ( #8041 )
...
I think this would have caught all the cases in
https://github.com/matrix-org/synapse/issues/7642 - and I think a 500 makes
more sense here than a 403
2020-08-10 12:29:47 +01:00
7f837959ea
Convert directory, e2e_room_keys, end_to_end_keys, monthly_active_users database to async ( #8042 )
2020-08-07 13:36:29 -04:00
f3fe6961b2
Convert additional database stores to async/await ( #8045 )
2020-08-07 12:17:17 -04:00
7620912d84
Add health check endpoint ( #8048 )
2020-08-07 14:21:24 +01:00
2ffd6783c7
Revert #7736 ( #8039 )
2020-08-06 17:15:35 +01:00
fe6cfc80ec
Convert some util functions to async ( #8035 )
2020-08-06 08:39:35 -04:00
d4a7829b12
Convert synapse.api to async/await ( #8031 )
2020-08-06 08:30:06 -04:00
66f24449dd
Improve performance of the register endpoint ( #8009 )
2020-08-06 08:09:55 -04:00
a7bdf98d01
Rename database classes to make some sense ( #8033 )
2020-08-05 21:38:57 +01:00
e19de43eb5
Convert streams to async. ( #8014 )
2020-08-04 07:21:47 -04:00
5d92a1428c
Prevent join->join membership transitions changing member count ( #7977 )
...
`StatsHandler` handles updates to the `current_state_delta_stream`, and updates room stats such as the amount of state events, joined users, etc.
However, it counts every new join membership as a new user entering a room (and that user being in another room), whereas it's possible for a user's membership status to go from join -> join, for instance when they change their per-room profile information.
This PR adds a check for join->join membership transitions, and bails out early, as none of the further checks are necessary at that point.
Due to this bug, membership stats in many rooms have ended up being wildly larger than their true values. I am not sure if we also want to include a migration step which recalculates these statistics (possibly using the `_populate_stats_process_rooms` bg update).
Bug introduced in the initial implementation https://github.com/matrix-org/synapse/pull/4338 .
2020-08-03 21:54:24 +01:00
6812509807
Implement handling of HTTP HEAD requests. ( #7999 )
2020-08-03 08:45:42 -04:00
2a89ce8cd4
Convert the crypto module to async/await. ( #8003 )
2020-08-03 08:29:01 -04:00
faba873d4b
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/add_rate_limiting_to_joins
2020-07-31 15:07:01 +01:00
18de00adb4
Add ratelimiting on joins
2020-07-31 15:06:56 +01:00
c978f6c451
Convert federation client to async/await. ( #7975 )
2020-07-30 08:01:33 -04:00
4cce8ef74e
Convert appservice to async. ( #7973 )
2020-07-30 07:27:39 -04:00
b3a97d6dac
Convert some of the data store to async. ( #7976 )
2020-07-30 07:20:41 -04:00
8dff4a1242
Re-implement unread counts ( #7736 )
2020-07-29 18:26:55 +01:00
3345c166a4
Convert storage layer to async/await. ( #7963 )
2020-07-28 16:09:53 -04:00
e866e3b896
Add an option to disable purge in delete room admin API ( #7964 )
...
Add option ```purge``` to ```POST /_synapse/admin/v1/rooms/<room_id>/delete```
Fixes : #3761
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-07-28 20:08:23 +01:00
3857de2194
Option to allow server admins to join complex rooms ( #7902 )
...
Fixes #7901 .
Signed-off-by: Niklas Tittjung <nik_t.01@web.de>
2020-07-28 13:41:44 +01:00
8553f46498
Convert a synapse.events to async/await. ( #7949 )
2020-07-27 13:40:22 -04:00
5f65e62681
Convert groups and visibility code to async / await. ( #7951 )
2020-07-27 12:32:08 -04:00
8144bc26a7
Convert push to async/await. ( #7948 )
2020-07-27 12:21:34 -04:00
c4268e3da6
Convert tests/rest/admin/test_room.py to unix file endings ( #7953 )
...
Converts tests/rest/admin/test_room.py to have unix file endings after they were accidentally changed in #7613 .
Keeping the same changelog as #7613 as it hasn't gone out in a release yet.
2020-07-27 13:22:52 +01:00
3fc8fdd150
Support oEmbed for media previews. ( #7920 )
...
Fixes previews of Twitter URLs by using their oEmbed endpoint to grab content.
2020-07-27 07:50:44 -04:00
b975fa2e99
Convert state resolution to async/await ( #7942 )
2020-07-24 10:59:51 -04:00
5ea29d7f85
Convert more of the media code to async/await ( #7873 )
2020-07-24 09:39:02 -04:00
6a080ea184
Return an empty body for OPTIONS requests. ( #7886 )
2020-07-24 07:08:07 -04:00
1ec688bf21
Downgrade warning on client disconnect to INFO ( #7928 )
...
Clients disconnecting before we finish processing the request happens from time
to time. We don't need to yell about it
2020-07-24 09:55:47 +01:00
68cd935826
Convert the federation agent and related code to async/await. ( #7874 )
2020-07-23 07:05:57 -04:00
cc9bb3dc3f
Convert the message handler to async/await. ( #7884 )
2020-07-22 12:29:15 -04:00
6b3ac3b8cd
Convert device handler to async/await ( #7871 )
2020-07-17 07:09:25 -04:00
6fca1b3506
Convert _base, profile, and _receipts handlers to async/await ( #7860 )
2020-07-17 07:08:30 -04:00
fff483ea96
Add admin endpoint to get members in a room. ( #7842 )
2020-07-16 16:43:23 -04:00
649a7ead5c
Add ability to run multiple pusher instances ( #7855 )
...
This reuses the same scheme as federation sender sharding
2020-07-16 14:06:28 +01:00
97e1159ac1
Merge branch 'erikj/faster_typing' of github.com:matrix-org/synapse into develop
2020-07-15 16:54:30 +01:00
8c7d0f163d
Allow accounts to be re-activated from the admin APIs. ( #7847 )
2020-07-15 11:00:21 -04:00
9006e125af
Fix tests
2020-07-15 15:47:27 +01:00
f13061d515
Fix client reader sharding tests ( #7853 )
...
* Fix client reader sharding tests
* Newsfile
* Fix typing
* Update changelog.d/7853.misc
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Move mocking of http_client to tests
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2020-07-15 15:27:35 +01:00
b11450dedc
Convert E2E key and room key handlers to async/await. ( #7851 )
2020-07-15 08:48:58 -04:00
111e70d75c
Return the proper 403 Forbidden error during errors with JWT logins. ( #7844 )
2020-07-15 07:10:21 -04:00
491f0dab1b
Add delete room admin endpoint ( #7613 )
...
The Delete Room admin API allows server admins to remove rooms from server
and block these rooms.
`DELETE /_synapse/admin/v1/rooms/<room_id>`
It is a combination and improvement of "[Shutdown room](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/shutdown_room.md )" and "[Purge room](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/purge_room.md )" API.
Fixes : #6425
It also fixes a bug in [synapse/storage/data_stores/main/room.py](synapse/storage/data_stores/main/room.py) in ` get_room_with_stats`.
It should return `None` if the room is unknown. But it returns an `IndexError`.
901b1fa561/synapse/storage/data_stores/main/room.py (L99-L105)#5575
- https://github.com/Awesome-Technologies/synapse-admin/issues/17
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-07-14 12:36:23 +01:00
77d2c05410
Add the option to validate the `iss` and `aud` claims for JWT logins. ( #7827 )
2020-07-14 07:16:43 -04:00
457096e6df
Support handling registration requests across multiple client readers. ( #7830 )
2020-07-13 13:31:46 -04:00
66a4af8d96
Do not use canonicaljson to magically handle decoding bytes from JSON. ( #7802 )
2020-07-10 14:30:08 -04:00
f299441cc6
Add ability to shard the federation sender ( #7798 )
2020-07-10 18:26:36 +01:00
38e1fac886
Fix some spelling mistakes / typos. ( #7811 )
2020-07-09 09:52:58 -04:00
67593b1728
Add `HomeServer.signing_key` property ( #7805 )
...
... instead of duplicating `config.signing_key[0]` everywhere
2020-07-08 17:51:56 +01:00
76dbd7b8d6
Stop populating unused table `local_invites`. ( #7793 )
...
This table is no longer used, so we may as well stop populating it. Removing it
would prevent people rolling back to older releases of Synapse, so that can
happen in a future release.
2020-07-07 14:20:40 +01:00
6d687ebba1
Convert the appservice handler to async/await. ( #7775 )
2020-07-06 07:40:35 -04:00
57feeab364
Don't ignore `set_tweak` actions with no explicit `value`. ( #7766 )
...
* Fix spec compliance; tweaks without values are valid
(default to True, which is only concretely specified for
`highlight`, but it seems only reasonable to generalise)
* Changelog for 7766.
* Add documentation to `tweaks_for_actions`
May as well tidy up when I'm here.
* Add a test for `tweaks_for_actions`
2020-07-06 11:43:41 +01:00
62b1ce8539
isort 5 compatibility ( #7786 )
...
The CI appears to use the latest version of isort, which is a problem when isort gets a major version bump. Rather than try to pin the version, I've done the necessary to make isort5 happy with synapse.
2020-07-05 16:32:02 +01:00
5cdca53aa0
Merge different Resource implementation classes ( #7732 )
2020-07-03 19:02:19 +01:00
21a212f8e5
Fix inconsistent handling of upper and lower cases of email addresses. ( #7021 )
...
fixes #7016
2020-07-03 14:03:13 +01:00
e5808c4cfb
Hack to add push priority to push notifications ( #7765 )
...
* Remove obsolete comment about ancient temporary code
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Implement hack to set push priority
based on whether the tweaks indicate the event might cause
effects.
* Changelog for 7765
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Antilint
* Add tests for push priority
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Update synapse/push/httppusher.py
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Antilint
* Remove needless invites from tests.
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2020-07-01 17:02:31 +01:00
74d3e177f0
Back out MSC2625 implementation ( #7761 )
2020-07-01 11:08:25 +01:00
71cccf1593
Additional configuration options for auto-join rooms ( #7763 )
2020-06-30 15:41:36 -04:00
0e0a2817a2
Yield during large v2 state res. ( #7735 )
...
State res v2 across large data sets can be very CPU intensive, and if
all the relevant events are in the cache the algorithm will run from
start to finish within a single reactor tick. This can result in
blocking the reactor tick for several seconds, which can have major
repercussions on other requests.
To fix this we simply add the occaisonal `sleep(0)` during iterations to
yield execution until the next reactor tick. The aim is to only do this
for large data sets so that we don't impact otherwise quick resolutions.=
2020-06-24 18:48:18 +01:00
6920e58136
add org.matrix.login.jwt so that m.login.jwt can be deprecated ( #7675 )
2020-06-24 10:23:55 +01:00
95e41f368b
Allow local media to be marked as safe from being quarantined. ( #7718 )
2020-06-22 08:04:14 -04:00
3630825612
Convert the typing handler to async/await. ( #7679 )
2020-06-17 10:37:59 -04:00
434716e1d3
Fetch from the r0 media path instead of the unspecced v1. ( #7714 )
2020-06-17 08:36:46 -04:00
46613aaf79
Implement unread counter (MSC2625) ( #7673 )
...
Implementation of https://github.com/matrix-org/matrix-doc/pull/2625
2020-06-17 10:58:32 +01:00
ac51bd581a
Include a user agent in federation requests. ( #7677 )
2020-06-16 10:43:29 -04:00
a3f11567d9
Replace all remaining six usage with native Python 3 equivalents ( #7704 )
2020-06-16 08:51:47 -04:00
03619324fc
Create a ListenerConfig object ( #7681 )
...
This ended up being a bit more invasive than I'd hoped for (not helped by
generic_worker duplicating some of the code from homeserver), but hopefully
it's an improvement.
The idea is that, rather than storing unstructured `dict`s in the config for
the listener configurations, we instead parse it into a structured
`ListenerConfig` object.
2020-06-16 12:44:07 +01:00
789606577a
Merge branch 'master' into develop
2020-06-16 10:36:29 +01:00
3e6b5bba71
Wrap register_device coroutine in an ensureDeferred ( #7684 )
...
Fixes https://github.com/matrix-org/synapse/issues/7683
Broke in: #7649
We had a `yield` acting on a coroutine. To be fair this one is a bit difficult to notice as there's a function in the middle that just passes the coroutine along.
2020-06-16 10:13:59 +01:00
cc32fa7358
Ensure the body is a string before comparing push rules. ( #7701 )
2020-06-15 16:20:34 -04:00
2b2344652b
Ensure etag is a string for GET room_keys/version response ( #7691 )
2020-06-15 13:42:44 -04:00
6efb2b0ad4
Merge branch 'develop' into babolivier/mark_unread
2020-06-15 16:37:52 +01:00
c2b4621630
Merge branch 'develop' into babolivier/mark_unread
2020-06-15 16:37:08 +01:00
7d2532be36
Discard RDATA from already seen positions. ( #7648 )
2020-06-15 08:44:54 -04:00
7e80c84902
Lint
2020-06-12 11:31:11 +01:00
Brendan Abolivier