Commit Graph

11845 Commits

Author SHA1 Message Date
Richard van der Hoff e6027562e2 remove `builtins.buffer` code from storage code
this is no longer needed on python 3
2020-05-15 19:37:41 +01:00
Richard van der Hoff 91f51c611c remove redundant `__func__`
this is a no-op under python 3
2020-05-15 19:37:41 +01:00
Richard van der Hoff 65902e08c3 remove to_ascii
this is a no-op on python 3.
2020-05-15 19:12:03 +01:00
Richard van der Hoff 08fa96f030 Remove `exception_to_unicode`
this is a no-op on python 3.
2020-05-15 19:07:24 +01:00
Patrick Cloke e9f3de0bab
Update the room member handler to use async/await. (#7507) 2020-05-15 09:32:13 -04:00
Patrick Cloke 08bc80ef09
Implement room version 6 (MSC2240). (#7506) 2020-05-15 09:30:10 -04:00
Andrew Morgan 02d97fc3ba
Ignore incoming presence updates when presence is disabled (#7508) 2020-05-15 11:44:00 +01:00
Patrick Cloke 56b66db78a
Strictly enforce canonicaljson requirements in a new room version (#7381) 2020-05-14 13:24:01 -04:00
Patrick Cloke fef3ff5cc4
Enforce MSC2209: auth rules for notifications in power level event (#7502)
In a new room version, the "notifications" key of power level events are
subject to restricted auth rules.
2020-05-14 12:38:17 -04:00
Andrew Morgan 5611644519
Workaround for failure to wrap reason in Failure (#7473) 2020-05-14 17:07:24 +01:00
Richard van der Hoff eafd103fc7
Fix b'GET' in prometheus metrics (#7503) 2020-05-14 17:01:34 +01:00
Andrew Morgan 225c165087
Allow expired accounts to logout (#7443) 2020-05-14 16:32:49 +01:00
Erik Johnston 4734a7bbe4
Move EventStream handling into default ReplicationDataHandler (#7493)
This is so that the logic can happen on both master and workers when we move event persistence out.
2020-05-14 14:01:39 +01:00
Erik Johnston 1de36407d1
Add `instance_map` config and route replication calls (#7495) 2020-05-14 14:00:58 +01:00
Richard van der Hoff dede23ff1e Synapse 1.13.0rc2 (2020-05-14)
==============================
 
 Bugfixes
 --------
 
 - Fix a long-standing bug which could cause messages not to be sent over federation, when state events with state keys matching user IDs (such as custom user statuses) were received. ([\#7376](https://github.com/matrix-org/synapse/issues/7376))
 - Restore compatibility with non-compliant clients during the user interactive authentication process, fixing a problem introduced in v1.13.0rc1. ([\#7483](https://github.com/matrix-org/synapse/issues/7483))
 
 Internal Changes
 ----------------
 
 - Fix linting errors in new version of Flake8. ([\#7470](https://github.com/matrix-org/synapse/issues/7470))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEv27Axt/F4vrTL/8QOSor00I9eP8FAl69IQ8ACgkQOSor00I9
 eP87lAf8DK+v6cs2U0BoD5opzQ7ZazJT6JYTmnMBaTzHU6Wx20V2ttkF7Vpwm3WU
 Zsz0048tdYtHFyYBQ1kF5RNIBBJwV8SA/QUcPkR7FVpwZMLR2q4aJn0EE7kC9OMf
 tYsmdbHeBdyfLXpXzazxWlgHquLyEIt52ykAcCphjx/Jl2fAExFEhtfsxpECoJ2f
 8Dqhjg3WFjd6QWU6AFkElbwHUYCdIWdJOcsC8N1p8OvBmDz5QXv/RlYipHE00Cpx
 QQQOgEjdRc6dlz2mbetMklnfII3p2kO9bzNdmEpOzT0Zt7nFaGdntW4I1QA0yJfa
 gows9bYMzhqYk7YSiyTYOZ4qyavVtw==
 =N/zZ
 -----END PGP SIGNATURE-----

Merge tag 'v1.13.0rc2' into develop

Synapse 1.13.0rc2 (2020-05-14)
==============================

Bugfixes
--------

- Fix a long-standing bug which could cause messages not to be sent over federation, when state events with state keys matching user IDs (such as custom user statuses) were received. ([\#7376](https://github.com/matrix-org/synapse/issues/7376))
- Restore compatibility with non-compliant clients during the user interactive authentication process, fixing a problem introduced in v1.13.0rc1. ([\#7483](https://github.com/matrix-org/synapse/issues/7483))

Internal Changes
----------------

- Fix linting errors in new version of Flake8. ([\#7470](https://github.com/matrix-org/synapse/issues/7470))
2020-05-14 11:46:38 +01:00
Richard van der Hoff 623abbeb8d 1.13.0rc2 2020-05-14 11:01:24 +01:00
Richard van der Hoff a0e063387d
Stop `get_joined_users` corruption from custom statuses (#7376)
Fix a bug where the `get_joined_users` cache could be corrupted by custom
status events (or other state events with a state_key matching the user ID).

The bug was introduced by #2229, but has largely gone unnoticed since then.

Fixes #7099, #7373.
2020-05-14 10:07:54 +01:00
Patrick Cloke 5d64fefd6c
Do not validate that the client dict is stable during UI Auth. (#7483)
This backs out some of the validation for the client dictionary and logs if
this changes during a user interactive authentication session instead.
2020-05-13 14:26:44 -04:00
Erik Johnston 1124111a12
Allow censoring of events to happen on workers. (#7492)
This is safe as we can now write to cache invalidation stream on workers, and is required for when we move event persistence off master.
2020-05-13 17:15:40 +01:00
Paul Tötterman 46cb2550bb
Fix copypasted comment (#7477)
Signed-off-by: Paul Tötterman <paul.totterman@iki.fi>
2020-05-13 16:55:43 +01:00
Erik Johnston 00ba9c48bf Spelling 2020-05-13 13:38:51 +01:00
Erik Johnston 782e4e64df
Shuffle persist event data store functions. (#7440)
The aim here is to get to a stage where we have a `PersistEventStore` that holds all the write methods used during event persistence, so that we can take that class out of the `DataStore` mixin and instansiate it separately. This will allow us to instansiate it on processes other than master, while also ensuring it is only available on processes that are configured to write to events stream.

This is a bit of an architectural change, where we end up with multiple classes per data store (rather than one per data store we have now). We end up having:

1. Storage classes that provide high level APIs that can talk to multiple data stores.
2. Data store modules that consist of classes that must point at the same database instance.
3. Classes in a data store that can be instantiated on processes depending on config.
2020-05-13 13:38:22 +01:00
Patrick Cloke edd3b0747c
Fix new flake8 errors (#7489)
This is a cherry-pick of 1a1da60ad2 (#7470)
to the release-v1.13.0 branch.
2020-05-13 08:24:50 -04:00
Erik Johnston 7ee24c5674
Have all instances correctly respond to REPLICATE command. (#7475)
Before all streams were only written to from master, so only master needed to respond to `REPLICATE` commands.

Before all instances wrote to the cache invalidation stream, but didn't respond to `REPLICATE`. This was a bug, which could lead to missed rows from cache invalidation stream if an instance is restarted, however all the caches would be empty in that case so it wasn't a problem.
2020-05-13 10:27:02 +01:00
Erik Johnston 8ca79613e6
Fix Redis reconnection logic (#7482)
Proactively send out `POSITION` commands (as if we had just received a `REPLICATE`) when we connect to Redis. This is important as other instances won't notice we've connected to issue a `REPLICATE` command (unlike for direct TCP connections). This is only currently an issue if master process reconnects without restarting (if it restarts then it won't have written anything and so other instances probably won't have missed anything).
2020-05-13 09:57:15 +01:00
Erik Johnston 1a1da60ad2
Fix new flake8 errors (#7470) 2020-05-12 11:20:48 +01:00
Patrick Cloke 8c8858e124
Convert federation handler to async/await. (#7459) 2020-05-11 15:12:46 -04:00
Patrick Cloke be309d99cf
Convert search code to async/await. (#7460) 2020-05-11 15:12:39 -04:00
Amber Brown 7cb8b4bc67
Allow configuration of Synapse's cache without using synctl or environment variables (#6391) 2020-05-11 18:45:23 +01:00
Andrew Morgan a8580c5f19
Remove unused store method get_hosts_in_room (#7448) 2020-05-11 16:55:57 +01:00
Andrew Morgan 5cf758cdd6 Merge branch 'release-v1.13.0' into develop
* release-v1.13.0:
  Don't UPGRADE database rows
  RST indenting
  Put rollback instructions in upgrade notes
  Fix changelog typo
  Oh yeah, RST
  Absolute URL it is then
  Fix upgrade notes link
  Provide summary of upgrade issues in changelog. Fix )
  Move next version notes from changelog to upgrade notes
  Changelog fixes
  1.13.0rc1
  Documentation on setting up redis (#7446)
  Rework UI Auth session validation for registration (#7455)
  Fix errors from malformed log line (#7454)
  Drop support for redis.dbid (#7450)
2020-05-11 16:46:33 +01:00
Andrew Morgan 20ffaa7209 1.13.0rc1 2020-05-11 14:54:38 +01:00
Patrick Cloke 0ad6d28b0d
Rework UI Auth session validation for registration (#7455)
Be less strict about validation of UI authentication sessions during
registration to match client expecations.
2020-05-08 16:08:58 -04:00
Andrew Morgan 67feea8044
Extend spam checker to allow for multiple modules (#7435) 2020-05-08 19:25:48 +01:00
Quentin Gliech 616af44137
Implement OpenID Connect-based login (#7256) 2020-05-08 08:30:40 -04:00
Manuel Stahl a4a5ec4096
Add room details admin endpoint (#7317) 2020-05-07 15:33:07 -04:00
Richard van der Hoff aa5aa6f96a
Fix errors from malformed log line (#7454) 2020-05-07 19:51:38 +01:00
Richard van der Hoff da9b2db3af
Drop support for redis.dbid (#7450)
Since we only use pubsub, the dbid is irrelevant.
2020-05-07 16:46:15 +01:00
Brendan Abolivier 5bb26b7c4f Merge branch 'release-v1.13.0' into develop 2020-05-07 17:31:19 +02:00
Patrick Cloke 9e0384dd3f
Fixes typo (bellow -> below) (#7449) 2020-05-07 09:31:06 -04:00
Patrick Cloke 22246919e3
Add more type hints to SAML handler. (#7445) 2020-05-07 09:30:45 -04:00
Erik Johnston d7983b63a6
Support any process writing to cache invalidation stream. (#7436) 2020-05-07 13:51:08 +01:00
Brendan Abolivier d9b8d27494
Add a configuration setting for the dummy event threshold (#7422)
Add dummy_events_threshold which allows configuring the number of forward extremities a room needs for Synapse to send forward extremities in it.
2020-05-07 10:35:23 +01:00
Patrick Cloke d7c2df2fa3
Improve per-block CPU and DB usage metrics (#7426) 2020-05-06 16:43:39 -04:00
Andrew Morgan 4162c39dcf
Port group attestation renewal slow down from matrix-org-hotfixes (#7442) 2020-05-06 20:21:38 +01:00
Richard van der Hoff e053c86a96
Make redis go faster with hiredis (#7439)
For the record, the reason we need this is as follows:

each RDATA command comes down the redis pipe as a subscription message. txredisapi as written needs at least three reactor ticks to read each subscription message from the tcp buffer. Hence, once the process gets loaded, it starts getting behind, and eventually redis knifes the connection. it then takes ages for the master to work its way through the backlog, before it reconnects again, during which any commands from any workers are dropped.
2020-05-06 17:36:46 +01:00
Richard van der Hoff 62ee862119 Merge branch 'release-v1.13.0' into develop 2020-05-06 15:56:03 +01:00
Andrew Morgan aee9130a83
Stop Auth methods from polling the config on every req. (#7420) 2020-05-06 15:54:58 +01:00
Richard van der Hoff fa0b2bd28d
Merge pull request #7428 from matrix-org/rav/cross_signing_keys_cache
Make get_e2e_cross_signing_key delegate to get_e2e_cross_signing_keys_bulk
2020-05-06 12:00:01 +01:00
Richard van der Hoff 16b67c404d Make get_e2e_cross_signing_key delegate to get_e2e_cross_signing_keys_bulk
... mostly because the latter has a cache.
2020-05-06 11:59:19 +01:00