Commit Graph

12348 Commits

Author SHA1 Message Date
Andrew Morgan 920dd1083e 1.20.1 2020-09-24 16:25:33 +01:00
Andrew Morgan 3f4a2a7064
Hotfix: disable autoescape by default when rendering Jinja2 templates (#8394)
#8037 changed the default `autoescape` option when rendering Jinja2 templates from `False` to `True`. This caused some bugs, noticeably around redirect URLs being escaped in SAML2 auth confirmation templates, causing those URLs to break for users.

This change returns the previous behaviour as it stood. We may want to look at each template individually and see whether autoescaping is a good idea at some point, but for now lets just fix the breakage.
2020-09-24 16:24:08 +01:00
Andrew Morgan 55bb5fda33 1.20.0 2020-09-22 15:18:31 +01:00
Patrick Cloke d5f7182ba1 1.20.0rc5 2020-09-18 10:56:50 -04:00
Patrick Cloke 88e67d1adb 1.19.3
Synapse 1.19.3 (2020-09-18)
 ===========================
 
 Bugfixes
 --------
 
 - Partially mitigate bug where newly joined servers couldn't get past
 events in a room when there is a malformed event.
 ([\#8350](https://github.com/matrix-org/synapse/issues/8350))
 -----BEGIN PGP SIGNATURE-----
 
 iQJHBAABCAAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAl9kxN4THGFuZHJld0Bh
 bW9yZ2FuLnh5egAKCRCIhIgNLv5f9LznEACSm7ZL0GjVDcDjGEu+QjKIi3KUiFq3
 i8EXEZWT3w5NNNER0Jey5BHxaBKtnPsvon0k3U4bp5KKOA6BGa9L4NDGZYJa3p0k
 A1uc3DCgGG1aVazpIzfhWRA0va13T+zRKdz52GdjzksH0WGl6w3UoJhloWOmHyxz
 K4UxGwOqJMSBxseBHOFcXdomPtsNYUqsrOcZYWjh3hWN0GMV6H+WrcbKVYl49V0F
 6aVHuaxit35iAGYER41mnTA34ZNuC1Qkp83mAaE+Z8i39qBWPMRErUNAyZQ/mCKz
 QrF98p7F2kFgSzDagtZiUPZj3w3XwfZf05bqnyd9cxBEQdIYFLAL0lokEXcoY1os
 q7gKwGuwicuvYEQrt+gSFlkoUaSvy7/b4cmFqvT0NGnBNZoYl6MX4MXP2CNHuaFk
 yljZoTecKEmhInY10S4uy+Hp0JNHuZWEOYGKy7CrQaqRo8MhBLk5LWBPjUOayPLP
 uvDNv6MShQ8SpCiKvsoCBiX9G3LEo1yHPo5oX57nOr+IHawH0PPkXVKL3b+K+7s0
 eXah/9n/wQYO5K+ReqTFd9ZCegN0/hW/NAT9aX/gEYASkS4ANvGALWwXbZSOG5IG
 2glXiewbJSOaVutPRpIVI3XGDSdm3/8VpO+cAKotZ+pR1V6nsxtVwLRmAhxqhNFD
 3AULCLMt2yKzDw==
 =a9VC
 -----END PGP SIGNATURE-----

Merge tag 'v1.19.3' into release-v1.20.0

1.19.3

Synapse 1.19.3 (2020-09-18)
===========================

Bugfixes
--------

- Partially mitigate bug where newly joined servers couldn't get past
events in a room when there is a malformed event.
([\#8350](https://github.com/matrix-org/synapse/issues/8350))
2020-09-18 10:53:01 -04:00
Andrew Morgan 5b70acb44c 1.19.3 2020-09-18 15:00:07 +01:00
Andrew Morgan 27c1abc7b8
Use _check_sigs_and_hash_and_fetch to validate backfill requests (#8350)
This is a bit of a hack, as `_check_sigs_and_hash_and_fetch` is intended
for attempting to pull an event from the database/(re)pull it from the
server that originally sent the event if checking the signature of the
event fails.

During backfill we *know* that we won't have the event in our database,
however it is still useful to be able to query the original sending
server as the server we're backfilling from may be acting maliciously.

The main benefit and reason for this change however is that
`_check_sigs_and_hash_and_fetch` will drop an event during backfill if
it cannot be successfully validated, whereas the current code will
simply fail the backfill request - resulting in the client's /messages
request silently being dropped.

This is a quick patch to fix backfilling rooms that contain malformed
events. A better implementation in planned in future.
2020-09-18 14:51:11 +01:00
Erik Johnston 43f2b67e4d
Intelligently select extremities used in backfill. (#8349)
Instead of just using the most recent extremities let's pick the
ones that will give us results that the pagination request cares about,
i.e. pick extremities only if they have a smaller depth than the
pagination token.

This is useful when we fail to backfill an extremity, as we no longer
get stuck requesting that same extremity repeatedly.
2020-09-18 14:25:52 +01:00
Patrick Cloke 9db4c1b175
Add flags to /versions about whether new rooms are encrypted by default. (#8343) 2020-09-18 07:56:20 -04:00
Erik Johnston 14b5b48a22
Fix ratelimiting for federation `/send` requests. (#8342)
c.f. #8295 for rationale
2020-09-18 10:49:29 +01:00
Patrick Cloke 7141057e85 1.20.0rc4 2020-09-16 08:54:30 -04:00
Erik Johnston 5ffd68dca1 1.19.2 2020-09-16 13:37:03 +01:00
Erik Johnston 9c8ef134fe Merge branch 'erikj/fix_origin_check' into release-v1.20.0 2020-09-16 12:42:42 +01:00
Erik Johnston c570f24acc Don't assume that an event has an origin field
This fixes #8319.
2020-09-16 11:56:23 +01:00
Patrick Cloke 2832ef5bb7 1.20.0rc3 2020-09-11 08:14:15 -04:00
Patrick Cloke b86764662b
Fix the exception that is raised when invalid JSON is encountered. (#8291) 2020-09-10 14:55:25 -04:00
Richard van der Hoff 536f4a2482 1.20.0rc2 2020-09-09 17:08:33 +01:00
Brendan Abolivier d4daff9b59
Fix /notifications and pushers misbehaving because of unread counts (#8280) 2020-09-08 15:26:06 +01:00
Richard van der Hoff 525efab612 1.20.0rc1 2020-09-08 12:58:37 +01:00
Brendan Abolivier d8762cc116
Only add rows to the push actions table if the event notifies or should be marked unread (#8274) 2020-09-07 16:56:27 +01:00
Richard van der Hoff ef2804d27c
Avoid table-scanning users at startup (#8271)
This takes about 10 seconds in the best case; often more.
2020-09-07 16:48:52 +01:00
Brendan Abolivier a55e2707d7
Fix unread count failing on NULL values (#8270)
Fix unread counts making sync fail if the value of the `unread_count`
column in `event_push_summary` is `None`.
2020-09-07 15:15:06 +01:00
Richard van der Hoff 0dae7d80bf
Add more logging to debug slow startup (#8264)
I'm hoping this will provide some pointers for debugging
https://github.com/matrix-org/synapse/issues/7968.
2020-09-07 13:36:02 +01:00
Richard van der Hoff 96312536f2
Refuse to upgrade database on worker processes (#8266) 2020-09-07 13:04:10 +01:00
Richard van der Hoff f25af1f9c7
Add cross-signing sigs to the `keys` object (#8234)
All the callers want this info in the same place, so let's reduce the
duplication by doing it here.
2020-09-04 15:06:05 +01:00
Brendan Abolivier 041ee971c9
Unread counts fixes (#8254)
* Fixup `ALTER TABLE` database queries

Make the new columns nullable, because doing otherwise can wedge a
server with a big database, as setting a default value rewrites the
table.

* Switch back to using the notifications count in the push badge

Clients are likely to be confused if we send a push but the badge count
is the unread messages one, and not the notifications one.

* Changelog
2020-09-04 14:14:22 +01:00
Patrick Cloke db7de4d182
Fix a regression from calling read_templates. (#8252)
Regressed in #8037.
2020-09-04 09:10:33 -04:00
reivilibre e351298444
Fix type signature in simple_select_one_onecol and friends (#8241)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2020-09-04 12:02:29 +01:00
Patrick Cloke c619253db8
Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
Brendan Abolivier 9f8abdcc38
Revert "Add experimental support for sharding event persister. (#8170)" (#8242)
* Revert "Add experimental support for sharding event persister. (#8170)"

This reverts commit 82c1ee1c22.

* Changelog
2020-09-04 10:19:42 +01:00
Erik Johnston be16ee59a8
Add type hints to more handlers (#8244) 2020-09-03 22:02:29 +01:00
reivilibre 4535e849d7
Remove obsolete order field in `send_new_transaction` (#8245)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2020-09-03 19:23:07 +01:00
Richard van der Hoff f97f9485ee
Split fetching device keys and signatures into two transactions (#8233)
I think this is simpler (and moves stuff out of the db threads)
2020-09-03 18:27:26 +01:00
Erik Johnston 208e1d3eb3
Fix typing for `@cached` wrapped functions (#8240)
This requires adding a mypy plugin to fiddle with the type signatures a bit.
2020-09-03 15:38:32 +01:00
Patrick Cloke 2aa127c207
Revert pinning of setuptools (#8239) 2020-09-03 09:45:36 -04:00
Erik Johnston 5bfc79486d
Fix typing for SyncHandler (#8237) 2020-09-03 12:54:10 +01:00
Richard van der Hoff 6f6f371a87
wrap `_get_e2e_device_keys_and_signatures_txn` in a non-txn method (#8231)
We have three things which all call `_get_e2e_device_keys_and_signatures_txn`
with their own `runInteraction`. Factor out the common code.
2020-09-03 11:50:49 +01:00
Patrick Cloke c8758cb72f
Add an overload for simple_select_one_onecol_txn. (#8235) 2020-09-02 15:03:12 -04:00
Patrick Cloke 912e024913
Convert runInteraction to async/await (#8156) 2020-09-02 13:11:02 -04:00
Erik Johnston 112266eafd
Add StreamStore to mypy (#8232) 2020-09-02 17:52:38 +01:00
Brendan Abolivier 5a1dd297c3
Re-implement unread counts (again) (#8059) 2020-09-02 17:19:37 +01:00
Richard van der Hoff 0d4f614fda
Refactor `_get_e2e_device_keys_for_federation_query_txn` (#8225)
We can use the existing `_get_e2e_device_keys_and_signatures_txn` instead of
creating our own txn function
2020-09-02 15:53:26 +01:00
Erik Johnston 82c1ee1c22
Add experimental support for sharding event persister. (#8170)
This is *not* ready for production yet. Caveats:

1. We should write some tests...
2. The stream token that we use for events can get stalled at the minimum position of all writers. This means that new events may not be processed and e.g. sent down sync streams if a writer isn't writing or is slow.
2020-09-02 15:48:37 +01:00
Will Hunt b257c788c0
Add /user/{user_id}/shared_rooms/ api (#7785)
* Add shared_rooms api

* Add changelog

* Add .

* Wrap response in {"rooms": }

* linting

* Add unstable_features key

* Remove options from isort that aren't part of 5.x

`-y` and `-rc` are now default behaviour and no longer exist.

`dont-skip` is no longer required

https://timothycrosley.github.io/isort/CHANGELOG/#500-penny-july-4-2020

* Update imports to make isort happy

* Add changelog

* Update tox.ini file with correct invocation

* fix linting again for isort

* Vendor prefix unstable API

* Fix to match spec

* import Codes

* import Codes

* Use FORBIDDEN

* Update changelog.d/7785.feature

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

* Implement get_shared_rooms_for_users

* a comma

* trailing whitespace

* Handle the easy feedback

* Switch to using runInteraction

* Add tests

* Feedback

* Seperate unstable endpoint from v2

* Add upgrade node

* a line

* Fix style by adding a blank line at EOF.

* Update synapse/storage/databases/main/user_directory.py

Co-authored-by: Tulir Asokan <tulir@maunium.net>

* Update synapse/storage/databases/main/user_directory.py

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

* Update UPGRADE.rst

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

* Fix UPGRADE/CHANGELOG unstable paths

unstable unstable unstable

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: Tulir Asokan <tulir@maunium.net>

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Tulir Asokan <tulir@maunium.net>
2020-09-02 13:18:40 +01:00
Patrick Cloke 9356656e67
Do not try to store invalid data in the stats table (#8226) 2020-09-02 07:59:39 -04:00
Patrick Cloke d250521cf5
Convert the main methods run by the reactor to async. (#8213) 2020-09-02 07:44:50 -04:00
Richard van der Hoff abeab964d5
Make _get_e2e_device_keys_and_signatures_txn return an attrs (#8224)
this makes it a bit clearer what's going on.
2020-09-02 11:47:26 +01:00
Patrick Cloke b939251c37
Fix errors when updating the user directory with invalid data (#8223) 2020-09-01 13:02:41 -04:00
Patrick Cloke 37db6252b7
Convert additional databases to async/await part 3 (#8201) 2020-09-01 11:04:17 -04:00
Patrick Cloke 7d103a594e
Convert appservice code to async/await. (#8207) 2020-09-01 11:03:49 -04:00