Merge branch 'release-v1.55' into develop

2022-03-16 12:22:07 -04:00 · 2022-03-16 12:22:07 -04:00 · 3f7cfbc9e5
parent 6121056740 f70afbd565
commit 3f7cfbc9e5
67 changed files with 126 additions and 66 deletions
--- a/CHANGES.md
+++ b/CHANGES.md
@ -1,3 +1,88 @@
 Synapse 1.55.0rc1 (2022-03-15)
 ==============================
 This release removes a workaround introduced in Synapse 1.50.0 for Mjolnir compatibility. **This breaks compatibility with Mjolnir 1.3.1 and earlier. ([\#11700](https://github.com/matrix-org/synapse/issues/11700))**; Mjolnir users should upgrade Mjolnir before upgrading Synapse to this version.
 Features
 --------
 - Add third-party rules callbacks `check_can_shutdown_room` and `check_can_deactivate_user`. ([\#12028](https://github.com/matrix-org/synapse/issues/12028))
 - Improve performance of logging in for large accounts. ([\#12132](https://github.com/matrix-org/synapse/issues/12132))
 - Add experimental env var `SYNAPSE_ASYNC_IO_REACTOR` that causes Synapse to use the asyncio reactor for Twisted. ([\#12135](https://github.com/matrix-org/synapse/issues/12135))
 - Support the stable identifiers from [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440): threads. ([\#12151](https://github.com/matrix-org/synapse/issues/12151))
 - Add a new Jinja2 template filter to extract the local part of an email address. ([\#12212](https://github.com/matrix-org/synapse/issues/12212))
 Bugfixes
 --------
 - Use the proper serialization format for bundled thread aggregations. The bug has existed since Synapse v1.48.0. ([\#12090](https://github.com/matrix-org/synapse/issues/12090))
 - Fix a long-standing bug when redacting events with relations. ([\#12113](https://github.com/matrix-org/synapse/issues/12113), [\#12121](https://github.com/matrix-org/synapse/issues/12121), [\#12130](https://github.com/matrix-org/synapse/issues/12130), [\#12189](https://github.com/matrix-org/synapse/issues/12189))
 - Fix a bug introduced in Synapse 1.7.2 whereby background updates are never run with the default background batch size. ([\#12157](https://github.com/matrix-org/synapse/issues/12157))
 - Fix a bug where non-standard information was returned from the `/hierarchy` API. Introduced in Synapse v1.41.0. ([\#12175](https://github.com/matrix-org/synapse/issues/12175))
 - Fix a bug introduced in Synapse 1.54.0 that broke background updates on sqlite homeservers while search was disabled. ([\#12215](https://github.com/matrix-org/synapse/issues/12215))
 - Fix a long-standing bug when a `filter` argument with `event_fields` which did not include the `unsigned` field could result in a 500 error on `/sync`. ([\#12234](https://github.com/matrix-org/synapse/issues/12234))
 Improved Documentation
 ----------------------
 - Fix complexity checking config example in [Resource Constrained Devices](https://matrix-org.github.io/synapse/v1.54/other/running_synapse_on_single_board_computers.html) docs page. ([\#11998](https://github.com/matrix-org/synapse/issues/11998))
 - Improve documentation for demo scripts. ([\#12143](https://github.com/matrix-org/synapse/issues/12143))
 - Updates to the Room DAG concepts development document. ([\#12179](https://github.com/matrix-org/synapse/issues/12179))
 - Document that the `typing`, `to_device`, `account_data`, `receipts`, and `presence` stream writer can only be used on a single worker. ([\#12196](https://github.com/matrix-org/synapse/issues/12196))
 - Document that contributors can sign off privately by email. ([\#12204](https://github.com/matrix-org/synapse/issues/12204))
 Deprecations and Removals
 -------------------------
 - **Remove workaround introduced in Synapse 1.50.0 for Mjolnir compatibility. Breaks compatibility with Mjolnir 1.3.1 and earlier. ([\#11700](https://github.com/matrix-org/synapse/issues/11700))**
 - Remove backwards compatibilty with pagination tokens from the `/relations` and `/aggregations` endpoints generated from Synapse < v1.52.0. ([\#12138](https://github.com/matrix-org/synapse/issues/12138))
 - The groups/communities feature in Synapse has been deprecated. ([\#12200](https://github.com/matrix-org/synapse/issues/12200))
 Internal Changes
 ----------------
 - Simplify the `ApplicationService` class' set of public methods related to interest checking. ([\#11915](https://github.com/matrix-org/synapse/issues/11915))
 - Add config settings for background update parameters. ([\#11980](https://github.com/matrix-org/synapse/issues/11980))
 - Correct type hints for txredis. ([\#12042](https://github.com/matrix-org/synapse/issues/12042))
 - Limit the size of `aggregation_key` on annotations. ([\#12101](https://github.com/matrix-org/synapse/issues/12101))
 - Add type hints to tests files. ([\#12108](https://github.com/matrix-org/synapse/issues/12108), [\#12146](https://github.com/matrix-org/synapse/issues/12146), [\#12207](https://github.com/matrix-org/synapse/issues/12207), [\#12208](https://github.com/matrix-org/synapse/issues/12208))
 - Move scripts to Synapse package and expose as setuptools entry points. ([\#12118](https://github.com/matrix-org/synapse/issues/12118))
 - Add support for cancellation to `ReadWriteLock`. ([\#12120](https://github.com/matrix-org/synapse/issues/12120))
 - Fix data validation to compare to lists, not sequences. ([\#12128](https://github.com/matrix-org/synapse/issues/12128))
 - Fix CI not attaching source distributions and wheels to the GitHub releases. ([\#12131](https://github.com/matrix-org/synapse/issues/12131))
 - Remove unused mocks from `test_typing`. ([\#12136](https://github.com/matrix-org/synapse/issues/12136))
 - Give `scripts-dev` scripts suffixes for neater CI config. ([\#12137](https://github.com/matrix-org/synapse/issues/12137))
 - Move `synctl` into `synapse._scripts` and expose as an entry point. ([\#12140](https://github.com/matrix-org/synapse/issues/12140))
 - Move the snapcraft configuration file to `contrib`. ([\#12142](https://github.com/matrix-org/synapse/issues/12142))
 - Enable [MSC3030](https://github.com/matrix-org/matrix-doc/pull/3030) Complement tests in CI. ([\#12144](https://github.com/matrix-org/synapse/issues/12144))
 - Enable [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) Complement tests in CI. ([\#12145](https://github.com/matrix-org/synapse/issues/12145))
 - Add test for `ObservableDeferred`'s cancellation behaviour. ([\#12149](https://github.com/matrix-org/synapse/issues/12149))
 - Use `ParamSpec` in type hints for `synapse.logging.context`. ([\#12150](https://github.com/matrix-org/synapse/issues/12150))
 - Prune unused jobs from `tox` config. ([\#12152](https://github.com/matrix-org/synapse/issues/12152))
 - Move CI checks out of tox, to facilitate a move to using poetry. ([\#12153](https://github.com/matrix-org/synapse/issues/12153))
 - Avoid generating state groups for local out-of-band leaves. ([\#12154](https://github.com/matrix-org/synapse/issues/12154))
 - Avoid trying to calculate the state at outlier events. ([\#12155](https://github.com/matrix-org/synapse/issues/12155), [\#12173](https://github.com/matrix-org/synapse/issues/12173), [\#12202](https://github.com/matrix-org/synapse/issues/12202))
 - Fix some type annotations. ([\#12156](https://github.com/matrix-org/synapse/issues/12156))
 - Add type hints for `ObservableDeferred` attributes. ([\#12159](https://github.com/matrix-org/synapse/issues/12159))
 - Use a prebuilt Action for the `tests-done` CI job. ([\#12161](https://github.com/matrix-org/synapse/issues/12161))
 - Reduce number of DB queries made during processing of `/sync`. ([\#12163](https://github.com/matrix-org/synapse/issues/12163))
 - Add `delay_cancellation` utility function, which behaves like `stop_cancellation` but waits until the original `Deferred` resolves before raising a `CancelledError`. ([\#12180](https://github.com/matrix-org/synapse/issues/12180))
 - Retry HTTP replication failures, this should prevent 502's when restarting stateful workers (main, event persisters, stream writers). Contributed by Nick @ Beeper. ([\#12182](https://github.com/matrix-org/synapse/issues/12182))
 - Add cancellation support to `@cached` and `@cachedList` decorators. ([\#12183](https://github.com/matrix-org/synapse/issues/12183))
 - Remove unused variables. ([\#12187](https://github.com/matrix-org/synapse/issues/12187))
 - Add combined test for HTTP pusher and push rule. Contributed by Nick @ Beeper. ([\#12188](https://github.com/matrix-org/synapse/issues/12188))
 - Rename `HomeServer.get_tcp_replication` to `get_replication_command_handler`. ([\#12192](https://github.com/matrix-org/synapse/issues/12192))
 - Remove some dead code. ([\#12197](https://github.com/matrix-org/synapse/issues/12197))
 - Fix a misleading comment in the function `check_event_for_spam`. ([\#12203](https://github.com/matrix-org/synapse/issues/12203))
 - Remove unnecessary `pass` statements. ([\#12206](https://github.com/matrix-org/synapse/issues/12206))
 - Update the SSO username picker template to comply with SIWA guidelines. ([\#12210](https://github.com/matrix-org/synapse/issues/12210))
 - Improve code documentation for the typing stream over replication. ([\#12211](https://github.com/matrix-org/synapse/issues/12211))
 Synapse 1.54.0 (2022-03-08)
 ===========================
--- a/changelog.d/11700.removal
+++ b/changelog.d/11700.removal
@ -1 +0,0 @@
 Remove workaround introduced in Synapse 1.50.0 for Mjolnir compatibility. Breaks compatibility with Mjolnir 1.3.1 and earlier.
--- a/changelog.d/11915.misc
+++ b/changelog.d/11915.misc
@ -1 +0,0 @@
 Simplify the `ApplicationService` class' set of public methods related to interest checking.
--- a/changelog.d/11980.misc
+++ b/changelog.d/11980.misc
@ -1 +0,0 @@
 Add config settings for background update parameters.
--- a/changelog.d/11998.doc
+++ b/changelog.d/11998.doc
@ -1 +0,0 @@
 Fix complexity checking config example in [Resource Constrained Devices](https://matrix-org.github.io/synapse/v1.54/other/running_synapse_on_single_board_computers.html) docs page.
--- a/changelog.d/12028.feature
+++ b/changelog.d/12028.feature
@ -1 +0,0 @@
 Add third-party rules rules callbacks `check_can_shutdown_room` and `check_can_deactivate_user`.
--- a/changelog.d/12042.misc
+++ b/changelog.d/12042.misc
@ -1 +0,0 @@
 Correct type hints for txredis.
--- a/changelog.d/12090.bugfix
+++ b/changelog.d/12090.bugfix
@ -1 +0,0 @@
 Use the proper serialization format for bundled thread aggregations. The bug has existed since Synapse v1.48.0.
--- a/changelog.d/12101.misc
+++ b/changelog.d/12101.misc
@ -1 +0,0 @@
 Limit the size of `aggregation_key` on annotations.
--- a/changelog.d/12108.misc
+++ b/changelog.d/12108.misc
@ -1 +0,0 @@
 Add type hints to tests files.
--- a/changelog.d/12113.bugfix
+++ b/changelog.d/12113.bugfix
@ -1 +0,0 @@
 Fix a long-standing bug when redacting events with relations.
--- a/changelog.d/12118.misc
+++ b/changelog.d/12118.misc
@ -1 +0,0 @@
 Move scripts to Synapse package and expose as setuptools entry points.
--- a/changelog.d/12120.misc
+++ b/changelog.d/12120.misc
@ -1 +0,0 @@
 Add support for cancellation to `ReadWriteLock`.
--- a/changelog.d/12121.bugfix
+++ b/changelog.d/12121.bugfix
@ -1 +0,0 @@
 Fix a long-standing bug when redacting events with relations.
--- a/changelog.d/12128.misc
+++ b/changelog.d/12128.misc
@ -1 +0,0 @@
 Fix data validation to compare to lists, not sequences.
--- a/changelog.d/12130.bugfix
+++ b/changelog.d/12130.bugfix
@ -1 +0,0 @@
 Fix a long-standing bug when redacting events with relations.
--- a/changelog.d/12131.misc
+++ b/changelog.d/12131.misc
@ -1 +0,0 @@
 Fix CI not attaching source distributions and wheels to the GitHub releases.
--- a/changelog.d/12132.feature
+++ b/changelog.d/12132.feature
@ -1 +0,0 @@
 Improve performance of logging in for large accounts.
--- a/changelog.d/12135.feature
+++ b/changelog.d/12135.feature
@ -1 +0,0 @@
 Add experimental env var `SYNAPSE_ASYNC_IO_REACTOR` that causes Synapse to use the asyncio reactor for Twisted.
--- a/changelog.d/12136.misc
+++ b/changelog.d/12136.misc
@ -1 +0,0 @@
 Remove unused mocks from `test_typing`.
--- a/changelog.d/12137.misc
+++ b/changelog.d/12137.misc
@ -1 +0,0 @@
 Give `scripts-dev` scripts suffixes for neater CI config.
--- a/changelog.d/12138.removal
+++ b/changelog.d/12138.removal
@ -1 +0,0 @@
 Remove backwards compatibilty with pagination tokens from the `/relations` and `/aggregations` endpoints generated from Synapse < v1.52.0.
--- a/changelog.d/12140.misc
+++ b/changelog.d/12140.misc
@ -1 +0,0 @@
 Move `synctl` into `synapse._scripts` and expose as an entry point.
--- a/changelog.d/12142.misc
+++ b/changelog.d/12142.misc
@ -1 +0,0 @@
 Move the snapcraft configuration file to `contrib`.
--- a/changelog.d/12143.doc
+++ b/changelog.d/12143.doc
@ -1 +0,0 @@
 Improve documentation for demo scripts.
--- a/changelog.d/12144.misc
+++ b/changelog.d/12144.misc
@ -1 +0,0 @@
 Enable [MSC3030](https://github.com/matrix-org/matrix-doc/pull/3030) Complement tests in CI.
--- a/changelog.d/12145.misc
+++ b/changelog.d/12145.misc
@ -1 +0,0 @@
 Enable [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) Complement tests in CI.
--- a/changelog.d/12146.misc
+++ b/changelog.d/12146.misc
@ -1 +0,0 @@
 Add type hints to tests files.
--- a/changelog.d/12149.misc
+++ b/changelog.d/12149.misc
@ -1 +0,0 @@
 Add test for `ObservableDeferred`'s cancellation behaviour.
--- a/changelog.d/12150.misc
+++ b/changelog.d/12150.misc
@ -1 +0,0 @@
 Use `ParamSpec` in type hints for `synapse.logging.context`.
--- a/changelog.d/12151.feature
+++ b/changelog.d/12151.feature
@ -1 +0,0 @@
 Support the stable identifiers from [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440): threads.
--- a/changelog.d/12152.misc
+++ b/changelog.d/12152.misc
@ -1 +0,0 @@
 Prune unused jobs from `tox` config.
--- a/changelog.d/12153.misc
+++ b/changelog.d/12153.misc
@ -1 +0,0 @@
 Move CI checks out of tox, to facilitate a move to using poetry.
--- a/changelog.d/12154.misc
+++ b/changelog.d/12154.misc
@ -1 +0,0 @@
 Avoid generating state groups for local out-of-band leaves.
--- a/changelog.d/12155.misc
+++ b/changelog.d/12155.misc
@ -1 +0,0 @@
 Avoid trying to calculate the state at outlier events.
--- a/changelog.d/12156.misc
+++ b/changelog.d/12156.misc
@ -1 +0,0 @@
 Fix some type annotations.
--- a/changelog.d/12157.bugfix
+++ b/changelog.d/12157.bugfix
@ -1 +0,0 @@
 Fix a bug introduced in #4864 whereby background updates are never run with the default background batch size.
--- a/changelog.d/12159.misc
+++ b/changelog.d/12159.misc
@ -1 +0,0 @@
 Add type hints for `ObservableDeferred` attributes.
--- a/changelog.d/12161.misc
+++ b/changelog.d/12161.misc
@ -1 +0,0 @@
 Use a prebuilt Action for the `tests-done` CI job.
--- a/changelog.d/12163.misc
+++ b/changelog.d/12163.misc
@ -1 +0,0 @@
 Reduce number of DB queries made during processing of `/sync`.
--- a/changelog.d/12173.misc
+++ b/changelog.d/12173.misc
@ -1 +0,0 @@
 Avoid trying to calculate the state at outlier events.
--- a/changelog.d/12175.bugfix
+++ b/changelog.d/12175.bugfix
@ -1 +0,0 @@
 Fix a bug where non-standard information was returned from the `/hierarchy` API. Introduced in Synapse v1.41.0.
--- a/changelog.d/12179.doc
+++ b/changelog.d/12179.doc
@ -1 +0,0 @@
 Updates to the Room DAG concepts development document.
--- a/changelog.d/12180.misc
+++ b/changelog.d/12180.misc
@ -1 +0,0 @@
 Add `delay_cancellation` utility function, which behaves like `stop_cancellation` but waits until the original `Deferred` resolves before raising a `CancelledError`.
--- a/changelog.d/12182.misc
+++ b/changelog.d/12182.misc
@ -1 +0,0 @@
 Retry HTTP replication failures, this should prevent 502's when restarting stateful workers (main, event persisters, stream writers). Contributed by Nick @ Beeper.
--- a/changelog.d/12183.misc
+++ b/changelog.d/12183.misc
@ -1 +0,0 @@
 Add cancellation support to `@cached` and `@cachedList` decorators.
--- a/changelog.d/12187.misc
+++ b/changelog.d/12187.misc
@ -1 +0,0 @@
 Remove unused variables.
--- a/changelog.d/12188.misc
+++ b/changelog.d/12188.misc
@ -1 +0,0 @@
 Add combined test for HTTP pusher and push rule. Contributed by Nick @ Beeper.
--- a/changelog.d/12189.bugfix
+++ b/changelog.d/12189.bugfix
@ -1 +0,0 @@
 Fix a long-standing bug when redacting events with relations.
--- a/changelog.d/12192.misc
+++ b/changelog.d/12192.misc
@ -1 +0,0 @@
 Rename `HomeServer.get_tcp_replication` to `get_replication_command_handler`.
--- a/changelog.d/12196.doc
+++ b/changelog.d/12196.doc
@ -1 +0,0 @@
 Document that the `typing`, `to_device`, `account_data`, `receipts`, and `presence` stream writer can only be used on a single worker.
--- a/changelog.d/12197.misc
+++ b/changelog.d/12197.misc
@ -1 +0,0 @@
 Remove some dead code.
--- a/changelog.d/12200.removal
+++ b/changelog.d/12200.removal
@ -1 +0,0 @@
 The groups/communities feature in Synapse has been deprecated.
--- a/changelog.d/12202.misc
+++ b/changelog.d/12202.misc
@ -1 +0,0 @@
 Avoid trying to calculate the state at outlier events.
--- a/changelog.d/12203.misc
+++ b/changelog.d/12203.misc
@ -1 +0,0 @@
 Fix a misleading comment in the function `check_event_for_spam`.
--- a/changelog.d/12204.doc
+++ b/changelog.d/12204.doc
@ -1 +0,0 @@
 Document that contributors can sign off privately by email.
--- a/changelog.d/12206.misc
+++ b/changelog.d/12206.misc
@ -1 +0,0 @@
 Remove unnecessary `pass` statements.
--- a/changelog.d/12207.misc
+++ b/changelog.d/12207.misc
@ -1 +0,0 @@
 Add type hints to tests files.
--- a/changelog.d/12208.misc
+++ b/changelog.d/12208.misc
@ -1 +0,0 @@
 Add type hints to tests files.
--- a/changelog.d/12210.misc
+++ b/changelog.d/12210.misc
@ -1 +0,0 @@
 Update the SSO username picker template to comply with SIWA guidelines.
--- a/changelog.d/12211.misc
+++ b/changelog.d/12211.misc
@ -1 +0,0 @@
 Improve code documentation for the typing stream over replication.
--- a/changelog.d/12212.feature
+++ b/changelog.d/12212.feature
@ -1 +0,0 @@
 Add a new Jinja2 template filter to extract the local part of an email address.
--- a/changelog.d/12215.bugfix
+++ b/changelog.d/12215.bugfix
@ -1 +0,0 @@
 Fix a bug introduced in 1.54.0 that broke background updates on sqlite homeservers while search was disabled.
--- a/debian/changelog
+++ b/debian/changelog
@ -1,3 +1,9 @@
 matrix-synapse-py3 (1.55.0~rc1) stable; urgency=medium
  * New synapse release 1.55.0~rc1.
 -- Synapse Packaging team <packages@matrix.org>  Tue, 15 Mar 2022 10:59:31 +0000
 matrix-synapse-py3 (1.54.0) stable; urgency=medium
  * New synapse release 1.54.0.
--- a/synapse/init.py
+++ b/synapse/init.py
@ -68,7 +68,7 @@ try:
 except ImportError:
    pass
-__version__ = "1.54.0"
+__version__ = "1.55.0rc1"
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
    # We import here so that we don't have to install a bunch of deps when
--- a/synapse/events/utils.py
+++ b/synapse/events/utils.py
@ -530,9 +530,12 @@ class EventClientSerializer:
        # Include the bundled aggregations in the event.
        if serialized_aggregations:
-            serialized_event["unsigned"].setdefault("m.relations", {}).update(
+            # There is likely already an "unsigned" field, but a filter might
-                serialized_aggregations
+            # have stripped it off (via the event_fields option). The server is
-            )
+            # allowed to return additional fields, so add it back.
            serialized_event.setdefault("unsigned", {}).setdefault(
                "m.relations", {}
            ).update(serialized_aggregations)
    def serialize_events(
        self,
--- a/tests/rest/client/test_relations.py
+++ b/tests/rest/client/test_relations.py
@ -981,6 +981,34 @@ class RelationsTestCase(BaseRelationsTestCase):
            [annotation_event_id_good, thread_event_id],
        )
    def test_bundled_aggregations_with_filter(self) -> None:
        """
        If "unsigned" is an omitted field (due to filtering), adding the bundled
        aggregations should not break.
        Note that the spec allows for a server to return additional fields beyond
        what is specified.
        """
        self._send_relation(RelationTypes.ANNOTATION, "m.reaction", "a")
        # Note that the sync filter does not include "unsigned" as a field.
        filter = urllib.parse.quote_plus(
            b'{"event_fields": ["content", "event_id"], "room": {"timeline": {"limit": 3}}}'
        )
        channel = self.make_request(
            "GET", f"/sync?filter={filter}", access_token=self.user_token
        )
        self.assertEqual(200, channel.code, channel.json_body)
        # Ensure the timeline is limited, find the parent event.
        room_timeline = channel.json_body["rooms"]["join"][self.room]["timeline"]
        self.assertTrue(room_timeline["limited"])
        parent_event = self._find_event_in_chunk(room_timeline["events"])
        # Ensure there's bundled aggregations on it.
        self.assertIn("unsigned", parent_event)
        self.assertIn("m.relations", parent_event["unsigned"])
 class RelationPaginationTestCase(BaseRelationsTestCase):
    def test_basic_paginate_relations(self) -> None:
		`@ -1 +0,0 @@`
			`Remove workaround introduced in Synapse 1.50.0 for Mjolnir compatibility. Breaks compatibility with Mjolnir 1.3.1 and earlier.`
		`@ -1 +0,0 @@`
			Simplify the `ApplicationService` class' set of public methods related to interest checking.
		`@ -1 +0,0 @@`
			`Add config settings for background update parameters.`
		`@ -1 +0,0 @@`
			`Fix complexity checking config example in [Resource Constrained Devices](https://matrix-org.github.io/synapse/v1.54/other/running_synapse_on_single_board_computers.html) docs page.`
		`@ -1 +0,0 @@`
			Add third-party rules rules callbacks `check_can_shutdown_room` and `check_can_deactivate_user`.
		`@ -1 +0,0 @@`
			`Use the proper serialization format for bundled thread aggregations. The bug has existed since Synapse v1.48.0.`
		`@ -1 +0,0 @@`
			Limit the size of `aggregation_key` on annotations.
		`@ -1 +0,0 @@`
			`Fix a long-standing bug when redacting events with relations.`
		`@ -1 +0,0 @@`
			`Move scripts to Synapse package and expose as setuptools entry points.`
		`@ -1 +0,0 @@`
			Add support for cancellation to `ReadWriteLock`.