Merge remote-tracking branch 'origin/develop' into erikj/destination_retry_cache

2018-09-25 12:03:54 +01:00 · 2018-09-25 12:03:54 +01:00 · a1cd37390f
parent 19dc676d1a 4c3e7eeec5
commit a1cd37390f
51 changed files with 128 additions and 62 deletions
--- a/CHANGES.md
+++ b/CHANGES.md
@ -1,3 +1,77 @@
+Synapse 0.33.5.1 (2018-09-25)
+=============================
+
+Internal Changes
+----------------
+
+- Fix incompatibility with older Twisted version in tests. Thanks 
+  @OlegGirko! ([\#3940](https://github.com/matrix-org/synapse/issues/3940))
+
+
+Synapse 0.33.5 (2018-09-24)
+===========================
+
+No significant changes.
+
+
+Synapse 0.33.5rc1 (2018-09-17)
+==============================
+
+Features
+--------
+
+- Python 3.5 and 3.6 support is now in beta. ([\#3576](https://github.com/matrix-org/synapse/issues/3576))
+- Implement `event_format` filter param in `/sync` ([\#3790](https://github.com/matrix-org/synapse/issues/3790))
+- Add synapse_admin_mau:registered_reserved_users metric to expose number of real reaserved users ([\#3846](https://github.com/matrix-org/synapse/issues/3846))
+
+
+Bugfixes
+--------
+
+- Remove connection ID for replication prometheus metrics, as it creates a large number of new series. ([\#3788](https://github.com/matrix-org/synapse/issues/3788))
+- guest users should not be part of mau total ([\#3800](https://github.com/matrix-org/synapse/issues/3800))
+- Bump dependency on pyopenssl 16.x, to avoid incompatibility with recent Twisted. ([\#3804](https://github.com/matrix-org/synapse/issues/3804))
+- Fix existing room tags not coming down sync when joining a room ([\#3810](https://github.com/matrix-org/synapse/issues/3810))
+- Fix jwt import check ([\#3824](https://github.com/matrix-org/synapse/issues/3824))
+- fix VOIP crashes under Python 3 (#3821) ([\#3835](https://github.com/matrix-org/synapse/issues/3835))
+- Fix manhole so that it works with latest openssh clients ([\#3841](https://github.com/matrix-org/synapse/issues/3841))
+- Fix outbound requests occasionally wedging, which can result in federation breaking between servers. ([\#3845](https://github.com/matrix-org/synapse/issues/3845))
+- Show heroes if room name/canonical alias has been deleted ([\#3851](https://github.com/matrix-org/synapse/issues/3851))
+- Fix handling of redacted events from federation ([\#3859](https://github.com/matrix-org/synapse/issues/3859))
+-  ([\#3874](https://github.com/matrix-org/synapse/issues/3874))
+- Mitigate outbound federation randomly becoming wedged ([\#3875](https://github.com/matrix-org/synapse/issues/3875))
+
+
+Internal Changes
+----------------
+
+- CircleCI tests now run on the potential merge of a PR. ([\#3704](https://github.com/matrix-org/synapse/issues/3704))
+- http/ is now ported to Python 3. ([\#3771](https://github.com/matrix-org/synapse/issues/3771))
+- Improve human readable error messages for threepid registration/account update ([\#3789](https://github.com/matrix-org/synapse/issues/3789))
+- Make /sync slightly faster by avoiding needless copies ([\#3795](https://github.com/matrix-org/synapse/issues/3795))
+- handlers/ is now ported to Python 3. ([\#3803](https://github.com/matrix-org/synapse/issues/3803))
+- Limit the number of PDUs/EDUs per federation transaction ([\#3805](https://github.com/matrix-org/synapse/issues/3805))
+- Only start postgres instance for postgres tests on Travis CI ([\#3806](https://github.com/matrix-org/synapse/issues/3806))
+- tests/ is now ported to Python 3. ([\#3808](https://github.com/matrix-org/synapse/issues/3808))
+- crypto/ is now ported to Python 3. ([\#3822](https://github.com/matrix-org/synapse/issues/3822))
+- rest/ is now ported to Python 3. ([\#3823](https://github.com/matrix-org/synapse/issues/3823))
+- add some logging for the keyring queue ([\#3826](https://github.com/matrix-org/synapse/issues/3826))
+- speed up lazy loading by 2-3x ([\#3827](https://github.com/matrix-org/synapse/issues/3827))
+- Improved Dockerfile to remove build requirements after building reducing the image size. ([\#3834](https://github.com/matrix-org/synapse/issues/3834))
+- Disable lazy loading for incremental syncs for now ([\#3840](https://github.com/matrix-org/synapse/issues/3840))
+- federation/ is now ported to Python 3. ([\#3847](https://github.com/matrix-org/synapse/issues/3847))
+- Log when we retry outbound requests ([\#3853](https://github.com/matrix-org/synapse/issues/3853))
+- Removed some excess logging messages. ([\#3855](https://github.com/matrix-org/synapse/issues/3855))
+- Speed up purge history for rooms that have been previously purged ([\#3856](https://github.com/matrix-org/synapse/issues/3856))
+- Refactor some HTTP timeout code. ([\#3857](https://github.com/matrix-org/synapse/issues/3857))
+- Fix running merged builds on CircleCI ([\#3858](https://github.com/matrix-org/synapse/issues/3858))
+- Fix typo in replication stream exception. ([\#3860](https://github.com/matrix-org/synapse/issues/3860))
+- Add in flight real time metrics for Measure blocks ([\#3871](https://github.com/matrix-org/synapse/issues/3871))
+- Disable buffering and automatic retrying in treq requests to prevent timeouts. ([\#3872](https://github.com/matrix-org/synapse/issues/3872))
+- mention jemalloc in the README ([\#3877](https://github.com/matrix-org/synapse/issues/3877))
+- Remove unmaintained "nuke-room-from-db.sh" script ([\#3888](https://github.com/matrix-org/synapse/issues/3888))
+
+
 Synapse 0.33.4 (2018-09-07)
 ===========================

--- a/changelog.d/3576.feature
+++ b/changelog.d/3576.feature
@ -1 +0,0 @@
-Python 3.5+ is now supported.
--- a/changelog.d/3578.bugfix
+++ b/changelog.d/3578.bugfix
@ -0,0 +1 @@
+Fix problem when playing media from Chrome using direct URL (thanks @remjey!)
--- a/changelog.d/3704.misc
+++ b/changelog.d/3704.misc
@ -1 +0,0 @@
-CircleCI tests now run on the potential merge of a PR.
--- a/changelog.d/3771.misc
+++ b/changelog.d/3771.misc
@ -1 +0,0 @@
-http/ is now ported to Python 3.
--- a/changelog.d/3788.bugfix
+++ b/changelog.d/3788.bugfix
@ -1 +0,0 @@
-Remove connection ID for replication prometheus metrics, as it creates a large number of new series.
--- a/changelog.d/3789.misc
+++ b/changelog.d/3789.misc
@ -1 +0,0 @@
-Improve human readable error messages for threepid registration/account update
--- a/changelog.d/3790.feature
+++ b/changelog.d/3790.feature
@ -1 +0,0 @@
-Implement `event_format` filter param in `/sync`
--- a/changelog.d/3795.misc
+++ b/changelog.d/3795.misc
@ -1 +0,0 @@
-Make /sync slightly faster by avoiding needless copies
--- a/changelog.d/3800.bugfix
+++ b/changelog.d/3800.bugfix
@ -1 +0,0 @@
-guest users should not be part of mau total
--- a/changelog.d/3803.misc
+++ b/changelog.d/3803.misc
@ -1 +0,0 @@
-handlers/ is now ported to Python 3.
--- a/changelog.d/3804.bugfix
+++ b/changelog.d/3804.bugfix
@ -1 +0,0 @@
-Bump dependency on pyopenssl 16.x, to avoid incompatibility with recent Twisted.
--- a/changelog.d/3805.misc
+++ b/changelog.d/3805.misc
@ -1 +0,0 @@
-Limit the number of PDUs/EDUs per federation transaction
--- a/changelog.d/3806.misc
+++ b/changelog.d/3806.misc
@ -1 +0,0 @@
-Only start postgres instance for postgres tests on Travis CI
--- a/changelog.d/3808.misc
+++ b/changelog.d/3808.misc
@ -1 +0,0 @@
-tests/ is now ported to Python 3.
--- a/changelog.d/3810.bugfix
+++ b/changelog.d/3810.bugfix
@ -1 +0,0 @@
-Fix existing room tags not coming down sync when joining a room
--- a/changelog.d/3822.misc
+++ b/changelog.d/3822.misc
@ -1 +0,0 @@
-crypto/ is now ported to Python 3.
--- a/changelog.d/3823.misc
+++ b/changelog.d/3823.misc
@ -1 +0,0 @@
-rest/ is now ported to Python 3.
--- a/changelog.d/3824.bugfix
+++ b/changelog.d/3824.bugfix
@ -1 +0,0 @@
-Fix jwt import check
--- a/changelog.d/3826.misc
+++ b/changelog.d/3826.misc
@ -1 +0,0 @@
-add some logging for the keyring queue
--- a/changelog.d/3827.misc
+++ b/changelog.d/3827.misc
@ -1 +0,0 @@
-speed up lazy loading by 2-3x
--- a/changelog.d/3834.misc
+++ b/changelog.d/3834.misc
@ -1 +0,0 @@
-Improved Dockerfile to remove build requirements after building reducing the image size.
--- a/changelog.d/3835.bugfix
+++ b/changelog.d/3835.bugfix
@ -1 +0,0 @@
-fix VOIP crashes under Python 3 (#3821)
--- a/changelog.d/3840.misc
+++ b/changelog.d/3840.misc
@ -1 +0,0 @@
-Disable lazy loading for incremental syncs for now
--- a/changelog.d/3841.bugfix
+++ b/changelog.d/3841.bugfix
@ -1 +0,0 @@
-Fix manhole so that it works with latest openssh clients
--- a/changelog.d/3845.bugfix
+++ b/changelog.d/3845.bugfix
@ -1 +0,0 @@
-Fix outbound requests occasionally wedging, which can result in federation breaking between servers.
--- a/changelog.d/3846.feature
+++ b/changelog.d/3846.feature
@ -1 +0,0 @@
-Add synapse_admin_mau:registered_reserved_users metric to expose number of real reaserved users 
--- a/changelog.d/3847.misc
+++ b/changelog.d/3847.misc
@ -1 +0,0 @@
-federation/ is now ported to Python 3.
--- a/changelog.d/3851.bugfix
+++ b/changelog.d/3851.bugfix
@ -1 +0,0 @@
-Show heroes if room name/canonical alias has been deleted
--- a/changelog.d/3853.misc
+++ b/changelog.d/3853.misc
@ -1 +0,0 @@
-Log when we retry outbound requests
--- a/changelog.d/3855.misc
+++ b/changelog.d/3855.misc
@ -1 +0,0 @@
-Removed some excess logging messages.
--- a/changelog.d/3856.misc
+++ b/changelog.d/3856.misc
@ -1 +0,0 @@
-Speed up purge history for rooms that have been previously purged
--- a/changelog.d/3857.misc
+++ b/changelog.d/3857.misc
@ -1 +0,0 @@
-Refactor some HTTP timeout code.
--- a/changelog.d/3858.misc
+++ b/changelog.d/3858.misc
@ -1 +0,0 @@
-Fix running merged builds on CircleCI
--- a/changelog.d/3859.bugfix
+++ b/changelog.d/3859.bugfix
@ -1 +0,0 @@
-Fix handling of redacted events from federation
--- a/changelog.d/3860.misc
+++ b/changelog.d/3860.misc
@ -1 +0,0 @@
-Fix typo in replication stream exception.
--- a/changelog.d/3871.misc
+++ b/changelog.d/3871.misc
@ -1 +0,0 @@
-Add in flight real time metrics for Measure blocks
--- a/changelog.d/3872.misc
+++ b/changelog.d/3872.misc
@ -1 +0,0 @@
-Disable buffering and automatic retrying in treq requests to prevent timeouts.
--- a/changelog.d/3874.bugfix
+++ b/changelog.d/3874.bugfix
--- a/changelog.d/3875.bugfix
+++ b/changelog.d/3875.bugfix
@ -1 +0,0 @@
-Mitigate outbound federation randomly becoming wedged
--- a/changelog.d/3877.misc
+++ b/changelog.d/3877.misc
@ -1 +0,0 @@
-mention jemalloc in the README
--- a/changelog.d/3888.misc
+++ b/changelog.d/3888.misc
@ -1 +0,0 @@
-Remove unmaintained "nuke-room-from-db.sh" script
--- a/changelog.d/3925.misc
+++ b/changelog.d/3925.misc
@ -0,0 +1 @@
+Fix spurious exceptions when remote http client closes conncetion
--- a/changelog.d/3936.bugfix
+++ b/changelog.d/3936.bugfix
@ -0,0 +1 @@
+Fix out-of-bounds error when LLing yourself
--- a/changelog.d/3947.misc
+++ b/changelog.d/3947.misc
@ -0,0 +1 @@
+Require attrs 16.0.0 or later
--- a/synapse/init.py
+++ b/synapse/init.py
@ -27,4 +27,4 @@ try:
 except ImportError:
    pass

-__version__ = "0.33.4"
+__version__ = "0.33.5.1"
--- a/synapse/handlers/sync.py
+++ b/synapse/handlers/sync.py
@ -713,10 +713,6 @@ class SyncHandler(object):
                    )
                ]

-                # always make sure we LL ourselves so we know we're in the room
-                # (if we are), to fix https://github.com/vector-im/riot-web/issues/7209
-                types.append((EventTypes.Member, sync_config.user.to_string()))
-
                # only apply the filtering to room members
                filtered_types = [EventTypes.Member]

@ -726,6 +722,13 @@ class SyncHandler(object):
            }

            if full_state:
+                if lazy_load_members:
+                    # always make sure we LL ourselves so we know we're in the room
+                    # (if we are) to fix https://github.com/vector-im/riot-web/issues/7209
+                    # We only need apply this on full state syncs given we disabled
+                    # LL for incr syncs in #3840.
+                    types.append((EventTypes.Member, sync_config.user.to_string()))
+
                if batch:
                    current_state_ids = yield self.store.get_state_ids_for_event(
                        batch.events[-1].event_id, types=types,
@ -794,7 +797,7 @@ class SyncHandler(object):
            else:
                state_ids = {}
                if lazy_load_members:
-                    if types:
+                    if types and batch.events:
                        # We're returning an incremental sync, with no
                        # "gap" since the previous sync, so normally there would be
                        # no state to return.
--- a/synapse/http/server.py
+++ b/synapse/http/server.py
@ -84,6 +84,17 @@ def wrap_json_request_handler(h):
            logger.info(
                "%s SynapseError: %s - %s", request, code, e.msg
            )
+
+            # Only respond with an error response if we haven't already started
+            # writing, otherwise lets just kill the connection
+            if request.startedWriting:
+                if request.transport:
+                    try:
+                        request.transport.abortConnection()
+                    except Exception:
+                        # abortConnection throws if the connection is already closed
+                        pass
+            else:
                respond_with_json(
                    request, code, e.error_dict(), send_cors=True,
                    pretty_print=_request_user_agent_is_curl(request),
@ -100,6 +111,16 @@ def wrap_json_request_handler(h):
                request,
                f.getTraceback().rstrip(),
            )
+            # Only respond with an error response if we haven't already started
+            # writing, otherwise lets just kill the connection
+            if request.startedWriting:
+                if request.transport:
+                    try:
+                        request.transport.abortConnection()
+                    except Exception:
+                        # abortConnection throws if the connection is already closed
+                        pass
+            else:
                respond_with_json(
                    request,
                    500,
--- a/synapse/python_dependencies.py
+++ b/synapse/python_dependencies.py
@ -58,7 +58,9 @@ REQUIREMENTS = {
    "phonenumbers>=8.2.0": ["phonenumbers"],
    "six": ["six"],
    "prometheus_client": ["prometheus_client"],
-    "attrs": ["attr"],
+
+    # we use attr.s(slots), which arrived in 16.0.0
+    "attrs>=16.0.0": ["attr>=16.0.0"],
    "netaddr>=0.7.18": ["netaddr"],
 }

--- a/synapse/rest/media/v1/download_resource.py
+++ b/synapse/rest/media/v1/download_resource.py
@ -52,6 +52,7 @@ class DownloadResource(Resource):
            b" script-src 'none';"
            b" plugin-types application/pdf;"
            b" style-src 'unsafe-inline';"
+            b" media-src 'self';"
            b" object-src 'self';"
        )
        server_name, media_id, name = parse_media_id(request)
--- a/tests/http/test_fedclient.py
+++ b/tests/http/test_fedclient.py
@ -54,7 +54,7 @@ class FederationClientTests(HomeserverTestCase):
    def test_client_never_connect(self):
        """
        If the HTTP request is not connected and is timed out, it'll give a
-        ConnectingCancelledError.
+        ConnectingCancelledError or TimeoutError.
        """
        d = self.cl.get_json("testserv:8008", "foo/bar", timeout=10000)

@ -76,7 +76,7 @@ class FederationClientTests(HomeserverTestCase):
        self.reactor.advance(10.5)
        f = self.failureResultOf(d)

-        self.assertIsInstance(f.value, ConnectingCancelledError)
+        self.assertIsInstance(f.value, (ConnectingCancelledError, TimeoutError))

    def test_client_connect_no_response(self):
        """
				`@ -0,0 +1 @@`
				`Fix problem when playing media from Chrome using direct URL (thanks @remjey!)`
				`@ -1 +0,0 @@`
				`CircleCI tests now run on the potential merge of a PR.`
				`@ -1 +0,0 @@`
				`Remove connection ID for replication prometheus metrics, as it creates a large number of new series.`
				`@ -1 +0,0 @@`
				`Improve human readable error messages for threepid registration/account update`
				`@ -1 +0,0 @@`
				Implement `event_format` filter param in `/sync`
				`@ -1 +0,0 @@`
				`Make /sync slightly faster by avoiding needless copies`
				`@ -1 +0,0 @@`
				`Bump dependency on pyopenssl 16.x, to avoid incompatibility with recent Twisted.`
				`@ -1 +0,0 @@`
				`Limit the number of PDUs/EDUs per federation transaction`
				`@ -1 +0,0 @@`
				`Only start postgres instance for postgres tests on Travis CI`
				`@ -1 +0,0 @@`
				`Fix existing room tags not coming down sync when joining a room`