Merge branch 'develop' of github.com:matrix-org/synapse into anoa/room_upgrade_federatable

CHANGES.md

@@ -1,3 +1,86 @@
+Synapse 0.99.0rc2 (2019-01-30)
+==============================
+
+Bugfixes
+--------
+
+- Fix bug when rejecting remote invites. ([\#4527](https://github.com/matrix-org/synapse/issues/4527))
+- Fix incorrect rendering of server capabilities. ([81b7e7eed](https://github.com/matrix-org/synapse/commit/81b7e7eed323f55d6550e7a270a9dc2c4c7b0fe0))
+
+Improved Documentation
+----------------------
+
+- Add documentation on enabling ACME support when upgrading to v0.99. ([\#4528](https://github.com/matrix-org/synapse/issues/4528))
+
+
+Synapse 0.99.0rc1 (2019-01-30)
+==============================
+
+Synapse v0.99.x is a precursor to the upcoming Synapse v1.0 release. It contains foundational changes to room architecture and the federation security model necessary to support the upcoming r0 release of the Server to Server API.
+
+Features
+--------
+
+- Synapse's cipher string has been updated to require ECDH key exchange. Configuring and generating dh_params is no longer required, and they will be ignored. ([\#4229](https://github.com/matrix-org/synapse/issues/4229))
+- Synapse can now automatically provision TLS certificates via ACME (the protocol used by CAs like Let's Encrypt). ([\#4384](https://github.com/matrix-org/synapse/issues/4384), [\#4492](https://github.com/matrix-org/synapse/issues/4492), [\#4525](https://github.com/matrix-org/synapse/issues/4525))
+- Implement MSC1708 (.well-known routing for server-server federation) ([\#4408](https://github.com/matrix-org/synapse/issues/4408), [\#4409](https://github.com/matrix-org/synapse/issues/4409), [\#4426](https://github.com/matrix-org/synapse/issues/4426), [\#4427](https://github.com/matrix-org/synapse/issues/4427), [\#4428](https://github.com/matrix-org/synapse/issues/4428), [\#4464](https://github.com/matrix-org/synapse/issues/4464), [\#4468](https://github.com/matrix-org/synapse/issues/4468), [\#4487](https://github.com/matrix-org/synapse/issues/4487), [\#4488](https://github.com/matrix-org/synapse/issues/4488), [\#4489](https://github.com/matrix-org/synapse/issues/4489), [\#4497](https://github.com/matrix-org/synapse/issues/4497), [\#4511](https://github.com/matrix-org/synapse/issues/4511), [\#4516](https://github.com/matrix-org/synapse/issues/4516), [\#4520](https://github.com/matrix-org/synapse/issues/4520), [\#4521](https://github.com/matrix-org/synapse/issues/4521))
+- Search now includes results from predecessor rooms after a room upgrade. ([\#4415](https://github.com/matrix-org/synapse/issues/4415))
+- Config option to disable requesting MSISDN on registration. ([\#4423](https://github.com/matrix-org/synapse/issues/4423))
+- Add a metric for tracking event stream position of the user directory. ([\#4445](https://github.com/matrix-org/synapse/issues/4445))
+- Support exposing server capabilities in CS API (MSC1753, MSC1804) ([\#4472](https://github.com/matrix-org/synapse/issues/4472))
+- Add support for room version 3 ([\#4483](https://github.com/matrix-org/synapse/issues/4483), [\#4499](https://github.com/matrix-org/synapse/issues/4499), [\#4515](https://github.com/matrix-org/synapse/issues/4515), [\#4523](https://github.com/matrix-org/synapse/issues/4523))
+- Synapse will now reload TLS certificates from disk upon SIGHUP. ([\#4495](https://github.com/matrix-org/synapse/issues/4495), [\#4524](https://github.com/matrix-org/synapse/issues/4524))
+
+
+Bugfixes
+--------
+
+- Prevent users with access tokens predating the introduction of device IDs from creating spurious entries in the user_ips table. ([\#4369](https://github.com/matrix-org/synapse/issues/4369))
+- Fix typo in ALL_USER_TYPES definition to ensure type is a tuple ([\#4392](https://github.com/matrix-org/synapse/issues/4392))
+- Fix high CPU usage due to remote devicelist updates ([\#4397](https://github.com/matrix-org/synapse/issues/4397))
+- Fix potential bug where creating or joining a room could fail ([\#4404](https://github.com/matrix-org/synapse/issues/4404))
+- Fix bug when rejecting remote invites ([\#4405](https://github.com/matrix-org/synapse/issues/4405))
+- Fix incorrect logcontexts after a Deferred was cancelled ([\#4407](https://github.com/matrix-org/synapse/issues/4407))
+- Ensure encrypted room state is persisted across room upgrades. ([\#4411](https://github.com/matrix-org/synapse/issues/4411))
+- Copy over whether a room is a direct message and any associated room tags on room upgrade. ([\#4412](https://github.com/matrix-org/synapse/issues/4412))
+- Fix None guard in calling config.server.is_threepid_reserved ([\#4435](https://github.com/matrix-org/synapse/issues/4435))
+- Don't send IP addresses as SNI ([\#4452](https://github.com/matrix-org/synapse/issues/4452))
+- Fix UnboundLocalError in post_urlencoded_get_json ([\#4460](https://github.com/matrix-org/synapse/issues/4460))
+- Add a timeout to filtered room directory queries. ([\#4461](https://github.com/matrix-org/synapse/issues/4461))
+- Workaround for login error when using both LDAP and internal authentication. ([\#4486](https://github.com/matrix-org/synapse/issues/4486))
+- Fix a bug where setting a relative consent directory path would cause a crash. ([\#4512](https://github.com/matrix-org/synapse/issues/4512))
+
+
+Deprecations and Removals
+-------------------------
+
+- Synapse no longer generates self-signed TLS certificates when generating a configuration file. ([\#4509](https://github.com/matrix-org/synapse/issues/4509))
+
+
+Internal Changes
+----------------
+
+- Synapse will now take advantage of native UPSERT functionality in PostgreSQL 9.5+ and SQLite 3.24+. ([\#4306](https://github.com/matrix-org/synapse/issues/4306), [\#4459](https://github.com/matrix-org/synapse/issues/4459), [\#4466](https://github.com/matrix-org/synapse/issues/4466), [\#4471](https://github.com/matrix-org/synapse/issues/4471), [\#4477](https://github.com/matrix-org/synapse/issues/4477), [\#4505](https://github.com/matrix-org/synapse/issues/4505))
+- Update README to use the new virtualenv everywhere ([\#4342](https://github.com/matrix-org/synapse/issues/4342))
+- Add better logging for unexpected errors while sending transactions ([\#4368](https://github.com/matrix-org/synapse/issues/4368))
+- Apply a unique index to the user_ips table, preventing duplicates. ([\#4370](https://github.com/matrix-org/synapse/issues/4370), [\#4432](https://github.com/matrix-org/synapse/issues/4432), [\#4434](https://github.com/matrix-org/synapse/issues/4434))
+- Silence travis-ci build warnings by removing non-functional python3.6 ([\#4377](https://github.com/matrix-org/synapse/issues/4377))
+- Fix a comment in the generated config file ([\#4387](https://github.com/matrix-org/synapse/issues/4387))
+- Add ground work for implementing future federation API versions ([\#4390](https://github.com/matrix-org/synapse/issues/4390))
+- Update dependencies on msgpack and pymacaroons to use the up-to-date packages. ([\#4399](https://github.com/matrix-org/synapse/issues/4399))
+- Tweak codecov settings to make them less loud. ([\#4400](https://github.com/matrix-org/synapse/issues/4400))
+- Implement server support for MSC1794 - Federation v2 Invite API ([\#4402](https://github.com/matrix-org/synapse/issues/4402))
+- debian package: symlink to explicit python version ([\#4433](https://github.com/matrix-org/synapse/issues/4433))
+- Add infrastructure to support different event formats ([\#4437](https://github.com/matrix-org/synapse/issues/4437), [\#4447](https://github.com/matrix-org/synapse/issues/4447), [\#4448](https://github.com/matrix-org/synapse/issues/4448), [\#4470](https://github.com/matrix-org/synapse/issues/4470), [\#4481](https://github.com/matrix-org/synapse/issues/4481), [\#4482](https://github.com/matrix-org/synapse/issues/4482), [\#4493](https://github.com/matrix-org/synapse/issues/4493), [\#4494](https://github.com/matrix-org/synapse/issues/4494), [\#4496](https://github.com/matrix-org/synapse/issues/4496), [\#4510](https://github.com/matrix-org/synapse/issues/4510), [\#4514](https://github.com/matrix-org/synapse/issues/4514))
+- Generate the debian config during build ([\#4444](https://github.com/matrix-org/synapse/issues/4444))
+- Clarify documentation for the `public_baseurl` config param ([\#4458](https://github.com/matrix-org/synapse/issues/4458), [\#4498](https://github.com/matrix-org/synapse/issues/4498))
+- Fix quoting for allowed_local_3pids example config ([\#4476](https://github.com/matrix-org/synapse/issues/4476))
+- Remove deprecated --process-dependency-links option from UPGRADE.rst ([\#4485](https://github.com/matrix-org/synapse/issues/4485))
+- Make it possible to set the log level for tests via an environment variable ([\#4506](https://github.com/matrix-org/synapse/issues/4506))
+- Reduce the log level of linearizer lock acquirement to DEBUG. ([\#4507](https://github.com/matrix-org/synapse/issues/4507))
+- Fix code to comply with linting in PyFlakes 3.7.1. ([\#4519](https://github.com/matrix-org/synapse/issues/4519))
+
+
 Synapse 0.34.1.1 (2019-01-11)
 =============================
 

UPGRADE.rst

@@ -48,6 +48,38 @@ returned by the Client-Server API:
     # configured on port 443.
     curl -kv https://<host.name>/_matrix/client/versions 2>&1 | grep "Server:"
 
+Upgrading to v0.99.0
+====================
+
+In preparation for Synapse v1.0, you must update your TLS certificates from
+self-signed ones to verifiable ones signed by a trusted root CA.
+
+If you do not already have a certificate for your domain, the easiest way to get
+one is with Synapse's new ACME support, which will use the ACME protocol to
+provision a certificate automatically. By default, certificates will be obtained
+from the publicly trusted CA Let's Encrypt.
+
+For a sample configuration, please inspect the new ACME section in the example
+generated config by running the ``generate-config`` executable. For example::
+
+  ~/synapse/env3/bin/generate-config
+
+You will need to provide Let's Encrypt (or other ACME provider) access to your
+Synapse ACME challenge responder on port 80, at the domain of your homeserver.
+This requires you either change the port of the ACME listener provided by
+Synapse to a high port and reverse proxy to it, or use a tool like authbind to
+allow Synapse to listen on port 80 without root access. (Do not run Synapse with
+root permissions!)
+
+You will need to back up or delete your self signed TLS certificate
+(``example.com.tls.crt`` and ``example.com.tls.key``), Synapse's ACME
+implementation will not overwrite them.
+
+You may wish to use alternate methods such as Certbot to obtain a certificate
+from Let's Encrypt, depending on your server configuration. Of course, if you
+already have a valid certificate for your homeserver's domain, that can be
+placed in Synapse's config directory without the need for ACME.
+
 Upgrading to v0.34.0
 ====================
 

changelog.d/3902.feature

@@ -0,0 +1 @@
+Include m.room.encryption on invites by default

changelog.d/4229.feature

@@ -1 +0,0 @@
-Synapse's cipher string has been updated to require ECDH key exchange. Configuring and generating dh_params is no longer required, and they will be ignored.

changelog.d/4306.misc

@@ -1 +0,0 @@
-Synapse will now take advantage of native UPSERT functionality in PostgreSQL 9.5+ and SQLite 3.24+.

changelog.d/4342.misc

@@ -1 +0,0 @@
-Update README to use the new virtualenv everywhere

changelog.d/4368.misc

@@ -1 +0,0 @@
-Add better logging for unexpected errors while sending transactions

changelog.d/4369.bugfix

@@ -1 +0,0 @@
-Prevent users with access tokens predating the introduction of device IDs from creating spurious entries in the user_ips table.

changelog.d/4370.misc

@@ -1 +0,0 @@
-Apply a unique index to the user_ips table, preventing duplicates.

changelog.d/4377.misc

@@ -1 +0,0 @@
-Silence travis-ci build warnings by removing non-functional python3.6

changelog.d/4384.feature

@@ -1 +0,0 @@
-Synapse can now automatically provision TLS certificates via ACME (the protocol used by CAs like Let's Encrypt).

changelog.d/4387.misc

@@ -1 +0,0 @@
-Fix a comment in the generated config file

changelog.d/4390.misc

@@ -1 +0,0 @@
-Add ground work for implementing future federation API versions

changelog.d/4392.bugfix

@@ -1 +0,0 @@
-Fix typo in ALL_USER_TYPES definition to ensure type is a tuple

changelog.d/4397.bugfix

@@ -1 +0,0 @@
-Fix high CPU usage due to remote devicelist updates

changelog.d/4399.misc

@@ -1 +0,0 @@
-Update dependencies on msgpack and pymacaroons to use the up-to-date packages.

changelog.d/4400.misc

@@ -1 +0,0 @@
-Tweak codecov settings to make them less loud.

changelog.d/4402.misc

@@ -1 +0,0 @@
-Implement server support for MSC1794 - Federation v2 Invite API

changelog.d/4404.bugfix

@@ -1 +0,0 @@
-Fix potential bug where creating or joining a room could fail

changelog.d/4405.bugfix

@@ -1 +0,0 @@
-Fix bug when rejecting remote invites

changelog.d/4407.bugfix

@@ -1 +0,0 @@
-Fix incorrect logcontexts after a Deferred was cancelled

changelog.d/4408.feature

@@ -1 +0,0 @@
-Implement MSC1708 (.well-known routing for server-server federation)

changelog.d/4409.feature

@@ -1 +0,0 @@
-Implement MSC1708 (.well-known routing for server-server federation)

changelog.d/4411.bugfix

@@ -1 +0,0 @@
-Ensure encrypted room state is persisted across room upgrades.

changelog.d/4412.bugfix

@@ -1 +0,0 @@
-Copy over whether a room is a direct message and any associated room tags on room upgrade.

changelog.d/4415.feature

@@ -1 +0,0 @@
-Search now includes results from predecessor rooms after a room upgrade.

changelog.d/4423.feature

@@ -1 +0,0 @@
-Config option to disable requesting MSISDN on registration.

changelog.d/4426.feature

@@ -1 +0,0 @@
-Implement MSC1708 (.well-known routing for server-server federation)

changelog.d/4427.feature

@@ -1 +0,0 @@
-Implement MSC1708 (.well-known routing for server-server federation)

changelog.d/4428.feature

@@ -1 +0,0 @@
-Implement MSC1708 (.well-known routing for server-server federation)

changelog.d/4432.misc

@@ -1 +0,0 @@
-Apply a unique index to the user_ips table, preventing duplicates.

changelog.d/4433.misc

@@ -1 +0,0 @@
-debian package: symlink to explicit python version

changelog.d/4434.misc

@@ -1 +0,0 @@
-Apply a unique index to the user_ips table, preventing duplicates.

changelog.d/4435.bugfix

@@ -1 +0,0 @@
-Fix None guard in calling config.server.is_threepid_reserved

changelog.d/4437.misc

@@ -1 +0,0 @@
-Add infrastructure to support different event formats

changelog.d/4444.misc

@@ -1 +0,0 @@
-Generate the debian config during build

changelog.d/4445.feature

@@ -1 +0,0 @@
-Add a metric for tracking event stream position of the user directory.

changelog.d/4447.misc

@@ -1 +0,0 @@
-Add infrastructure to support different event formats

changelog.d/4448.misc

@@ -1 +0,0 @@
-Add infrastructure to support different event formats

changelog.d/4452.bugfix

@@ -1 +0,0 @@
-Don't send IP addresses as SNI

changelog.d/4458.misc

@@ -1 +0,0 @@
-Clarify documentation for the `public_baseurl` config param

changelog.d/4459.misc

@@ -1 +0,0 @@
-Synapse will now take advantage of native UPSERT functionality in PostgreSQL 9.5+ and SQLite 3.24+.

changelog.d/4460.bugfix

@@ -1 +0,0 @@
-Fix UnboundLocalError in post_urlencoded_get_json

changelog.d/4461.bugfix

@@ -1 +0,0 @@
-Add a timeout to filtered room directory queries.

changelog.d/4464.feature

@@ -1 +0,0 @@
-Implement MSC1708 (.well-known routing for server-server federation)

changelog.d/4466.misc

@@ -1 +0,0 @@
-Synapse will now take advantage of native UPSERT functionality in PostgreSQL 9.5+ and SQLite 3.24+.

changelog.d/4468.feature

@@ -1 +0,0 @@
-Implement MSC1708 (.well-known routing for server-server federation)

changelog.d/4470.misc

@@ -1 +0,0 @@
-Add infrastructure to support different event formats

changelog.d/4471.misc

@@ -1 +0,0 @@
- Synapse will now take advantage of native UPSERT functionality in PostgreSQL 9.5+ and SQLite 3.24+.

changelog.d/4472.feature

@@ -1 +0,0 @@
-Support exposing server capabilities in CS API (MSC1753, MSC1804)

changelog.d/4476.misc

@@ -1 +0,0 @@
-Fix quoting for allowed_local_3pids example config

changelog.d/4477.misc

@@ -1 +0,0 @@
-Synapse will now take advantage of native UPSERT functionality in PostgreSQL 9.5+ and SQLite 3.24+.

changelog.d/4481.misc

@@ -1 +0,0 @@
-Add infrastructure to support different event formats

changelog.d/4482.misc

@@ -1 +0,0 @@
-Add infrastructure to support different event formats

changelog.d/4483.feature

@@ -1 +0,0 @@
-Add support for room version 3

changelog.d/4485.misc

@@ -1 +0,0 @@
-Remove deprecated --process-dependency-links option from UPGRADE.rst

changelog.d/4486.bugfix

@@ -1 +0,0 @@
-Workaround for login error when using both LDAP and internal authentication.

changelog.d/4487.feature

@@ -1 +0,0 @@
-Implement MSC1708 (.well-known routing for server-server federation)

changelog.d/4488.feature

@@ -1 +0,0 @@
-Implement MSC1708 (.well-known routing for server-server federation)

changelog.d/4489.feature

@@ -1 +0,0 @@
-Implement MSC1708 (.well-known routing for server-server federation)

changelog.d/4492.feature

@@ -1 +0,0 @@
- Synapse can now automatically provision TLS certificates via ACME (the protocol used by CAs like Let's Encrypt).

changelog.d/4493.misc

@@ -1 +0,0 @@
-Add infrastructure to support different event formats

changelog.d/4494.misc

@@ -1 +0,0 @@
-Add infrastructure to support different event formats

changelog.d/4495.feature

@@ -1 +0,0 @@
-Synapse will now reload TLS certificates from disk upon SIGHUP.

changelog.d/4496.misc

@@ -1 +0,0 @@
-Add infrastructure to support different event formats

changelog.d/4497.feature

@@ -1 +0,0 @@
-Implement MSC1708 (.well-known routing for server-server federation)

changelog.d/4498.misc

@@ -1 +0,0 @@
-Clarify documentation for the `public_baseurl` config param

changelog.d/4499.feature

@@ -1 +0,0 @@
-Add support for room version 3

changelog.d/4505.misc

@@ -1 +0,0 @@
- Synapse will now take advantage of native UPSERT functionality in PostgreSQL 9.5+ and SQLite 3.24+.

changelog.d/4506.misc

@@ -1 +0,0 @@
-Make it possible to set the log level for tests via an environment variable

changelog.d/4507.misc

@@ -1 +0,0 @@
-Reduce the log level of linearizer lock acquirement to DEBUG.

changelog.d/4509.removal

@@ -1 +0,0 @@
-Synapse no longer generates self-signed TLS certificates when generating a configuration file.

changelog.d/4510.misc

@@ -1 +0,0 @@
-Add infrastructure to support different event formats

changelog.d/4511.feature

@@ -1 +0,0 @@
-Implement MSC1708 (.well-known routing for server-server federation)

changelog.d/4512.bugfix

@@ -1 +0,0 @@
-Fix a bug where setting a relative consent directory path would cause a crash.

changelog.d/4513.misc

@@ -0,0 +1 @@
+Reject federation transactions if they include more than 50 PDUs or 100 EDUs.

changelog.d/4514.misc

@@ -1 +0,0 @@
-Add infrastructure to support different event formats

changelog.d/4515.feature

@@ -1 +0,0 @@
-Add support for room version 3

changelog.d/4516.feature

@@ -1 +0,0 @@
-Implement MSC1708 (.well-known routing for server-server federation)

changelog.d/4519.misc

@@ -1 +0,0 @@
-Fix code to comply with linting in PyFlakes 3.7.1.

changelog.d/4520.feature

@@ -1 +0,0 @@
-Implement MSC1708 (.well-known routing for server-server federation)

changelog.d/4521.feature

@@ -1 +0,0 @@
-Implement MSC1708 (.well-known routing for server-server federation)

changelog.d/4523.feature

@@ -1 +0,0 @@
-Add support for room version 3

changelog.d/4524.feature

@@ -1 +0,0 @@
-Synapse will now reload TLS certificates from disk upon SIGHUP.

changelog.d/4525.feature

@@ -1 +0,0 @@
- Synapse can now automatically provision TLS certificates via ACME (the protocol used by CAs like Let's Encrypt).

synapse/__init__.py

@@ -27,4 +27,4 @@ try:
 except ImportError:
     pass
 
-__version__ = "0.34.1.1"
+__version__ = "0.99.0rc2"

synapse/api/constants.py

@@ -73,6 +73,7 @@ class EventTypes(object):
     RoomHistoryVisibility = "m.room.history_visibility"
     CanonicalAlias = "m.room.canonical_alias"
     RoomAvatar = "m.room.avatar"
+    RoomEncryption = "m.room.encryption"
     GuestAccess = "m.room.guest_access"
 
     # These are used for validation
@@ -109,7 +110,7 @@ class RoomVersions(object):
 
 
 class RoomDisposition(object):
-    STABLE = "stable",
+    STABLE = "stable"
     UNSTABLE = "unstable"
 
 

synapse/config/api.py

@@ -24,6 +24,7 @@ class ApiConfig(Config):
             EventTypes.JoinRules,
             EventTypes.CanonicalAlias,
             EventTypes.RoomAvatar,
+            EventTypes.RoomEncryption,
             EventTypes.Name,
         ])
 
@@ -36,5 +37,6 @@ class ApiConfig(Config):
             - "{JoinRules}"
             - "{CanonicalAlias}"
             - "{RoomAvatar}"
+            - "{RoomEncryption}"
             - "{Name}"
         """.format(**vars(EventTypes))

synapse/federation/federation_server.py

@@ -148,6 +148,22 @@ class FederationServer(FederationBase):
 
         logger.debug("[%s] Transaction is new", transaction.transaction_id)
 
+        # Reject if PDU count > 50 and EDU count > 100
+        if (len(transaction.pdus) > 50
+                or (hasattr(transaction, "edus") and len(transaction.edus) > 100)):
+
+            logger.info(
+                "Transaction PDU or EDU count too large. Returning 400",
+            )
+
+            response = {}
+            yield self.transaction_actions.set_response(
+                origin,
+                transaction,
+                400, response
+            )
+            defer.returnValue((400, response))
+
         received_pdus_counter.inc(len(transaction.pdus))
 
         origin_host, _ = parse_server_name(origin)

synapse/handlers/sync.py

@@ -1473,10 +1473,22 @@ class SyncHandler(object):
                 if since_token and since_token.is_after(leave_token):
                     continue
 
+                # If this is an out of band message, like a remote invite
+                # rejection, we include it in the recents batch. Otherwise, we
+                # let _load_filtered_recents handle fetching the correct
+                # batches.
+                #
+                # This is all screaming out for a refactor, as the logic here is
+                # subtle and the moving parts numerous.
+                if leave_event.internal_metadata.is_out_of_band_membership():
+                    batch_events = [leave_event]
+                else:
+                    batch_events = None
+
                 room_entries.append(RoomSyncResultBuilder(
                     room_id=room_id,
                     rtype="archived",
-                    events=None,
+                    events=batch_events,
                     newly_joined=room_id in newly_joined_rooms,
                     full_state=False,
                     since_token=since_token,