Merge branch 'release-v0.34.0' into develop

.circleci/merge_base_branch.sh

@@ -20,7 +20,7 @@ else
 fi
 
 # Show what we are before
-git show -s
+git --no-pager show -s
 
 # Set up username so it can do a merge
 git config --global user.email bot@matrix.org
@@ -31,4 +31,4 @@ git fetch -u origin $GITBASE
 git merge --no-edit origin/$GITBASE
 
 # Show what we are after.
-git show -s
+git --no-pager show -s

CHANGES.md

@@ -1,3 +1,73 @@
+Synapse 0.34.0rc2 (2018-12-11)
+==============================
+
+Features
+--------
+
+- Add a welcome page for the client API port. Credit to @krombel! ([\#4289](https://github.com/matrix-org/synapse/issues/4289))
+- Remove Matrix console from the default distribution ([\#4290](https://github.com/matrix-org/synapse/issues/4290))
+
+
+Internal Changes
+----------------
+
+- Disable pager when running git-show in CI ([\#4291](https://github.com/matrix-org/synapse/issues/4291))
+
+
+Synapse 0.34.0rc1 (2018-12-04)
+==============================
+
+Synapse 0.34 is the first release to fully support Python 3. We recommend
+upgrading to Python 3, but make sure to read the
+[upgrade notes](UPGRADE.rst#upgrading-to-v0340) when doing so.
+
+Features
+--------
+
+- Add option to track MAU stats (but not limit people) ([\#3830](https://github.com/matrix-org/synapse/issues/3830))
+- Add an option to enable recording IPs for appservice users ([\#3831](https://github.com/matrix-org/synapse/issues/3831))
+- Rename login type m.login.cas to m.login.sso ([\#4220](https://github.com/matrix-org/synapse/issues/4220))
+- Add an option to disable search for homeservers that may not be interested in it. ([\#4230](https://github.com/matrix-org/synapse/issues/4230))
+
+
+Bugfixes
+--------
+
+- Pushrules can now again be made with non-ASCII rule IDs. ([\#4165](https://github.com/matrix-org/synapse/issues/4165))
+- The media repository now no longer fails to decode UTF-8 filenames when downloading remote media. ([\#4176](https://github.com/matrix-org/synapse/issues/4176))
+- URL previews now correctly decode non-UTF-8 text if the header contains a `<meta http-equiv="Content-Type"` header. ([\#4183](https://github.com/matrix-org/synapse/issues/4183))
+- Fix an issue where public consent URLs had two slashes. ([\#4192](https://github.com/matrix-org/synapse/issues/4192))
+- Fallback auth now accepts the session parameter on Python 3. ([\#4197](https://github.com/matrix-org/synapse/issues/4197))
+- Remove riot.im from the list of trusted Identity Servers in the default configuration ([\#4207](https://github.com/matrix-org/synapse/issues/4207))
+- fix start up failure when mau_limit_reserved_threepids set and db is postgres ([\#4211](https://github.com/matrix-org/synapse/issues/4211))
+- Fix auto join failures for servers that require user consent ([\#4223](https://github.com/matrix-org/synapse/issues/4223))
+- Fix exception caused by non-ascii event IDs ([\#4241](https://github.com/matrix-org/synapse/issues/4241))
+- Pushers can now be unsubscribed from on Python 3. ([\#4250](https://github.com/matrix-org/synapse/issues/4250))
+- Fix UnicodeDecodeError when postgres is configured to give non-English errors ([\#4253](https://github.com/matrix-org/synapse/issues/4253))
+
+
+Internal Changes
+----------------
+
+- A coveragerc file, as well as the py36-coverage tox target, have been added. ([\#4180](https://github.com/matrix-org/synapse/issues/4180))
+- Add a GitHub pull request template and add multiple issue templates ([\#4182](https://github.com/matrix-org/synapse/issues/4182))
+- Update README to reflect the fact that #1491 is fixed ([\#4188](https://github.com/matrix-org/synapse/issues/4188))
+- Run the AS senders as background processes to fix warnings ([\#4189](https://github.com/matrix-org/synapse/issues/4189))
+- Add some diagnostics to the tests to detect logcontext problems ([\#4190](https://github.com/matrix-org/synapse/issues/4190))
+- Add missing `jpeg` package prerequisite for OpenBSD in README. ([\#4193](https://github.com/matrix-org/synapse/issues/4193))
+- Add a note saying you need to manually reclaim disk space after using the Purge History API ([\#4200](https://github.com/matrix-org/synapse/issues/4200))
+- More logcontext checking in unittests ([\#4205](https://github.com/matrix-org/synapse/issues/4205))
+- Ignore __pycache__ directories in the database schema folder ([\#4214](https://github.com/matrix-org/synapse/issues/4214))
+- Add note to UPGRADE.rst about removing riot.im from list of trusted identity servers ([\#4224](https://github.com/matrix-org/synapse/issues/4224))
+- Added automated coverage reporting to CI. ([\#4225](https://github.com/matrix-org/synapse/issues/4225))
+- Garbage-collect after each unit test to fix logcontext leaks ([\#4227](https://github.com/matrix-org/synapse/issues/4227))
+- add more detail to logging regarding "More than one row matched" error ([\#4234](https://github.com/matrix-org/synapse/issues/4234))
+- Drop sent_transactions table ([\#4244](https://github.com/matrix-org/synapse/issues/4244))
+- Add a basic .editorconfig ([\#4257](https://github.com/matrix-org/synapse/issues/4257))
+- Update README.rst and UPGRADE.rst for Python 3. ([\#4260](https://github.com/matrix-org/synapse/issues/4260))
+- Remove obsolete `verbose` and `log_file` settings from `homeserver.yaml` for Docker image. ([\#4261](https://github.com/matrix-org/synapse/issues/4261))
+
+
 Synapse 0.33.9 (2018-11-19)
 ===========================
 
@@ -71,7 +141,7 @@ Synapse 0.33.8rc2 (2018-10-31)
 Bugfixes
 --------
 
-- Searches that request profile info now no longer fail with a 500. Fixes 
+- Searches that request profile info now no longer fail with a 500. Fixes
   a regression in 0.33.8rc1. ([\#4122](https://github.com/matrix-org/synapse/issues/4122))
 
 

README.rst

@@ -289,10 +289,6 @@ go back in your web client and proceed further.
 If all goes well you should at least be able to log in, create a room, and
 start sending messages.
 
-(The homeserver runs a web client by default at https://localhost:8448/, though
-as of the time of writing it is somewhat outdated and not really recommended -
-https://github.com/matrix-org/synapse/issues/1527).
-
 .. _`client-user-reg`:
 
 Registering a new user from a client
@@ -372,10 +368,7 @@ ArchLinux
 
 The quickest way to get up and running with ArchLinux is probably with the community package
 https://www.archlinux.org/packages/community/any/matrix-synapse/, which should pull in most of
-the necessary dependencies. If the default web client is to be served (enabled by default in
+the necessary dependencies.
-the generated config),
-https://www.archlinux.org/packages/community/any/python2-matrix-angular-sdk/ will also need to
-be installed.
 
 pip may be outdated (6.0.7-1 and needs to be upgraded to 6.0.8-1 )::
 

UPGRADE.rst

@@ -115,6 +115,10 @@ Upgrading to v0.34.0
    case a hypothetical future identity server was put there. If you don't
    remove it, users may be unable to deactivate their accounts.
 
+3. This release no longer installs the (unmaintained) Matrix Console web client
+   as part of the default installation. It is possible to re-enable it by
+   installing it separately and setting the ``web_client_location`` config
+   option, but please consider switching to another client.
 
 Upgrading to v0.33.7
 ====================

changelog.d/3830.feature

@@ -1 +0,0 @@
-Add option to track MAU stats (but not limit people)

changelog.d/3831.feature

@@ -1 +0,0 @@
-Add an option to enable recording IPs for appservice users

changelog.d/4165.bugfix

@@ -1 +0,0 @@
-Pushrules can now again be made with non-ASCII rule IDs.

changelog.d/4176.bugfix

@@ -1 +0,0 @@
-The media repository now no longer fails to decode UTF-8 filenames when downloading remote media.

changelog.d/4180.misc

@@ -1 +0,0 @@
-A coveragerc file, as well as the py36-coverage tox target, have been added.

changelog.d/4182.misc

@@ -1 +0,0 @@
-Add a GitHub pull request template and add multiple issue templates

changelog.d/4183.bugfix

@@ -1 +0,0 @@
-URL previews now correctly decode non-UTF-8 text if the header contains a `<meta http-equiv="Content-Type"` header.

changelog.d/4188.misc

@@ -1 +0,0 @@
-Update README to reflect the fact that #1491 is fixed

changelog.d/4189.misc

@@ -1,2 +0,0 @@
-Run the AS senders as background processes to fix warnings
-

changelog.d/4190.misc

@@ -1 +0,0 @@
-Add some diagnostics to the tests to detect logcontext problems

changelog.d/4192.bugfix

@@ -1 +0,0 @@
-Fix an issue where public consent URLs had two slashes.

changelog.d/4193.misc

@@ -1 +0,0 @@
-Add missing `jpeg` package prerequisite for OpenBSD in README.

changelog.d/4197.bugfix

@@ -1 +0,0 @@
-Fallback auth now accepts the session parameter on Python 3.

changelog.d/4200.misc

@@ -1 +0,0 @@
-Add a note saying you need to manually reclaim disk space after using the Purge History API

changelog.d/4204.misc

@@ -1 +0,0 @@
-Fix logcontext leaks in EmailPusher and in tests

changelog.d/4205.misc

@@ -1 +0,0 @@
-More logcontext checking in unittests

changelog.d/4207.bugfix

@@ -1 +0,0 @@
-Remove riot.im from the list of trusted Identity Servers in the default configuration

changelog.d/4209.misc

@@ -1 +0,0 @@
-Fix logcontext leaks in EmailPusher and in tests

changelog.d/4211.bugfix

@@ -1,2 +0,0 @@
-fix start up failure when mau_limit_reserved_threepids set and db is postgres 
-

changelog.d/4214.misc

@@ -1 +0,0 @@
-Ignore __pycache__ directories in the database schema folder

changelog.d/4220.feature

@@ -1 +0,0 @@
-Rename login type m.login.cas to m.login.sso

changelog.d/4223.bugfix

@@ -1 +0,0 @@
-Fix auto join failures for servers that require user consent

changelog.d/4224.misc

@@ -1 +0,0 @@
-Add note to UPGRADE.rst about removing riot.im from list of trusted identity servers

changelog.d/4225.misc

@@ -1 +0,0 @@
-Added automated coverage reporting to CI.

changelog.d/4227.misc

@@ -1 +0,0 @@
-Garbage-collect after each unit test to fix logcontext leaks

changelog.d/4230.feature

@@ -1 +0,0 @@
-Add an option to disable search for homeservers that may not be interested in it.

changelog.d/4234.misc

@@ -1 +0,0 @@
-add more detail to logging regarding "More than one row matched" error

changelog.d/4241.bugfix

@@ -1 +0,0 @@
-Fix exception caused by non-ascii event IDs

changelog.d/4244.misc

@@ -1 +0,0 @@
-Drop sent_transactions table

changelog.d/4250.bugfix

@@ -1 +0,0 @@
-Pushers can now be unsubscribed from on Python 3.

changelog.d/4253.bugfix

@@ -1 +0,0 @@
-Fix UnicodeDecodeError when postgres is configured to give non-English errors

changelog.d/4257.misc

@@ -1 +0,0 @@
-Add a basic .editorconfig

changelog.d/4260.misc

@@ -1 +0,0 @@
-Update README.rst and UPGRADE.rst for Python 3.

changelog.d/4261.misc

@@ -1 +0,0 @@
-Remove obsolete `verbose` and `log_file` settings from `homeserver.yaml` for Docker image.

synapse/__init__.py

@@ -27,4 +27,4 @@ try:
 except ImportError:
     pass
 
-__version__ = "0.33.9"
+__version__ = "0.34.0rc2"

synapse/app/homeserver.py

@@ -54,7 +54,7 @@ from synapse.metrics import RegistryProxy
 from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.metrics.resource import METRICS_PREFIX, MetricsResource
 from synapse.module_api import ModuleApi
-from synapse.python_dependencies import CONDITIONAL_REQUIREMENTS, check_requirements
+from synapse.python_dependencies import check_requirements
 from synapse.replication.http import REPLICATION_PREFIX, ReplicationRestResource
 from synapse.replication.tcp.resource import ReplicationStreamProtocolFactory
 from synapse.rest import ClientRestResource
@@ -80,36 +80,6 @@ def gz_wrap(r):
     return EncodingResourceWrapper(r, [GzipEncoderFactory()])
 
 
-def build_resource_for_web_client(hs):
-    webclient_path = hs.get_config().web_client_location
-    if not webclient_path:
-        try:
-            import syweb
-        except ImportError:
-            quit_with_error(
-                "Could not find a webclient.\n\n"
-                "Please either install the matrix-angular-sdk or configure\n"
-                "the location of the source to serve via the configuration\n"
-                "option `web_client_location`\n\n"
-                "To install the `matrix-angular-sdk` via pip, run:\n\n"
-                "    pip install '%(dep)s'\n"
-                "\n"
-                "You can also disable hosting of the webclient via the\n"
-                "configuration option `web_client`\n"
-                % {"dep": CONDITIONAL_REQUIREMENTS["web_client"].keys()[0]}
-            )
-        syweb_path = os.path.dirname(syweb.__file__)
-        webclient_path = os.path.join(syweb_path, "webclient")
-    # GZip is disabled here due to
-    # https://twistedmatrix.com/trac/ticket/7678
-    # (It can stay enabled for the API resources: they call
-    # write() with the whole body and then finish() straight
-    # after and so do not trigger the bug.
-    # GzipFile was removed in commit 184ba09
-    # return GzipFile(webclient_path)  # TODO configurable?
-    return File(webclient_path)  # TODO configurable?
-
-
 class SynapseHomeServer(HomeServer):
     DATASTORE_CLASS = DataStore
 
@@ -138,8 +108,11 @@ class SynapseHomeServer(HomeServer):
             handler = handler_cls(config, module_api)
             resources[path] = AdditionalResource(self, handler.handle_request)
 
+        # try to find something useful to redirect '/' to
         if WEB_CLIENT_PREFIX in resources:
             root_resource = RootRedirect(WEB_CLIENT_PREFIX)
+        elif STATIC_PREFIX in resources:
+            root_resource = RootRedirect(STATIC_PREFIX)
         else:
             root_resource = NoResource()
 
@@ -243,7 +216,16 @@ class SynapseHomeServer(HomeServer):
             resources[SERVER_KEY_V2_PREFIX] = KeyApiV2Resource(self)
 
         if name == "webclient":
-            resources[WEB_CLIENT_PREFIX] = build_resource_for_web_client(self)
+            webclient_path = self.get_config().web_client_location
+
+            if webclient_path is None:
+                logger.warning(
+                    "Not enabling webclient resource, as web_client_location is unset."
+                )
+            else:
+                # GZip is disabled here due to
+                # https://twistedmatrix.com/trac/ticket/7678
+                resources[WEB_CLIENT_PREFIX] = File(webclient_path)
 
         if name == "metrics" and self.get_config().enable_metrics:
             resources[METRICS_PREFIX] = MetricsResource(RegistryProxy)

synapse/config/server.py

@@ -34,7 +34,6 @@ class ServerConfig(Config):
             raise ConfigError(str(e))
 
         self.pid_file = self.abspath(config.get("pid_file"))
-        self.web_client = config["web_client"]
         self.web_client_location = config.get("web_client_location", None)
         self.soft_file_limit = config["soft_file_limit"]
         self.daemonize = config.get("daemonize")
@@ -128,6 +127,9 @@ class ServerConfig(Config):
             elif not bind_addresses:
                 bind_addresses.append('')
 
+        if not self.web_client_location:
+            _warn_if_webclient_configured(self.listeners)
+
         self.gc_thresholds = read_gc_thresholds(config.get("gc_thresholds", None))
 
         bind_port = config.get("bind_port")
@@ -136,8 +138,6 @@ class ServerConfig(Config):
             bind_host = config.get("bind_host", "")
             gzip_responses = config.get("gzip_responses", True)
 
-            names = ["client", "webclient"] if self.web_client else ["client"]
-
             self.listeners.append({
                 "port": bind_port,
                 "bind_addresses": [bind_host],
@@ -145,7 +145,7 @@ class ServerConfig(Config):
                 "type": "http",
                 "resources": [
                     {
-                        "names": names,
+                        "names": ["client"],
                         "compress": gzip_responses,
                     },
                     {
@@ -164,7 +164,7 @@ class ServerConfig(Config):
                     "type": "http",
                     "resources": [
                         {
-                            "names": names,
+                            "names": ["client"],
                             "compress": gzip_responses,
                         },
                         {
@@ -247,13 +247,9 @@ class ServerConfig(Config):
         #
         # cpu_affinity: 0xFFFFFFFF
 
-        # Whether to serve a web client from the HTTP/HTTPS root resource.
+        # The path to the web client which will be served at /_matrix/client/
-        web_client: True
+        # if 'webclient' is configured under the 'listeners' configuration.
-
+        #
-        # The root directory to server for the above web client.
-        # If left undefined, synapse will serve the matrix-angular-sdk web client.
-        # Make sure matrix-angular-sdk is installed with pip if web_client is True
-        # and web_client_location is undefined
         # web_client_location: "/path/to/web/root"
 
         # The public-facing base URL for the client API (not including _matrix/...)
@@ -320,8 +316,8 @@ class ServerConfig(Config):
               -
                 # List of resources to host on this listener.
                 names:
-                  - client     # The client-server APIs, both v1 and v2
+                  - client       # The client-server APIs, both v1 and v2
-                  - webclient  # The bundled webclient.
+                  # - webclient  # A web client. Requires web_client_location to be set.
 
                 # Should synapse compress HTTP responses to clients that support it?
                 # This should be disabled if running synapse behind a load balancer
@@ -348,7 +344,7 @@ class ServerConfig(Config):
             x_forwarded: false
 
             resources:
-              - names: [client, webclient]
+              - names: [client]
                 compress: true
               - names: [federation]
                 compress: false
@@ -452,3 +448,19 @@ def read_gc_thresholds(thresholds):
         raise ConfigError(
             "Value of `gc_threshold` must be a list of three integers if set"
         )
+
+
+NO_MORE_WEB_CLIENT_WARNING = """
+Synapse no longer includes a web client. To enable a web client, configure
+web_client_location. To remove this warning, remove 'webclient' from the 'listeners'
+configuration.
+"""
+
+
+def _warn_if_webclient_configured(listeners):
+    for listener in listeners:
+        for res in listener.get("resources", []):
+            for name in res.get("names", []):
+                if name == 'webclient':
+                    logger.warning(NO_MORE_WEB_CLIENT_WARNING)
+                    return

synapse/python_dependencies.py

@@ -68,9 +68,6 @@ REQUIREMENTS = {
 }
 
 CONDITIONAL_REQUIREMENTS = {
-    "web_client": {
-        "matrix_angular_sdk>=0.6.8": ["syweb>=0.6.8"],
-    },
     "email.enable_notifs": {
         "Jinja2>=2.8": ["Jinja2>=2.8"],
         "bleach>=1.4.2": ["bleach>=1.4.2"],

synapse/static/index.html

@@ -0,0 +1,26 @@
+<html>
+   <head>
+       <title>Synapse is running</title>
+       <style>
+           body {
+               width: 30em;
+               margin: 0 auto;
+               font-family: Tahoma, Verdana, Arial, sans-serif;
+           }
+           h1 {
+               text-align: center;
+           }
+       </style>
+   </head>
+   <body>
+       <h1>Synapse is running</h1>
+       <p>Congratulations!</p>
+       <p>Your Synapse server is listening on this port and is ready for messages.</p>
+       <p>To use this server you'll need a client - e.g. one of
+       <a href="https://matrix.org/docs/projects/try-matrix-now.html#clients">this list of Matrix clients</a>.</p>
+       <p>You can find (federated) rooms that might be of interest to you on
+       <a href="https://view.matrix.org/">view.matrix.org</a>.</p>
+       <p>Or you just start creating your own rooms with your friends.</p>
+       <p>Welcome to the Matrix universe :)</p>
+   </body>
+</html>