Merge remote-tracking branch 'origin/develop' into room-initial-sync

README.rst

@@ -4,9 +4,9 @@ Introduction
 Matrix is an ambitious new ecosystem for open federated Instant Messaging and
 VoIP.  The basics you need to know to get up and running are:
 
-- Chatrooms are distributed and do not exist on any single server.  Rooms
+- Everything in Matrix happens in a room.  Rooms are distributed and do not
-  can be found using aliases like ``#matrix:matrix.org`` or
+  exist on any single server.  Rooms can be located using convenience aliases 
-  ``#test:localhost:8008`` or they can be ephemeral.
+  like ``#matrix:matrix.org`` or ``#test:localhost:8008``.
 
 - Matrix user IDs look like ``@matthew:matrix.org`` (although in the future
   you will normally refer to yourself and others using a 3PID: email
@@ -17,56 +17,12 @@ The overall architecture is::
       client <----> homeserver <=====================> homeserver <----> client
              https://somewhere.org/_matrix      https://elsewhere.net/_matrix
 
-WARNING
+``#matrix:matrix.org`` is the official support room for Matrix, and can be
-=======
+accessed by the web client at http://matrix.org/alpha or via an IRC bridge at
-
+irc://irc.freenode.net/matrix.
-**Synapse is currently in a state of rapid development, and not all features
-are yet functional. Critically, some security features are still in
-development, which means Synapse can *not* be considered secure or reliable at
-this point.**  For instance:
-
-- **SSL Certificates used by server-server federation are not yet validated.**
-- **Room permissions are not yet enforced on traffic received via federation.**
-- **Homeservers do not yet cryptographically sign their events to avoid
-  tampering**
-- Default configuration provides open signup to the service from the internet
-
-Despite this, we believe Synapse is more than useful as a way for experimenting
-and exploring Synapse, and the missing features will land shortly. **Until
-then, please do *NOT* use Synapse for any remotely important or secure
-communication.**
-
-
-Quick Start
-===========
-
-System requirements:
-- POSIX-compliant system (tested on Linux & OSX)
-- Python 2.7
-
-To get up and running:
-
-- To simply play with an **existing** homeserver you can
-  just go straight to http://matrix.org/alpha.
-
-- To run your own **private** homeserver on localhost:8008, generate a basic
-  config file: ``./synctl start`` will give you instructions on how to do this.
-  For this purpose, you can use 'localhost' or your hostname as a server name.
-  Once you've done so, running ``./synctl start`` again will start your private
-  home server. You will find a webclient running at http://localhost:8008.
-  Please use a recent Chrome or Firefox for now (or Safari if you don't need
-  VoIP support).
-
-- To run a **public** homeserver and let it exchange messages with other
-  homeservers and participate in the global Matrix federation, you must expose
-  port 8448 to the internet and edit homeserver.yaml to specify server_name
-  (the public DNS entry for this server) and then run ``synctl start``. If you
-  changed the server_name, you may need to move the old database
-  (homeserver.db) out of the way first. Then come join ``#matrix:matrix.org``
-  and say hi! :)
-
-For more detailed setup instructions, please see further down this document.
 
+Synapse is currently in rapid development, but as of version 0.5 we believe it 
+is sufficiently stable to be run as an internet-facing service for real usage!
 
 About Matrix
 ============
@@ -76,10 +32,10 @@ which handle:
 
 - Creating and managing fully distributed chat rooms with no
   single points of control or failure
-- Eventually-consistent cryptographically secure[1] synchronisation of room
+- Eventually-consistent cryptographically secure synchronisation of room
   state across a global open network of federated servers and services
 - Sending and receiving extensible messages in a room with (optional)
-  end-to-end encryption[2]
+  end-to-end encryption[1]
 - Inviting, joining, leaving, kicking, banning room members
 - Managing user accounts (registration, login, logout)
 - Using 3rd Party IDs (3PIDs) such as email addresses, phone numbers,
@@ -111,25 +67,31 @@ Synapse ships with two basic demo Matrix clients: webclient (a basic group chat
 web client demo implemented in AngularJS) and cmdclient (a basic Python
 command line utility which lets you easily see what the JSON APIs are up to).
 
-We'd like to invite you to take a look at the Matrix spec, try to run a
+Meanwhile, iOS and Android SDKs and clients are currently in development and available from:
-homeserver, and join the existing Matrix chatrooms already out there,
-experiment with the APIs and the demo clients, and let us know your thoughts at
-https://github.com/matrix-org/synapse/issues or at matrix@matrix.org.
 
-Thanks for trying Matrix!
+ * https://github.com/matrix-org/matrix-ios-sdk
+ * https://github.com/matrix-org/matrix-android-sdk
 
-[1] Cryptographic signing of messages isn't turned on yet
+We'd like to invite you to join #matrix:matrix.org (via http://matrix.org/alpha), run a homeserver, take a look at the Matrix spec at
+http://matrix.org/docs/spec, experiment with the APIs and the demo
+clients, and report any bugs via http://matrix.org/jira.
 
-[2] End-to-end encryption is currently in development
+Thanks for using Matrix!
+
+[1] End-to-end encryption is currently in development
 
 Homeserver Installation
 =======================
 
+System requirements:
+- POSIX-compliant system (tested on Linux & OSX)
+- Python 2.7
+
 Synapse is written in python but some of the libraries is uses are written in
 C. So before we can install synapse itself we need a working C compiler and the
 header files for python C extensions.
 
-Installing prerequisites on Ubuntu::
+Installing prerequisites on Ubuntu or Debian::
 
     $ sudo apt-get install build-essential python2.7-dev libffi-dev \
                            python-pip python-setuptools
@@ -138,7 +100,61 @@ Installing prerequisites on Mac OS X::
 
     $ xcode-select --install
     
-Synapse uses NaCl (http://nacl.cr.yp.to/) for encryption and digital signatures.
+To install the synapse homeserver run::
+
+    $ pip install --user --process-dependency-links https://github.com/matrix-org/synapse/tarball/master
+
+This installs synapse, along with the libraries it uses, into
+``$HOME/.local/lib/`` on Linux or ``$HOME/Library/Python/2.7/lib/`` on OSX.
+
+Troubleshooting Installation
+----------------------------
+
+Synapse requires pip 1.7 or later, so if your OS provides too old a version and 
+you get errors about ``error: no such option: --process-dependency-links`` you 
+may need to manually upgrade it::
+
+    $ sudo pip install --upgrade pip
+    
+If pip crashes mid-installation for reason (e.g. lost terminal), pip may
+refuse to run until you remove the temporary installation directory it
+created. To reset the installation::
+
+    $ rm -rf /tmp/pip_install_matrix
+    
+pip seems to leak *lots* of memory during installation.  For instance, a Linux 
+host with 512MB of RAM may run out of memory whilst installing Twisted.  If this 
+happens, you will have to individually install the dependencies which are 
+failing, e.g.:
+
+    $ pip install --user twisted
+
+Running Your Homeserver
+=======================
+
+To actually run your new homeserver, pick a working directory for Synapse to run 
+(e.g. ``~/.synapse``), and::
+
+    $ mkdir ~/.synapse
+    $ cd ~/.synapse
+    
+    $ # on Linux
+    $ ~/.local/bin/synctl start
+    
+    $ # on OSX
+    $ ~/Library/Python/2.7/bin/synctl start
+
+Troubleshooting Running
+-----------------------
+
+If ``synctl`` fails with ``pkg_resources.DistributionNotFound`` errors you may 
+need a newer version of setuptools than that provided by your OS.
+
+    $ sudo pip install setuptools --upgrade
+
+If synapse fails with ``missing "sodium.h"`` crypto errors, you may need 
+to manually upgrade PyNaCL, as synapse uses NaCl (http://nacl.cr.yp.to/) for 
+encryption and digital signatures.
 Unfortunately PyNACL currently has a few issues
 (https://github.com/pyca/pynacl/issues/53) and
 (https://github.com/pyca/pynacl/issues/79) that mean it may not install
@@ -151,21 +167,6 @@ fix try re-installing from PyPI or directly from
     $ # Install from github
     $ pip install --user https://github.com/pyca/pynacl/tarball/master
 
-On OSX, if you encounter ``clang: error: unknown argument: '-mno-fused-madd'``
-you will need to ``export CFLAGS=-Qunused-arguments``.
-
-To install the synapse homeserver run::
-
-    $ pip install --user --process-dependency-links https://github.com/matrix-org/synapse/tarball/master
-
-This installs synapse, along with the libraries it uses, into
-``$HOME/.local/lib/``.
-
-To actually run your new homeserver, pick a working directory for Synapse to run (e.g. ``~/.synapse``), and::
-
-    $ mkdir ~/.synapse
-    $ cd ~/.synapse
-    $ synctl start
 
 Homeserver Development
 ======================
@@ -200,8 +201,8 @@ This should end with a 'PASSED' result::
 Upgrading an existing homeserver
 ================================
 
-Before upgrading an existing homeserver to a new version, please refer to
+IMPORTANT: Before upgrading an existing homeserver to a new version, please
-UPGRADE.rst for any additional instructions.
+refer to UPGRADE.rst for any additional instructions.
 
 
 Setting up Federation
@@ -230,10 +231,7 @@ For the first form, simply pass the required hostname (of the machine) as the
         --generate-config
     $ python -m synapse.app.homeserver --config-path homeserver.config
 
-Alternatively, you can run synapse via synctl - running ``synctl start`` to
+Alternatively, you can run ``synctl start`` to guide you through the process.
-generate a homeserver.yaml config file, where you can then edit server-name to
-specify machine.my.domain.name, and then set the actual server running again
-with synctl start.
 
 For the second form, first create your SRV record and publish it in DNS. This
 needs to be named _matrix._tcp.YOURDOMAIN, and point at at least one hostname
@@ -271,6 +269,8 @@ http://localhost:8080. Simply run::
 
     $ demo/start.sh
     
+This is mainly useful just for development purposes.
+
 Running The Demo Web Client
 ===========================
 
@@ -328,13 +328,14 @@ time.
 Where's the spec?!
 ==================
 
-For now, please go spelunking in the ``docs/`` directory to find out.
+The source of the matrix spec lives at https://github.com/matrix-org/matrix-doc.  
+A recent HTML snapshot of this lives at http://matrix.org/docs/spec
 
 
 Building Internal API Documentation
 ===================================
 
-Before building internal API documentation install spinx and
+Before building internal API documentation install sphinx and
 sphinxcontrib-napoleon::
 
     $ pip install sphinx

synapse/api/auth.py

@@ -21,7 +21,7 @@ from synapse.api.constants import Membership, JoinRules
 from synapse.api.errors import AuthError, StoreError, Codes, SynapseError
 from synapse.api.events.room import (
     RoomMemberEvent, RoomPowerLevelsEvent, RoomRedactionEvent,
-    RoomJoinRulesEvent, RoomCreateEvent,
+    RoomJoinRulesEvent, RoomCreateEvent, RoomAliasesEvent,
 )
 from synapse.util.logutils import log_function
 from syutil.base64util import encode_base64
@@ -63,6 +63,10 @@ class Auth(object):
                     # FIXME
                     return True
 
+                # FIXME: Temp hack
+                if event.type == RoomAliasesEvent.TYPE:
+                    return True
+
                 if event.type == RoomMemberEvent.TYPE:
                     allowed = self.is_membership_change_allowed(event)
                     if allowed:
@@ -144,6 +148,17 @@ class Auth(object):
 
     @log_function
     def is_membership_change_allowed(self, event):
+        membership = event.content["membership"]
+
+        # Check if this is the room creator joining:
+        if len(event.prev_events) == 1 and Membership.JOIN == membership:
+            # Get room creation event:
+            key = (RoomCreateEvent.TYPE, "", )
+            create = event.old_state_events.get(key)
+            if event.prev_events[0][0] == create.event_id:
+                if create.content["creator"] == event.state_key:
+                    return True
+
         target_user_id = event.state_key
 
         # get info about the caller
@@ -159,8 +174,6 @@ class Auth(object):
 
         target_in_room = target and target.membership == Membership.JOIN
 
-        membership = event.content["membership"]
-
         key = (RoomJoinRulesEvent.TYPE, "", )
         join_rule_event = event.old_state_events.get(key)
         if join_rule_event:
@@ -255,6 +268,12 @@ class Auth(object):
             level = power_level_event.content.get("users", {}).get(user_id)
             if not level:
                 level = power_level_event.content.get("users_default", 0)
+        else:
+            key = (RoomCreateEvent.TYPE, "", )
+            create_event = event.old_state_events.get(key)
+            if (create_event is not None and
+                create_event.content["creator"] == user_id):
+                return 100
 
         return level
 

synapse/app/homeserver.py

@@ -184,15 +184,7 @@ class SynapseHomeServer(HomeServer):
             logger.info("Synapse now listening on port %d", unsecure_port)
 
 
-def setup():
+def setup(config, run_http=True):
-    config = HomeServerConfig.load_config(
-        "Synapse Homeserver",
-        sys.argv[1:],
-        generate_section="Homeserver"
-    )
-
-    config.setup_logging()
-
     logger.info("Server hostname: %s", config.server_name)
 
     if re.search(":[0-9]+$", config.server_name):
@@ -212,12 +204,13 @@ def setup():
         content_addr=config.content_addr,
     )
 
-    hs.register_servlets()
+    if run_http:
+        hs.register_servlets()
 
-    hs.create_resource_tree(
+        hs.create_resource_tree(
-        web_client=config.webclient,
+            web_client=config.webclient,
-        redirect_root_to_web_client=True,
+            redirect_root_to_web_client=True,
-    )
+        )
 
     db_name = hs.get_db_name()
 
@@ -237,11 +230,18 @@ def setup():
         f.namespace['hs'] = hs
         reactor.listenTCP(config.manhole, f, interface='127.0.0.1')
 
-    bind_port = config.bind_port
+    if run_http:
-    if config.no_tls:
+        bind_port = config.bind_port
-        bind_port = None
+        if config.no_tls:
-    hs.start_listening(bind_port, config.unsecure_port)
+            bind_port = None
+        hs.start_listening(bind_port, config.unsecure_port)
 
+    hs.config = config
+
+    return hs
+
+
+def run(config):
     if config.daemonize:
         print config.pid_file
         daemon = Daemonize(
@@ -257,13 +257,26 @@ def setup():
     else:
         reactor.run()
 
-def run():
-    with LoggingContext("run"):
-        reactor.run()
 
-def main():
+def main(args, run_http=True):
     with LoggingContext("main"):
-        setup()
+        config = HomeServerConfig.load_config(
+            "Synapse Homeserver",
+            args,
+            generate_section="Homeserver"
+        )
+
+        config.setup_logging()
+
+        hs = setup(config, run_http=run_http)
+
+        def r():
+            run(config)
+        hs.run = r
+
+        return hs
+
 
 if __name__ == '__main__':
-    main()
+    hs = main(sys.argv[1:])
+    hs.run()

synapse/app/synctl.py

@@ -56,8 +56,8 @@ def main():
     elif action == "stop":
         stop()
     elif action == "restart":
-        start()
         stop()
+        start()
     else:
         sys.stderr.write("Usage: %s [start|stop|restart]\n" % (sys.argv[0],))
         sys.exit(1)

synapse/handlers/directory.py

@@ -21,8 +21,6 @@ from synapse.api.errors import SynapseError
 from synapse.api.events.room import RoomAliasesEvent
 
 import logging
-import sqlite3
-
 
 logger = logging.getLogger(__name__)
 
@@ -56,17 +54,11 @@ class DirectoryHandler(BaseHandler):
         if not servers:
             raise SynapseError(400, "Failed to get server list")
 
-        try:
+        yield self.store.create_room_alias_association(
-            yield self.store.create_room_alias_association(
+            room_alias,
-                room_alias,
+            room_id,
-                room_id,
+            servers
-                servers
+        )
-            )
-        except sqlite3.IntegrityError:
-            defer.returnValue("Already exists")
-
-        # TODO: Send the room event.
-        yield self._update_room_alias_events(user_id, room_id)
 
     @defer.inlineCallbacks
     def delete_association(self, user_id, room_alias):
@@ -136,7 +128,7 @@ class DirectoryHandler(BaseHandler):
         })
 
     @defer.inlineCallbacks
-    def _update_room_alias_events(self, user_id, room_id):
+    def send_room_alias_update_event(self, user_id, room_id):
         aliases = yield self.store.get_aliases_for_room(room_id)
 
         event = self.event_factory.create_event(

synapse/handlers/room.py

@@ -106,20 +106,37 @@ class RoomCreationHandler(BaseHandler):
             if not room_id:
                 raise StoreError(500, "Couldn't generate a room ID.")
 
+        if room_alias:
+            directory_handler = self.hs.get_handlers().directory_handler
+            yield directory_handler.create_association(
+                user_id=user_id,
+                room_id=room_id,
+                room_alias=room_alias,
+                servers=[self.hs.hostname],
+            )
+
         user = self.hs.parse_userid(user_id)
         creation_events = self._create_events_for_new_room(
             user, room_id, is_public=is_public
         )
 
+        room_member_handler = self.hs.get_handlers().room_member_handler
+
         @defer.inlineCallbacks
         def handle_event(event):
             snapshot = yield self.store.snapshot_room(event)
 
             logger.debug("Event: %s", event)
 
-            yield self._on_new_room_event(
+            if event.type == RoomMemberEvent.TYPE:
-                event, snapshot, extra_users=[user], suppress_auth=True
+                yield room_member_handler.change_membership(
-            )
+                    event,
+                    do_auth=True
+                )
+            else:
+                yield self._on_new_room_event(
+                    event, snapshot, extra_users=[user], suppress_auth=True
+                )
 
         for event in creation_events:
             yield handle_event(event)
@@ -146,16 +163,6 @@ class RoomCreationHandler(BaseHandler):
 
             yield handle_event(topic_event)
 
-        content = {"membership": Membership.JOIN}
-        join_event = self.event_factory.create_event(
-            etype=RoomMemberEvent.TYPE,
-            state_key=user_id,
-            room_id=room_id,
-            user_id=user_id,
-            membership=Membership.JOIN,
-            content=content
-        )
-
         content = {"membership": Membership.INVITE}
         for invitee in invite_list:
             invite_event = self.event_factory.create_event(
@@ -165,35 +172,22 @@ class RoomCreationHandler(BaseHandler):
                 user_id=user_id,
                 content=content
             )
-
+            yield handle_event(invite_event)
-            yield self.hs.get_handlers().room_member_handler.change_membership(
-                invite_event,
-                do_auth=False
-            )
-
-        yield self.hs.get_handlers().room_member_handler.change_membership(
-            join_event,
-            do_auth=False
-        )
 
         result = {"room_id": room_id}
 
         if room_alias:
             result["room_alias"] = room_alias.to_string()
-            directory_handler = self.hs.get_handlers().directory_handler
+            directory_handler.send_room_alias_update_event(user_id, room_id)
-            yield directory_handler.create_association(
-                user_id=user_id,
-                room_id=room_id,
-                room_alias=room_alias,
-                servers=[self.hs.hostname],
-            )
 
         defer.returnValue(result)
 
     def _create_events_for_new_room(self, creator, room_id, is_public=False):
+        creator_id = creator.to_string()
+
         event_keys = {
             "room_id": room_id,
-            "user_id": creator.to_string(),
+            "user_id": creator_id,
         }
 
         def create(etype, **content):
@@ -208,6 +202,16 @@ class RoomCreationHandler(BaseHandler):
             creator=creator.to_string(),
         )
 
+        join_event = self.event_factory.create_event(
+            etype=RoomMemberEvent.TYPE,
+            state_key=creator_id,
+            content={
+                "membership": Membership.JOIN,
+            },
+            **event_keys
+        )
+
+
         power_levels_event = self.event_factory.create_event(
             etype=RoomPowerLevelsEvent.TYPE,
             content={
@@ -236,6 +240,7 @@ class RoomCreationHandler(BaseHandler):
 
         return [
             creation_event,
+            join_event,
             power_levels_event,
             join_rules_event,
         ]

synapse/rest/directory.py

@@ -70,9 +70,11 @@ class ClientDirectoryServer(RestServlet):
         dir_handler = self.handlers.directory_handler
 
         try:
+            user_id = user.to_string()
             yield dir_handler.create_association(
-                user.to_string(), room_alias, room_id, servers
+                user_id, room_alias, room_id, servers
             )
+            yield dir_handler.send_room_alias_update_event(user_id, room_id)
         except SynapseError as e:
             raise e
         except:

synapse/storage/directory.py

@@ -14,10 +14,15 @@
 # limitations under the License.
 
 from ._base import SQLBaseStore
+
+from synapse.api.errors import SynapseError
+
 from twisted.internet import defer
 
 from collections import namedtuple
 
+import sqlite3
+
 
 RoomAliasMapping = namedtuple(
     "RoomAliasMapping",
@@ -75,13 +80,18 @@ class DirectoryStore(SQLBaseStore):
         Returns:
             Deferred
         """
-        yield self._simple_insert(
+        try:
-            "room_aliases",
+            yield self._simple_insert(
-            {
+                "room_aliases",
-                "room_alias": room_alias.to_string(),
+                {
-                "room_id": room_id,
+                    "room_alias": room_alias.to_string(),
-            },
+                    "room_id": room_id,
-        )
+                },
+            )
+        except sqlite3.IntegrityError:
+            raise SynapseError(
+                409, "Room alias %s already exists" % room_alias.to_string()
+            )
 
         for server in servers:
             # TODO(erikj): Fix this to bulk insert