Commit Graph

847 Commits

Author SHA1 Message Date
Amber Brown 32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Brendan Abolivier 26b62796c2
Merge pull request #5363 from matrix-org/babolivier/account_validity_send_mail_auth
Don't check whether the user's account is expired on /send_mail requests
2019-06-10 11:57:02 +01:00
Brendan Abolivier 028f674cd3
Better wording 2019-06-10 11:35:54 +01:00
Brendan Abolivier 4914a88829
Doc 2019-06-10 11:34:45 +01:00
Andrew Morgan 3719680ee4
Add ability to perform password reset via email without trusting the identity server (#5377)
Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option.

This PR is a culmination of 3 smaller PRs which have each been separately reviewed:

* #5308
* #5345
* #5368
2019-06-06 17:34:07 +01:00
Brendan Abolivier fe13bd52ac
Don't check whether the user's account is expired on /send_mail requests 2019-06-05 16:35:05 +01:00
Richard van der Hoff d18e4ea0d4 Implement room v5 which enforces signing key validity
Implements [MSC2077](https://github.com/matrix-org/matrix-doc/pull/2077) and
fixes #5247 and #4364.
2019-06-05 11:00:47 +01:00
Richard van der Hoff 14f13babb0
Add a test room version where we enforce key validity (#5348) 2019-06-05 10:38:25 +01:00
Amber Brown 46c8f7a517
Implement the SHHS complexity API (#5216) 2019-05-30 01:47:16 +10:00
Andrew Morgan 6368150a74
Add config option for setting homeserver's default room version (#5223)
Replaces DEFAULT_ROOM_VERSION constant with a method that first checks the config, then returns a hardcoded value if the option is not present.

That hardcoded value is now located in the server.py config file.
2019-05-23 15:00:20 +01:00
Amber Brown 4a30e4acb4
Room Statistics (#4338) 2019-05-21 11:36:50 -05:00
Richard van der Hoff 17f6804837
Introduce room v4 which updates event ID format. (#5217)
Implements https://github.com/matrix-org/matrix-doc/pull/2002.
2019-05-21 16:22:54 +01:00
Richard van der Hoff 04d53794d6
Fix error handling for rooms whose versions are unknown. (#5219)
If we remove support for a particular room version, we should behave more
gracefully. This should make client requests fail with a 400 rather than a 500,
and will ignore individiual PDUs in a federation transaction, rather than the
whole transaction.
2019-05-21 13:47:25 +01:00
Richard van der Hoff 5206648a4a
Add a test room version which updates event ID format (#5210)
Implements MSC1884
2019-05-20 15:54:42 +01:00
Erik Johnston 1dff859d6a Rename relation types to match MSC 2019-05-20 14:31:19 +01:00
Erik Johnston 57ba3451b6
Merge pull request #5209 from matrix-org/erikj/reactions_base
Land basic reaction and edit support.
2019-05-20 14:06:40 +01:00
ReidAnderson 3787133c9e Limit UserIds to a length that fits in a state key (#5198) 2019-05-20 11:20:08 +01:00
Richard van der Hoff 5f027a315f
Drop support for v2_alpha API prefix (#5190) 2019-05-15 17:37:46 +01:00
Erik Johnston efe3c7977a Add simple send_relation API and track in DB 2019-05-15 13:36:51 +01:00
Brendan Abolivier 1473058b5e
Do checks on aliases for incoming m.room.aliases events (#5128)
Follow-up to #5124

Also added a bunch of checks to make sure everything (both the stuff added on #5124 and this PR) works as intended.
2019-05-08 17:01:30 +01:00
Richard van der Hoff 0836cbb9f5
Factor out an "assert_requester_is_admin" function (#5120)
Rather than copying-and-pasting the same four lines hundreds of times
2019-05-02 10:45:52 +01:00
Erik Johnston ca90336a69 Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/account_expiration 2019-04-17 19:44:40 +01:00
Brendan Abolivier eaf41a943b Add management endpoints for account validity 2019-04-17 19:34:45 +01:00
Brendan Abolivier 91934025b9
Merge pull request #5047 from matrix-org/babolivier/account_expiration
Send out emails with links to extend an account's validity period
2019-04-17 14:57:39 +01:00
Brendan Abolivier 20f0617e87 Send out emails with links to extend an account's validity period 2019-04-17 14:42:20 +01:00
Andrew Morgan caa76e6021
Remove periods from copyright headers (#5046) 2019-04-11 17:08:13 +01:00
Brendan Abolivier bfc8fdf1fc
Merge pull request #5027 from matrix-org/babolivier/account_expiration
Add time-based account expiration
2019-04-09 17:02:41 +01:00
Brendan Abolivier 747aa9f8ca Add account expiration feature 2019-04-09 16:46:04 +01:00
Andrew Morgan 66e78700a2
Transfer related groups on room upgrade (#4990)
Transfers the m.room.related_groups state event on room upgrade.
2019-04-02 17:15:24 +01:00
Richard van der Hoff 54a87a7b08
Collect room-version variations into one place (#4969)
Collect all the things that make room-versions different to one another into
one place, so that it's easier to define new room versions.
2019-04-01 10:24:38 +01:00
Andrew Morgan bbd244c7b2
Support 3PID login in password providers (#4931)
Adds a new method, check_3pid_auth, which gives password providers
the chance to allow authentication with third-party identifiers such
as email or msisdn.
2019-03-26 17:48:30 +00:00
Richard van der Hoff 0dbfae03f9 Enforce hs_disabled_message correctly
Fixes a bug where hs_disabled_message was not enforced for 3pid-based requests
if there was no server_notices_mxid configured.
2019-03-19 11:30:54 +00:00
Brendan Abolivier 899e523d6d
Add ratelimiting on login (#4821)
Add two ratelimiters on login (per-IP address and per-userID).
2019-03-15 17:46:16 +00:00
Brendan Abolivier a4c3a361b7
Add rate-limiting on registration (#4735)
* Rate-limiting for registration

* Add unit test for registration rate limiting

* Add config parameters for rate limiting on auth endpoints

* Doc

* Fix doc of rate limiting function

Co-Authored-By: babolivier <contact@brendanabolivier.com>

* Incorporate review

* Fix config parsing

* Fix linting errors

* Set default config for auth rate limiting

* Fix tests

* Add changelog

* Advance reactor instead of mocked clock

* Move parameters to registration specific config and give them more sensible default values

* Remove unused config options

* Don't mock the rate limiter un MAU tests

* Rename _register_with_store into register_with_store

* Make CI happy

* Remove unused import

* Update sample config

* Fix ratelimiting test for py2

* Add non-guest test
2019-03-05 14:25:33 +00:00
Richard van der Hoff 6fba9fd20c Merge remote-tracking branch 'origin/release-v0.99.0' into develop 2019-01-30 17:02:32 +00:00
Matthew Hodgson ad7ac8853c by default include m.room.encryption on invites (#3902)
* by default include m.room.encryption on invites

* fix constant

* changelog
2019-01-30 16:26:13 +00:00
Neil Johnson 81b7e7eed3
Update constants.py
remove trailing ,
2019-01-30 16:11:36 +00:00
Neil Johnson ee4df7fd7a
Merge branch 'develop' into neilj/room_capabilities 2019-01-30 10:28:08 +00:00
Neil Johnson c5a0f82cca define room dispositions for use in exposing room capabilities 2019-01-30 10:24:24 +00:00
Neil Johnson c7837dce24 reflect that rooms v3 is a stable room version 2019-01-30 09:33:30 +00:00
Erik Johnston a1b0e1879b Enable room version v3 2019-01-29 23:09:10 +00:00
Erik Johnston a696c48133 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/redactions_eiah 2019-01-29 22:00:33 +00:00
Erik Johnston 47e2dd1994 Drop vdh support 2019-01-29 21:24:34 +00:00
Erik Johnston 84af577356 Implement event format v2 2019-01-29 18:06:11 +00:00
Erik Johnston b82a76c384 Finish comment... 2019-01-29 13:50:59 +00:00
Erik Johnston 64c1bd1d21 Remove dead function 2019-01-29 12:06:28 +00:00
Erik Johnston 7709d2bd16 Implement rechecking of redactions 2019-01-29 11:56:20 +00:00
Erik Johnston 7a3ec5b022 Add RoomVersions.V3 constant, without enabling it
We add the constant, but don't add it to the known room versions. This
lets us start adding V3 logic, but the servers will never join or create
V3 rooms
2019-01-29 11:55:33 +00:00
Erik Johnston 5180f12bae Replace usage of builder.user_id with builder.sender
`.user_id` is proxed to `.sender` in FrozenEvent, so this has no
functional change
2019-01-29 11:18:38 +00:00
Erik Johnston ae2a957dba Pass through room version to event auth 2019-01-25 18:31:41 +00:00
Neil Johnson 893107be78 backout v3 2019-01-25 17:27:36 +00:00
Neil Johnson 53ef4da8c2 track unstable room v3 2019-01-25 11:04:11 +00:00
Andrew Morgan b1b6dba2d2
Merge pull request #4415 from matrix-org/anoa/full_search_upgraded_rooms
Ability to search entire room history after upgrading room
2019-01-25 10:49:58 +00:00
Erik Johnston 664b7a2920
Merge pull request #4437 from matrix-org/erikj/event_format_version_v2
Add support for persisting event format versions
2019-01-24 18:32:30 +00:00
Richard van der Hoff 8ea509a935
Update synapse/api/filtering.py
Co-Authored-By: anoadragon453 <1342360+anoadragon453@users.noreply.github.com>
2019-01-24 17:21:35 +00:00
Erik Johnston c5a296b10c Add support for persisting event format versions
Currently we only have the one event format version defined, but this
adds the necessary infrastructure to persist and fetch the format
versions alongside the events.

We specify the format version rather than the room version as:

1. We don't necessarily know the room version, existing events may be
   either v1 or v2.
2. We'd need to be careful to prevent/handle correctly if different
   events in the same room reported to be of different versions, which
   sounds annoying.
2019-01-23 11:30:01 +00:00
Neil Johnson d619b113ed Fix None guard in config.server.is_threepid_reserved 2019-01-22 16:52:29 +00:00
Andrew Morgan c433f61091 Ensure new filter is actually created 2019-01-22 12:06:36 +00:00
Andrew Morgan c9bfb058d8 Fix a bug with single-room search searching all rooms
* Create a new method for getting predecessor rooms
* Remove formatting change
2019-01-22 12:00:41 +00:00
Erik Johnston 35e1d67b4e Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_v2_invite_server 2019-01-21 14:04:19 +00:00
Erik Johnston 5f54765587
Merge pull request #4390 from matrix-org/erikj/versioned_fed_apis
Add groundwork for new versions of federation APIs
2019-01-21 11:44:05 +00:00
Andrew Morgan 702c4b750c Migrate encryption state on room upgrade (#4411)
* Migrate encryption state on room upgrade

Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>

* Add changelog file
2019-01-21 20:42:58 +11:00
Andrew Morgan df3a661e4a Search for messages across predecessor rooms
Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>
2019-01-18 11:19:20 +00:00
Neil Johnson 9ec56d6935 ALL_USER_TYPES should be a tuple 2019-01-15 14:38:15 +00:00
Erik Johnston 4a4d2e17bc Add /v2/invite federation API 2019-01-15 13:22:44 +00:00
Erik Johnston bb63e7ca4f Add groundwork for new versions of federation APIs 2019-01-15 11:14:34 +00:00
Richard van der Hoff e0910d0145 Merge branch rav/macaroon_key_fix_0.34 into rav/macaroon_key_fix_0.34.1
Fixes #4371
2019-01-10 14:12:50 +00:00
Richard van der Hoff aa70d24125 Merge branch 'rav/macaroon_key_fix' into rav/macaroon_key_fix_0.34 2019-01-10 12:58:33 +00:00
Richard van der Hoff 566947ff34 Skip macaroon check for access tokens in the db 2019-01-10 12:57:21 +00:00
Erik Johnston b970cb0e96 Refactor request sending to have better excpetions (#4358)
* Correctly retry and back off if we get a HTTPerror response

* Refactor request sending to have better excpetions

MatrixFederationHttpClient blindly reraised exceptions to the caller
without differentiating "expected" failures (e.g. connection timeouts
etc) versus more severe problems (e.g. programming errors).

This commit adds a RequestSendFailed exception that is raised when
"expected" failures happen, allowing the TransactionQueue to log them as
warnings while allowing us to log other exceptions as actual exceptions.
2019-01-08 11:04:28 +00:00
Michael Telatynski a27e501b09 fix the check for whether `is_url` to match all the other ones in codebase (#3405)
Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>
2019-01-06 21:43:25 +00:00
Richard van der Hoff cd018e3543
Merge pull request #4307 from matrix-org/erikj/v2_rooms
Add v2 room version
2018-12-24 10:48:59 +01:00
Erik Johnston df89f8afb8 Add v2 room version 2018-12-18 18:10:37 +00:00
Neil Johnson d2f7c4e6b1
create support user (#4141)
Allow for the creation of a support user.

A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.
2018-12-14 18:20:59 +00:00
Travis Ralston 1737753a62 Add an option to enable recording IPs for appservice users (#3831) 2018-12-04 12:44:41 +01:00
Travis Ralston c68aab1536
Merge pull request #4004 from matrix-org/travis/login-terms
Add m.login.terms to the registration flow
2018-11-01 11:03:38 -06:00
Erik Johnston b3dd6fa981 Add STATE_V2_TEST room version 2018-11-01 11:43:46 +00:00
Travis Ralston d1e7b9c44c Merge branch 'develop' into travis/login-terms 2018-10-31 13:15:14 -06:00
Amber Brown f79f454485
Remove deprecated v1 key exchange endpoint (#4119) 2018-10-31 22:29:02 +11:00
Richard van der Hoff 0f7d1c9906 Basic initial support for room upgrades
Currently just creates a new, empty, room, and sends a tombstone in the old
room.
2018-10-25 19:10:24 +01:00
Travis Ralston 54def42c19 Merge branch 'develop' into travis/login-terms 2018-10-24 13:22:59 -06:00
Richard van der Hoff 7e07d25ed6 Allow backslashes in event field filters
Fixes a bug introduced in https://github.com/matrix-org/synapse/pull/1783 which
meant that single backslashes were not allowed in event field filters.

The intention here is to allow single-backslashes, but disallow
double-backslashes.
2018-10-24 11:11:24 +01:00
Travis Ralston f293d124b6 Merge branch 'develop' into travis/login-terms 2018-10-15 14:44:32 -06:00
David Baker 83e72bb2f0 PR feedback pt. 1 2018-10-12 11:26:18 +01:00
David Baker b8d9e108be Fix mergefail 2018-10-09 18:04:21 +01:00
David Baker dc045ef202 Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backups 2018-10-09 10:05:02 +01:00
Travis Ralston fd99787162 Incorporate Dave's work for GDPR login flows
As per https://github.com/vector-im/riot-web/issues/7168#issuecomment-419996117
2018-10-03 15:57:42 -06:00
Amber Brown 6e05fd032c
Fix userconsent on Python 3 (#3938) 2018-10-02 00:11:58 +10:00
Schnuffle dc5db01ff2 Replaced all occurences of e.message with str(e)
Signed-off-by: Schnuffle  <schnuffle@github.com>
2018-09-27 13:38:50 +02:00
Hubert Chathi 3801b8aa03 try to make flake8 and isort happy 2018-09-06 11:35:19 -04:00
Richard van der Hoff 87c18d12ee Implement 'event_format' filter param in /sync
This has been specced and part-implemented; let's implement it for /sync (but
no other endpoints yet :/).
2018-09-04 15:20:09 +01:00
Neil Johnson 301cb60d0b assert rather than warn 2018-08-31 17:29:35 +01:00
Neil Johnson 0b01281e77 move threepid checker to config, add missing yields 2018-08-31 17:11:11 +01:00
Neil Johnson e8e540630e fix reference to is_threepid_reserved 2018-08-31 16:09:15 +01:00
Neil Johnson 09f3cf1a7e ensure post registration auth checks do not fail erroneously 2018-08-31 15:42:51 +01:00
Neil Johnson ea068d6f3c fix bug where preserved threepid user comes to sign up and server is mau blocked 2018-08-31 10:49:14 +01:00
Erik Johnston 05077e06fa Change admin_uri to admin_contact in config and errors 2018-08-24 16:51:27 +01:00
Hubert Chathi 83caead95a
Merge branch 'develop' into e2e_backups 2018-08-24 11:44:26 -04:00
Erik Johnston cd77270a66 Implement trail users 2018-08-23 19:17:19 +01:00
Erik Johnston fd2dbf1836 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-22 17:06:10 +01:00
Erik Johnston 9643a6f7f2 Update notice format 2018-08-22 17:00:29 +01:00
Erik Johnston 3bf8bab8f9
Merge pull request #3673 from matrix-org/erikj/refactor_state_handler
Refactor state module to support multiple room versions
2018-08-22 10:04:55 +01:00
Amber Brown 324525f40c
Port over enough to get some sytests running on Python 3 (#3668) 2018-08-20 23:54:49 +10:00
Erik Johnston 4d664278af Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler 2018-08-20 14:49:43 +01:00
Neil Johnson e07970165f rename error code 2018-08-18 14:39:45 +01:00
Neil Johnson c5171bf171 special case server_notices_mxid 2018-08-18 12:33:07 +01:00
Neil Johnson 3ee57bdcbb Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-17 14:34:10 +01:00
Neil Johnson b5f638f1f4 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-17 14:04:15 +01:00
Neil Johnson 69c49d3fa3
Merge branch 'develop' into neilj/limit_exceeded_error 2018-08-17 12:44:26 +00:00
Neil Johnson bcfeb44afe call reap on start up and fix under reaping bug 2018-08-16 22:55:32 +01:00
Neil Johnson 13ad9930c8 add new error type ResourceLimit 2018-08-16 18:02:02 +01:00
Neil Johnson fc5d937550 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-15 16:31:40 +01:00
Neil Johnson c24fc9797b add new event types 2018-08-15 15:04:30 +01:00
Neil Johnson 87a824bad1 clean up AuthError 2018-08-15 11:58:03 +01:00
Neil Johnson b8429c7c81 update error codes for resource limiting 2018-08-15 10:19:48 +01:00
Neil Johnson ab035bdeac replace admin_email with admin_uri for greater flexibility 2018-08-15 10:16:41 +01:00
Neil Johnson 19b433e3f4 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/admin_email 2018-08-14 17:44:46 +01:00
Neil Johnson 2545993ce4 make comments clearer 2018-08-14 15:48:12 +01:00
Neil Johnson 9b75c78b4d support server notice state events for resource limits 2018-08-14 11:20:41 +01:00
Neil Johnson f4b49152e2 support admin_email config and pass through into blocking errors, return AuthError in all cases 2018-08-13 21:09:47 +01:00
Neil Johnson ce7de9ae6b Revert "support admin_email config and pass through into blocking errors, return AuthError in all cases"
This reverts commit 0d43f991a1.
2018-08-13 18:06:18 +01:00
Neil Johnson 0d43f991a1 support admin_email config and pass through into blocking errors, return AuthError in all cases 2018-08-13 18:00:23 +01:00
Matthew Hodgson 0abb205b47 blindly incorporate PR review - needs testing & fixing 2018-08-12 19:14:31 -04:00
Matthew Hodgson 8ae64b270f implement /room_keys/version too (untested) 2018-08-12 19:14:31 -04:00
Neil Johnson 885ea9c602 rename _user_last_seen_monthly_active 2018-08-09 18:02:12 +01:00
Neil Johnson c1f9dec92a fix errant parenthesis 2018-08-09 17:43:26 +01:00
Neil Johnson 09cf130898 only block on sync where user is not part of the mau cohort 2018-08-09 17:39:12 +01:00
Erik Johnston 152c0aa58e Add constants for room versions 2018-08-09 14:55:47 +01:00
Neil Johnson 839a317c96
fix pep8 too many lines 2018-08-08 17:39:04 +01:00
Neil Johnson 5298d79fb5
Merge branch 'develop' into neilj/disable_hs 2018-08-08 16:13:03 +00:00
Richard van der Hoff 8521ae13e3
Merge pull request #3654 from matrix-org/rav/room_versions
Support for room versioning
2018-08-08 17:10:53 +01:00
Richard van der Hoff 3523f5432a Don't expose default_room_version as config opt 2018-08-07 12:51:57 +01:00
Neil Johnson 7bcf126b18 Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/disable_hs 2018-08-06 21:39:44 +01:00
Neil Johnson 1911c037cb update comments to reflect new sig 2018-08-06 18:01:46 +01:00
Neil Johnson 42c6823827 disable HS from config 2018-08-04 22:07:04 +01:00
Neil Johnson e10830e976 wip commit - tests failing 2018-08-03 17:55:50 +01:00
Richard van der Hoff 0d63d93ca8 Enforce compatibility when processing make_join requests
Reject make_join requests from servers which do not support the room version.

Also include the room version in the response.
2018-08-03 16:08:32 +01:00
Richard van der Hoff 0ca459ea33 Basic support for room versioning
This is the first tranche of support for room versioning. It includes:
 * setting the default room version in the config file
 * new room_version param on the createRoom API
 * storing the version of newly-created rooms in the m.room.create event
 * fishing the version of existing rooms out of the m.room.create event
2018-08-03 16:08:32 +01:00
Neil Johnson 897c51d274 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker 2018-08-03 13:40:47 +01:00
Richard van der Hoff 1fa98495d0
Merge pull request #3639 from matrix-org/rav/refactor_error_handling
Clean up handling of errors from outbound requests
2018-08-02 17:38:24 +01:00
Neil Johnson 74b1d46ad9 do mau checks based on monthly_active_users table 2018-08-02 16:57:35 +01:00
Neil Johnson 00f99f74b1 insertion into monthly_active_users 2018-08-02 13:47:19 +01:00
Neil Johnson 085435e13a
Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits
Initial impl of capping MAU
2018-08-01 15:58:45 +00:00
Richard van der Hoff 01e93f48ed Kill off MatrixCodeMessageException
This code brings the SimpleHttpClient into line with the
MatrixFederationHttpClient by having it raise HttpResponseExceptions when a
request fails (rather than trying to parse for matrix errors and maybe raising
MatrixCodeMessageException).

Then, whenever we were checking for MatrixCodeMessageException and turning them
into SynapseErrors, we now need to check for HttpResponseExceptions and call
to_synapse_error.
2018-08-01 16:02:46 +01:00
Richard van der Hoff 018d75a148 Refactor code for turning HttpResponseException into SynapseError
This commit replaces SynapseError.from_http_response_exception with
HttpResponseException.to_synapse_error.

The new method actually returns a ProxiedRequestError, which allows us to pass
through additional metadata from the API call.
2018-08-01 16:02:46 +01:00
Richard van der Hoff fa7dc889f1 Be more careful which errors we send back over the C-S API
We really shouldn't be sending all CodeMessageExceptions back over the C-S API;
it will include things like 401s which we shouldn't proxy.

That means that we need to explicitly turn a few HttpResponseExceptions into
SynapseErrors in the federation layer.

The effect of the latter is that the matrix errcode will get passed through
correctly to calling clients, which might help with some of the random
M_UNKNOWN errors when trying to join rooms.
2018-08-01 16:02:38 +01:00
Amber Brown da7785147d
Python 3: Convert some unicode/bytes uses (#3569) 2018-08-02 00:54:06 +10:00
Neil Johnson 251e6c1210 limit register and sign in on number of monthly users 2018-07-30 15:55:57 +01:00
Matthew Hodgson a75231b507
Deduplicate redundant lazy-loaded members (#3331)
* attempt at deduplicating lazy-loaded members

as per the proposal; we can deduplicate redundant lazy-loaded members
which are sent in the same sync sequence. we do this heuristically
rather than requiring the client to somehow tell us which members it
has chosen to cache, by instead caching the last N members sent to
a client, and not sending them again.  For now we hardcode N to 100.
Each cache for a given (user,device) tuple is in turn cached for up to
X minutes (to avoid the caches building up).  For now we hardcode X to 30.

* add include_redundant_members filter option & make it work

* remove stale todo

* add tests for _get_some_state_from_cache

* incorporate review
2018-07-26 22:51:30 +01:00
Matthew Hodgson d19fba3655 Merge branch 'develop' into matthew/filter_members 2018-07-24 12:39:54 +01:00
Erik Johnston 536bc63a4e
Merge branch 'develop' into erikj/client_apis_move 2018-07-24 09:57:05 +01:00
Matthew Hodgson adfe29ec0b Merge branch 'develop' into matthew/filter_members 2018-07-23 19:21:37 +01:00
Erik Johnston e42510ba63 Use new getters 2018-07-23 13:17:22 +01:00
Erik Johnston 0ecf68aedc Move check_in_room_or_world_readable to Auth 2018-07-20 15:30:59 +01:00
Matthew Hodgson be3adfc331 merge develop pydoc for _get_state_for_groups 2018-07-19 11:26:04 +01:00
Amber Brown 33b60c01b5
Make auth & transactions more testable (#3499) 2018-07-14 07:34:49 +10:00
Amber Brown 49af402019 run isort 2018-07-09 16:09:20 +10:00
Richard van der Hoff 3cf3e08a97 Implementation of server_acls
... as described at
https://docs.google.com/document/d/1EttUVzjc2DWe2ciw4XPtNpUpIl9lWXGEsy2ewDS7rtw.
2018-07-04 19:06:20 +01:00
Matthew Hodgson f4f1cda928 add ip_range_whitelist parameter to limit where ASes can connect from 2018-06-28 20:32:00 +01:00
Amber Brown 6350bf925e
Attempt to be more performant on PyPy (#3462) 2018-06-28 14:49:57 +01:00
Travis Ralston ec1e799e17 Don't print invalid access tokens in the logs
Tokens shouldn't be appearing the logs, valid or invalid. 

Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-06-24 12:17:01 -06:00
Richard van der Hoff 557b686eac Refactor get_send_level to take a power_levels event
it makes it easier for me to reason about
2018-06-14 11:26:27 +01:00
Matthew Hodgson 28f09fcdd5 Merge branch 'develop' into matthew/filter_members 2018-06-04 00:09:17 +03:00
Amber Brown c936a52a9e
Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307) 2018-05-31 19:03:47 +10:00
Matthew Hodgson 9bbb9f5556 add lazy_load_members to the filter json schema 2018-05-29 04:26:10 +01:00
Matthew Hodgson 7a6df013cc merge develop 2018-05-29 00:25:22 +01:00
Amber Brown 754826a830 Merge remote-tracking branch 'origin/develop' into 3218-official-prom 2018-05-28 18:57:23 +10:00
Richard van der Hoff 08bfc48abf custom error code for not leaving server notices room 2018-05-22 17:27:27 +01:00
Amber Brown a8990fa2ec Merge remote-tracking branch 'origin/develop' into 3218-official-prom 2018-05-22 10:50:26 -05:00
Richard van der Hoff a5e2941aad Reject attempts to send event before privacy consent is given
Returns an M_CONSENT_NOT_GIVEN error (cf
https://github.com/matrix-org/matrix-doc/issues/1252) if consent is not yet
given.
2018-05-22 12:00:47 +01:00
Amber Brown df9f72d9e5 replacing portions 2018-05-21 19:47:37 -05:00
Richard van der Hoff 33f469ba19 Apply some limits to depth to counter abuse
* When creating a new event, cap its depth to 2^63 - 1
* When receiving events, reject any without a sensible depth

As per https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI
2018-05-01 17:54:19 +01:00
Adrian Tschira 36c59ce669 Use six.itervalues in some places
There's more where that came from

Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 20:39:43 +02:00
Adrian Tschira 6168351877 Add b prefixes to some strings that are bytes in py3
This has no effect on python2

Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-04 13:48:51 +02:00
Richard van der Hoff fcfe7f6ad3 Use simplejson throughout
Let's use simplejson rather than json, for consistency.
2018-03-29 22:45:52 +01:00
Erik Johnston fa72803490 Merge branch 'master' of github.com:matrix-org/synapse into develop 2018-03-19 11:41:01 +00:00
Matthew Hodgson b2f2282947 make lazy_load_members configurable in filters 2018-03-19 01:15:13 +00:00
Erik Johnston 926ba76e23 Replace ujson with simplejson 2018-03-15 23:43:31 +00:00
Matthew Hodgson ab9f844aaf
Add federation_domain_whitelist option (#2820)
Add federation_domain_whitelist

gives a way to restrict which domains your HS is allowed to federate with.
useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network
2018-01-22 19:11:18 +01:00
Matthew Hodgson 28a6ccb49c add registrations_require_3pid
lets homeservers specify a whitelist for 3PIDs that users are allowed to associate with.
Typically useful for stopping people from registering with non-work emails
2018-01-19 00:19:58 +00:00
Richard van der Hoff d5f9fb06b0 Refactor UI auth implementation
Instead of returning False when auth is incomplete, throw an exception which
can be caught with a wrapper.
2017-12-05 09:40:05 +00:00
Richard van der Hoff da562bd6a1 Improve comments on get_user_by_access_token
because I have to reverse-engineer this every time.
2017-11-29 15:52:41 +00:00
Richard van der Hoff aa620d09a0 Add a config option to block all room invites (#2457)
- allows sysadmins the ability to lock down their servers so that people can't
send their users room invites.
2017-09-19 16:08:14 +01:00
Erik Johnston ed9a7f5436 Merge pull request #2309 from matrix-org/erikj/user_ip_repl
Fix up user_ip replication commands
2017-07-06 14:33:14 +01:00
Erik Johnston 2c365f4723 Cache macaroon parse and validation
Turns out this can be quite expensive for requests, and is easily
cachable. We don't cache the lookup to the DB so invalidation still
works.
2017-06-29 14:50:18 +01:00
Erik Johnston 8c23221666 Fix up 2017-06-27 15:53:45 +01:00
Erik Johnston ed3d0170d9 Batch upsert user ips 2017-06-27 13:37:04 +01:00
Erik Johnston 0185b75381 Change is_host_joined to use current_state table
This bypasses a bug where using the state groups to figure out if a host
is in a room sometimes errors if the servers isn't in the room. (For
example when the server rejected an invite to a remote room)
2017-06-09 10:52:26 +01:00
David Baker 1a9255c12e Use CodeMessageException subclass instead
Parse json errors from get_json client methods and throw special
errors.
2017-04-25 19:30:55 +01:00
pik 566641a0b5 use jsonschema.FormatChecker for RoomID and UserID strings
* use a valid filter in rest/client/v2_alpha test

Signed-off-by: pik <alexander.maznev@gmail.com>
2017-03-23 11:42:41 -03:00
pik acafcf1c5b Add valid filter tests, flake8, fix typo
Signed-off-by: pik <alexander.maznev@gmail.com>
2017-03-23 11:42:10 -03:00
pik e56c79c114 check_valid_filter using JSONSchema
* add invalid filter tests

Signed-off-by: pik <alexander.maznev@gmail.com>
2017-03-23 11:42:07 -03:00
Richard van der Hoff 19b9366d73 Fix a couple of logcontext leaks
Use preserve_fn to correctly manage the logcontexts around things we don't want
to yield on.
2017-03-23 00:17:46 +00:00
Erik Johnston a8f96c63aa Comment 2017-03-15 16:01:01 +00:00
Erik Johnston e892457a03 Comment 2017-03-15 15:01:39 +00:00
Erik Johnston 6c82de5100 Format presence events on the edges instead of reformatting them multiple times 2017-03-15 14:27:34 +00:00
Erik Johnston 7827251daf Merge pull request #1994 from matrix-org/dbkr/msisdn_signin_2
Phone number registration / login support v2
2017-03-15 09:59:54 +00:00
Richard van der Hoff 1d09586599 Address review comments
- don't blindly proxy all HTTPRequestExceptions
- log unexpected exceptions at error
- avoid `isinstance`
- improve docs on `from_http_response_exception`
2017-03-14 14:15:37 +00:00
Richard van der Hoff 7f237800e9 re-refactor exception heirarchy
Give CodeMessageException back its `msg` attribute, and use that to hold the
HTTP status message for HttpResponseException.
2017-03-14 14:15:37 +00:00
David Baker 73a5f06652 Support registration / login with phone number
Changes from https://github.com/matrix-org/synapse/pull/1971
2017-03-13 17:27:51 +00:00
Richard van der Hoff 170ccc9de5 Fix routing loop when fetching remote media
When we proxy a media request to a remote server, add a query-param, which will
tell the remote server to 404 if it doesn't recognise the server_name.

This should fix a routing loop where the server keeps forwarding back to
itself.

Also improves the error handling on remote media fetches, so that we don't
always return a rather obscure 502.
2017-03-13 16:30:36 +00:00
Erik Johnston 7eae6eaa2f Revert "Support registration & login with phone number" 2017-03-13 09:59:33 +00:00
David Baker ce3e583d94 WIP support for msisdn 3pid proxy methods 2017-02-14 15:05:55 +00:00
David Baker 063a1251a9 Remove a few aspirational but unused constants
from the Kegan era
2017-02-08 11:36:08 +00:00
Erik Johnston 5f027d1fc5 Change resolve_state_groups call site logging to DEBUG 2017-01-17 17:07:15 +00:00
Erik Johnston e178feca3f Remove unused function 2017-01-13 15:16:45 +00:00
Erik Johnston 8b2fa38256 Split event auth code into seperate module 2017-01-13 15:07:32 +00:00
Erik Johnston 7e6c2937c3 Split out static auth methods from Auth object 2017-01-10 18:16:54 +00:00
Mark Haines c18f7fc410 Fix flake8 and update changelog 2017-01-05 13:50:22 +00:00
Matthew Hodgson d79d165761 add logging for all the places we call resolve_state_groups. my kingdom for a backtrace that actually works. 2017-01-05 13:40:39 +00:00
Richard van der Hoff 1529c19675 Prevent user tokens being used as guest tokens (#1675)
Make sure that a user cannot pretend to be a guest by adding 'guest = True'
caveats.
2016-12-06 15:31:37 +00:00
Richard van der Hoff aa09d6b8f0 Rip out more refresh_token code
We might as well treat all refresh_tokens as invalid. Just return a 403 from
/tokenrefresh, so that we don't have a load of dead, untestable code hanging
around.

Still TODO: removing the table from the schema.
2016-11-30 17:40:18 +00:00
Richard van der Hoff 321fe5c44c Merge pull request #1656 from matrix-org/rav/remove_time_caveat
Stop putting a time caveat on access tokens
2016-11-30 16:53:20 +00:00
Richard van der Hoff 4febfe47f0 Comments
Update comments in verify_macaroon
2016-11-30 07:36:32 +00:00
Richard van der Hoff 77eca2487c Merge pull request #1653 from matrix-org/rav/guest_e2e
Implement E2E for guests
2016-11-29 17:41:35 +00:00
Richard van der Hoff 1c4f05db41 Stop putting a time caveat on access tokens
The 'time' caveat on the access tokens was something of a lie, since we weren't
enforcing it; more pertinently its presence stops us ever adding useful time
caveats.

Let's move in the right direction by not lying in our caveats.
2016-11-29 16:49:41 +00:00
Richard van der Hoff b6146537d2 Merge pull request #1655 from matrix-org/rav/remove_redundant_macaroon_checks
Remove redundant list of known caveat prefixes
2016-11-25 16:57:19 +00:00
Richard van der Hoff 7f02e4d008 Give guest users a device_id
We need to create devices for guests so that they can use e2e, but we don't
have anywhere to store it, so just use a fixed one.
2016-11-25 15:25:30 +00:00
Richard van der Hoff e1d7c96814 Remove redundant list of known caveat prefixes
Also add some comments.
2016-11-24 12:38:17 +00:00
Kegan Dougal 83bcdcee61 Return early on /sync code paths if a '*' filter is used
This is currently very conservative in that it only does this if there is no
`since` token. This limits the risk to clients likely to be doing one-off
syncs (like bridges), but does mean that normal human clients won't benefit
from the time savings here. If the savings are large enough, I would consider
generalising this to just check the filter.
2016-11-22 16:38:35 +00:00
Kegsay d4a459f7cb Merge pull request #1638 from matrix-org/kegan/sync-event-fields
Implement "event_fields" in filters
2016-11-22 14:02:38 +00:00
Kegan Dougal cea4e4e7b2 Glue only_event_fields into the sync rest servlet 2016-11-22 10:14:05 +00:00
Kegan Dougal f97511a1f3 Move event_fields filtering to serialize_event
Also make it an inclusive not exclusive filter, as the spec demands.
2016-11-21 17:42:16 +00:00
Kegan Dougal e90fcd9edd Add filter_event_fields and filter_field to FilterCollection 2016-11-21 15:18:18 +00:00
Kegan Dougal a2a6c1c22f Fail with a coherent error message if `/sync?filter=` is invalid 2016-11-21 13:15:25 +00:00
Mark Haines 177f104432 Merge pull request #1098 from matrix-org/markjh/bearer_token
Allow clients to supply access_tokens as headers
2016-10-25 17:33:15 +01:00
Luke Barnard e01a1bc92d Merge pull request #1175 from matrix-org/luke/feature-configurable-as-rate-limiting
Allow Configurable Rate Limiting Per AS
2016-10-20 16:21:10 +01:00
Luke Barnard 07caa749bf Closing brace on following line 2016-10-20 12:07:16 +01:00
Luke Barnard f09db236b1 as_user->app_service, less redundant comments, better positioned comments 2016-10-20 12:04:54 +01:00
Luke Barnard 8bfd01f619 flake8 2016-10-20 11:52:46 +01:00
Luke Barnard 1b17d1a106 Use real AS object by passing it through the requester
This means synapse does not have to check if the AS is interested, but instead it effectively re-uses what it already knew about the requesting user
2016-10-20 11:43:05 +01:00
Erik Johnston 550308c7a1 Check whether to ratelimit sooner to avoid work 2016-10-19 10:45:24 +01:00
Erik Johnston a2f2516199 Merge pull request #1157 from Rugvip/nolimit
Remove rate limiting from app service senders and fix get_or_create_user requester
2016-10-11 11:20:54 +01:00
Patrik Oldsberg 9bfc617791 storage/appservice: make appservice methods only relying on the cache synchronous 2016-10-06 15:24:59 +02:00
Patrik Oldsberg 24a70e19c7 api/auth: fix for not being allowed to set your own state_key
Signed-off-by: Patrik Oldsberg <patrik.oldsberg@ericsson.com>
2016-09-30 13:08:25 +02:00
Erik Johnston f96020550f Update comments 2016-09-22 12:54:22 +01:00
Erik Johnston 2e9ee30969 Add comments 2016-09-22 11:59:46 +01:00
Erik Johnston a61e4522b5 Shuffle things around to make unit tests work 2016-09-22 11:08:12 +01:00
Erik Johnston 1168cbd54d Allow invites via 3pid to bypass sender sig check
When a server sends a third party invite another server may be the one
that the inviting user registers with. In this case it is that remote
server that will issue an actual invitation, and wants to do it "in the
name of" the original invitee. However, the new proper invite will not
be signed by the original server, and thus other servers would reject
the invite if it was seen as coming from the original user.

To fix this, a special case has been added to the auth rules whereby
another server can send an invite "in the name of" another server's
user, so long as that user had previously issued a third party invite
that is now being accepted.
2016-09-22 10:56:53 +01:00
Mark Haines ec609f8094 Fix unit tests 2016-09-12 10:46:02 +01:00
Mark Haines 8e01263587 Allow clients to supply access_tokens as headers
Clients can continue to supply access tokens as query parameters
or can supply the token as a header:

   Authorization: Bearer <access_token_goes_here>

This matches the ouath2 format of
https://tools.ietf.org/html/rfc6750#section-2.1
2016-09-09 18:17:42 +01:00
Mark Haines 8aee5aa068 Add helper function for getting access_tokens from requests
Rather than reimplementing the token parsing in the various places.
This will make it easier to change the token parsing to allow access_tokens
in HTTP headers.
2016-09-09 16:33:15 +01:00
Erik Johnston ed7a703d4c Handle the fact that workers can't generate state groups 2016-08-31 15:53:19 +01:00
Erik Johnston c10cb581c6 Correctly handle the difference between prev and current state 2016-08-31 14:26:22 +01:00
Erik Johnston 1ccdc1e93a Cache check_host_in_room 2016-08-26 10:59:40 +01:00
Erik Johnston 25414b44a2 Add measure on check_host_in_room 2016-08-26 10:47:00 +01:00
Erik Johnston 30961182f2 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_ids 2016-08-26 09:48:13 +01:00
Paul "LeoNerd" Evans 1294d4a329 Move ThirdPartyEntityKind into api.constants so the expectation becomes that the value is significant 2016-08-25 18:34:47 +01:00
Erik Johnston 0e1900d819 Pull out full state less 2016-08-25 18:15:51 +01:00
Paul "LeoNerd" Evans 142983b4ea APP_SERVICE_PREFIX is never used; don't bother 2016-08-25 18:06:05 +01:00