Commit Graph

10012 Commits

Author SHA1 Message Date
Erik Johnston d16f5574b6
Merge pull request #5220 from matrix-org/erikj/dont_bundle_live_events
Don't bundle aggregations with events in /sync or /events or state queries
2019-05-24 10:36:31 +01:00
Erik Johnston 4cb577c23f Don't bundle aggs for /state and /members etc APIs 2019-05-24 09:52:33 +01:00
Erik Johnston 8c41c04ee4
Merge pull request #5244 from matrix-org/rav/server_keys/00-factor-out-fetchers
Factor out KeyFetchers from KeyRing
2019-05-23 16:28:51 +01:00
Richard van der Hoff 753b1270da Require sig from origin server on perspectives responses 2019-05-23 15:01:09 +01:00
Andrew Morgan 6368150a74
Add config option for setting homeserver's default room version (#5223)
Replaces DEFAULT_ROOM_VERSION constant with a method that first checks the config, then returns a hardcoded value if the option is not present.

That hardcoded value is now located in the server.py config file.
2019-05-23 15:00:20 +01:00
Richard van der Hoff ec24108cc2 Fix remote_key_resource 2019-05-23 14:52:13 +01:00
Richard van der Hoff 895b79ac2e Factor out KeyFetchers from KeyRing
Rather than have three methods which have to have the same interface,
factor out a separate interface which is provided by three implementations.

I find it easier to grok the code this way.
2019-05-23 13:46:47 +01:00
Richard van der Hoff b75537beaf Store key validity time in the storage layer
This is a first step to checking that the key is valid at the required moment.

The idea here is that, rather than passing VerifyKey objects in and out of the
storage layer, we instead pass FetchKeyResult objects, which simply wrap the
VerifyKey and add a valid_until_ts field.
2019-05-23 11:52:22 +01:00
Richard van der Hoff 84660d91b2
Simplify process_v2_response (#5236)
* Pass time_added_ms into process_v2_response

* Simplify process_v2_response

We can merge old_verify_keys into verify_keys, and reduce the number of dicts
flying around.
2019-05-23 11:51:39 +01:00
Richard van der Hoff cc187f9337
Remove unused VerifyKey.expired and .time_added fields (#5235)
These were never used, and poking arbitary data into objects from other
packages seems confusing at best.
2019-05-23 11:46:05 +01:00
Richard van der Hoff 2e052110ee
Rewrite store_server_verify_key to store several keys at once (#5234)
Storing server keys hammered the database a bit. This replaces the
implementation which stored a single key, with one which can do many updates at
once.
2019-05-23 11:45:39 +01:00
Richard van der Hoff 85d1e03b9d
Simplifications and comments in do_auth (#5227)
I was staring at this function trying to figure out wtf it was actually
doing. This is (hopefully) a non-functional refactor which makes it a bit
clearer.
2019-05-23 11:17:42 +01:00
Richard van der Hoff 1a94de60e8
Run black on synapse.crypto.keyring (#5232) 2019-05-22 18:39:33 +01:00
Neil Johnson 73f1de31d1 Merge branch 'master' into develop 2019-05-22 17:59:43 +01:00
Neil Johnson 3d5bba581b 0.99.5.1 2019-05-22 17:52:44 +01:00
Neil Johnson 006bd8f4f6 Revert "0.99.5"
This reverts commit c31e375ade.
2019-05-22 17:49:53 +01:00
Neil Johnson c31e375ade 0.99.5 2019-05-22 17:45:44 +01:00
Marcus Hoffmann 62388a1e44 remove urllib3 pin (#5230)
requests 2.22.0 as been released supporting urllib3 1.25.2

Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
2019-05-22 16:48:12 +01:00
Neil Johnson ae5521be9c Merge branch 'master' into develop 2019-05-22 15:56:55 +01:00
Neil Johnson 8031a6f3d5 0.99.5 2019-05-22 15:40:28 +01:00
Neil Johnson 66b75e2d81
Neilj/ensure get profileinfo available in client reader slaved store (#5213)
* expose SlavedProfileStore to ClientReaderSlavedStore
2019-05-22 13:55:32 +01:00
Richard van der Hoff b898a5600a Merge branch 'master' into develop 2019-05-22 11:38:27 +01:00
Amber Brown 4a30e4acb4
Room Statistics (#4338) 2019-05-21 11:36:50 -05:00
Richard van der Hoff f3ff64e000 Merge commit 'f4c80d70f' into release-v0.99.5 2019-05-21 17:35:31 +01:00
Erik Johnston f4c80d70f8
Merge pull request #5203 from matrix-org/erikj/aggregate_by_sender
Only count aggregations from distinct senders
2019-05-21 17:10:48 +01:00
Erik Johnston 9526aa96a6
Merge pull request #5212 from matrix-org/erikj/deny_multiple_reactions
Block attempts to annotate the same event twice
2019-05-21 17:08:14 +01:00
Richard van der Hoff 959550b645 0.99.5rc1 2019-05-21 16:51:49 +01:00
Erik Johnston 44b8ba484e Fix words 2019-05-21 16:51:45 +01:00
Richard van der Hoff 17f6804837
Introduce room v4 which updates event ID format. (#5217)
Implements https://github.com/matrix-org/matrix-doc/pull/2002.
2019-05-21 16:22:54 +01:00
Richard van der Hoff c4aef549ad
Exclude soft-failed events from fwd-extremity candidates. (#5146)
When considering the candidates to be forward-extremities, we must exclude soft
failures.

Hopefully fixes #5090.
2019-05-21 16:10:54 +01:00
Richard van der Hoff bab3eddac4
Pin eliot to <1.8 on python 3.5.2 (#5218)
* Pin eliot to <1.8 on python 3.5.2

Fixes https://github.com/matrix-org/synapse/issues/5199

* Add support for 'markers' to python_dependencies

* tell xargs not to strip quotes
2019-05-21 15:58:01 +01:00
Brendan Abolivier 6a5a70edf0
Merge pull request #5204 from matrix-org/babolivier/account_validity_expiration_date
Add startup background job for account validity
2019-05-21 14:55:15 +01:00
Brendan Abolivier 384122efa8
Doc 2019-05-21 14:39:36 +01:00
Erik Johnston de7672b78f Don't bundle events in /sync or /events
As we'll send down the annotations too anyway, so this just ends up
confusing clients.
2019-05-21 13:54:09 +01:00
Richard van der Hoff 04d53794d6
Fix error handling for rooms whose versions are unknown. (#5219)
If we remove support for a particular room version, we should behave more
gracefully. This should make client requests fail with a 400 rather than a 500,
and will ignore individiual PDUs in a federation transaction, rather than the
whole transaction.
2019-05-21 13:47:25 +01:00
Brendan Abolivier 5ceee46c6b
Do the select and insert in a single transaction 2019-05-21 13:38:51 +01:00
Erik Johnston c7ec06e8a6 Block attempts to annotate the same event twice 2019-05-20 17:39:05 +01:00
Richard van der Hoff 24b93b9c76 Revert "expose SlavedProfileStore to ClientReaderSlavedStore (#5200)"
This reverts commit ce5bcefc60.

This caused:

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/synapse/src/synapse/app/client_reader.py", line 32, in <module>
    from synapse.replication.slave.storage import SlavedProfileStore
ImportError: cannot import name 'SlavedProfileStore' from 'synapse.replication.slave.storage' (/home/synapse/src/synapse/replication/slave/storage/__init__.py)
error starting synapse.app.client_reader('/home/synapse/config/workers/client_reader.yaml') (exit code: 1); see above for logs
```
2019-05-20 16:21:34 +01:00
Richard van der Hoff 5206648a4a
Add a test room version which updates event ID format (#5210)
Implements MSC1884
2019-05-20 15:54:42 +01:00
Erik Johnston 1dff859d6a Rename relation types to match MSC 2019-05-20 14:31:19 +01:00
Erik Johnston 57ba3451b6
Merge pull request #5209 from matrix-org/erikj/reactions_base
Land basic reaction and edit support.
2019-05-20 14:06:40 +01:00
Erik Johnston 935af0da38 Correctly update aggregation counts after redaction 2019-05-20 12:09:27 +01:00
ReidAnderson 3787133c9e Limit UserIds to a length that fits in a state key (#5198) 2019-05-20 11:20:08 +01:00
Brendan Abolivier ad5b4074e1
Add startup background job for account validity
If account validity is enabled in the server's configuration, this job will run at startup as a background job and will stick an expiration date to any registered account missing one.
2019-05-17 19:37:31 +01:00
Erik Johnston b63cc325a9 Only count aggregations from distinct senders
As a user isn't allowed to send a single emoji more than once.
2019-05-17 18:03:10 +01:00
bytepoets-blo 291e1eea5e fix mapping of return values for get_or_register_3pid_guest (#5177)
* fix mapping of return values for get_or_register_3pid_guest
2019-05-17 17:27:14 +01:00
Erik Johnston 85ece3df46
Merge pull request #5191 from matrix-org/erikj/refactor_pagination_bounds
Make generating SQL bounds for pagination generic
2019-05-17 17:24:36 +01:00
Erik Johnston 8dd9cca8ea Spelling and clarifications 2019-05-17 16:40:51 +01:00
Erik Johnston 5dbff34509 Fixup bsaed on review comments 2019-05-17 15:48:04 +01:00
Neil Johnson ce5bcefc60
expose SlavedProfileStore to ClientReaderSlavedStore (#5200)
* expose SlavedProfileStore to ClientReaderSlavedStore
2019-05-17 13:27:19 +01:00
Brendan Abolivier fdeac1e984
Merge pull request #5196 from matrix-org/babolivier/per_room_profiles
Add an option to disable per-room profiles
2019-05-17 12:10:49 +01:00
PauRE f89f688a55 Fix image orientation when generating thumbnail (#5039) 2019-05-16 19:04:26 +01:00
David Baker 07cff7b121
Merge pull request #5174 from matrix-org/dbkr/add_dummy_flow_to_recaptcha_only
Re-order registration stages to do msisdn & email auth last
2019-05-16 17:27:39 +01:00
Erik Johnston d46aab3fa8 Add basic editing support 2019-05-16 16:54:45 +01:00
Erik Johnston 895179a4dc Update docstring 2019-05-16 16:41:05 +01:00
Brendan Abolivier cd32375846 Add option to disable per-room profiles 2019-05-16 14:34:28 +01:00
Erik Johnston 7a7eba8302 Move parsing of tokens out of storage layer 2019-05-16 14:26:23 +01:00
Erik Johnston 2c662ddde4 Indirect tuple conversion 2019-05-16 14:21:39 +01:00
Erik Johnston 95f3fcda3c Check that event is visible in new APIs 2019-05-16 14:19:06 +01:00
Matthew Hodgson 4a6d5de98c Make /sync attempt to return device updates for both joined and invited users (#3484) 2019-05-16 13:23:43 +01:00
David Baker fafb936de5
Merge pull request #5187 from matrix-org/dbkr/only_check_threepid_not_in_use_if_actually_registering
Only check 3pids not in use when registering
2019-05-16 10:58:09 +01:00
Erik Johnston b5c62c6b26 Fix relations in worker mode 2019-05-16 10:38:13 +01:00
Erik Johnston 33453419b0 Add cache to relations 2019-05-16 10:02:14 +01:00
Erik Johnston a0603523d2 Add aggregations API 2019-05-16 09:37:20 +01:00
Amber Brown f1e5b41388
Make all the rate limiting options more consistent (#5181) 2019-05-15 12:06:04 -05:00
Richard van der Hoff 5f027a315f
Drop support for v2_alpha API prefix (#5190) 2019-05-15 17:37:46 +01:00
Erik Johnston 5be34fc3e3 Actually check for None rather falsey 2019-05-15 17:30:23 +01:00
Erik Johnston e6459c26b4 Actually implement idempotency 2019-05-15 17:28:33 +01:00
Richard van der Hoff 1757e2d7c3 Merge branch 'master' into develop 2019-05-15 14:09:30 +01:00
Richard van der Hoff 4a926f528e 0.99.4 2019-05-15 13:58:45 +01:00
Erik Johnston b50641e357 Add simple pagination API 2019-05-15 13:36:51 +01:00
Erik Johnston efe3c7977a Add simple send_relation API and track in DB 2019-05-15 13:36:51 +01:00
Erik Johnston 54d77107c1 Make generating SQL bounds for pagination generic
This will allow us to reuse the same structure when we paginate e.g.
relations
2019-05-15 11:30:05 +01:00
Erik Johnston 0aba6c8251
Merge pull request #5183 from matrix-org/erikj/async_serialize_event
Allow client event serialization to be async
2019-05-15 10:36:30 +01:00
Erik Johnston d94544051b
Merge pull request #5184 from matrix-org/erikj/expose_get_events_as_array
Expose DataStore._get_events as get_events_as_list
2019-05-15 10:17:38 +01:00
Erik Johnston 8ed2f182f7
Update docstring with correct return type
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-05-15 09:52:52 +01:00
Erik Johnston 52ddc6c0ed
Update docstring with correct type
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-05-15 09:52:15 +01:00
David Baker 6ca88c4693 Only check 3pids not in use when registering
We checked that 3pids were not already in use before we checked if
we were going to return the account previously registered in the
same UI auth session, in which case the 3pids will definitely
be in use.

https://github.com/vector-im/riot-web/issues/9586
2019-05-14 19:04:59 +01:00
Richard van der Hoff daa2fb6317 comment about user_joined_room 2019-05-14 18:53:09 +01:00
Erik Johnston 4fb44fb5b9 Expose DataStore._get_events as get_events_as_list
This is in preparation for reaction work which requires it.
2019-05-14 13:37:44 +01:00
Erik Johnston a80e6b53f9 Newsfile 2019-05-14 13:12:23 +01:00
Erik Johnston b54b03f9e1 Allow client event serialization to be async 2019-05-14 11:58:01 +01:00
Amber Brown df2ebd75d3
Migrate all tests to use the dict-based config format instead of hanging items off HomeserverConfig (#5171) 2019-05-13 15:01:14 -05:00
Andrew Morgan 5a4b328f52 Add ability to blacklist ip ranges for federation traffic (#5043) 2019-05-13 19:05:06 +01:00
Brendan Abolivier 2e1129b5f7 0.99.4rc1 2019-05-13 16:11:21 +01:00
David Baker 516a5fb64b Merge remote-tracking branch 'origin/develop' into dbkr/add_dummy_flow_to_recaptcha_only 2019-05-13 15:54:25 +01:00
David Baker 9e99143c47 Merge remote-tracking branch 'origin/develop' into dbkr/add_dummy_flow_to_recaptcha_only 2019-05-13 15:37:03 +01:00
Brendan Abolivier bb93757b32 Fix CI after new release of isort 2019-05-13 15:19:44 +01:00
Andrew Morgan 2f48c4e1ae
URL preview blacklisting fixes (#5155)
Prevents a SynapseError being raised inside of a IResolutionReceiver and instead opts to just return 0 results. This thus means that we have to lump a failed lookup and a blacklisted lookup together with the same error message, but the substitute should be generic enough to cover both cases.
2019-05-10 10:32:44 -07:00
David Baker 04299132af Re-order flows so that email auth is done last
It's more natural for the user if the bit that takes them away
from the registration flow comes last. Adding the dummy stage allows
us to do the stages in this order without the ambiguity.
2019-05-10 13:58:03 +01:00
David Baker 9c61dce3c8 Comment 2019-05-10 11:14:55 +01:00
David Baker 8714ff6d51 Add a DUMMY stage to captcha-only registration flow
This allows the client to complete the email last which is more
natual for the user. Without this stage, if the client would
complete the recaptcha (and terms, if enabled) stages and then the
registration request would complete because you've now completed a
flow, even if you were intending to complete the flow that's the
same except has email auth at the end.

Adding a dummy auth stage to the recaptcha-only flow means it's
always unambiguous which flow the client was trying to complete.
Longer term we should think about changing the protocol so the
client explicitly says which flow it's trying to complete.

vector-im/riot-web#9586
2019-05-10 11:09:53 +01:00
David Baker c2bb7476c9 Revert 085ae346ac
Accidentally went straight to develop
2019-05-10 11:08:01 +01:00
David Baker 085ae346ac Add a DUMMY stage to captcha-only registration flow
This allows the client to complete the email last which is more
natual for the user. Without this stage, if the client would
complete the recaptcha (and terms, if enabled) stages and then the
registration request would complete because you've now completed a
flow, even if you were intending to complete the flow that's the
same except has email auth at the end.

Adding a dummy auth stage to the recaptcha-only flow means it's
always unambiguous which flow the client was trying to complete.
Longer term we should think about changing the protocol so the
client explicitly says which flow it's trying to complete.

https://github.com/vector-im/riot-web/issues/9586
2019-05-10 10:52:24 +01:00
Richard van der Hoff 130f932cbc Run `black` on per_destination_queue
... mostly to fix pep8 fails
2019-05-09 16:27:02 +01:00
Quentin Dufour 11ea16777f Limit the number of EDUs in transactions to 100 as expected by receiver (#5138)
Fixes #3951.
2019-05-09 11:01:41 +01:00
Matthew Hodgson c0e0740bef add options to require an access_token to GET /profile and /publicRooms on CS API (#5083)
This commit adds two config options:

* `restrict_public_rooms_to_local_users`

Requires auth to fetch the public rooms directory through the CS API and disables fetching it through the federation API.

* `require_auth_for_profile_requests`

When set to `true`, requires that requests to `/profile` over the CS API are authenticated, and only returns the user's profile if the requester shares a room with the profile's owner, as per MSC1301.

MSC1301 also specifies a behaviour for federation (only returning the profile if the server asking for it shares a room with the profile's owner), but that's currently really non-trivial to do in a not too expensive way. Next step is writing down a MSC that allows a HS to specify which user sent the profile query. In this implementation, Synapse won't send a profile query over federation if it doesn't believe it already shares a room with the profile's owner, though.

Groups have been intentionally omitted from this commit.
2019-05-08 18:26:56 +01:00
Erik Johnston c8c069db92
Merge pull request #5037 from matrix-org/erikj/limit_inflight_dns
Limit in flight DNS requests
2019-05-08 17:11:03 +01:00
Brendan Abolivier 1473058b5e
Do checks on aliases for incoming m.room.aliases events (#5128)
Follow-up to #5124

Also added a bunch of checks to make sure everything (both the stuff added on #5124 and this PR) works as intended.
2019-05-08 17:01:30 +01:00
Erik Johnston de655e669a
Merge pull request #5104 from matrix-org/erikj/ratelimit_3pid_invite
Ratelimit 3pid invites
2019-05-07 10:12:49 +01:00