Commit Graph

3056 Commits

Author SHA1 Message Date
Erik Johnston c413540fb9 Fix bug sending federation transactions with lots of EDUs
If we try and send a transaction with lots of EDUs and we run out of
space, we call get_new_device_msgs_for_remote with a limit of 0, which
then failed.
2019-06-10 16:22:45 +01:00
Erik Johnston 43badd2cd4 Fix key verification when key stored with null valid_until_ms
Some keys are stored in the synapse database with a null valid_until_ms
which caused an exception to be thrown when using that key. We fix this
by treating nulls as zeroes, i.e. they keys will match verification
requests with a minimum_valid_until_ms of zero (i.e. don't validate ts)
but will not match requests with a non-zero minimum_valid_until_ms.

Fixes #5391.
2019-06-10 15:37:10 +01:00
Andrew Morgan 2d1d7b7e6f Prevent multiple device list updates from breaking a batch send (#5156)
fixes #5153
2019-06-06 23:54:00 +01:00
Erik Johnston 1b8cb64393
Merge pull request #5320 from matrix-org/hawkowl/full-schema-v1
Make a full SQL schema
2019-06-06 18:35:38 +01:00
Amber H. Brown ed872db8df fix maybe 2019-06-07 02:53:47 +10:00
Andrew Morgan 3719680ee4
Add ability to perform password reset via email without trusting the identity server (#5377)
Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option.

This PR is a culmination of 3 smaller PRs which have each been separately reviewed:

* #5308
* #5345
* #5368
2019-06-06 17:34:07 +01:00
Amber H. Brown 837e32ef55 just user dir? 2019-06-07 01:49:25 +10:00
Amber H. Brown 3b6645d3bf remove background updates that arent needed 2019-06-07 01:20:58 +10:00
Erik Johnston 89d3d7b2c0
Merge pull request #5221 from matrix-org/erikj/fix_worker_sytest
Fix get_max_topological_token to never return None
2019-06-06 13:54:46 +01:00
Erik Johnston 7f08a3523a Better words 2019-06-06 11:09:38 +01:00
Amber Brown 6362e3af14 add more comments 2019-06-06 04:20:35 +10:00
Erik Johnston 75538813fc Fix background updates to handle redactions/rejections (#5352)
* Fix background updates to handle redactions/rejections

In background updates based on current state delta stream we need to
handle that we may not have all the events (or at least that
`get_events` may raise an exception).
2019-06-06 00:45:46 +10:00
Erik Johnston 0a56966f7d Fix 2019-06-03 17:42:52 +01:00
Erik Johnston fa4b54aca5 Ignore room state with null bytes in for room stats 2019-06-03 17:06:54 +01:00
Amber Brown 2198b7ce2a add stuff in bg updates 2019-06-04 01:06:00 +10:00
Amber Brown 4e75c5e02a WHY IS THIS CALLED A SLIGHTLY DIFFERENT THING 2019-06-03 22:42:12 +10:00
Amber Brown be452fc9ac more fix 2019-06-03 22:24:23 +10:00
Amber Brown 7f81b967ca fix schemas 2019-06-03 22:23:40 +10:00
Amber Brown dc72b90cd6 full schema 2019-06-03 22:03:28 +10:00
Amber Brown 37057d5d60 prepare 2019-06-03 22:02:47 +10:00
Erik Johnston 58cce39f3a
Merge pull request #5276 from matrix-org/babolivier/account_validity_job_delta
Allow configuring a range for the account validity startup job
2019-05-31 12:11:56 +01:00
Brendan Abolivier 4d794dae21 Move delta from +10% to -10% 2019-05-31 11:09:39 +01:00
Erik Johnston 31d44ec4bd
Merge pull request #5294 from matrix-org/erikj/speed_up_room_stats
Speed up room stats background update
2019-05-31 10:48:51 +01:00
Erik Johnston 5037326d66 Add indices. Remove room_ids accidentally added
We have to do this by re-inserting a background update and recreating
tables, as the tables only get created during a background update and
will later be deleted.

We also make sure that we remove any entries that should have been
removed but weren't due to a race that has been fixed in a previous
commit.
2019-05-31 10:26:56 +01:00
Brendan Abolivier 847b9dcd1c Make max_delta equal to period * 10% 2019-05-31 09:54:46 +01:00
Erik Johnston 5ac75fc9a2 Join against events to use its room_id index 2019-05-30 15:26:55 +01:00
Erik Johnston e2c46ed851 Move deletion from table inside txn 2019-05-30 15:26:38 +01:00
Erik Johnston 04710cc2d7 Fetch membership counts all at once 2019-05-30 15:25:41 +01:00
Erik Johnston 54d50fbfdf Get events all at once 2019-05-30 15:15:13 +01:00
Erik Johnston 6cdfb0207e Add index to temp table 2019-05-30 15:02:31 +01:00
Erik Johnston e9e5d3392d
Merge pull request #5278 from matrix-org/erikj/cleanup_bad_extremities
Add DB bg update to cleanup extremities.
2019-05-30 14:28:26 +01:00
Erik Johnston cb967e2346
Update synapse/storage/events_bg_updates.py
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-05-30 14:06:42 +01:00
Erik Johnston 468bd090ff Rename constant 2019-05-30 11:24:42 +01:00
Erik Johnston 5c1ece0ffc Move event background updates to a separate file 2019-05-30 11:22:59 +01:00
Erik Johnston 640fcbb07f Fixup comments and logging 2019-05-30 10:55:55 +01:00
Amber Brown 46c8f7a517
Implement the SHHS complexity API (#5216) 2019-05-30 01:47:16 +10:00
Erik Johnston 7e8e683754 Log actual number of entries deleted 2019-05-29 15:11:28 +01:00
Erik Johnston d79c9994f4 Add DB bg update to cleanup extremities.
Due to #5269 we may have extremities in our DB that we shouldn't have,
so lets add a cleanup task such to remove those.
2019-05-29 15:11:26 +01:00
Erik Johnston 58c8ed5b0d
Correctly filter out extremities with soft failed prevs (#5274)
When we receive a soft failed event we, correctly, *do not* update the
forward extremity table with the event. However, if we later receive an
event that references the soft failed event we then need to remove the
soft failed events prev events from the forward extremities table,
otherwise we just build up forward extremities.

Fixes #5269
2019-05-29 11:56:24 +01:00
Brendan Abolivier 52839886d6
Allow configuring a range for the account validity startup job
When enabling the account validity feature, Synapse will look at startup for registered account without an expiration date, and will set one equals to 'now + validity_period' for them. On large servers, it can mean that a large number of users will have the same expiration date, which means that they will all be sent a renewal email at the same time, which isn't ideal.
In order to mitigate this, this PR allows server admins to define a 'max_delta' so that the expiration date is a random value in the [now + validity_period ; now + validity_period + max_delta] range. This allows renewal emails to be progressively sent over a configured period instead of being sent all in one big batch.
2019-05-28 16:52:45 +01:00
Brendan Abolivier a97d4e218a
Merge pull request #5268 from matrix-org/babolivier/account_validity_fix_schema
Fix schema update for account validity
2019-05-28 10:30:07 +01:00
Brendan Abolivier ba17de7fbc
Fix schema update for account validity 2019-05-28 10:11:38 +01:00
Travis Ralston bc4b2ecf70 Fix logging for room stats background update 2019-05-25 12:02:48 -06:00
Richard van der Hoff b75537beaf Store key validity time in the storage layer
This is a first step to checking that the key is valid at the required moment.

The idea here is that, rather than passing VerifyKey objects in and out of the
storage layer, we instead pass FetchKeyResult objects, which simply wrap the
VerifyKey and add a valid_until_ts field.
2019-05-23 11:52:22 +01:00
Richard van der Hoff 2e052110ee
Rewrite store_server_verify_key to store several keys at once (#5234)
Storing server keys hammered the database a bit. This replaces the
implementation which stored a single key, with one which can do many updates at
once.
2019-05-23 11:45:39 +01:00
Richard van der Hoff 85d1e03b9d
Simplifications and comments in do_auth (#5227)
I was staring at this function trying to figure out wtf it was actually
doing. This is (hopefully) a non-functional refactor which makes it a bit
clearer.
2019-05-23 11:17:42 +01:00
Amber Brown 4a30e4acb4
Room Statistics (#4338) 2019-05-21 11:36:50 -05:00
Erik Johnston f4c80d70f8
Merge pull request #5203 from matrix-org/erikj/aggregate_by_sender
Only count aggregations from distinct senders
2019-05-21 17:10:48 +01:00
Erik Johnston 9526aa96a6
Merge pull request #5212 from matrix-org/erikj/deny_multiple_reactions
Block attempts to annotate the same event twice
2019-05-21 17:08:14 +01:00
Richard van der Hoff c4aef549ad
Exclude soft-failed events from fwd-extremity candidates. (#5146)
When considering the candidates to be forward-extremities, we must exclude soft
failures.

Hopefully fixes #5090.
2019-05-21 16:10:54 +01:00