b6dbf89fae
Change more stream_ordering columns to BIGINT ( #10286 )
2021-06-30 17:27:20 +01:00
859dc05b36
Rebuild other indexes using `stream_ordering` ( #10282 )
...
We need to rebuild *all* of the indexes that use the current `stream_ordering`
column.
2021-06-30 15:01:24 +01:00
785bceef72
Merge branch 'release-v1.37' into develop
2021-06-29 20:25:47 +01:00
c54db67d0e
Handle inbound events from federation asynchronously ( #10272 )
...
Fixes #9490
This will break a couple of SyTest that are expecting failures to be added to the response of a federation /send, which obviously doesn't happen now that things are asynchronous.
Two drawbacks:
Currently there is no logic to handle any events left in the staging area after restart, and so they'll only be handled on the next incoming event in that room. That can be fixed separately.
We now only process one event per room at a time. This can be fixed up further down the line.
2021-06-29 19:55:22 +01:00
85d237eba7
Add a distributed lock ( #10269 )
...
This adds a simple best effort locking mechanism that works cross workers.
2021-06-29 19:15:47 +01:00
60efc51a2b
Migrate stream_ordering to a bigint ( #10264 )
...
* Move background update names out to a separate class
`EventsBackgroundUpdatesStore` gets inherited and we don't really want to
further pollute the namespace.
* Migrate stream_ordering to a bigint
* changelog
2021-06-29 11:25:34 +01:00
bd4919fb72
MSC2918 Refresh tokens implementation ( #9450 )
...
This implements refresh tokens, as defined by MSC2918
This MSC has been implemented client side in Hydrogen Web: vector-im/hydrogen-web#235
The basics of the MSC works: requesting refresh tokens on login, having the access tokens expire, and using the refresh token to get a new one.
Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
2021-06-24 14:33:20 +01:00
33701dc116
Fix schema delta to not take as long on large servers ( #10227 )
...
Introduced in #6739
2021-06-22 12:00:45 +01:00
c1b9922498
Support for database schema version ranges ( #9933 )
...
This is essentially an implementation of the proposal made at https://hackmd.io/@richvdh/BJYXQMQHO , though the details have ended up looking slightly different.
2021-06-11 14:45:53 +01:00
d936371b69
Implement knock feature ( #6739 )
...
This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403
Signed-off-by: Sorunome mail@sorunome.de
Signed-off-by: Andrew Morgan andrewm@element.io
2021-06-09 19:39:51 +01:00
4d6e5a5e99
Use a database table to hold the users that should have full presence sent to them, instead of something in-memory ( #9823 )
2021-05-18 14:13:45 +01:00
25f43faa70
Reorganise the database schema directories ( #9932 )
...
The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate.
It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.
2021-05-07 10:22:05 +01:00
7b608cf468
Only run one background update at a time
2020-03-31 17:43:58 +01:00
3aa2a90556
Move schema delta files to the correct data store.
...
They were put in the global schema delta directory due to a bad merge.
2019-10-24 16:45:03 +01:00
bb6264be0b
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_stores
2019-10-22 10:41:18 +01:00
c66a06ac6b
Move storage classes into a main "data store".
...
This is in preparation for having multiple data stores that offer
different functionality, e.g. splitting out state or event storage.
2019-10-21 16:05:06 +01:00
cfc28325a6
Merge branch 'develop' into uhoreg/e2e_cross-signing_merged
2019-10-18 11:39:40 +01:00
91f43dca39
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/disable_sql_bytes
2019-10-10 13:10:57 +01:00
562b4e51dd
Rewrite the user_filter migration again ( #6184 )
...
you can't plausibly ALTER TABLE in sqlite, so we create the new table with the
right schema to start with.
2019-10-10 11:28:23 +01:00
7f18b3d526
Do the update as a background index
2019-10-09 16:03:24 +01:00
def5413480
Merge pull request #6185 from matrix-org/erikj/fix_censored_evnets
...
Fix inserting bytes as text in `censor_redactions`
2019-10-09 15:39:13 +01:00
e7631d84e6
Fix existing hex encoded json values in DB
2019-10-08 16:18:43 +01:00
c8e6c308c6
Fix unique_user_filter_index schema update
2019-10-07 13:15:35 +01:00
13c4345c84
Update `user_filters` table to have a unique index, and non-null columns ( #1172 )
2019-10-04 10:34:16 +01:00
a5166e4d5f
Land improved room list based on room stats ( #6019 )
...
Use room_stats and room_state for room directory search
2019-10-02 14:08:35 +01:00
ecd254bc49
Merge branch 'release-v1.4.0' of github.com:matrix-org/synapse into develop
2019-10-02 11:08:07 +01:00
f44f1d2e83
Fix errors storing large retry intervals.
...
We have set the max retry interval to a value larger than a postgres or
sqlite int can hold, which caused exceptions when updating the
destinations table.
To fix postgres we need to change the column to a bigint, and for sqlite
we lower the max interval to 2**62 (which is still incredibly long).
2019-10-02 10:36:27 +01:00
898dde981b
Add received_ts column to redactions.
...
This will allow us to efficiently search for uncensored redactions in
the DB before a given time.
2019-10-01 13:43:48 +01:00
16cb9a71b8
Drop unused tables ( #6115 )
...
These tables are unused since #5893 (as amended by #6047 ), so we can now drop
them.
Fixes #6048 .
2019-09-30 09:38:41 +01:00
d2bd0bc6b1
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips
2019-09-25 17:16:28 +01:00
ed8b92f0d2
Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work
2019-09-24 12:57:32 +01:00
ed80231ade
Add BG update to populate devices last seen info
2019-09-23 16:59:45 +01:00
2ade05dca3
Add last seen info to devices table.
...
This allows us to purge old user_ips entries without having to preserve
the latest last seen info for active devices.
2019-09-23 16:59:35 +01:00
b65327ff66
Merge branch 'develop' into rav/saml_mapping_work
2019-09-19 18:13:31 +01:00
bcd9132869
Undo the deletion of some tables ( #6047 )
...
This is a partial revert of #5893 . The problem is that if we drop these tables
in the same release as removing the code that writes to them, it prevents users
users from being able to roll back to a previous release.
So let's leave the tables in place for now, and remember to drop them in a
subsequent release.
(Note that these tables haven't been *read* for *years*, so any missing rows
resulting from a temporary upgrade to vNext won't cause a problem.)
2019-09-19 15:06:27 +01:00
1e19ce00bf
Add 'failure_ts' column to 'destinations' table ( #6016 )
...
Track the time that a server started failing at, for general analysis purposes.
2019-09-17 11:41:54 +01:00
a8ac40445c
Record mappings from saml users in an external table
...
We want to assign unique mxids to saml users based on an incrementing
suffix. For that to work, we need to record the allocated mxid in a separate
table.
2019-09-13 16:01:46 +01:00
591d82f06b
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/censor_redactions
2019-09-05 17:27:46 +01:00
faf72a4c40
Merge branch 'develop' into cross-signing_keys
2019-09-04 19:12:29 -04:00
6e834e94fc
Fix and refactor room and user stats ( #5971 )
...
Previously the stats were not being correctly populated.
2019-09-04 13:04:27 +01:00
a4bf72c30c
Censor redactions in DB after a month
2019-08-30 11:29:17 +01:00
e3d3fbf63f
Merge branch 'uhoreg/e2e_cross-signing_merged' into cross-signing_keys
2019-08-28 17:36:46 -07:00
72d296a7f3
Merge branch 'develop' into uhoreg/e2e_cross-signing_merged
2019-08-28 17:33:44 -07:00
3b0b22cb05
use stream ID generator instead of timestamp
2019-08-28 17:17:21 -07:00
8767b63a82
Propagate opentracing contexts through EDUs ( #5852 )
...
Propagate opentracing contexts through EDUs
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-08-22 18:21:10 +01:00
c9f11d09fc
Add missing index on users_in_public_rooms. ( #5894 )
2019-08-22 10:43:13 +01:00
4dab867288
Drop some unused tables. ( #5893 )
...
These tables are never used, so we may as well drop them.
2019-08-21 13:16:28 +01:00
b7f7cc7ace
add the version field to the index for e2e_room_keys
2019-08-14 17:14:40 -07:00
f63ba7a795
Cross-signing [1/4] -- hidden devices ( #5759 )
...
* allow devices to be marked as "hidden"
This is a prerequisite for cross-signing, as it allows us to create other things
that live within the device namespace, so they can be used for signatures.
2019-08-12 15:14:37 -07:00
fac1cdc562
make changes from PR review
2019-08-01 21:51:19 -04:00
336c546d6a
Merge branch 'cross-signing_hidden' into cross-signing_keys
2019-08-01 16:31:40 -04:00
c1f0a5636c
Merge branch 'develop' into cross-signing_hidden
2019-07-30 23:33:06 -04:00
bc95890822
Merge branch 'cross-signing_hidden' into cross-signing_keys
2019-07-30 23:21:26 -04:00
781ade836b
apply changes from PR review
2019-07-30 23:09:50 -04:00
123c04daa7
Don't recreate column
2019-07-30 16:42:48 +01:00
15056ca208
Fix current_state_events membership background update.
...
Turns out not all rooms are in `rooms`, so lets fetch the room list from
`current_state_events`. We move the delta file to force it to be run
again.
2019-07-30 14:51:41 +01:00
105e7f6ed3
Remove lost comment
2019-07-29 16:09:48 +01:00
c659b9f94f
allow uploading keys for cross-signing
2019-07-25 11:08:24 -04:00
d1c7c2a98a
allow devices to be marked as "hidden"
...
This is a prerequisite for cross-signing, as it allows us to create other things
that live within the device namespace, so they can be used for signatures.
2019-07-24 23:21:52 -04:00
32768e96d4
Add function to get all forgotten rooms for user
...
This will allow us to efficiently filter out rooms that have been
forgotten in other queries without having to join against the
`room_memberships` table.
2019-07-24 11:44:23 +01:00
ebc5ed1296
Update comment for new column
2019-07-19 13:29:02 +01:00
c618a5d348
Add background update for current_state_events.membership column
2019-07-18 14:16:01 +01:00
6de09e07a6
Add membership column to current_state_events table.
...
It turns out that doing a join is surprisingly expensive for the DB to
do when room_membership table is larger than the disk cache.
2019-07-18 14:15:57 +01:00
5f158ec039
Implement access token expiry ( #5660 )
...
Record how long an access token is valid for, and raise a soft-logout once it
expires.
2019-07-12 17:26:02 +01:00
32e7c9e7f2
Run Black. ( #5482 )
2019-06-20 19:32:02 +10:00
d0530382ee
Track deactivated accounts in the database ( #5378 )
2019-06-14 13:18:24 +01:00
1b8cb64393
Merge pull request #5320 from matrix-org/hawkowl/full-schema-v1
...
Make a full SQL schema
2019-06-06 18:35:38 +01:00
ed872db8df
fix maybe
2019-06-07 02:53:47 +10:00
3719680ee4
Add ability to perform password reset via email without trusting the identity server ( #5377 )
...
Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option.
This PR is a culmination of 3 smaller PRs which have each been separately reviewed:
* #5308
* #5345
* #5368
2019-06-06 17:34:07 +01:00
837e32ef55
just user dir?
2019-06-07 01:49:25 +10:00
3b6645d3bf
remove background updates that arent needed
2019-06-07 01:20:58 +10:00
6362e3af14
add more comments
2019-06-06 04:20:35 +10:00
2198b7ce2a
add stuff in bg updates
2019-06-04 01:06:00 +10:00
4e75c5e02a
WHY IS THIS CALLED A SLIGHTLY DIFFERENT THING
2019-06-03 22:42:12 +10:00
be452fc9ac
more fix
2019-06-03 22:24:23 +10:00
7f81b967ca
fix schemas
2019-06-03 22:23:40 +10:00
dc72b90cd6
full schema
2019-06-03 22:03:28 +10:00
31d44ec4bd
Merge pull request #5294 from matrix-org/erikj/speed_up_room_stats
...
Speed up room stats background update
2019-05-31 10:48:51 +01:00
5037326d66
Add indices. Remove room_ids accidentally added
...
We have to do this by re-inserting a background update and recreating
tables, as the tables only get created during a background update and
will later be deleted.
We also make sure that we remove any entries that should have been
removed but weren't due to a race that has been fixed in a previous
commit.
2019-05-31 10:26:56 +01:00
6cdfb0207e
Add index to temp table
2019-05-30 15:02:31 +01:00
640fcbb07f
Fixup comments and logging
2019-05-30 10:55:55 +01:00
d79c9994f4
Add DB bg update to cleanup extremities.
...
Due to #5269 we may have extremities in our DB that we shouldn't have,
so lets add a cleanup task such to remove those.
2019-05-29 15:11:26 +01:00
a97d4e218a
Merge pull request #5268 from matrix-org/babolivier/account_validity_fix_schema
...
Fix schema update for account validity
2019-05-28 10:30:07 +01:00
ba17de7fbc
Fix schema update for account validity
2019-05-28 10:11:38 +01:00
b75537beaf
Store key validity time in the storage layer
...
This is a first step to checking that the key is valid at the required moment.
The idea here is that, rather than passing VerifyKey objects in and out of the
storage layer, we instead pass FetchKeyResult objects, which simply wrap the
VerifyKey and add a valid_until_ts field.
2019-05-23 11:52:22 +01:00
4a30e4acb4
Room Statistics ( #4338 )
2019-05-21 11:36:50 -05:00
efe3c7977a
Add simple send_relation API and track in DB
2019-05-15 13:36:51 +01:00
91934025b9
Merge pull request #5047 from matrix-org/babolivier/account_expiration
...
Send out emails with links to extend an account's validity period
2019-04-17 14:57:39 +01:00
20f0617e87
Send out emails with links to extend an account's validity period
2019-04-17 14:42:20 +01:00
e446921def
Merge pull request #5033 from matrix-org/erikj/fix_schema_delta
...
Fix schema upgrade when dropping tables
2019-04-10 10:22:35 +01:00
bfc8fdf1fc
Merge pull request #5027 from matrix-org/babolivier/account_expiration
...
Add time-based account expiration
2019-04-09 17:02:41 +01:00
747aa9f8ca
Add account expiration feature
2019-04-09 16:46:04 +01:00
50d2a3059d
Fix schema upgrade when dropping tables
...
We need to drop tables in the correct order due to foreign table
constraints (on `application_services`), otherwise the DROP TABLE
command will fail.
Introduced in #4992 .
2019-04-09 14:39:18 +01:00
3352baac4b
Remove unused server_tls_certificates functions ( #5028 )
...
These have been unused since #4120 , and with the demise of perspectives, it is
unlikely that they will ever be used again.
2019-04-08 21:50:18 +01:00
2d951686a7
drop tables listed in #1830 ( #4992 )
...
Tables dropped:
* application_services,
* application_services_regex,
* transaction_id_to_pdu,
* stats_reporting
* current_state_resets
* event_content_hashes
* event_destinations
* event_edge_hashes
* event_signatures
* feedback
* room_hosts
* state_forward_extremities
2019-04-08 15:37:26 +01:00
e8419554ff
Remove presence lists ( #4989 )
...
Remove presence list support as per MSC 1819
2019-04-03 11:11:15 +01:00
c75e2017f1
Fixup docstrings
2019-04-02 11:20:06 +01:00
3715c124b3
Grandfather in existing user threepids
...
We assume, as we did before, that users bound their threepid to one of
the trusted identity servers. So we simply fill the new table with all
threepids in `user_threepids` joined with the trusted identity servers.
2019-04-01 15:25:19 +01:00
1666c0696a
Track IS used to bind 3PIDs
...
This will then be used to know which IS to default to when unbinding the
threepid.
2019-04-01 15:23:01 +01:00
e9eeca1314
Fix user directory background update ( #4887 )
2019-03-19 22:13:53 +11:00
Richard van der Hoff