Commit Graph

759 Commits

Author SHA1 Message Date
Richard van der Hoff 859dc05b36
Rebuild other indexes using `stream_ordering` (#10282)
We need to rebuild *all* of the indexes that use the current `stream_ordering`
column.
2021-06-30 15:01:24 +01:00
Richard van der Hoff 785bceef72 Merge branch 'release-v1.37' into develop 2021-06-29 20:25:47 +01:00
Erik Johnston c54db67d0e
Handle inbound events from federation asynchronously (#10272)
Fixes #9490

This will break a couple of SyTest that are expecting failures to be added to the response of a federation /send, which obviously doesn't happen now that things are asynchronous.

Two drawbacks:

    Currently there is no logic to handle any events left in the staging area after restart, and so they'll only be handled on the next incoming event in that room. That can be fixed separately.
    We now only process one event per room at a time. This can be fixed up further down the line.
2021-06-29 19:55:22 +01:00
Erik Johnston 85d237eba7
Add a distributed lock (#10269)
This adds a simple best effort locking mechanism that works cross workers.
2021-06-29 19:15:47 +01:00
Richard van der Hoff 60efc51a2b
Migrate stream_ordering to a bigint (#10264)
* Move background update names out to a separate class

`EventsBackgroundUpdatesStore` gets inherited and we don't really want to
further pollute the namespace.

* Migrate stream_ordering to a bigint

* changelog
2021-06-29 11:25:34 +01:00
Quentin Gliech bd4919fb72
MSC2918 Refresh tokens implementation (#9450)
This implements refresh tokens, as defined by MSC2918

This MSC has been implemented client side in Hydrogen Web: vector-im/hydrogen-web#235

The basics of the MSC works: requesting refresh tokens on login, having the access tokens expire, and using the refresh token to get a new one.

Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
2021-06-24 14:33:20 +01:00
Erik Johnston 33701dc116
Fix schema delta to not take as long on large servers (#10227)
Introduced in #6739
2021-06-22 12:00:45 +01:00
Richard van der Hoff c1b9922498
Support for database schema version ranges (#9933)
This is essentially an implementation of the proposal made at https://hackmd.io/@richvdh/BJYXQMQHO, though the details have ended up looking slightly different.
2021-06-11 14:45:53 +01:00
Sorunome d936371b69
Implement knock feature (#6739)
This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403

Signed-off-by: Sorunome mail@sorunome.de
Signed-off-by: Andrew Morgan andrewm@element.io
2021-06-09 19:39:51 +01:00
Andrew Morgan 4d6e5a5e99
Use a database table to hold the users that should have full presence sent to them, instead of something in-memory (#9823) 2021-05-18 14:13:45 +01:00
Richard van der Hoff 25f43faa70
Reorganise the database schema directories (#9932)
The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate.

It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.
2021-05-07 10:22:05 +01:00
Richard van der Hoff 7b608cf468 Only run one background update at a time 2020-03-31 17:43:58 +01:00
Erik Johnston 3aa2a90556 Move schema delta files to the correct data store.
They were put in the global schema delta directory due to a bad merge.
2019-10-24 16:45:03 +01:00
Erik Johnston bb6264be0b Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_stores 2019-10-22 10:41:18 +01:00
Erik Johnston c66a06ac6b Move storage classes into a main "data store".
This is in preparation for having multiple data stores that offer
different functionality, e.g. splitting out state or event storage.
2019-10-21 16:05:06 +01:00
Hubert Chathi cfc28325a6 Merge branch 'develop' into uhoreg/e2e_cross-signing_merged 2019-10-18 11:39:40 +01:00
Erik Johnston 91f43dca39 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/disable_sql_bytes 2019-10-10 13:10:57 +01:00
Richard van der Hoff 562b4e51dd
Rewrite the user_filter migration again (#6184)
you can't plausibly ALTER TABLE in sqlite, so we create the new table with the
right schema to start with.
2019-10-10 11:28:23 +01:00
Erik Johnston 7f18b3d526 Do the update as a background index 2019-10-09 16:03:24 +01:00
Erik Johnston def5413480
Merge pull request #6185 from matrix-org/erikj/fix_censored_evnets
Fix inserting bytes as text in `censor_redactions`
2019-10-09 15:39:13 +01:00
Erik Johnston e7631d84e6 Fix existing hex encoded json values in DB 2019-10-08 16:18:43 +01:00
Brendan Abolivier c8e6c308c6
Fix unique_user_filter_index schema update 2019-10-07 13:15:35 +01:00
Alexander Maznev 13c4345c84 Update `user_filters` table to have a unique index, and non-null columns (#1172) 2019-10-04 10:34:16 +01:00
Erik Johnston a5166e4d5f
Land improved room list based on room stats (#6019)
Use room_stats and room_state for room directory search
2019-10-02 14:08:35 +01:00
Erik Johnston ecd254bc49 Merge branch 'release-v1.4.0' of github.com:matrix-org/synapse into develop 2019-10-02 11:08:07 +01:00
Erik Johnston f44f1d2e83 Fix errors storing large retry intervals.
We have set the max retry interval to a value larger than a postgres or
sqlite int can hold, which caused exceptions when updating the
destinations table.

To fix postgres we need to change the column to a bigint, and for sqlite
we lower the max interval to 2**62 (which is still incredibly long).
2019-10-02 10:36:27 +01:00
Erik Johnston 898dde981b Add received_ts column to redactions.
This will allow us to efficiently search for uncensored redactions in
the DB before a given time.
2019-10-01 13:43:48 +01:00
Richard van der Hoff 16cb9a71b8
Drop unused tables (#6115)
These tables are unused since #5893 (as amended by #6047), so we can now drop
them.

Fixes #6048.
2019-09-30 09:38:41 +01:00
Erik Johnston d2bd0bc6b1 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips 2019-09-25 17:16:28 +01:00
Richard van der Hoff ed8b92f0d2 Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work 2019-09-24 12:57:32 +01:00
Erik Johnston ed80231ade Add BG update to populate devices last seen info 2019-09-23 16:59:45 +01:00
Erik Johnston 2ade05dca3 Add last seen info to devices table.
This allows us to purge old user_ips entries without having to preserve
the latest last seen info for active devices.
2019-09-23 16:59:35 +01:00
Richard van der Hoff b65327ff66 Merge branch 'develop' into rav/saml_mapping_work 2019-09-19 18:13:31 +01:00
Richard van der Hoff bcd9132869
Undo the deletion of some tables (#6047)
This is a partial revert of #5893. The problem is that if we drop these tables
in the same release as removing the code that writes to them, it prevents users
users from being able to roll back to a previous release.

So let's leave the tables in place for now, and remember to drop them in a
subsequent release.

(Note that these tables haven't been *read* for *years*, so any missing rows
resulting from a temporary upgrade to vNext won't cause a problem.)
2019-09-19 15:06:27 +01:00
Richard van der Hoff 1e19ce00bf
Add 'failure_ts' column to 'destinations' table (#6016)
Track the time that a server started failing at, for general analysis purposes.
2019-09-17 11:41:54 +01:00
Richard van der Hoff a8ac40445c Record mappings from saml users in an external table
We want to assign unique mxids to saml users based on an incrementing
suffix. For that to work, we need to record the allocated mxid in a separate
table.
2019-09-13 16:01:46 +01:00
Erik Johnston 591d82f06b Merge branch 'develop' of github.com:matrix-org/synapse into erikj/censor_redactions 2019-09-05 17:27:46 +01:00
Hubert Chathi faf72a4c40 Merge branch 'develop' into cross-signing_keys 2019-09-04 19:12:29 -04:00
Erik Johnston 6e834e94fc
Fix and refactor room and user stats (#5971)
Previously the stats were not being correctly populated.
2019-09-04 13:04:27 +01:00
Erik Johnston a4bf72c30c Censor redactions in DB after a month 2019-08-30 11:29:17 +01:00
Hubert Chathi e3d3fbf63f Merge branch 'uhoreg/e2e_cross-signing_merged' into cross-signing_keys 2019-08-28 17:36:46 -07:00
Hubert Chathi 72d296a7f3 Merge branch 'develop' into uhoreg/e2e_cross-signing_merged 2019-08-28 17:33:44 -07:00
Hubert Chathi 3b0b22cb05 use stream ID generator instead of timestamp 2019-08-28 17:17:21 -07:00
Jorik Schellekens 8767b63a82
Propagate opentracing contexts through EDUs (#5852)
Propagate opentracing contexts through EDUs
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-08-22 18:21:10 +01:00
Richard van der Hoff c9f11d09fc
Add missing index on users_in_public_rooms. (#5894) 2019-08-22 10:43:13 +01:00
Richard van der Hoff 4dab867288
Drop some unused tables. (#5893)
These tables are never used, so we may as well drop them.
2019-08-21 13:16:28 +01:00
Hubert Chathi b7f7cc7ace add the version field to the index for e2e_room_keys 2019-08-14 17:14:40 -07:00
Hubert Chathi f63ba7a795
Cross-signing [1/4] -- hidden devices (#5759)
* allow devices to be marked as "hidden"

This is a prerequisite for cross-signing, as it allows us to create other things
that live within the device namespace, so they can be used for signatures.
2019-08-12 15:14:37 -07:00
Hubert Chathi fac1cdc562 make changes from PR review 2019-08-01 21:51:19 -04:00
Hubert Chathi 336c546d6a Merge branch 'cross-signing_hidden' into cross-signing_keys 2019-08-01 16:31:40 -04:00
Hubert Chathi c1f0a5636c Merge branch 'develop' into cross-signing_hidden 2019-07-30 23:33:06 -04:00
Hubert Chathi bc95890822 Merge branch 'cross-signing_hidden' into cross-signing_keys 2019-07-30 23:21:26 -04:00
Hubert Chathi 781ade836b apply changes from PR review 2019-07-30 23:09:50 -04:00
Erik Johnston 123c04daa7 Don't recreate column 2019-07-30 16:42:48 +01:00
Erik Johnston 15056ca208 Fix current_state_events membership background update.
Turns out not all rooms are in `rooms`, so lets fetch the room list from
`current_state_events`. We move the delta file to force it to be run
again.
2019-07-30 14:51:41 +01:00
Erik Johnston 105e7f6ed3 Remove lost comment 2019-07-29 16:09:48 +01:00
Hubert Chathi c659b9f94f allow uploading keys for cross-signing 2019-07-25 11:08:24 -04:00
Hubert Chathi d1c7c2a98a allow devices to be marked as "hidden"
This is a prerequisite for cross-signing, as it allows us to create other things
that live within the device namespace, so they can be used for signatures.
2019-07-24 23:21:52 -04:00
Erik Johnston 32768e96d4 Add function to get all forgotten rooms for user
This will allow us to efficiently filter out rooms that have been
forgotten in other queries without having to join against the
`room_memberships` table.
2019-07-24 11:44:23 +01:00
Erik Johnston ebc5ed1296 Update comment for new column 2019-07-19 13:29:02 +01:00
Erik Johnston c618a5d348 Add background update for current_state_events.membership column 2019-07-18 14:16:01 +01:00
Erik Johnston 6de09e07a6 Add membership column to current_state_events table.
It turns out that doing a join is surprisingly expensive for the DB to
do when room_membership table is larger than the disk cache.
2019-07-18 14:15:57 +01:00
Richard van der Hoff 5f158ec039
Implement access token expiry (#5660)
Record how long an access token is valid for, and raise a soft-logout once it
expires.
2019-07-12 17:26:02 +01:00
Amber Brown 32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Brendan Abolivier d0530382ee
Track deactivated accounts in the database (#5378) 2019-06-14 13:18:24 +01:00
Erik Johnston 1b8cb64393
Merge pull request #5320 from matrix-org/hawkowl/full-schema-v1
Make a full SQL schema
2019-06-06 18:35:38 +01:00
Amber H. Brown ed872db8df fix maybe 2019-06-07 02:53:47 +10:00
Andrew Morgan 3719680ee4
Add ability to perform password reset via email without trusting the identity server (#5377)
Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option.

This PR is a culmination of 3 smaller PRs which have each been separately reviewed:

* #5308
* #5345
* #5368
2019-06-06 17:34:07 +01:00
Amber H. Brown 837e32ef55 just user dir? 2019-06-07 01:49:25 +10:00
Amber H. Brown 3b6645d3bf remove background updates that arent needed 2019-06-07 01:20:58 +10:00
Amber Brown 6362e3af14 add more comments 2019-06-06 04:20:35 +10:00
Amber Brown 2198b7ce2a add stuff in bg updates 2019-06-04 01:06:00 +10:00
Amber Brown 4e75c5e02a WHY IS THIS CALLED A SLIGHTLY DIFFERENT THING 2019-06-03 22:42:12 +10:00
Amber Brown be452fc9ac more fix 2019-06-03 22:24:23 +10:00
Amber Brown 7f81b967ca fix schemas 2019-06-03 22:23:40 +10:00
Amber Brown dc72b90cd6 full schema 2019-06-03 22:03:28 +10:00
Erik Johnston 31d44ec4bd
Merge pull request #5294 from matrix-org/erikj/speed_up_room_stats
Speed up room stats background update
2019-05-31 10:48:51 +01:00
Erik Johnston 5037326d66 Add indices. Remove room_ids accidentally added
We have to do this by re-inserting a background update and recreating
tables, as the tables only get created during a background update and
will later be deleted.

We also make sure that we remove any entries that should have been
removed but weren't due to a race that has been fixed in a previous
commit.
2019-05-31 10:26:56 +01:00
Erik Johnston 6cdfb0207e Add index to temp table 2019-05-30 15:02:31 +01:00
Erik Johnston 640fcbb07f Fixup comments and logging 2019-05-30 10:55:55 +01:00
Erik Johnston d79c9994f4 Add DB bg update to cleanup extremities.
Due to #5269 we may have extremities in our DB that we shouldn't have,
so lets add a cleanup task such to remove those.
2019-05-29 15:11:26 +01:00
Brendan Abolivier a97d4e218a
Merge pull request #5268 from matrix-org/babolivier/account_validity_fix_schema
Fix schema update for account validity
2019-05-28 10:30:07 +01:00
Brendan Abolivier ba17de7fbc
Fix schema update for account validity 2019-05-28 10:11:38 +01:00
Richard van der Hoff b75537beaf Store key validity time in the storage layer
This is a first step to checking that the key is valid at the required moment.

The idea here is that, rather than passing VerifyKey objects in and out of the
storage layer, we instead pass FetchKeyResult objects, which simply wrap the
VerifyKey and add a valid_until_ts field.
2019-05-23 11:52:22 +01:00
Amber Brown 4a30e4acb4
Room Statistics (#4338) 2019-05-21 11:36:50 -05:00
Erik Johnston efe3c7977a Add simple send_relation API and track in DB 2019-05-15 13:36:51 +01:00
Brendan Abolivier 91934025b9
Merge pull request #5047 from matrix-org/babolivier/account_expiration
Send out emails with links to extend an account's validity period
2019-04-17 14:57:39 +01:00
Brendan Abolivier 20f0617e87 Send out emails with links to extend an account's validity period 2019-04-17 14:42:20 +01:00
Erik Johnston e446921def
Merge pull request #5033 from matrix-org/erikj/fix_schema_delta
Fix schema upgrade when dropping tables
2019-04-10 10:22:35 +01:00
Brendan Abolivier bfc8fdf1fc
Merge pull request #5027 from matrix-org/babolivier/account_expiration
Add time-based account expiration
2019-04-09 17:02:41 +01:00
Brendan Abolivier 747aa9f8ca Add account expiration feature 2019-04-09 16:46:04 +01:00
Erik Johnston 50d2a3059d Fix schema upgrade when dropping tables
We need to drop tables in the correct order due to foreign table
constraints (on `application_services`), otherwise the DROP TABLE
command will fail.

Introduced in #4992.
2019-04-09 14:39:18 +01:00
Richard van der Hoff 3352baac4b
Remove unused server_tls_certificates functions (#5028)
These have been unused since #4120, and with the demise of perspectives, it is
unlikely that they will ever be used again.
2019-04-08 21:50:18 +01:00
Neil Johnson 2d951686a7
drop tables listed in #1830 (#4992)
Tables dropped: 
 * application_services, 
 * application_services_regex, 
 * transaction_id_to_pdu, 
 * stats_reporting
 * current_state_resets
 * event_content_hashes
 * event_destinations
 * event_edge_hashes
 * event_signatures
 * feedback
 * room_hosts
 * state_forward_extremities
2019-04-08 15:37:26 +01:00
Neil Johnson e8419554ff
Remove presence lists (#4989)
Remove presence list support as per MSC 1819
2019-04-03 11:11:15 +01:00
Erik Johnston c75e2017f1 Fixup docstrings 2019-04-02 11:20:06 +01:00
Erik Johnston 3715c124b3 Grandfather in existing user threepids
We assume, as we did before, that users bound their threepid to one of
the trusted identity servers. So we simply fill the new table with all
threepids in `user_threepids` joined with the trusted identity servers.
2019-04-01 15:25:19 +01:00
Erik Johnston 1666c0696a Track IS used to bind 3PIDs
This will then be used to know which IS to default to when unbinding the
threepid.
2019-04-01 15:23:01 +01:00
Erik Johnston e9eeca1314 Fix user directory background update (#4887) 2019-03-19 22:13:53 +11:00
Amber Brown 282c97327f
Migrate the user directory initial population to a background task (#4864) 2019-03-19 04:50:24 +11:00
Amber Brown 797b6a63fc fixup 2019-03-13 01:17:51 +11:00
Amber Brown 10480c4348 fixup 2019-03-12 21:47:14 +11:00
Amber Brown 30a8deeb68 Merge remote-tracking branch 'origin/develop' into hawkowl/userdir-search 2019-03-11 21:12:30 +11:00
Amber Brown 4abc988c6a
initial 2019-03-11 21:11:36 +11:00
Erik Johnston fe6c12e6cd Add comment to schema 2019-03-08 16:38:23 +00:00
Amber Brown f6135d06cf
Rewrite userdir to be faster (#4537) 2019-03-07 01:22:53 -08:00
Erik Johnston 483ba85c7a Analyze user_ips before running deduplication
Due to the table locks taken out by the naive upsert, the table
statistics may be out of date. During deduplication it is important that
the correct index is used as otherwise a full table scan may be
incorrectly used, which can end up thrashing the database badly.
2019-02-12 11:55:27 +00:00
Erik Johnston c5a296b10c Add support for persisting event format versions
Currently we only have the one event format version defined, but this
adds the necessary infrastructure to persist and fetch the format
versions alongside the events.

We specify the format version rather than the room version as:

1. We don't necessarily know the room version, existing events may be
   either v1 or v2.
2. We'd need to be careful to prevent/handle correctly if different
   events in the same room reported to be of different versions, which
   sounds annoying.
2019-01-23 11:30:01 +00:00
Amber Brown a35c66a00b
Remove duplicates in the user_ips table and add an index (#4370) 2019-01-12 06:21:50 +11:00
Neil Johnson d2f7c4e6b1
create support user (#4141)
Allow for the creation of a support user.

A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.
2018-12-14 18:20:59 +00:00
Aaron Raimist 704c5298f0
Drop sent_transactions
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2018-12-01 23:07:35 -06:00
David Baker bca3b91c2d Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backup_versions_are_numbers 2018-11-09 18:35:02 +00:00
Richard van der Hoff 350f654e7b Add unique indexes to a couple of tables
The indexes on device_lists_remote_extremeties can be unique, and they
therefore should, to ensure that the db remains consistent.
2018-11-02 10:36:13 +00:00
David Baker 64fa557f80 Try & make it work on postgres 2018-10-30 09:51:04 +00:00
David Baker 563f9b61b1 Make e2e backup versions numeric in the DB
We were doing max(version) which does not do what we wanted
on a column of type TEXT.
2018-10-29 21:01:22 +00:00
Erik Johnston 056f099126 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups 2018-10-19 15:48:59 +01:00
David Baker 497444f1fd Don't reuse backup versions
Since we don't actually delete the keys, just mark the versions
as deleted in the db rather than actually deleting them, then we
won't reuse versions.

Fixes https://github.com/vector-im/riot-web/issues/7448
2018-10-05 15:08:36 +01:00
Erik Johnston 4917ff5523 Add state_group index to event_to_state_groups
This is needed to efficiently check for unreferenced state groups during
purge.
2018-10-04 16:03:08 +01:00
David Baker bc74925c5b WIP e2e key backups
Continues from uhoreg's branch

This just fixed the errcode on /room_keys/version if no backup and
updates the schema delta to be on the latest so it gets run
2018-09-13 17:02:59 +01:00
Hubert Chathi 83caead95a
Merge branch 'develop' into e2e_backups 2018-08-24 11:44:26 -04:00
Matthew Hodgson 15d513f16f fix idiocies and so make tests pass 2018-08-12 19:14:31 -04:00
Matthew Hodgson 0abb205b47 blindly incorporate PR review - needs testing & fixing 2018-08-12 19:14:31 -04:00
Matthew Hodgson 69e51c7ba4 make /room_keys/version work 2018-08-12 19:14:31 -04:00
Matthew Hodgson 8ae64b270f implement /room_keys/version too (untested) 2018-08-12 19:14:31 -04:00
Matthew Hodgson 6b8c07abc2 make it work and fix pep8 2018-08-12 19:13:09 -04:00
Matthew Hodgson 53ace904b2 total WIP skeleton for /room_keys API 2018-08-12 18:23:10 -04:00
Neil Johnson 312ae74746 typos 2018-08-08 13:33:16 +01:00
Neil Johnson 16d78be315 make use of _simple_select_one_onecol, improved comments 2018-08-06 17:51:15 +01:00
Neil Johnson 165e067033 Revert "change monthly_active_users table to be a single column"
This reverts commit ec716a35b2.
2018-08-02 10:59:58 +01:00
Neil Johnson ec716a35b2 change monthly_active_users table to be a single column 2018-08-01 17:54:37 +01:00
Neil Johnson f9f5559971 fix comment 2018-08-01 12:03:42 +01:00
Neil Johnson 6ef983ce5c api into monthly_active_users table 2018-07-31 16:36:24 +01:00
Neil Johnson df2235e7fa coding style 2018-07-31 13:16:20 +01:00
Richard van der Hoff 51d7df1915 Create the column nullable
There's no real point in ever making the column non-nullable, and doing so
breaks the sytests.
2018-07-26 14:54:04 +01:00
Richard van der Hoff 5c1d301fd9 Stop populating events.content
This field is no longer read from, so we should stop populating it. Once we're
happy that this doesn't break everything, and a rollback is unlikely, we can
think about dropping the column.
2018-07-26 14:43:02 +01:00
Richard van der Hoff 1b4d73fa52 comment on event_edges 2018-07-26 12:53:51 +01:00
Amber Brown 49af402019 run isort 2018-07-09 16:09:20 +10:00
Erik Johnston 244484bf3c Revert "Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility""
This reverts commit 1d009013b3.
2018-06-25 13:42:55 +01:00
Richard van der Hoff 1d009013b3 Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility"
This reverts commit ce0d911156, reversing
changes made to b4a5d767a9.
2018-06-22 16:35:10 +01:00
Richard van der Hoff 3ff8a619f5 UserErasureStore
to store which users have been erased
2018-06-12 09:52:22 +01:00
Neil Johnson 4986b084f8 remove unnecessary INSERT 2018-06-01 10:50:40 +01:00
Neil Johnson ab0ef31dc7 create users index on creation_ts 2018-05-29 17:51:08 +01:00
Richard van der Hoff 9ea219c514 Send users a server notice about consent
When a user first syncs, we will send them a server notice asking them to
consent to the privacy policy if they have not already done so.
2018-05-22 11:54:51 +01:00
Neil Johnson dc8930ea9e
Merge pull request #3163 from matrix-org/cohort_analytics
user visit data
2018-05-16 10:09:24 +00:00
Richard van der Hoff 8030a825c8
Merge pull request #3213 from matrix-org/rav/consent_handler
ConsentResource to gather policy consent from users
2018-05-16 07:19:18 +01:00
Richard van der Hoff 47815edcfa ConsentResource to gather policy consent from users
Hopefully there are enough comments and docs in this that it makes sense on its
own.
2018-05-15 15:11:59 +01:00
Neil Johnson 977765bde2 Merge branch 'develop' of https://github.com/matrix-org/synapse into cohort_analytics 2018-05-14 09:31:42 +01:00
David Baker 46df23f581 Add the schema file 2018-05-09 15:07:54 +01:00
Neil Johnson d0857702e8 add inidexes based on usage 2018-05-01 12:12:57 +01:00
Matthew Hodgson 99a54bf2af
Merge pull request #3129 from matrix-org/matthew/fix_group_dups
remove duplicates from groups tables
2018-04-30 11:47:25 +01:00
Matthew Hodgson adaf3ec87f fix missing import 2018-04-28 22:39:15 +01:00
Matthew Hodgson 006e18b6bb pep8 2018-04-28 22:32:24 +01:00
Matthew Hodgson 42c89c8215 make it work with sqlite 2018-04-28 22:27:30 +01:00
Adrian Tschira d82b6ea9e6 Move more xrange to six
plus a bonus next()

Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:57:00 +02:00
Neil Johnson 617bf40924 Generate user daily stats 2018-04-25 17:37:29 +01:00
Matthew Hodgson e3a373f002 remove duplicates from groups tables
and rename inconsistently named indexes.
Based on https://github.com/matrix-org/synapse/pull/3128 - thanks @vurpo\!
2018-04-25 14:58:43 +01:00
Luke Barnard e089100c62
Merge pull request #3045 from matrix-org/dbkr/group_joinable
Add joinability for groups
2018-04-05 15:57:49 +01:00
Neil Johnson 68b0ee4e8d
Merge pull request #3041 from matrix-org/r30_stats
R30 stats
2018-04-05 15:37:37 +01:00
Luke Barnard 917380e89d NON NULL -> NOT NULL 2018-04-05 14:32:12 +01:00
Luke Barnard 700e5e7198 Use DEFAULT join_policy of "invite" in db 2018-04-05 14:01:17 +01:00
Luke Barnard eb8d8d6f57 Use join_policy API instead of joinable
The API is now under
 /groups/$group_id/setting/m.join_policy

and expects a JSON blob of the shape

```json
{
  "m.join_policy": {
    "type": "invite"
  }
}
```

where "invite" could alternatively be "open".
2018-04-03 16:16:40 +01:00
Richard van der Hoff fcfe7f6ad3 Use simplejson throughout
Let's use simplejson rather than json, for consistency.
2018-03-29 22:45:52 +01:00
David Baker 929b34963d OK, smallint it is then 2018-03-28 14:53:55 +01:00
David Baker a838444a70 Grr. Copy the definition from is_admin 2018-03-28 14:50:30 +01:00
David Baker a164270833 Make column definition that works on both dbs 2018-03-28 14:23:00 +01:00
David Baker 352e1ff9ed Add schema delta file 2018-03-28 14:07:57 +01:00
Neil Johnson 788e69098c Add user_ips last seen index 2018-03-28 12:03:13 +01:00
Erik Johnston fa72803490 Merge branch 'master' of github.com:matrix-org/synapse into develop 2018-03-19 11:41:01 +00:00
Erik Johnston 926ba76e23 Replace ujson with simplejson 2018-03-15 23:43:31 +00:00
Erik Johnston 012e8e142a Comments 2018-02-16 11:35:01 +00:00
Erik Johnston acac21248c Store push actions in staging area 2018-02-15 15:47:04 +00:00
Richard van der Hoff ddb6a79b68 Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin 2018-02-13 16:45:36 +00:00
Richard van der Hoff 0b27ae8dc3 move search reindex to schema 47
We're up to schema v47 on develop now, so this will have to go in there to have
an effect.

This might cause an error if somebody has already run it in the v46 guise, and
runs it again in the v47 guise, because it will cause a duplicate entry in the
bbackground_updates table. On the other hand, the entry is removed once it is
complete, and it is unlikely that anyone other than matrix.org has run it on
v46. The update itself is harmless to re-run because it deliberately copes with
the index already existing.
2018-02-13 16:44:46 +00:00
Erik Johnston 3d33eef6fc
Store state groups separately from events (#2784)
* Split state group persist into seperate storage func

* Add per database engine code for state group id gen

* Move store_state_group to StateReadStore

This allows other workers to use it, and so resolve state.

* Hook up store_state_group

* Fix tests

* Rename _store_mult_state_groups_txn

* Rename StateGroupReadStore

* Remove redundant _have_persisted_state_group_txn

* Update comments

* Comment compute_event_context

* Set start val for state_group_id_seq

... otherwise we try to recreate old state groups

* Update comments

* Don't store state for outliers

* Update comment

* Update docstring as state groups are ints
2018-02-06 14:31:24 +00:00
Erik Johnston 2fb3a28c98
Remove lost comment 2018-01-17 14:59:44 +00:00
Erik Johnston 05f98a2224 Keep track of last access time for local media 2018-01-17 10:24:43 +00:00
Matthew Hodgson a66f489678 fix GIST->GIN switch 2018-01-09 16:55:51 +00:00
Matthew Hodgson f397153dfc Merge branch 'develop' into matthew/search-all-local-users 2017-11-30 01:51:38 +00:00
Matthew Hodgson f61e107f63 remove null constraint on user_dir.room_id 2017-11-30 01:43:50 +00:00
Richard van der Hoff 7098b65cb8 Fix error on sqlite 3.7
Create the url_cache index on local_media_repository as a background update, so
that we can detect whether we are on sqlite or not and create a partial or
complete index accordingly.

To avoid running the cleanup job before we have built the index, add a bailout
which will defer the cleanup if the bg updates are still running.

Fixes https://github.com/matrix-org/synapse/issues/2572.
2017-11-21 11:14:17 +00:00
Richard van der Hoff 06e5bcfc83 Avoid locking for upsert on pushers tables
* replace the upsert into deleted_pushers with an insert
* no need to lock for upsert on pusher_throttle
2017-11-16 17:52:23 +00:00
Matthew Hodgson 631fa4a1b7 create new indexes before dropping old ones to keep safetynet in place 2017-11-07 10:41:55 +00:00
Matthew Hodgson 4ad883398f s/users_in_pubic_room/users_in_public_rooms/g 2017-11-04 19:39:40 +00:00
Matthew Hodgson d802e8ca6a s/users_in_pubic_room/users_in_public_rooms/g 2017-11-04 19:38:13 +00:00
Matthew Hodgson a100700630 fix copyright.... 2017-11-04 19:35:49 +00:00
Matthew Hodgson b6b075fd49 s/popualte/populate/ 2017-11-04 19:35:33 +00:00
Richard van der Hoff 356bcafc44 Remove the last vestiges of refresh_tokens 2017-10-31 20:35:58 +00:00
Richard van der Hoff 1650eb5847 DB schema interface for password auth providers
Provide an interface by which password auth providers can register db schema
files to be run at startup
2017-10-31 14:01:53 +00:00
Luke Barnard 007cd48af6 Recreate groups table instead of adding column
Adding a column with non-constant default not possible in sqlite3
2017-10-26 17:55:22 +01:00
Luke Barnard 713e60b9b6 Awful hack to get default true 2017-10-26 17:38:14 +01:00
Luke Barnard 9b2feef9eb Add is_public to groups table to allow for private groups
Prevent group API access to non-members for private groups

Also make all the group code paths consistent with `requester_user_id` always being the User ID of the requesting user.
2017-10-26 16:51:32 +01:00
Richard van der Hoff eaaabc6c4f replace 'except:' with 'except Exception:'
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Erik Johnston c2c47550f9 Fix schema delta versions 2017-10-11 13:23:15 +01:00
David Baker 27955056e0 Merge branch 'develop' into erikj/groups_merged 2017-10-02 16:20:41 +01:00
Erik Johnston 4dc07e93a8 Add old indices 2017-09-28 14:10:33 +01:00
Erik Johnston ae79764fe5 Change expires column to expires_ts 2017-09-28 12:37:53 +01:00
Erik Johnston 77f1d24de3 More brackets 2017-09-28 12:23:15 +01:00
Erik Johnston 9ccb4226ba Delete expired url cache data 2017-09-28 12:18:06 +01:00
Erik Johnston a8e2a3df32 Add unique index to group_rooms table 2017-09-26 15:39:21 +01:00
Erik Johnston 27ebc5c8f2 Add remote profile cache 2017-08-25 11:25:47 +01:00