Commit Graph

764 Commits

Author SHA1 Message Date
Patrick Cloke cabd577460
Drop unused tables & unneeded access token ID for events. (#16268)
Drop the event_txn_id table and the tables related to MSC2716,
which is no longer supported in Synapse.
2023-10-06 08:29:33 -04:00
Erik Johnston 329597022e
Some minor performance fixes for task schedular (#16313) 2023-09-14 16:20:47 +01:00
Patrick Cloke 16ef6f1e3c
Stop purging tables which are slated for removal. (#16273) 2023-09-12 07:12:31 -04:00
Mathieu Velten 4f1840a88a
Delete device messages asynchronously and in staged batches (#16240) 2023-09-06 09:30:53 +02:00
Patrick Cloke ebd8374fb5
Stop writing to the event_txn_id table (#16175) 2023-08-30 11:10:56 +01:00
Erik Johnston 18279631e9
Fix rare deadlock when using read/write locks (#16169) 2023-08-23 16:24:30 +01:00
Erik Johnston 4adaba9acf
Fix rare deadlock when using read/write locks (#16133) 2023-08-23 13:45:25 +01:00
Erik Johnston 7cd79ce051
Reduce DB contention on worker locks (#16160) 2023-08-23 13:45:19 +01:00
Erik Johnston 3b3fed7229
Increase perf of read/write locks (#16149)
We do this by marking the tables as `UNLOGGED` in PostgreSQL.
2023-08-23 09:23:22 +01:00
Mathieu Velten 358896e1b8
Implements a task scheduler for resumable potentially long running tasks (#15891) 2023-08-21 14:17:13 +02:00
Mathieu Velten dac97642e4
Implements admin API to lock an user (MSC3939) (#15870) 2023-08-10 09:10:55 +00:00
Patrick Cloke d98a43d922
Stabilize support for MSC3970: updated transaction semantics (scope to `device_id`) (#15629)
For now this maintains compatible with old Synapses by falling back
to using transaction semantics on a per-access token. A future version
of Synapse will drop support for this.
2023-08-04 07:47:18 -04:00
Mathieu Velten 8ebfd577e2
Bump DB version to 79 since synapse v1.88 was already there (#15998) 2023-07-26 14:51:44 +02:00
Erik Johnston 19796e20aa
Fix bad merge of #15933 (#15958)
This was because we reverted the bump of the schema version, so we were not applying the new deltas.
2023-07-19 12:17:08 +00:00
Erik Johnston 40a3583ba1
Fix race in triggers for read/write locks. (#15933) 2023-07-19 12:06:38 +01:00
Shay cb6e2c6cc7
Fix background schema updates failing over a large upgrade gap (#15887) 2023-07-18 16:59:27 -07:00
Shay e625c3dca0
Revert "Stop writing to column `user_id` of tables `profiles` and `user_filters`. (#15953)
* Revert "Stop writing to column `user_id` of tables `profiles` and `user_filters` (#15787)"

This reverts commit f25b0f8808.

* newsfragement
2023-07-18 11:44:09 +01:00
Erik Johnston e55a9b3e41
Fix downgrading to previous version of Synapse (#15907)
We do this by marking the constraint as deferrable.
2023-07-10 16:24:42 +01:00
Shay f25b0f8808
Stop writing to column `user_id` of tables `profiles` and `user_filters` (#15787) 2023-07-07 09:23:27 -07:00
Erik Johnston 39d131b016
Add basic read/write lock (#15782) 2023-07-05 17:25:00 +01:00
Erik Johnston 95a96b21eb
Add foreign key constraint to `event_forward_extremities`. (#15751) 2023-07-05 09:43:19 +00:00
Shay 78cfa55dad
Fix sqlite `user_filters` upgrade (#15817) 2023-06-27 09:41:42 +01:00
Erik Johnston 8934c11935 Merge branch 'master' into develop 2023-06-07 14:45:19 +01:00
Erik Johnston f7c6553ebc
Fix schema delta error in 1.85 (#15739)
Some users seem to have multiple rows per user / room with a null thread
ID, which we need to handle.
2023-06-07 13:02:42 +01:00
Erik Johnston a701c089fa
Fix schema delta error in 1.85 (#15738)
There appears to be a race where you can end up with entries in
`event_push_summary` with both a `NULL` and `main` thread ID.

Fixes #15736

Introduced in #15597
2023-06-07 10:50:32 +01:00
Shay d0c4257f14
`N + 3`: Read from column `full_user_id` rather than `user_id` of tables `profiles` and `user_filters` (#15649) 2023-06-02 17:24:13 -07:00
reivilibre 11e15d79b8
Fix a performance issue introduced in Synapse v1.83.0 which meant that purging rooms was very slow and database-intensive. (#15693)
* Add indices required to efficiently validate new foreign key constraints on stream_ordering

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-05-31 14:59:56 +01:00
Patrick Cloke 2ad91ec628
Set thread_id column to non-null for event_push_{actions,actions_staging,summary} (#15597)
Updates the database schema to require a thread_id (by adding a
constraint that the column is non-null) for event_push_actions,
event_push_actions_staging, and event_push_actions_summary.

For PostgreSQL we add the constraint as NOT VALID, then
VALIDATE the constraint a background job to avoid locking
the table during an upgrade.

Each table is updated as a separate schema delta to avoid
deadlocks between them.

For SQLite we simply rebuild the table & copy the data.
2023-05-26 13:16:08 -04:00
Patrick Cloke 375b0a8a11
Update code to refer to "workers". (#15606)
A bunch of comments and variables are out of date and use
obsolete terms.
2023-05-16 15:56:38 -04:00
Shay 9f6ff6a0eb
Add not null constraint to column `full_user_id` of tables `profiles` and `user_filters` (#15537) 2023-05-16 10:57:39 -07:00
Erik Johnston 808105bd31
Revert "Set thread_id column to non-null for event_push_{actions,actions_staging,summary} (#15437)" (#15580)
This reverts commit a7b3e9ce65.
2023-05-12 11:38:16 +01:00
Patrick Cloke a7b3e9ce65
Set thread_id column to non-null for event_push_{actions,actions_staging,summary} (#15437)
Updates the database schema to require a thread_id (by adding a
constraint that the column is non-null) for event_push_actions,
event_push_actions_staging, and event_push_actions_summary.

For PostgreSQL we add the constraint as NOT VALID, then
VALIDATE the constraint a background job to avoid locking
the table during an upgrade.

For SQLite we simply rebuild the table & copy the data.
2023-05-03 07:49:03 -04:00
Sean Quah 04e79e6a18
Add config option to forget rooms automatically when users leave them (#15224)
This is largely based off the stats and user directory updater code.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-05-03 12:27:33 +01:00
Shay 89f6fb0d5a
Add an admin API endpoint to support per-user feature flags (#15344) 2023-04-28 11:33:45 -07:00
Patrick Cloke 6efa674004
Add type hints to schema deltas (#15497)
Cleans-up the schema delta files:

* Removes no-op functions.
* Adds missing type hints to function parameters.
* Fixes any issues with type hints.

This also renames one (very old) schema delta to avoid a conflict
that mypy complains about.
2023-04-27 12:44:53 +00:00
Shay 301b4156d5
Add column `full_user_id` to tables `profiles` and `user_filters`. (#15458) 2023-04-26 16:03:26 -07:00
Nick Mills-Barrett c55293c230
Re re introduce membership tables event stream ordering (#15356) 2023-04-25 09:44:29 +01:00
Quentin Gliech 8b3a502996
Experimental support for MSC3970: per-device transaction IDs (#15318) 2023-04-25 09:37:09 +01:00
Shay 6b23d74ad1
Delete server-side backup keys when deactivating an account. (#15181) 2023-04-04 20:16:08 +00:00
Erik Johnston 79d2e2e79c
Speed up membership queries for users with forgotten rooms (#15385) 2023-04-04 14:11:34 +01:00
Olivier Wilkinson (reivilibre) 72d2ceaa9a Revert "Set thread_id column to non-null for event_push_{actions,actions_staging,summary} (#15350)"
This reverts commit 2a234b788e.

See #15359 for context.
2023-03-31 12:10:10 +01:00
Patrick Cloke 2a234b788e
Set thread_id column to non-null for event_push_{actions,actions_staging,summary} (#15350)
Clean-up from adding the thread_id column, which was initially
null but backfilled with values. It is desirable to require it to now
be non-null.

In addition to altering this column to be non-null, we clean up
obsolete background jobs, indexes, and just-in-time updating
code.
2023-03-30 15:11:31 -04:00
Erik Johnston 5350b5d04d
Revert "Reintroduce membership tables event stream ordering (#15128)" (#15347)
This reverts commit e6af49fbea.
2023-03-29 13:24:28 +01:00
Quentin Gliech 5b70f240cf
Make cleaning up pushers depend on the device_id instead of the token_id (#15280)
This makes it so that we rely on the `device_id` to delete pushers on logout,
instead of relying on the `access_token_id`. This ensures we're not removing
pushers on token refresh, and prepares for a world without access token IDs
(also known as the OIDC).

This actually runs the `set_device_id_for_pushers` background update, which
was forgotten in #13831.

Note that for backwards compatibility it still deletes pushers based on the
`access_token` until the background update finishes.
2023-03-24 11:09:39 -04:00
Nick Mills-Barrett e6af49fbea
Reintroduce membership tables event stream ordering (#15128)
* Add `event_stream_ordering` column to membership state tables

Specifically this adds the column to `current_state_events`,
`local_current_membership` and `room_memberships`. Each of these tables
is regularly joined with the `events` table to get the stream ordering
and denormalising this into each table will yield significant query
performance improvements once used.

* Make denormalised `event_stream_ordering` columns foreign keys
* Add comment in schema file explaining new denormalised columns
* Add triggers to enforce consistency of `event_stream_ordering` columns
* Re-order purge room tables to account for foreign keys
* Bump schema version to 75

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2023-03-24 11:44:01 +00:00
reivilibre f54f877f27
Preparatory work to fix the user directory assuming that any remote membership state events represent a profile change. [rei:userdirpriv] (#14755)
* Remove special-case method for new memberships only, use more generic method

* Only collect profiles from state events in public rooms

* Add a table to track stale remote user profiles

* Add store methods to set and delete rows in this new table

* Mark remote profiles as stale when a member state event comes in to a private room

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* Simplify by removing Optionality of `event_id`

* Replace names and avatars with None if they're set to dodgy things

I think this makes more sense anyway.

* Move schema delta to 74 (I missed the boat?)

* Turns out these can be None after all

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-03-16 09:55:19 +00:00
reivilibre 63d87c08c8
Add schema comments about the `destinations` and `destination_rooms` tables. (#15247) 2023-03-15 09:25:58 +00:00
David Robertson cd2484dc2e
Bump schema version (#15036)
* Bump schema version

This should have been included in
f10caa73ee (and #14979).

* Changelog
2023-02-09 15:28:26 +00:00
David Robertson 9cd7610f86
Revert "Add `event_stream_ordering` column to membership state tables (#14979)"
This reverts commit 5fdc12f482.
2023-02-07 15:26:55 +00:00
Nick Mills-Barrett 5fdc12f482
Add `event_stream_ordering` column to membership state tables (#14979)
This adds an `event_stream_ordering` column to `current_state_events`,
`local_current_membership` and `room_memberships`. Each of these tables
is regularly joined with the `events` table to get the stream ordering
and denormalising this into each table will yield significant query
performance improvements once used. Includes a background job to
populate these values from the `events` table.

Same idea as https://github.com/matrix-org/synapse/pull/13703.

Signed off by Nick @ Beeper (@fizzadar).
2023-02-07 00:10:54 +00:00