Commit Graph

4537 Commits

Author SHA1 Message Date
Erik Johnston 578a5e24a9
Use upserts for updating `event_push_summary` (#13153) 2022-07-05 13:51:04 +01:00
Andrew Morgan 6180e1bc4b Synapse 1.62.0rc3 (2022-07-04)
==============================
 
 Bugfixes
 --------
 
 - Update the version of the [ldap3 plugin](https://github.com/matrix-org/matrix-synapse-ldap3/) included in the `matrixdotorg/synapse` DockerHub images and the Debian packages hosted on `packages.matrix.org` to 0.2.1. This fixes [a bug](https://github.com/matrix-org/matrix-synapse-ldap3/pull/163) with usernames containing uppercase characters. ([\#13156](https://github.com/matrix-org/synapse/issues/13156))
 - Fix a bug introduced in Synapse 1.62.0rc1 affecting unread counts for users on small servers. ([\#13168](https://github.com/matrix-org/synapse/issues/13168))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmLDDVgACgkQiISIDS7+
 X/Q+KQ//WuWB9hfAW8XEyYHWox95zaITsAzY/TTG1IXygAMjgEk2+9utdRaX3wbk
 YDaZCeEw+vbK/3w/lt1RzI30K3uVCZVcW2DTQr1Qi4B+UWLOlCsVfOT9LcMvNoJe
 ww/cOK6RpPgTlqk5ij0MtdjfWkAJeToi7ESMooORxhWFm3Zd8e5BpbNv89WUBZhk
 zCqCjIdjSF+Mwk8NwmU1iJi5JQY/+Xl51uk2+wGIAe4vtgPTz7PJmoPF1E6nGGVF
 9OYdlWU4H7u6js8n05QL2jKtX34uszCo2hwoW2aFPPmF0B2CFEV6WFBiDOppLZ1g
 ZMJv1s/34RXoBu8pAuJnq2BZkWxu99LRmPV+f/R+S0jDT1MH9tdSdhfcGu7iH/Y9
 uguGX3OOlxnkUb5o825Xt3mvBcVaTGY+sspFtB12RtXmWRdll/Hq6w11ZN5f6qDy
 Nr/DuoPjMAH7kzelFn/GpP6K8zX8iYjf0lLCyrbYV7OYAI6/I+Vao+sT2ctHD1T8
 s4aTTx1bEl23mo/RiqH2fRHaPhBjZKW0uv6iRNqDE2ThYPAXinVtt7MiUU0QGco5
 vMca/RZBkEj0Lov0AleBx4XRXlBTyq5BX2V1frYLenKp42bDzN9sgsPAOPeKieHW
 qjr+Ti9i47wGADXs2GI/mke/C8jlONEKJm/v8mwXItn8Za7wBJc=
 =SpI6
 -----END PGP SIGNATURE-----

Merge tag 'v1.62.0rc3' into develop

Synapse 1.62.0rc3 (2022-07-04)
==============================

Bugfixes
--------

- Update the version of the [ldap3 plugin](https://github.com/matrix-org/matrix-synapse-ldap3/) included in the `matrixdotorg/synapse` DockerHub images and the Debian packages hosted on `packages.matrix.org` to 0.2.1. This fixes [a bug](https://github.com/matrix-org/matrix-synapse-ldap3/pull/163) with usernames containing uppercase characters. ([\#13156](https://github.com/matrix-org/synapse/issues/13156))
- Fix a bug introduced in Synapse 1.62.0rc1 affecting unread counts for users on small servers. ([\#13168](https://github.com/matrix-org/synapse/issues/13168))
2022-07-04 17:35:06 +01:00
Erik Johnston 723ce73d02
Fix stuck notification counts on small servers (#13168) 2022-07-04 16:02:21 +01:00
Patrick Cloke b0366853ca Merge remote-tracking branch 'origin/release-v1.62' into develop 2022-06-30 13:27:24 -04:00
Erik Johnston dbce28b2f1
Fix unread counts on large servers (#13140) 2022-06-30 15:08:40 +01:00
Erik Johnston a3a05c812d
Add index to help delete old push actions (#13141) 2022-06-30 14:05:49 +00:00
Brendan Abolivier 4d3b8fb23f
Don't actually one-line the SQL statements we send to the DB (#13129) 2022-06-30 10:43:24 +02:00
Šimon Brandner 13e359aec8
Implement MSC3827: Filtering of `/publicRooms` by room type (#13031)
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
2022-06-29 17:12:45 +00:00
Erik Johnston 92a0c18ef0
Improve performance of getting unread counts in rooms (#13119) 2022-06-29 10:32:38 +00:00
Erik Johnston 7469824d58
Fix serialization errors when rotating notifications (#13118) 2022-06-28 13:13:44 +01:00
reivilibre b26cbe3d45
Fix type error that made its way onto develop (#13098)
* Fix type error introduced accidentally by #13045

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-06-17 13:05:27 +01:00
Erik Johnston 5ef05c70c3
Rotate notifications more frequently (#13096) 2022-06-17 10:58:00 +00:00
Erik Johnston 5099b5ecc7
Use new `device_list_changes_in_room` table when getting device list changes (#13045) 2022-06-17 11:42:03 +01:00
Erik Johnston 8ceed5e6b5
Add desc to `get_earliest_token_for_stats` (#13085) 2022-06-16 17:50:46 +00:00
David Robertson 97e9fbe1b2
Type annotations in `synapse.databases.main.devices` (#13025)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-06-15 15:20:04 +00:00
Erik Johnston 0d1d3e0708
Speed up `get_unread_event_push_actions_by_room` (#13005)
Fixes #11887 hopefully.

The core change here is that `event_push_summary` now holds a summary of counts up until a much more recent point, meaning that the range of rows we need to count in `event_push_actions` is much smaller.

This needs two major changes:
1. When we get a receipt we need to recalculate `event_push_summary` rather than just delete it
2. The logic for deleting `event_push_actions` is now divorced from calculating `event_push_summary`.

In future it would be good to calculate `event_push_summary` while we persist a new event (it should just be a case of adding one to the relevant rows in `event_push_summary`), as that will further simplify the get counts logic and remove the need for us to periodically update `event_push_summary` in a background job.
2022-06-15 15:17:14 +00:00
Richard van der Hoff 75fb10ee45
Clean up schema for `event_edges` (#12893)
* Remove redundant references to `event_edges.room_id`

We don't need to care about the room_id here, because we are already checking
the event id.

* Clean up the event_edges table

We make a number of changes to `event_edges`:

 * We give the `room_id` and `is_state` columns defaults (null and false
   respectively) so that we can stop populating them.
 * We drop any rows that have `is_state` set true - they should no longer
   exist.
 * We drop any rows that do not exist in `events` - these should not exist
   either.
 * We drop the old unique constraint on all the colums, which wasn't much use.
 * We create a new unique index on `(event_id, prev_event_id)`.
 * We add a foreign key constraint to `events`.

These happen rather differently depending on whether we are on Postgres or
SQLite. For SQLite, we just rebuild the whole table, copying only the rows we
want to keep. For Postgres, we try to do things in the background as much as
possible.

* Stop populating `event_edges.room_id` and `is_state`

We can just rely on the defaults.
2022-06-15 12:29:42 +01:00
Patrick Cloke 5f4ecf759d
Rename delta to apply in the proper schema version. (#13050) 2022-06-14 14:34:04 +00:00
Patrick Cloke 53b77b203a
Replace noop background updates with DELETE. (#12954)
Removes the `register_noop_background_update` and deletes the background
updates directly in a delta file.
2022-06-13 14:06:27 -04:00
Richard van der Hoff 7c6b2204d1
Faster joins: add issue links to the TODOs (#13004)
... to help us keep track of these things
2022-06-09 10:13:03 +00:00
Nick Mills-Barrett 04ca3a52f6
Use READ COMMITTED isolation level when inserting read receipts (#12957) 2022-06-09 09:44:16 +01:00
David Robertson 586bfc6dc0
Use dummy fallback engines if imports fail (#12979) 2022-06-07 17:33:55 +01:00
Patrick Cloke d2fd7f7b5c
Fix a stale comment in get_room_version_id_txn. (#12969) 2022-06-07 07:44:31 -04:00
Patrick Cloke 9dc3293e0b
Consolidate the logic of delete_device/delete_devices. (#12970)
By always using delete_devices and sometimes passing a list
with a single device ID.

Previously these methods had gotten out of sync with each
other and it seems there's little benefit to the single-device
variant.
2022-06-07 07:43:35 -04:00
Andrew Morgan c51f5b9592
Prevent breaking old sqlite's when media retention is enabled (#12977) 2022-06-07 11:28:08 +00:00
Andrew Morgan a47636c570
Prevent local quarantined media from being claimed by media retention (#12972) 2022-06-07 10:53:47 +00:00
Patrick Cloke f7baffd8ec
Remove remaining pieces of groups code. (#12966)
* Remove an unused stream ID generator.
* Remove the now unused remote profile cache.
2022-06-06 13:20:05 -04:00
Erik Johnston 44de53bb79
Reduce state pulled from DB due to sending typing and receipts over federation (#12964)
Reducing the amount of state we pull from the DB is useful as fetching state is expensive in terms of DB, CPU and memory.
2022-06-06 16:46:11 +01:00
Erik Johnston e3163e2e11
Reduce the amount of state we pull from the DB (#12811) 2022-06-06 09:24:12 +01:00
Patrick Cloke 6b46c3eb3d
Remove groups code from synapse_port_db. (#12899) 2022-06-03 12:13:35 -04:00
Erik Johnston 888a29f412
Wait for lazy join to complete when getting current state (#12872) 2022-06-01 16:02:53 +01:00
Patrick Cloke 7bc08f3201
Remove remaining bits of groups code. (#12936)
* Update worker docs to remove group endpoints.
* Removes an unused parameter to `ApplicationService`.
* Break dependency between media repo and groups.
* Avoid copying `m.room.related_groups` state events during room upgrades.
2022-06-01 09:41:25 -04:00
Richard van der Hoff 79dadf7216
Fix 404 on `/sync` when the last event is a redaction of an unknown/purged event (#12905)
Currently, we try to pull the event corresponding to a sync token from the database. However, when
we fetch redaction events, we check the target of that redaction (because we aren't allowed to send
redactions to clients without validating them). So, if the sync token points to a redaction of an event
that we don't have, we have a problem.

It turns out we don't really need that event, and can just work with its ID and metadata, which
sidesteps the whole problem.
2022-06-01 11:29:51 +00:00
Patrick Cloke 2e8763ec96
Remove most groups datastore code. (#12895)
The remaining piece is a background update that is needed
for backwards compatibility.
2022-05-31 20:28:17 -04:00
Sean Quah 641908f72f
Faster room joins: Resume state re-syncing after a Synapse restart (#12813)
Signed-off-by: Sean Quah <seanq@matrix.org>
2022-05-31 15:15:08 +00:00
Erik Johnston c8684e6792
Reduce DB load of /sync when using presence (#12885)
While the query was fast, we were calling it *a lot*.
2022-05-31 13:01:05 +00:00
Richard van der Hoff 5e17922ef7
Stop reading from `event_edges.room_id`. (#12914)
event_edges.room_id is implied by the event id, so there is no need to join on the room id.
2022-05-31 13:51:49 +01:00
Erik Johnston 1e453053cb
Rename storage classes (#12913) 2022-05-31 12:17:50 +00:00
Mathieu Velten 7f92ac4c1c
Add a migration step to cleanup potential leftovers of bug 11833 (#12784)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-05-30 16:51:37 +02:00
DeepBlueV7.X b10211871f
Fix invite notifications for users without pushers (#12840)
Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
2022-05-30 13:14:43 +02:00
Richard van der Hoff 2480461879
Fix `get_metadata_for_events` (#12904)
This method was introduced in #12852. It is using the `state_key` column from
the `events` table, which is not (yet) reliable (see #11496).
2022-05-30 10:51:09 +01:00
Jonathan de Jong 6be4953b99
Mutual rooms: Remove dependency on user directory (#12836) 2022-05-30 10:05:31 +01:00
Sumner Evans bda4600399
LockStore: fix acquiring a lock via `LockStore.try_acquire_lock` (#12832)
Signed-off-by: Sumner Evans <sumner@beeper.com>
2022-05-30 09:41:13 +01:00
Brendan Abolivier 28989cb301
Add a background job to automatically delete stale devices (#12855)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-05-27 17:47:32 +02:00
Patrick Cloke 724e11d620
Clean-up some receipts code (#12888)
* Properly marks private methods as private.
* Adds missing docstrings.
* Rework inline methods.
2022-05-27 07:44:10 -04:00
Patrick Cloke c52abc1cfd
Additional constants for EDU types. (#12884)
Instead of hard-coding strings in many places.
2022-05-27 07:14:36 -04:00
Matt C a7da00d4f7
Add storage and module API methods to get monthly active users and their appservices (#12838) 2022-05-27 10:25:57 +00:00
Richard van der Hoff f1605b7447
Fix room deletion (#12889)
* Fix room deletion

ae7858f broke room deletion by attempting to delete the entry from `rooms`
before the tables that reference it.

* faster_joins: remove database rows on purge
2022-05-27 10:31:08 +01:00
Richard van der Hoff bc1beebc27
Refactor have_seen_events to reduce OOMs (#12886)
My server is currently OOMing in the middle of have_seen_events, so let's try
to fix that.
2022-05-27 10:27:33 +01:00
reivilibre e768644368
Fix ambiguous column name that would prevent use of MSC2716 History Import when using Postgres as a database. (#12843) 2022-05-26 11:19:01 +00:00