Commit Graph

24373 Commits

Author SHA1 Message Date
Eric Eastwood a9c0e27eb7
Sliding Sync: No need to sort if the range is large enough to cover all of the rooms (#17731)
No need to sort if the range is large enough to cover all of the rooms
in the list. Previously, we would only do this optimization if the range
was exactly large enough.

Follow-up to https://github.com/element-hq/synapse/pull/17672
2024-09-19 09:33:34 +01:00
Eric Eastwood faf5b40520
Sliding Sync: Fix `_bulk_get_max_event_pos(...)` being inefficient (#17728)
Fix `_bulk_get_max_event_pos(...)` being inefficient. It kept adding all
of the `batch_results` to the `results` over and over every time we
checked a single room in the batch.

I think we still ended up with the right answer before because we
accumulate `recheck_rooms` and actually recheck them to overwrite the
bad data we wrote to the `results` before.

Introduced in
https://github.com/element-hq/synapse/pull/17606/files#diff-cbd54e4b5a2a1646299d659a2d5884d6cb14e608efd2e1658e72b465bb66e31bR1481
2024-09-19 09:32:16 +01:00
Eric Eastwood af998e6c66
Sliding sync: Ignore invites from ignored users (#17729)
`m.ignored_user_list` in account data
2024-09-18 18:09:23 -05:00
Eric Eastwood 61b7c31772
Sliding Sync: Shortcut for checking if certain background updates have completed (#17724)
Shortcut for checking if certain background updates have completed

Pulling this change out from one of @erikjohnston's branches
(https://github.com/element-hq/synapse/compare/develop...erikj/ss_perf)

---------

Co-authored-by: Erik Johnston <erikj@element.io>
2024-09-18 13:12:14 -05:00
Kegan Dougal 3c8a116e1a
Sliding Sync: bugfix: ensure we can sync with SSS even with missing rooms (#17727)
Fixes https://github.com/element-hq/element-x-ios/issues/3300

Some rooms are missing from `sliding_sync_joined_rooms`. When this
happens, the first call will succeed, but any subsequent calls for this
room ID will cause the cache to return `None` for the room ID, rather
than not having the key at all. This then causes the `<=` check to
throw.

Root cause: https://github.com/element-hq/synapse/issues/17726

### Pull Request Checklist

<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->

* [x] Pull request is based on the develop branch
* [ ] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
  - Use markdown where necessary, mostly for `code blocks`.
  - End with either a period (.) or an exclamation mark (!).
  - Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [ ] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
2024-09-18 16:25:50 +00:00
Shay 51dd4df0a3
Add an Admin API endpoint to redact all a user's events (#17506) 2024-09-18 10:08:01 +00:00
Eric Eastwood 8881ad6d4b
Sliding Sync: Short-circuit `have_finished_sliding_sync_background_jobs` (#17723)
We only need to check it if returned bump stamp is `None`, which is rare.

Pulling this change out from one of @erikjohnston's branches
(https://github.com/element-hq/synapse/compare/develop...erikj/ss_perf)
2024-09-17 17:36:59 -05:00
Olivier 'reivilibre d40bc279ed Merge branch 'master' into develop 2024-09-17 15:47:32 +01:00
Olivier 'reivilibre d10872ee75 1.115.0 2024-09-17 14:32:29 +01:00
Eric Eastwood 03937a1cae
Sliding Sync: Return room tags in account data extension (#17707)
The account data extension was also updated to avoid copies when we pull
the data out of the cache.

Fix https://github.com/element-hq/synapse/issues/17694
2024-09-16 13:47:35 -05:00
dependabot[bot] 285de43e48
Bump anyhow from 1.0.87 to 1.0.89 (#17716)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-09-16 18:52:48 +01:00
dependabot[bot] 4900438712
Bump pyasn1 from 0.6.0 to 0.6.1 (#17714)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-09-16 18:52:10 +01:00
dependabot[bot] cf982d2e32
Bump ruff from 0.6.4 to 0.6.5 (#17715)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-09-16 18:51:33 +01:00
dependabot[bot] 7589565edd
Bump types-requests from 2.32.0.20240712 to 2.32.0.20240914 (#17713)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-09-16 18:32:39 +01:00
dependabot[bot] 7ed23e072e
Bump sentry-sdk from 2.13.0 to 2.14.0 (#17712) 2024-09-16 18:32:01 +01:00
David Baker 4ac783549c
Sliding Sync: Support filtering by 'tags' / 'not_tags' in SSS (#17662)
This appears to be enough to make Element Web work (or at least move it
on to the next hurdle)

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-09-12 20:18:19 -05:00
Erik Johnston 1cb84aaab5
Sliding Sync: Increase concurrency of sliding sync a bit (#17696)
For initial requests a typical page size is 20 rooms, so we may as well
do the batching as 20.

This should speed up bigger syncs a little bit.
2024-09-12 16:36:16 -05:00
Eric Eastwood 9b83fb7c16
Sliding Sync: Move filters tests to rest layer (#17703)
Move filters tests to rest layer in order to test the new (with sliding
sync tables) and fallback paths that Sliding Sync can use.

Also found a bug in the new path because it's not being tested which is
also fixed in this PR. We now take into account `has_known_state` when
filtering.

Spawning from
https://github.com/element-hq/synapse/pull/17662#discussion_r1755574791.
This should have been done when we started using the new sliding sync
tables in https://github.com/element-hq/synapse/pull/17630
2024-09-12 15:27:03 -05:00
Andrew Morgan c5b4be6d07 Merge branch 'release-v1.115' into develop 2024-09-12 13:05:43 +01:00
Andrew Morgan 4c66a7cbed 1.115.0rc2 2024-09-12 11:10:31 +01:00
Éloi Rivard ebad618bf0
import pydantic objects from the `_pydantic_compat` module (#17667)
This PR changes `from pydantic import BaseModel` to `from
synapse._pydantic_compat import BaseModel` (as well as `constr`,
`conbytes`, `conint`, `confloat`).

It allows `check_pydantic_models.py` to mock those pydantic objects only
in the synapse module, and not interfere with pydantic objects in
external dependencies.

This should solve the CI problems for #17144, which breaks because
`check_pydantic_models.py` patches pydantic models from
[scim2-models](https://scim2-models.readthedocs.io/).

/cc @DMRobertson @gotmax23
fixes #17659 


### Pull Request Checklist

<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->

* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
  - Use markdown where necessary, mostly for `code blocks`.
  - End with either a period (.) or an exclamation mark (!).
  - Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
2024-09-11 21:01:43 +00:00
Eric Eastwood 16af80b8fb
Sliding Sync: Use Sliding Sync tables for sorting (#17693)
Use Sliding Sync tables for sorting
(`bulk_get_last_event_pos_in_room_before_stream_ordering(...)` ->
`_bulk_get_max_event_pos(...)`)
2024-09-11 12:16:24 -05:00
Eric Eastwood e4a1f271b9
Sliding Sync: Make sure we get up-to-date information from `get_sliding_sync_rooms_for_user(...)` (#17692)
We need to bust the `get_sliding_sync_rooms_for_user`
cache when the room encryption is updated and any
other field that is used in the query.

Follow-up to https://github.com/element-hq/synapse/pull/17630

- Bust cache for membership change (cross-reference
`get_rooms_for_user`)
- Bust cache for room `encryption` (cross-reference
`get_room_encryption`)
- Bust cache for `forgotten` (cross-reference
`did_forget`/`get_forgotten_rooms_for_user`)
2024-09-11 12:13:54 -05:00
Erik Johnston 6b131a99fe Merge remote-tracking branch 'origin/release-v1.115' into develop 2024-09-11 16:43:07 +01:00
Erik Johnston 76f7c91e44 Sliding sync: don't fetch room summary for named rooms. (#17683)
For rooms with a name we can skip fetching a full room summary, as we
don't need to calculate heroes, and instead just fetch the room counts
directly.

This also changes things to not return counts and heroes for non-joined
rooms. For left/banned rooms we were returning zero values anyway, and
for invite/knock rooms we don't really want to leak such information
(even if some of is included in the stripped state).
2024-09-11 16:42:50 +01:00
Erik Johnston b732d13d4c Sliding sync: various fixups to the background update (#17652) 2024-09-11 16:42:15 +01:00
Erik Johnston 596b96411b
Sliding sync: various fixups to the background update (#17652) 2024-09-11 15:38:46 +01:00
Erik Johnston f6c2b0ec2e
Sliding sync: don't fetch room summary for named rooms. (#17683)
For rooms with a name we can skip fetching a full room summary, as we
don't need to calculate heroes, and instead just fetch the room counts
directly.

This also changes things to not return counts and heroes for non-joined
rooms. For left/banned rooms we were returning zero values anyway, and
for invite/knock rooms we don't really want to leak such information
(even if some of is included in the stripped state).
2024-09-11 13:16:57 +01:00
Travis Ralston a7fcac5648
Enable guest access on new media endpoints, per MSC4189 (#17675) 2024-09-10 18:29:24 +01:00
V02460 e06e3c4004
Add config option turn_shared_secret_path (#17690)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-09-10 17:27:46 +00:00
dependabot[bot] 60441059a3
Bump anyhow from 1.0.86 to 1.0.87 (#17685)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-09-10 18:05:31 +01:00
Jeremy Wright 1b197752b6
Fix minor misspelling in README.rst. (#17664) 2024-09-10 17:33:25 +01:00
dependabot[bot] 598a83d005
Bump cryptography from 43.0.0 to 43.0.1 (#17689)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-09-10 17:32:17 +01:00
dependabot[bot] be603de2cb
Bump serde_json from 1.0.127 to 1.0.128 (#17687)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-09-10 17:31:34 +01:00
dependabot[bot] 62523571ae
Bump serde from 1.0.209 to 1.0.210 (#17686) 2024-09-10 17:30:37 +01:00
Devon Hudson 5562a89168
Update changelog 2024-09-10 08:48:41 -06:00
Devon Hudson 59bcbcec0a
1.115.0rc1 2024-09-10 08:42:01 -06:00
dependabot[bot] d8b926d323
Bump idna from 3.7 to 3.8 (#17682) 2024-09-10 10:34:54 +01:00
dependabot[bot] 2efed1d4fb
Bump types-setuptools from 71.1.0.20240818 to 74.1.0.20240907 (#17681) 2024-09-10 10:34:45 +01:00
dependabot[bot] cd24bc2f36
Bump ruff from 0.6.2 to 0.6.4 (#17680) 2024-09-10 10:34:34 +01:00
dependabot[bot] a193d4a1b5
Bump authlib from 1.3.1 to 1.3.2 (#17679) 2024-09-10 10:34:26 +01:00
Erik Johnston b3047f3f17
Sliding sync: various fixups to the sliding sync joined room background job (#17673)
Follow-up to #17652, https://github.com/element-hq/synapse/pull/17641,
https://github.com/element-hq/synapse/pull/17634,
https://github.com/element-hq/synapse/pull/17631 and
https://github.com/element-hq/synapse/pull/17632 to fix-up
https://github.com/element-hq/synapse/pull/17512
2024-09-10 10:22:46 +01:00
Erik Johnston 9689ac3294
Sliding Sync: Look for `bump _stamp` in the room timeline (#17684)
This allows us to skip checking the database a lot of the time.

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-09-10 10:20:30 +01:00
Erik Johnston 588e5b521d
Sliding Sync: Retrieve fewer events from DB in sync (#17688)
When using timeline limit of 1 we end up fetching 2 events from the DB
purely to tell if the response was "limited" or not. Lets not do that.
2024-09-10 09:52:42 +01:00
Eric Eastwood 515c1cc0a1
Sliding Sync: Add comment to explain extra case where you can be invited -> banned -> unbanned (#17654)
Add comment to explain extra case where you can be
invited -> banned -> unbanned and we want to be able
to find the invite event.

Follow-up to https://github.com/element-hq/synapse/pull/17636#discussion_r1738993330
2024-09-09 17:55:59 -05:00
Eric Eastwood e1ed959a68
Sliding Sync: Get `bump_stamp` from new sliding sync tables because it's faster (#17658)
Get `bump_stamp` from [new sliding sync
tables](https://github.com/element-hq/synapse/pull/17512) which should
be faster (performance) than flipping through the latest events in the
room.
2024-09-09 16:41:25 +01:00
Erik Johnston 5c229415c4 Revert "Look for bump stamp in the room timeline"
This reverts commit a3c49565ff.
2024-09-09 11:58:50 +01:00
Erik Johnston a3c49565ff Look for bump stamp in the room timeline
This allows us to skip checking the database a lot of the time.
2024-09-09 11:58:18 +01:00
Eric Eastwood 5389374ef8
Sliding Sync: Speed up incremental sync by avoiding extra work (#17665)
Speed up incremental sync by avoiding extra work. We first look at the
state delta changes and only fetch and calculate further derived things
if they have changed.
2024-09-09 10:36:22 +01:00
Erik Johnston e5d07bb083
Fix bump stamp for non-joined rooms (#17674)
We should only look for bump stamps in joined rooms, otherwise we should
just use the membership stream ordering.
2024-09-06 11:44:37 +01:00