Commit Graph

871 Commits

Author SHA1 Message Date
Erik Johnston 52af16c561
Add a short sleep if the request is rate-limited (#17210)
This helps prevent clients from "tight-looping" retrying their request.
2024-05-18 12:03:30 +01:00
Eric Eastwood 28a948f04f
Removed `request_key` from the `SyncConfig` (moved outside as its own function parameter) (#17201)
Removed `request_key` from the `SyncConfig` (moved outside as its own function parameter) so it doesn't have to flow into `_generate_sync_entry_for_xxx` methods. This way we can separate the concerns of caching from generating the response and reuse the `_generate_sync_entry_for_xxx` functions as we see fit. Plus caching doesn't really have anything to do with the config of sync.

Split from https://github.com/element-hq/synapse/pull/17167

Spawning from https://github.com/element-hq/synapse/pull/17167#discussion_r1601497279
2024-05-16 11:54:46 -05:00
Erik Johnston 5e892671a7
Fix bug where push rules would be empty in `/sync` (#17142)
Fixes #16987

Some old accounts seem to have an entry in global account data table for
push rules, which we should ignore
2024-05-16 15:04:14 +01:00
Eric Eastwood d2d48cce85
Refactor Sync handler to be able to return different sync responses (`SyncVersion`) (#17200)
Refactor Sync handler to be able to be able to return different sync
responses (`SyncVersion`). Preparation to be able support sync v2 and a
new Sliding Sync `/sync/e2ee` endpoint which returns a subset of sync
v2.

Split upon request:
https://github.com/element-hq/synapse/pull/17167#discussion_r1601497279

Split from https://github.com/element-hq/synapse/pull/17167 where we
will add `SyncVersion.E2EE_SYNC` and a new type of sync response.
2024-05-16 11:36:54 +01:00
Till 47773232b0
Redact membership events if the user requested erasure upon deactivating (#17076)
Fixes #15355 by redacting all membership events before leaving rooms.
2024-04-25 14:25:31 +01:00
Kegan Dougal 259442fa4c
bugfix: make msc3967 idempotent (#16943)
MSC3967 was updated recently to make it more robust to network failures:

> there is an existing cross-signing master key and it exactly matches
the cross-signing master key provided in the request body. If there are
any additional keys provided in the request (self signing key, user
signing key) they MUST also match the existing keys stored on the
server. In other words, the request contains no new keys. If there are
new keys, UIA MUST be performed.


https://github.com/matrix-org/matrix-spec-proposals/blob/hughns/device-signing-upload-uia/proposals/3967-device-signing-upload-uia.md#proposal

This covers the case where the 200 OK is lost in transit so the client
retries the upload, only to then get UIA'd.

Complement tests: https://github.com/matrix-org/complement/pull/713 -
passing example
https://github.com/element-hq/synapse/actions/runs/7976948122/job/21778795094?pr=16943#step:7:8820

### Pull Request Checklist

<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->

* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
  - Use markdown where necessary, mostly for `code blocks`.
  - End with either a period (.) or an exclamation mark (!).
  - Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))

---------

Co-authored-by: reivilibre <oliverw@matrix.org>
2024-04-15 10:57:56 +00:00
Erik Johnston 89f1092284
Also check if first event matches the last in prev batch (#17066)
Refinement of #17064 

cc @richvdh
2024-04-09 14:01:12 +00:00
Richard van der Hoff 0e68e9b7f4
Fix bug in calculating state for non-gappy syncs (#16942)
Unfortunately, the optimisation we applied here for non-gappy syncs is
not actually valid.

Fixes https://github.com/element-hq/synapse/issues/16941.

~~Based on https://github.com/element-hq/synapse/pull/16930.~~
Requires https://github.com/matrix-org/sytest/pull/1374.
2024-04-04 16:15:35 +00:00
Richard van der Hoff 230b709d9d
`/sync`: fix bug in calculating `state` response (#16930)
Fix a long-standing issue which could cause state to be omitted from the
sync response if the last event was filtered out.

Fixes: https://github.com/element-hq/synapse/issues/16928
2024-04-04 12:14:24 +00:00
Richard van der Hoff 05957ac70f
Fix bug in `/sync` response for archived rooms (#16932)
This PR fixes a very, very niche edge-case, but I've got some more work
coming which will otherwise make the problem worse.

The bug happens when the syncing user leaves a room, and has a sync
filter which includes "left" rooms, but sets the timeline limit to 0. In
that case, the state returned in the `state` section is calculated
incorrectly.

The fix is to pass a token corresponding to the point that the user
leaves the room through to `compute_state_delta`.
2024-04-04 12:47:59 +01:00
Hanadi f7a3ebe44d
Fix reject knocks on deactivating account (#17010) 2024-03-21 18:05:54 +00:00
Shay 8fb5b0f335
Improve event validation (#16908)
As the title states.
2024-03-19 17:52:53 +00:00
Mathieu Velten cb562d73aa
Improve lock performance when a lot of locks are waiting (#16840)
When a lot of locks are waiting for a single lock, notifying all locks
independently with `call_later` on each release is really costly and
incurs some kind of async contention, where the CPU is spinning a lot
for not much.

The included test is taking around 30s before the change, and 0.5s
after.

It was found following failing tests with
https://github.com/element-hq/synapse/pull/16827.
2024-03-14 13:49:54 +00:00
Erik Johnston 23740eaa3d
Correctly mention previous copyright (#16820)
During the migration the automated script to update the copyright
headers accidentally got rid of some of the existing copyright lines.
Reinstate them.
2024-01-23 11:26:48 +00:00
Shay a68b48a5dd
Allow room creation but not publishing to continue if room publication rules are violated when creating a new room. (#16811)
Prior to this PR, if a request to create a public (public as in
published to the rooms directory) room violated the room list
publication rules set in the
[config](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#room_list_publication_rules),
the request to create the room was denied and the room was not created.

This PR changes the behavior such that when a request to create a room
published to the directory violates room list publication rules, the
room is still created but the room is not published to the directory.
2024-01-22 13:59:45 +00:00
reivilibre a83a337c4d
Filter out rooms from the room directory being served to other homeservers when those rooms block that homeserver by their Access Control Lists. (#16759)
The idea here being that the directory server shouldn't advertise rooms
to a requesting server is the requesting server would not be allowed to
join or participate in the room.

<!--
Fixes: # <!-- -->
<!--
Supersedes: # <!-- -->
<!--
Follows: # <!-- -->
<!--
Part of: # <!-- -->
Base: `develop` <!-- git-stack-base-branch:develop -->

<!--
This pull request is commit-by-commit review friendly. <!-- -->
<!--
This pull request is intended for commit-by-commit review. <!-- -->

Original commit schedule, with full messages:

<ol>
<li>

Pass `from_federation_origin` down into room list retrieval code 

</li>
<li>

Don't cache /publicRooms response for inbound federated requests 

</li>
<li>

fixup! Don't cache /publicRooms response for inbound federated requests 

</li>
<li>

Cap the number of /publicRooms entries to 100 

</li>
<li>

Simplify code now that you can't request unlimited rooms 

</li>
<li>

Filter out rooms from federated requests that don't have the correct ACL

</li>
<li>

Request a handful more when filtering ACLs so that we can try to avoid
shortchanging the requester

</li>
</ol>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2024-01-08 17:24:20 +00:00
Erik Johnston eaad9bb156 Merge remote-tracking branch 'gitlab/clokep/license-license' into new_develop 2023-12-13 15:11:56 +00:00
David Robertson 32a59a6495
Keep track of `user_ips` and `monthly_active_users` when delegating auth (#16672)
* Describe `insert_client_ip`
* Pull out client_ips and MAU tracking to BaseAuth
* Define HAS_AUTHLIB once in tests

sick of copypasting

* Track ips and token usage when delegating auth
* Test that we track MAU and user_ips
* Don't track `__oidc_admin`
2023-11-23 12:35:37 +00:00
Patrick Cloke 8e1e62c9e0 Update license headers 2023-11-21 15:29:58 -05:00
David Robertson 43d1aa75e8
Add an Admin API to temporarily grant the ability to update an existing cross-signing key without UIA (#16634) 2023-11-15 17:28:10 +00:00
Patrick Cloke f2f2c7c1f0
Use full GitHub links instead of bare issue numbers. (#16637) 2023-11-15 08:02:11 -05:00
Patrick Cloke ab3f1b3b53
Convert simple_select_one_txn and simple_select_one to return tuples. (#16612) 2023-11-09 11:13:31 -05:00
Patrick Cloke cfb6d38c47
Remove remaining usage of cursor_to_dict. (#16564) 2023-10-31 13:13:28 -04:00
David Robertson de981ae567
Claim local one-time-keys in bulk (#16565)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2023-10-30 21:25:21 +00:00
David Robertson fdce83ee60
Claim fallback keys in bulk (#16570) 2023-10-30 14:34:37 +00:00
Patrick Cloke 85e5f2dc25
Add a new module API to update user presence state. (#16544)
This adds a module API which allows a module to update a user's
presence state/status message. This is useful for controlling presence
from an external system.

To fully control presence from the module the presence.enabled config
parameter gains a new state of "untracked" which disables internal tracking
of presence changes via user actions, etc. Only updates from the module will
be persisted and sent down sync properly).
2023-10-26 15:11:24 -04:00
Patrick Cloke 9407d5ba78
Convert simple_select_list and simple_select_list_txn to return lists of tuples (#16505)
This should use fewer allocations and improves type hints.
2023-10-26 13:01:36 -04:00
Erik Johnston ba47fea528
Allow multiple workers to write to receipts stream. (#16432)
Fixes #16417
2023-10-25 16:16:19 +01:00
Patrick Cloke 3ab861ab9e
Fix type hint errors from Twisted trunk (#16526) 2023-10-23 14:28:05 -04:00
Patrick Cloke e3e0ae4ab1
Convert state delta processing from a dict to attrs. (#16469)
For improved type checking & memory usage.
2023-10-16 07:35:22 -04:00
Erik Johnston 009b47badf
Factor out `MultiWriter` token from `RoomStreamToken` (#16427) 2023-10-05 10:46:28 +01:00
Erik Johnston 80ec81dcc5
Some refactors around receipts stream (#16426) 2023-10-04 16:28:40 +01:00
Patrick Cloke 85bfd4735e
Return an immutable value from get_latest_event_ids_in_room. (#16326) 2023-09-18 09:29:05 -04:00
Hanadi 7afb5e0410
Fix using dehydrated devices (MSC2697) & refresh tokens (#16288)
Refresh tokens were not correctly moved to the rehydrated
device (similar to how the access token is currently handled).
This resulted in invalid refresh tokens after rehydration.
2023-09-13 08:33:39 -04:00
Patrick Cloke aa483cb4c9
Update ruff config (#16283)
Enable additional checks & clean-up unneeded configuration.
2023-09-08 11:24:36 -04:00
Aurélien Grimpard fe69e7f617
Handle "registration_enabled" parameter for CAS (#16262)
Similar to OIDC, CAS providers can now disable registration such
that only existing users are able to login via SSO.
2023-09-06 14:32:24 -04:00
Patrick Cloke 32fb264120 Merge remote-tracking branch 'origin/release-v1.92' into develop 2023-09-06 13:08:22 -04:00
Patrick Cloke 55c20da4a3 Merge remote-tracking branch 'origin/release-v1.91' into release-v1.92 2023-09-06 11:25:28 -04:00
Quentin Gliech 1940d990a3
Revert MSC3861 introspection cache, admin impersonation and account lock (#16258) 2023-09-06 15:19:51 +01:00
Mathieu Velten 4f1840a88a
Delete device messages asynchronously and in staged batches (#16240) 2023-09-06 09:30:53 +02:00
Will Hunt 1e571cd664
Fix appservices being unable to handle to_device messages for multiple users (#16251) 2023-09-05 15:46:57 -04:00
Patrick Cloke 8b5013dcbc
Time out busy presence status & test multi-device busy (#16174)
Add a (long) timeout to when a "busy" device is considered not online.
This does *not* match MSC3026, but is a reasonable thing for an
implementation to do.

Expands tests for the (unstable) busy presence with multiple devices.
2023-09-05 10:39:38 -04:00
Patrick Cloke ea75346f6a
Track presence state per-device and combine to a user state. (#16066)
Tracks presence on an individual per-device basis and combine
the per-device state into a per-user state. This should help in
situations where a user has multiple devices with conflicting status
(e.g. one is syncing with unavailable and one is syncing with online).

The tie-breaking is done by priority:

    BUSY > ONLINE > UNAVAILABLE > OFFLINE
2023-09-05 09:58:51 -04:00
Erik Johnston d35bed8369
Don't wake up destination transaction queue if they're not due for retry. (#16223) 2023-09-04 17:14:09 +01:00
Erik Johnston f84baecb6f
Don't reset retry timers on "valid" error codes (#16221) 2023-09-04 14:04:43 +01:00
Patrick Cloke ebd8374fb5
Stop writing to the event_txn_id table (#16175) 2023-08-30 11:10:56 +01:00
Patrick Cloke 9ec3da06da
Bump mypy-zope & mypy. (#16188) 2023-08-29 10:38:56 -04:00
Evilham 63b51ef3fb
Support IPv6-only SMTP servers (#16155)
Use Twisted HostnameEndpoint to connect to SMTP servers (instead
of connectTCP/connectSSL) which properly supports IPv6-only servers.
2023-08-29 09:33:58 -04:00
Patrick Cloke 40901af5e0
Pass the device ID around in the presence handler (#16171)
Refactoring to pass the device ID (in addition to the user ID) through
the presence handler (specifically the `user_syncing`, `set_state`,
and `bump_presence_active_time` methods and their replication
versions).
2023-08-28 13:08:49 -04:00
Patrick Cloke 1bf143699c
Combine logic about not overriding BUSY presence. (#16170)
Simplify some of the presence code by reducing duplicated code between
worker & non-worker modes.

The main change is to push some of the logic from `user_syncing` into
`set_state`. This is done by passing whether the user is setting the presence
via a `/sync` with a new `is_sync` flag to `set_state`. If this is `true` some
additional logic is performed:

* Don't override `busy` presence.
* Update the `last_user_sync_ts`.
* Never update the status message.
2023-08-28 11:03:23 -04:00