Commit Graph

8240 Commits

Author SHA1 Message Date
Erik Johnston a98cb87bee
Revert "Reduce device lists replication traffic." (#17360)
Reverts element-hq/synapse#17333

It looks like master was still sending out replication RDATA with the
old format... somehow
2024-06-25 09:57:34 +01:00
Eric Eastwood 6e8af83193
Add `is_invite` filtering to Sliding Sync `/sync` (#17335)
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
2024-06-24 19:07:56 -05:00
devonh 805e6c9a8f
Correct error in user_directory docs (#17348)
### Pull Request Checklist

<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->

* [X] Pull request is based on the develop branch
* [X] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
  - Use markdown where necessary, mostly for `code blocks`.
  - End with either a period (.) or an exclamation mark (!).
  - Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))

---------

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: reivilibre <oliverw@matrix.org>
2024-06-24 17:18:58 +00:00
devonh 3c61ddbbc9
Add default values for rc_invites per_issuer to docs (#17347)
A simple change to update the docs where default values were missing.

### Pull Request Checklist

<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->

* [X] Pull request is based on the develop branch
* [X] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
  - Use markdown where necessary, mostly for `code blocks`.
  - End with either a period (.) or an exclamation mark (!).
  - Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))

---------

Co-authored-by: Kim Brose <2803622+HarHarLinks@users.noreply.github.com>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-06-24 17:16:09 +00:00
Eric Eastwood ae4c236a6d
Fix room `type` typo in mailer (#17336)
Correct event content field is `EventContentFields.ROOM_TYPE` (`type`)  , not `room_type` 

Spec: https://spec.matrix.org/v1.10/client-server-api/#mroomcreate
2024-06-24 10:02:16 -05:00
Erik Johnston 930a64b6c1
Reintroduce #17291. (#17338)
This is #17291 (which got reverted), with some added fixups, and change
so that tests actually pick up the error.

The problem was that we were not calculating any new chain IDs due to a
missing `not` in a condition.
2024-06-24 14:40:28 +00:00
Shay 7a11c0ac4f
Add support for MSC3823 - Account Suspension Part 2 (#17255) 2024-06-24 14:21:22 +01:00
Erik Johnston cf711ac03c
Reduce device lists replication traffic. (#17333)
Reduce the replication traffic of device lists, by not sending every
destination that needs to be sent the device list update over
replication. Instead a "hosts to send to have been calculated"
notification over replication, and then federation senders read the
destinations from the DB.

For non federation senders this should heavily reduce the impact of a
user in many large rooms changing a device.
2024-06-24 14:15:13 +01:00
Denis Kasak 700d2cc4a0
Tidy up integer parsing (#17339)
The parse_integer function was previously made to reject negative values by
default in https://github.com/element-hq/synapse/pull/16920, but the
documentation stated otherwise. This fixes the documentation and also:

- Removes explicit negative=False parameters from call sites.
- Brings the negative default of parse_integer_from_args in alignment with
  parse_integer.
2024-06-24 15:12:14 +02:00
Erik Johnston 4243c1f074
Revert "Handle large chain calc better (#17291)" (#17334)
This reverts commit bdf82efea5  (#17291)

This seems to have stopped persisting auth chains for new events, and so
is causing state res to fall back to the slow methods
2024-06-19 17:39:33 +01:00
Erik Johnston 3239b7459c
Register sliding sync under a different path (#17331)
As the API is slightly incompatible.
2024-06-19 16:18:45 +00:00
Jörg Thalheim c99203d98c
register-new-matrix-user: add a flag to ignore already existing users (#17304)
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
2024-06-19 12:03:08 +01:00
Alexander Fechler 9104a9f0d0
Filter added to Admin-API GET /rooms (#17276) 2024-06-19 11:45:48 +01:00
Andrew Morgan a412a5829d
Update the README with Element branding and a few fixes (#17324)
Co-authored-by: Hugh Nimmo-Smith <hughns@element.io>
2024-06-19 10:58:22 +01:00
Aaron Dewes 7ef89b985d
Remove `expire_access_token` from Docker configuration (#17198)
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
2024-06-19 10:58:06 +01:00
Erik Johnston bdf82efea5
Handle large chain calc better (#17291)
We calculate the auth chain links outside of the main persist event
transaction to ensure that we do not block other event sending during
the calculation.
2024-06-19 10:33:53 +01:00
Andrew Morgan afaf2d9388
Require the 'from' parameter for `/notifications` be an integer (#17283)
Co-authored-by: Erik Johnston <erikj@element.io>
2024-06-19 10:05:39 +01:00
Andrew Morgan 199223062a
Revert "Support MSC3916 by adding a federation `/download` endpoint" (#17325) 2024-06-18 16:54:19 +01:00
Jörg Thalheim 97c3d98816
register_new_matrix_user: add password-file flag (#17294)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
2024-06-18 16:21:51 +01:00
Johannes Marbach 79767a1108
Add support for via query parameter from MSC4156 (#17322)
This adds support for the `via` query parameter from
https://github.com/matrix-org/matrix-spec-proposals/pull/4156.
2024-06-18 10:03:39 +00:00
Dirk Klimpel 1c7d85fdfe
fix missing quotes for exclude_rooms_from_sync (#17308)
We tried to configure rooms `exclude_rooms_from_sync`. If we do not
quote we get an error.
The example should be valid.
2024-06-18 10:37:02 +01:00
Eric Eastwood a5485437cf
Add `is_encrypted` filtering to Sliding Sync `/sync` (#17281)
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
2024-06-17 12:06:18 -05:00
Eric Eastwood e5b8a3e37f
Add `stream_ordering` sort to Sliding Sync `/sync` (#17293)
Sort is no longer configurable and we always sort rooms by the `stream_ordering` of the last event in the room or the point where the user can see up to in cases of leave/ban/invite/knock.
2024-06-17 11:27:14 -05:00
Richard van der Hoff 3aae60f17b
Enable cross-signing key upload without UIA (#17284)
Per MSC3967, which is now stable, we should not require UIA when
uploading cross-signing keys for the first time.

Fixes: #17227
2024-06-14 11:14:56 +01:00
Richard van der Hoff 2c36a679ae
Include user membership on events (#17282)
MSC4115 has now completed FCP, so we can enable it by default and switch
to the stable identifier.
2024-06-13 21:45:54 +00:00
Eric Eastwood c12ee0d5ba
Add `is_dm` filtering to Sliding Sync `/sync` (#17277)
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
2024-06-13 13:56:58 -05:00
Eric Eastwood 8aaff851b1
Fix `newly_left` rooms not appearing if we returned early (Sliding Sync) (#17301)
Fix `newly_left` rooms not appearing if we returned early when `membership_snapshot_token.is_before_or_eq(to_token.room_key)`. 

Introduced in https://github.com/element-hq/synapse/pull/17187 (part of Sliding Sync)

The tests didn't catch it because they had a small typo in it `room_id1` vs `room_id2`.

Found while working on https://github.com/element-hq/synapse/pull/17293
2024-06-13 11:36:57 -05:00
Eric Eastwood 8c58eb7f17
Add `event.internal_metadata.instance_name` (#17300)
Add `event.internal_metadata.instance_name` (the worker instance that persisted the event) to go alongside the existing `event.internal_metadata.stream_ordering`.

`instance_name` is useful to properly compare and query for events with a token since you need to compare both the `stream_ordering` and `instance_name` against the vector clock/`instance_map` in the `RoomStreamToken`.

This is pre-requisite work and may be used in https://github.com/element-hq/synapse/pull/17293

Adding `event.internal_metadata.instance_name` was first mentioned in the initial Sliding Sync PR while pairing with @erikjohnston, see 09609cb0db (diff-5cd773fb307aa754bd3948871ba118b1ef0303f4d72d42a2d21e38242bf4e096R405-R410)
2024-06-13 11:32:50 -05:00
Eric Eastwood ebdce69f6a
Fix `get_last_event_in_room_before_stream_ordering(...)` finding the wrong last event (#17295)
PR where this was introduced: https://github.com/matrix-org/synapse/pull/14817

### What does this affect?

`get_last_event_in_room_before_stream_ordering(...)` is used in Sync v2 in a lot of different state calculations.

`get_last_event_in_room_before_stream_ordering(...)`  is also used in `/rooms/{roomId}/members`
2024-06-13 11:00:52 -05:00
Andrew Morgan c6eb99c878
Bump `mypy` from 1.8.0 to 1.9.0 (#17297)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-13 15:50:10 +01:00
Quentin Gliech 5db3eec5bc
Clarify that MSC4151 is enabled on matrix.org (#17296)
This clarifies in the comments that the MSC is being used in matrix.org

See #17270
2024-06-13 13:49:00 +00:00
Travis Ralston f1c4dfb08b
Add report room API (MSC4151) (#17270)
https://github.com/matrix-org/matrix-spec-proposals/pull/4151

This is intended to be enabled by default for immediate use. When FCP is
complete, the unstable endpoint will be dropped and stable endpoint
supported instead - no backwards compatibility is expected for the
unstable endpoint.
2024-06-12 12:27:46 +02:00
Eric Eastwood dad1559721
Reorganize Pydantic models and types used in handlers (#17279)
Spawning from https://github.com/element-hq/synapse/pull/17187#discussion_r1619492779 around wanting to put `SlidingSyncBody` (parse the request in the rest layer), `SlidingSyncConfig` (from the rest layer, pass to the handler), `SlidingSyncResponse` (pass the response from the handler back to the rest layer to respond) somewhere that doesn't contaminate the imports and cause circular import issues.

 - Moved Pydantic parsing models to `synapse/types/rest`
 - Moved handler types to `synapse/types/handlers`
2024-06-10 15:03:50 -05:00
Eric Eastwood 265ee88f34
Wrong retention policy being used when filtering events (lint `ControlVarUsedAfterBlockViolation` `WPS441`) (#17272)
Fix loop var being used outside block.

Before this change, we were always using the last room_id's retention policy for all events being filtered.

I found this bug with the [new lint rule, `ControlVarUsedAfterBlockViolation` `WPS441`](https://github.com/astral-sh/ruff/pull/11769), that I re-implemented in `ruff`. Shout-out to @reivilibre for all the help in the beginning! 

### Pull Request Checklist

<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->

* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
  - Use markdown where necessary, mostly for `code blocks`.
  - End with either a period (.) or an exclamation mark (!).
  - Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
2024-06-10 09:48:38 -05:00
Shay ab94bce02c
Support MSC3916 by adding a federation `/download` endpoint (#17172) 2024-06-07 13:54:28 +01:00
reivilibre 17d6c28285
Add debug logging for when room keys are uploaded, including whether they are replacing other room keys. (#17266)
Fixes: #17013

Add logging for whether room keys are replaced
This is motivated by the Crypto team who need to diagnose crypto issues.

The existing opentracing logging is not enough because it is not enabled
for all users.
2024-06-07 12:01:21 +01:00
Eric Eastwood 4a7c58642c
Add Sliding Sync `/sync` endpoint (initial implementation) (#17187)
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync

This iteration only focuses on returning the list of room IDs in the sliding window API (without sorting/filtering).

Rooms appear in the Sliding sync response based on:

 - `invite`, `join`, `knock`, `ban` membership events
 - Kicks (`leave` membership events where `sender` is different from the `user_id`/`state_key`)
 - `newly_left` (rooms that were left during the given token range, > `from_token` and <= `to_token`)
 - In order for bans/kicks to not show up, you need to `/forget` those rooms. This doesn't modify the event itself though and only adds the `forgotten` flag to `room_memberships` in Synapse. There isn't a way to tell when a room was forgotten at the moment so we can't factor it into the from/to range.

### Example request

`POST http://localhost:8008/_matrix/client/unstable/org.matrix.msc3575/sync`

```json
{
  "lists": {
    "foo-list": {
      "ranges": [ [0, 99] ],
      "sort": [ "by_notification_level", "by_recency", "by_name" ],
      "required_state": [
        ["m.room.join_rules", ""],
        ["m.room.history_visibility", ""],
        ["m.space.child", "*"]
      ],
      "timeline_limit": 100
    }
  }
}
```

Response:
```json
{
  "next_pos": "s58_224_0_13_10_1_1_16_0_1",
  "lists": {
    "foo-list": {
      "count": 1,
      "ops": [
        {
          "op": "SYNC",
          "range": [0, 99],
          "room_ids": [
            "!MmgikIyFzsuvtnbvVG:my.synapse.linux.server"
          ]
        }
      ]
    }
  },
  "rooms": {},
  "extensions": {}
}
```
2024-06-06 14:44:32 -05:00
Erik Johnston ce9385819b
Handle OTK uploads off master (#17271)
And fallback keys uploads. Only device keys need handling on master
2024-06-06 17:47:02 +01:00
Erik Johnston a963f579de
Don't try and resync devices for down hosts (#17273)
It's just a waste of time if we won't even query the remote host as its
marked as down.
2024-06-06 17:46:52 +01:00
Erik Johnston 3f06bbc0ac
Always return OTK counts (#17275)
Broke in https://github.com/element-hq/synapse/pull/17215
2024-06-06 17:10:58 +01:00
Shay fcbc79bb87
Ratelimiting of remote media downloads (#17256) 2024-06-05 13:43:36 +01:00
Erik Johnston aabf577166
Handle hyphens in user dir search porperly (#17254)
c.f. #16675
2024-06-05 10:40:34 +01:00
Eric Eastwood 7d8f0ef351
Use fully-qualified `PersistedEventPosition` when returning `RoomsForUser` (#17265)
Use fully-qualified `PersistedEventPosition` (`instance_name` and `stream_ordering`) when returning `RoomsForUser` to facilitate proper comparisons and `RoomStreamToken` generation.

Spawning from https://github.com/element-hq/synapse/pull/17187 where we want to utilize this change
2024-06-04 12:58:03 -05:00
Andrew Morgan 18c1196893 1.109.0rc1 2024-06-04 09:46:09 +01:00
Erik Johnston 5c2a837e3c
Fix bug where typing replication breaks (#17252)
This can happen on restarts of the service, due to old rooms being
pruned.
2024-05-31 16:07:05 +01:00
Erik Johnston 64f5a4a353
Fix logging errors when receiving invalid User ID for key querys (#17250) 2024-05-31 11:27:56 +01:00
Erik Johnston 7dd14fadb1
Fix sentry default tags (#17251)
This was broken by the sentry 2.0 upgrade

Broke in v1.108.0
2024-05-31 11:27:47 +01:00
Erik Johnston 5624c8b961
In sync wait for worker to catch up since token (#17215)
Otherwise things will get confused.

An alternative would be to make sure that for lagging stream we don't
return anything (and make sure the returned next_batch token doesn't go
backwards). But that is a faff.
2024-05-30 14:03:49 +01:00
Erik Johnston 4e3868dc46
Fix deduplicating of membership events to not create unused state groups. (#17164)
We try and deduplicate in two places: 1) really early on, and 2) just
before we persist the event. The first case was broken due to it
occuring before the profile information was added, and so it thought the
event contents were different.

The second case did catch it and handle it correctly, however doing so
creates a redundant state group leading to bloat.

Fixes #3791
2024-05-30 11:33:48 +00:00
Erik Johnston d16910ca02
Replaces all usages of `StreamIdGenerator` with `MultiWriterIdGenerator` (#17229)
Replaces all usages of `StreamIdGenerator` with `MultiWriterIdGenerator`, which is safer.
2024-05-30 11:07:32 +00:00