Commit Graph

21083 Commits

Author SHA1 Message Date
Eric Eastwood 0f971ca68e
Update `get_pdu` to return the original, pristine `EventBase` (#13320)
Update `get_pdu` to return the untouched, pristine `EventBase` as it was originally seen over federation (no metadata added). Previously, we returned the same `event` reference that we stored in the cache which downstream code modified in place and added metadata like setting it as an `outlier`  and essentially poisoned our cache. Now we always return a copy of the `event` so the original can stay pristine in our cache and re-used for the next cache call.

Split out from https://github.com/matrix-org/synapse/pull/13205

As discussed at:

 - https://github.com/matrix-org/synapse/pull/13205#discussion_r918365746
 - https://github.com/matrix-org/synapse/pull/13205#discussion_r918366125

Related to https://github.com/matrix-org/synapse/issues/12584. This PR doesn't fix that issue because it hits [`get_event` which exists from the local database before it tries to `get_pdu`](7864f33e28/synapse/federation/federation_client.py (L581-L594)).
2022-07-20 15:58:51 -05:00
Shay a1b62af2af
Validate federation destinations and log an error if server name is invalid. (#13318) 2022-07-20 11:17:26 -07:00
Erik Johnston d3995049a8 Merge remote-tracking branch 'origin/master' into develop 2022-07-20 14:59:43 +01:00
Erik Johnston 93740cae57 1.63.1 2022-07-20 13:37:00 +01:00
Erik Johnston b4ae3b0d44
Don't include appservice users when calculating push rules (#13332)
This can cause a lot of extra load on servers with lots of appservice users. Introduced in #13078
2022-07-20 12:06:13 +01:00
Sean Quah 172ce29b14
Fix spurious warning when fetching state after a missing prev event (#13258) 2022-07-19 19:15:54 +01:00
Patrick Cloke a6895dd576
Add type annotations to `trace` decorator. (#13328)
Functions that are decorated with `trace` are now properly typed
and the type hints for them are fixed.
2022-07-19 14:14:30 -04:00
Brendan Abolivier 47822fd2e8
Merge branch 'master' into develop 2022-07-19 16:14:02 +02:00
Erik Johnston de70b25e84
Reduce memory usage of state group cache (#13323) 2022-07-19 14:40:37 +01:00
Patrick Cloke 1efe6b8c41
Stop building Ubuntu 21.10 (Impish Indri) which is end of life. (#13326) 2022-07-19 09:08:46 -04:00
Brendan Abolivier 6fccd72f42
Improve precision on validation improvements 2022-07-19 14:53:12 +02:00
Brendan Abolivier 097afd0e0b 1.63.0 2022-07-19 14:43:28 +02:00
Andrew Morgan 6faaf76a32
Remove 'anonymised' from the phone home stats documentation (#13321) 2022-07-19 12:38:29 +00:00
villepeh 84c5e6b1fd
Bash script for creating multiple stream writers (#13271)
Add another bash script to the contrib directory. It creates multiple stream writers and also prints out the example configuration for homeserver.yaml.

Signed-off-by: Ville Petteri Huh.
2022-07-19 12:37:20 +00:00
Jörg Behrmann 87a917e8c8
Add notes when config options were changed to config documentation (#13314)
Signed-off-by: Jörg Behrmann <behrmann@physik.fu-berlin.de>
2022-07-19 12:36:29 +00:00
David Robertson b977867358
Rate limit joins per-room (#13276) 2022-07-19 11:45:17 +00:00
Nick Mills-Barrett 2ee0b6ef4b
Safe async event cache (#13308)
Fix race conditions in the async cache invalidation logic, by separating
the async & local invalidation calls and ensuring any async call i
executed first.

Signed off by Nick @ Beeper (@Fizzadar).
2022-07-19 11:25:29 +00:00
Shay 7864f33e28
Increase batch size of `bulk_get_push_rules` and `_get_joined_profiles_from_event_ids`. (#13300) 2022-07-18 13:15:23 -07:00
Shay 15edf23626
Improve performance of query ` _get_subset_users_in_room_with_profiles` (#13299) 2022-07-18 12:35:45 -07:00
Sean Quah 5526f9fc4f
Fix overcounting of pushers when they are replaced (#13296)
Signed-off-by: Sean Quah <seanq@matrix.org>
2022-07-18 17:39:39 +01:00
Brendan Abolivier 8c60c572f0
Up the dependency on canonicaljson to ^1.5.0 (#13172)
Co-authored-by: David Robertson <davidr@element.io>
2022-07-18 17:30:59 +02:00
Andrew Morgan bb25dd81e3
Prevent #3679 from appearing in blame results (#13311) 2022-07-18 14:02:32 +00:00
Erik Johnston f721f1baba
Revert "Make all `process_replication_rows` methods async (#13304)" (#13312)
This reverts commit 5d4028f217.
2022-07-18 14:28:14 +01:00
Erik Johnston cf5fa5063d
Don't pull out full state when sending dummy events (#13310) 2022-07-18 14:19:11 +01:00
Nick Mills-Barrett 6785b0f39d
Use READ COMMITTED isolation level when purging rooms (#12942)
To close: #10294.

Signed off by Nick @ Beeper.
2022-07-18 14:17:24 +01:00
Andrew Morgan c5f487b7cb
Update expected DB query count when creating a room (#13307) 2022-07-18 13:02:25 +01:00
Erik Johnston c6a05063ff
Don't pull out the full state when creating an event (#13281) 2022-07-18 10:05:30 +01:00
Dirk Klimpel efee345b45
Remove unnecessary `json.dumps` from tests (#13303) 2022-07-17 22:28:45 +01:00
Nick Mills-Barrett 5d4028f217
Make all `process_replication_rows` methods async (#13304)
More prep work for asyncronous caching, also makes all process_replication_rows methods consistent (presence handler already is so).

Signed off by Nick @ Beeper (@Fizzadar)
2022-07-17 22:19:43 +01:00
Dirk Klimpel 96cf81e312
Use HTTPStatus constants in place of literals in tests. (#13297) 2022-07-15 19:31:27 +00:00
Eric Eastwood 7b67e93d49
Provide more info why we don't have any thumbnails to serve (#13038)
Fix https://github.com/matrix-org/synapse/issues/13016

## New error code and status

### Before

Previously, we returned a `404` for `/thumbnail` which isn't even in the spec.

```json
{
  "errcode": "M_NOT_FOUND",
  "error": "Not found [b'hs1', b'tefQeZhmVxoiBfuFQUKRzJxc']"
}
```

### After

What does the spec say?

> 400: The request does not make sense to the server, or the server cannot thumbnail the content. For example, the client requested non-integer dimensions or asked for negatively-sized images.
>
> *-- https://spec.matrix.org/v1.1/client-server-api/#get_matrixmediav3thumbnailservernamemediaid*

Now with this PR, we respond with a `400` when we don't have thumbnails to serve and we explain why we might not have any thumbnails.

```json
{
    "errcode": "M_UNKNOWN",
    "error": "Cannot find any thumbnails for the requested media ([b'example.com', b'12345']). This might mean the media is not a supported_media_format=(image/jpeg, image/jpg, image/webp, image/gif, image/png) or that thumbnailing failed for some other reason. (Dynamic thumbnails are disabled on this server.)",
}
```

> Cannot find any thumbnails for the requested media ([b'example.com', b'12345']). This might mean the media is not a supported_media_format=(image/jpeg, image/jpg, image/webp, image/gif, image/png) or that thumbnailing failed for some other reason. (Dynamic thumbnails are disabled on this server.)


---

We still respond with a 404 in many other places. But we can iterate on those later and maybe keep some in some specific places after spec updates/clarification: https://github.com/matrix-org/matrix-spec/issues/1122

We can also iterate on the bugs where Synapse doesn't thumbnail when it should in other issues/PRs.
2022-07-15 11:42:21 -05:00
David Robertson e9ce4d089b
Use and recommend poetry 1.1.14, up from 1.1.12 (#13285) 2022-07-15 16:18:47 +01:00
Erik Johnston 0731e0829c
Don't pull out the full state when storing state (#13274) 2022-07-15 12:59:45 +00:00
Patrick Cloke 3343035a06
Use a real room in the notification rotation tests. (#13260)
Instead of manually inserting fake data. This fixes some issues with
having to manually calculate stream orderings and other oddities.
2022-07-15 08:22:43 -04:00
David Robertson 7281591f4c
Use state before join to determine if we `_should_perform_remote_join` (#13270)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-07-15 12:20:47 +00:00
Sean Quah d765ada84f
Update locked frozendict version to 2.3.2 (#13284)
`frozendict` 2.3.2 includes a fix for a memory leak in
`frozendict.__hash__`. This likely has no impact outside of the
deprecated `/initialSync` endpoint, which uses `StreamToken`s,
containing `RoomStreamToken`s, containing `frozendict`s, as cache keys.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-07-15 13:18:51 +01:00
Richard van der Hoff b116d3ce00
Bg update to populate new `events` table columns (#13215)
These columns were added back in Synapse 1.52, and have been populated for new
events since then. It's now (beyond) time to back-populate them for existing
events.
2022-07-15 12:47:26 +01:00
Erik Johnston 7be954f59b
Fix a bug which could lead to incorrect state (#13278)
There are two fixes here:
1. A long-standing bug where we incorrectly calculated `delta_ids`; and
2. A bug introduced in #13267 where we got current state incorrect.
2022-07-15 11:06:41 +00:00
Richard van der Hoff 512486bbeb
Docker: copy postgres from base image (#13279)
When building the docker images for complement testing, copy a preinstalled
complement over from a base image, rather than apt installing it. This avoids
network traffic and is much faster.
2022-07-15 11:13:40 +01:00
Nick Mills-Barrett cc21a431f3
Async get event cache prep (#13242)
Some experimental prep work to enable external event caching based on #9379 & #12955. Doesn't actually move the cache at all, just lays the groundwork for async implemented caches.

Signed off by Nick @ Beeper (@Fizzadar)
2022-07-15 09:30:46 +00:00
Nick Mills-Barrett 21eeacc995
Federation Sender & Appservice Pusher Stream Optimisations (#13251)
* Replace `get_new_events_for_appservice` with `get_all_new_events_stream`

The functions were near identical and this brings the AS worker closer
to the way federation senders work which can allow for multiple workers
to handle AS traffic.

* Pull received TS alongside events when processing the stream

This avoids an extra query -per event- when both federation sender
and appservice pusher process events.
2022-07-15 09:36:56 +01:00
Richard van der Hoff fe15a865a5
Rip out auth-event reconciliation code (#12943)
There is a corner in `_check_event_auth` (long known as "the weird corner") where, if we get an event with auth_events which don't match those we were expecting, we attempt to resolve the diffence between our state and the remote's with a state resolution.

This isn't specced, and there's general agreement we shouldn't be doing it.

However, it turns out that the faster-joins code was relying on it, so we need to introduce something similar (but rather simpler) for that.
2022-07-14 21:52:26 +00:00
Richard van der Hoff df55b377be
CHANGES.md: fix link to upgrade notes 2022-07-14 15:07:52 +01:00
Erik Johnston 0ca4172b5d
Don't pull out state in `compute_event_context` for unconflicted state (#13267) 2022-07-14 13:57:02 +00:00
David Robertson 599c403d99
Allow rate limiters to passively record actions they cannot limit (#13253)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-07-13 19:09:42 +00:00
David Robertson 0eb7e69768
Notifier: accept callbacks to fire on room joins (#13254) 2022-07-13 19:48:24 +01:00
Jacek Kuśnierz cc1071598a
Call the v2 identity service `/3pid/unbind` endpoint, rather than v1. (#13240)
* Drop support for v1 unbind

Signed-off-by: Jacek Kusnierz <jacek.kusnierz@tum.de>

* Add changelog

Signed-off-by: Jacek Kusnierz <jacek.kusnierz@tum.de>

* Update changelog.d/13240.misc
2022-07-13 19:43:17 +01:00
Shay ad5761b65c
Add support for room version 10 (#13220) 2022-07-13 11:36:02 -07:00
jejo86 2341032cf2
Document advising against publicly exposing the Admin API and provide a usage example (#13231)
* Admin API request explanation improved

Pointed out, that the Admin API is not accessible by default from any remote computer, but only from the PC `matrix-synapse` is running on.
Added a full, working example, making sure to include the cURL flag `-X`, which needs to be prepended to `GET`, `POST`, `PUT` etc. and listing the full query string including protocol, IP address and port.

* Admin API request explanation improved

* Apply suggestions from code review

Update changelog. Reword prose.

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
2022-07-13 19:33:33 +01:00
Nick Mills-Barrett 982fe29655
Optimise room creation event lookups part 2 (#13224) 2022-07-13 19:32:46 +01:00