Commit Graph

21689 Commits

Author SHA1 Message Date
Erik Johnston 6fcdda68d3 Update lockfile 2022-10-19 14:26:21 +01:00
Erik Johnston 15a240f135 1.70.0rc1 2022-10-19 14:12:03 +01:00
Will Hunt 04d7f56f53
Use backend-meta edition of issue triage workflow (#14230) 2022-10-19 11:41:25 +01:00
Eric Eastwood fa8616e65c
Fix MSC3030 `/timestamp_to_event` returning `outliers` that it has no idea whether are near a gap or not (#14215)
Fix MSC3030 `/timestamp_to_event` endpoint returning `outliers` that it has no idea whether are near a gap or not (and therefore unable to determine whether it's actually the closest event). The reason Synapse doesn't know whether an `outlier` is next to a gap is because our gap checks rely on entries in the `event_edges`, `event_forward_extremeties`, and `event_backward_extremities` tables which is [not the case for `outliers`](2c63cdcc3f/docs/development/room-dag-concepts.md (outliers)).

Also fixes MSC3030 Complement `can_paginate_after_getting_remote_event_from_timestamp_to_event_endpoint` test flake.  Although this acted flakey in Complement, if `sync_partial_state` raced and beat us before `/timestamp_to_event`, then even if we retried the failing `/context` request it wouldn't work until we made this Synapse change. With this PR, Synapse will never return an `outlier` event so that test will always go and ask over federation.

Fix  https://github.com/matrix-org/synapse/issues/13944


### Why did this fail before? Why was it flakey?

Sleuthing the server logs on the [CI failure](https://github.com/matrix-org/synapse/actions/runs/3149623842/jobs/5121449357#step:5:5805), it looks like `hs2:/timestamp_to_event` found `$NP6-oU7mIFVyhtKfGvfrEQX949hQX-T-gvuauG6eurU` as an `outlier` event locally. Then when we went and asked for it via `/context`, since it's an `outlier`, it was filtered out of the results -> `You don't have permission to access that event.`

This is reproducible when `sync_partial_state` races and persists `$NP6-oU7mIFVyhtKfGvfrEQX949hQX-T-gvuauG6eurU` as an `outlier` before we evaluate `get_event_for_timestamp(...)`. To consistently reproduce locally, just add a delay at the [start of `get_event_for_timestamp(...)`](cb20b885cb/synapse/handlers/room.py (L1470-L1496)) so it always runs after `sync_partial_state` completes.

```py
from twisted.internet import task as twisted_task
d = twisted_task.deferLater(self.hs.get_reactor(), 3.5)
await d
```

In a run where it passes, on `hs2`, `get_event_for_timestamp(...)` finds a different event locally which is next to a gap and we request from a closer one from `hs1` which gets backfilled. And since the backfilled event is not an `outlier`, it's returned as expected during `/context`.

With this PR, Synapse will never return an `outlier` event so that test will always go and ask over federation.
2022-10-18 19:46:25 -05:00
Aaron Raimist 2a76a7369f
Fix hiding devices names over federation (#10015)
And don't include blank opentracing stuff in device list updates.

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2022-10-18 20:54:27 +00:00
Shay 1c777ef1e8
Fix docstring in EventContext (#14145) 2022-10-18 13:40:50 -07:00
MichaIng 06b0c4edfe
Add aarch64 wheels to CI (#14212)
Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
2022-10-18 17:12:21 +00:00
dependabot[bot] 85aa0f513b
Bump twisted from 22.4.0 to 22.8.0 (#14207)
* Bump twisted from 22.4.0 to 22.8.0

Bumps [twisted](https://github.com/twisted/twisted) from 22.4.0 to 22.8.0.
- [Release notes](https://github.com/twisted/twisted/releases)
- [Changelog](https://github.com/twisted/twisted/blob/trunk/NEWS.rst)
- [Commits](https://github.com/twisted/twisted/compare/twisted-22.4.0...twisted-22.8.0)

---
updated-dependencies:
- dependency-name: twisted
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-18 18:06:06 +01:00
Shay 847e2393f3
Prepatory work for adding power level event to batched events (#14214) 2022-10-18 09:58:47 -07:00
dependabot[bot] 2b940d2668
Bump pygithub from 1.55 to 1.56 (#14206)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-18 16:36:31 +00:00
dependabot[bot] f91b547a07
Bump types-setuptools from 65.4.0.0 to 65.5.0.1 (#14208)
* Bump types-setuptools from 65.4.0.0 to 65.5.0.1

Bumps [types-setuptools](https://github.com/python/typeshed) from 65.4.0.0 to 65.5.0.1.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-setuptools
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-18 16:53:34 +01:00
Hugh Nimmo-Smith 4eaf3eb840
Implementation of HTTP 307 response for MSC3886 POST endpoint (#14018)
Co-authored-by: reivilibre <olivier@librepush.net>
Co-authored-by: Andrew Morgan <andrewm@element.io>
2022-10-18 15:52:25 +00:00
David Robertson 844ce47b9b
Don't pin dev-deps in pyproject; use lower bounds (#14227)
* Don't pin dev-deps in pyproject; use lower bounds

This makes it slightly less tedious to update these things via
successive dependabot updates, by reducing the likelihood of a merge
conflict.

* Changelog

* Changelog
2022-10-18 16:44:43 +01:00
David Robertson b951d6bd4c
Fixes to release-artifacts warnings (#14224) 2022-10-18 15:40:05 +00:00
Patrick Cloke dbf18f514e
Update the thread_id right before use (in case the bg update hasn't finished) (#14222)
This avoids running a forced-update of a null thread_id rows.

An index is added (in the background) to hopefully make this
easier in the future.
2022-10-18 14:55:41 +00:00
Jonathan de Jong e440f9674a
Enable URL previews in complement homeserver config. (#14198) 2022-10-18 09:52:23 -04:00
David Robertson 8e50299d8b
Fix `track_memory_usage` on poetry-core 1.3.x installations (#14221)
* Fix `track_memory_usage` on poetry-core 1.3.x installations

The same kind of problem as discussed in #14085:

1. we defined an extra with an underscore
2. we look it up at runtime with an underscore
3. but poetry-core 1.3.x. installs it with a dash, causing (2) to fail.

Fix by using a dash everywhere.

* Changelog
2022-10-18 13:59:04 +01:00
David Robertson a8677bc9b8
Deal with some GHA deprecation warnings (#14216) 2022-10-18 13:45:34 +01:00
realtyem 6c5082f3e0
Flush stdout/err in Dockerfile-workers before replacing the current process (#14195)
Also update `subprocess.check_output` to the slightly newer `subprocess.run`.

Signed-off-by: Jason Little <realtyem@gmail.com>
2022-10-18 11:56:20 +00:00
David Robertson c3a4780080
When restarting a partial join resync, prioritise the server which actioned a partial join (#14126) 2022-10-18 12:33:18 +01:00
Ivan Shapovalov 4af93bd7f6
Allow poetry-core 1.3.2 (#14217)
Signed-off-by: Ivan Shapovalov <intelfx@intelfx.name>
2022-10-18 10:38:58 +01:00
Andrew Morgan dc02d9f8c5
Avoid checking the event cache when backfilling events (#14164) 2022-10-18 10:33:35 +01:00
Andrew Morgan 828b5502cf
Remove `_get_events_cache` check optimisation from `_have_seen_events_dict` (#14161) 2022-10-18 10:33:21 +01:00
Eric Eastwood 2c63cdcc3f
Add debug logs to figure out why an event was filtered (#14095)
Spawned while investigating https://github.com/matrix-org/synapse/issues/13944

This way we might get some more context whenever an `403 Forbidden - body: {"errcode":"M_FORBIDDEN","error":"You don't have permission to access that event."}` error is produced.

`log_config.yaml`
```yaml
loggers:
    synapse:
        level: INFO

    synapse.visibility:
        level: DEBUG
```
2022-10-17 16:02:39 -05:00
Erik Johnston 6fee2f49f3
Cache Rust build cache when building docker images (#14130) 2022-10-17 17:21:14 +00:00
dependabot[bot] c101fc6568
Bump click from 8.1.1 to 8.1.3 (#14201) 2022-10-17 17:54:11 +01:00
Erik Johnston 130668b66c Merge branch 'master' into develop 2022-10-17 17:04:40 +01:00
Patrick Cloke 4283bd1cf9
Support filtering the /messages API by relation type (MSC3874). (#14148)
Gated behind an experimental configuration flag.
2022-10-17 11:32:11 -04:00
dependabot[bot] 6b24235142
Bump psycopg2 from 2.9.3 to 2.9.4 (#14200) 2022-10-17 13:24:17 +00:00
dependabot[bot] 2fa1bf598d
Bump bcrypt from 3.2.0 to 4.0.1 (#14186) 2022-10-17 13:42:18 +01:00
David Robertson ccce8cdfc5
Use Pydantic when PUTting room aliases (#14179) 2022-10-17 13:39:12 +01:00
Nick Mills-Barrett 2c2c3f8b2c
Invalidate rooms for user caches when receiving membership events (#14155)
This should fix a race where the event notification comes in over
replication before the state replication, leaving a window during
which a sync may get an incorrect list of rooms for the user.
2022-10-17 13:27:51 +01:00
David Robertson cd01a1d3b5
Fix dead link to admin registration API (#14189)
* Fix dead link to admin registration API

* Changelog
2022-10-17 12:10:01 +01:00
Dirk Klimpel 1eb8dcf4c9
Remove not needed `replication` listener in docker compose example (#14107) 2022-10-17 12:00:09 +01:00
Erik Johnston 6b097a3e17 1.69.0 2022-10-17 11:31:15 +01:00
Eric Eastwood 40bb37eb27
Stop getting missing `prev_events` after we already know their signature is invalid (#13816)
While https://github.com/matrix-org/synapse/pull/13635 stops us from doing the slow thing after we've already done it once, this PR stops us from doing one of the slow things in the first place.

Related to
 - https://github.com/matrix-org/synapse/issues/13622
    - https://github.com/matrix-org/synapse/pull/13635
 - https://github.com/matrix-org/synapse/issues/13676

Part of https://github.com/matrix-org/synapse/issues/13356

Follow-up to https://github.com/matrix-org/synapse/pull/13815 which tracks event signature failures.

With this PR, we avoid the call to the costly `_get_state_ids_after_missing_prev_event` because the signature failure will count as an attempt before and we filter events based on the backoff before calling `_get_state_ids_after_missing_prev_event` now.

For example, this will save us 156s out of the 185s total that this `matrix.org` `/messages` request. If you want to see the full Jaeger trace of this, you can drag and drop this `trace.json` into your own Jaeger, https://gist.github.com/MadLittleMods/4b12d0d0afe88c2f65ffcc907306b761

To explain this exact scenario around `/messages` -> backfill, we call `/backfill` and first check the signatures of the 100 events. We see bad signature for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` and `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` (both member events). Then we process the 98 events remaining that have valid signatures but one of the events references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event`. So we have to do the whole `_get_state_ids_after_missing_prev_event` rigmarole which pulls in those same events which fail again because the signatures are still invalid.

 - `backfill`
    - `outgoing-federation-request` `/backfill`
    - `_check_sigs_and_hash_and_fetch`
       - `_check_sigs_and_hash_and_fetch_one` for each event received over backfill
          -  `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)`
          -  `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)`
   - `_process_pulled_events`
      - `_process_pulled_event` for each validated event
         -  Event `$Q0iMdqtz3IJYfZQU2Xk2WjB5NDF8Gg8cFSYYyKQgKJ0` references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event` which is missing so we try to get it
            - `_get_state_ids_after_missing_prev_event`
               - `outgoing-federation-request` `/state_ids`
               -  `get_pdu` for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` which fails the signature check again
               -  `get_pdu` for `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` which fails the signature check
2022-10-15 00:36:49 -05:00
dependabot[bot] 616dcc1d18
Bump types-bleach from 4.1.4 to 5.0.3 (#14188)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-14 23:57:55 +00:00
dependabot[bot] 8cd22674a1
Bump types-pillow from 9.2.2 to 9.2.2.1 (#14187)
* Bump types-pillow from 9.2.2 to 9.2.2.1

Bumps [types-pillow](https://github.com/python/typeshed) from 9.2.2 to 9.2.2.1.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-pillow
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-15 00:16:25 +01:00
realtyem c75836fe77
Strip whitespace from worker types in Dockerfile-workers (#14165) 2022-10-14 19:38:04 +00:00
Patrick Cloke bc2bd92b93 Merge remote-tracking branch 'origin/release-v1.69' into develop 2022-10-14 14:11:27 -04:00
Patrick Cloke d1bdeccb50
Accept threaded receipts for events related to the root event. (#14174)
The root node of a thread (and events related to it) are considered
"part of a thread" when validating receipts. This allows clients which
show the root node in both the main timeline and the threaded timeline
to easily send receipts in either.

Note that threaded notifications are not created for these events, these
events created notifications on the main timeline.
2022-10-14 18:05:25 +00:00
Erik Johnston b43be004b4 1.69.0rc4 2022-10-14 15:05:04 +01:00
Erik Johnston d241a1350d
Fix background update to use an index (#14181) 2022-10-14 13:46:23 +00:00
Patrick Cloke 022f25b309
Advertise support for Matrix 1.4. (#14184)
All features / changes in Matrix 1.4 are now supported in
Synapse.
2022-10-14 13:21:55 +00:00
Patrick Cloke 5a983cccfb Fix changelog for #14171 to match #13394/#14175. 2022-10-14 08:56:37 -04:00
Andrew Morgan 97b3d037c0
Don't require optional `invite_room_state` field on fed v2 invite (#14083) 2022-10-14 13:48:33 +01:00
Patrick Cloke 126a15794c
Do not allow a None-limit on PaginationConfig. (#14146)
The callers either set a default limit or manually handle a None-limit
later on (by setting a default value).

Update the callers to always instantiate PaginationConfig with a default
limit and then assume the limit is non-None.
2022-10-14 12:30:05 +00:00
realtyem c7446906bd
Set LD_PRELOAD to load jemalloc in Dockerfile-workers. (#14182) 2022-10-14 13:29:49 +01:00
Patrick Cloke c3e4edb4d6
Stabilize the threads API. (#14175)
Stabilize the threads API (MSC3856) by supporting (only) the v1
path for the endpoint.

This also marks the API as safe for workers since it is a read-only
API.
2022-10-14 07:16:50 -04:00
Patrick Cloke 9ff4155f6c
Properly invalidate get_thread_id cache. (#14163)
This was missed in 2b6d41ebd6 (#13824).
2022-10-14 07:10:44 -04:00