Commit Graph

21646 Commits

Author SHA1 Message Date
Patrick Cloke 581b37b5d6
Revert behavior change for bundling edits of non-message events (#14283) 2022-10-24 17:07:16 +01:00
David Robertson 1e73effebf
Fix typo in cibuildwheel conf introduced in #14253 2022-10-24 13:56:18 +01:00
Erik Johnston 09b588854e
Fix `TypeError: 'dict_keys' object is not reversible` (#14280) 2022-10-24 13:05:14 +01:00
Erik Johnston 1c642156d7
Only build aarch64 wheels for cpython manylinux (#14259) 2022-10-21 14:22:26 +00:00
Erik Johnston 5f77b74215
Try building ABI3 wheels for cpython (#14253) 2022-10-21 14:17:19 +00:00
Patrick Cloke 4dd7aa371b
Properly update the threads table when thread events are redacted. (#14248)
When the last event in a thread is redacted we need to update
the threads table:

* Find the new latest event in the thread and store it into the table; or
* Remove the thread from the table if it is no longer a thread (i.e. all
  events in the thread were redacted).
2022-10-21 09:11:19 -04:00
David Robertson 7fe3b908a5
Update changelog entry for #10015.
Per https://github.com/matrix-org/synapse/pull/10015#issuecomment-1282773048
2022-10-20 17:42:23 +01:00
Erik Johnston 7f6f3ff337 Remove poetry.lock updates 2022-10-19 15:10:55 +01:00
Erik Johnston 6fcdda68d3 Update lockfile 2022-10-19 14:26:21 +01:00
Erik Johnston 15a240f135 1.70.0rc1 2022-10-19 14:12:03 +01:00
Eric Eastwood fa8616e65c
Fix MSC3030 `/timestamp_to_event` returning `outliers` that it has no idea whether are near a gap or not (#14215)
Fix MSC3030 `/timestamp_to_event` endpoint returning `outliers` that it has no idea whether are near a gap or not (and therefore unable to determine whether it's actually the closest event). The reason Synapse doesn't know whether an `outlier` is next to a gap is because our gap checks rely on entries in the `event_edges`, `event_forward_extremeties`, and `event_backward_extremities` tables which is [not the case for `outliers`](2c63cdcc3f/docs/development/room-dag-concepts.md (outliers)).

Also fixes MSC3030 Complement `can_paginate_after_getting_remote_event_from_timestamp_to_event_endpoint` test flake.  Although this acted flakey in Complement, if `sync_partial_state` raced and beat us before `/timestamp_to_event`, then even if we retried the failing `/context` request it wouldn't work until we made this Synapse change. With this PR, Synapse will never return an `outlier` event so that test will always go and ask over federation.

Fix  https://github.com/matrix-org/synapse/issues/13944


### Why did this fail before? Why was it flakey?

Sleuthing the server logs on the [CI failure](https://github.com/matrix-org/synapse/actions/runs/3149623842/jobs/5121449357#step:5:5805), it looks like `hs2:/timestamp_to_event` found `$NP6-oU7mIFVyhtKfGvfrEQX949hQX-T-gvuauG6eurU` as an `outlier` event locally. Then when we went and asked for it via `/context`, since it's an `outlier`, it was filtered out of the results -> `You don't have permission to access that event.`

This is reproducible when `sync_partial_state` races and persists `$NP6-oU7mIFVyhtKfGvfrEQX949hQX-T-gvuauG6eurU` as an `outlier` before we evaluate `get_event_for_timestamp(...)`. To consistently reproduce locally, just add a delay at the [start of `get_event_for_timestamp(...)`](cb20b885cb/synapse/handlers/room.py (L1470-L1496)) so it always runs after `sync_partial_state` completes.

```py
from twisted.internet import task as twisted_task
d = twisted_task.deferLater(self.hs.get_reactor(), 3.5)
await d
```

In a run where it passes, on `hs2`, `get_event_for_timestamp(...)` finds a different event locally which is next to a gap and we request from a closer one from `hs1` which gets backfilled. And since the backfilled event is not an `outlier`, it's returned as expected during `/context`.

With this PR, Synapse will never return an `outlier` event so that test will always go and ask over federation.
2022-10-18 19:46:25 -05:00
Aaron Raimist 2a76a7369f
Fix hiding devices names over federation (#10015)
And don't include blank opentracing stuff in device list updates.

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2022-10-18 20:54:27 +00:00
Shay 1c777ef1e8
Fix docstring in EventContext (#14145) 2022-10-18 13:40:50 -07:00
MichaIng 06b0c4edfe
Add aarch64 wheels to CI (#14212)
Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
2022-10-18 17:12:21 +00:00
dependabot[bot] 85aa0f513b
Bump twisted from 22.4.0 to 22.8.0 (#14207)
* Bump twisted from 22.4.0 to 22.8.0

Bumps [twisted](https://github.com/twisted/twisted) from 22.4.0 to 22.8.0.
- [Release notes](https://github.com/twisted/twisted/releases)
- [Changelog](https://github.com/twisted/twisted/blob/trunk/NEWS.rst)
- [Commits](https://github.com/twisted/twisted/compare/twisted-22.4.0...twisted-22.8.0)

---
updated-dependencies:
- dependency-name: twisted
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-18 18:06:06 +01:00
Shay 847e2393f3
Prepatory work for adding power level event to batched events (#14214) 2022-10-18 09:58:47 -07:00
dependabot[bot] 2b940d2668
Bump pygithub from 1.55 to 1.56 (#14206)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-18 16:36:31 +00:00
dependabot[bot] f91b547a07
Bump types-setuptools from 65.4.0.0 to 65.5.0.1 (#14208)
* Bump types-setuptools from 65.4.0.0 to 65.5.0.1

Bumps [types-setuptools](https://github.com/python/typeshed) from 65.4.0.0 to 65.5.0.1.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-setuptools
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-18 16:53:34 +01:00
Hugh Nimmo-Smith 4eaf3eb840
Implementation of HTTP 307 response for MSC3886 POST endpoint (#14018)
Co-authored-by: reivilibre <olivier@librepush.net>
Co-authored-by: Andrew Morgan <andrewm@element.io>
2022-10-18 15:52:25 +00:00
David Robertson 844ce47b9b
Don't pin dev-deps in pyproject; use lower bounds (#14227)
* Don't pin dev-deps in pyproject; use lower bounds

This makes it slightly less tedious to update these things via
successive dependabot updates, by reducing the likelihood of a merge
conflict.

* Changelog

* Changelog
2022-10-18 16:44:43 +01:00
David Robertson b951d6bd4c
Fixes to release-artifacts warnings (#14224) 2022-10-18 15:40:05 +00:00
Patrick Cloke dbf18f514e
Update the thread_id right before use (in case the bg update hasn't finished) (#14222)
This avoids running a forced-update of a null thread_id rows.

An index is added (in the background) to hopefully make this
easier in the future.
2022-10-18 14:55:41 +00:00
Jonathan de Jong e440f9674a
Enable URL previews in complement homeserver config. (#14198) 2022-10-18 09:52:23 -04:00
David Robertson 8e50299d8b
Fix `track_memory_usage` on poetry-core 1.3.x installations (#14221)
* Fix `track_memory_usage` on poetry-core 1.3.x installations

The same kind of problem as discussed in #14085:

1. we defined an extra with an underscore
2. we look it up at runtime with an underscore
3. but poetry-core 1.3.x. installs it with a dash, causing (2) to fail.

Fix by using a dash everywhere.

* Changelog
2022-10-18 13:59:04 +01:00
David Robertson a8677bc9b8
Deal with some GHA deprecation warnings (#14216) 2022-10-18 13:45:34 +01:00
realtyem 6c5082f3e0
Flush stdout/err in Dockerfile-workers before replacing the current process (#14195)
Also update `subprocess.check_output` to the slightly newer `subprocess.run`.

Signed-off-by: Jason Little <realtyem@gmail.com>
2022-10-18 11:56:20 +00:00
David Robertson c3a4780080
When restarting a partial join resync, prioritise the server which actioned a partial join (#14126) 2022-10-18 12:33:18 +01:00
Ivan Shapovalov 4af93bd7f6
Allow poetry-core 1.3.2 (#14217)
Signed-off-by: Ivan Shapovalov <intelfx@intelfx.name>
2022-10-18 10:38:58 +01:00
Andrew Morgan dc02d9f8c5
Avoid checking the event cache when backfilling events (#14164) 2022-10-18 10:33:35 +01:00
Andrew Morgan 828b5502cf
Remove `_get_events_cache` check optimisation from `_have_seen_events_dict` (#14161) 2022-10-18 10:33:21 +01:00
Eric Eastwood 2c63cdcc3f
Add debug logs to figure out why an event was filtered (#14095)
Spawned while investigating https://github.com/matrix-org/synapse/issues/13944

This way we might get some more context whenever an `403 Forbidden - body: {"errcode":"M_FORBIDDEN","error":"You don't have permission to access that event."}` error is produced.

`log_config.yaml`
```yaml
loggers:
    synapse:
        level: INFO

    synapse.visibility:
        level: DEBUG
```
2022-10-17 16:02:39 -05:00
Erik Johnston 6fee2f49f3
Cache Rust build cache when building docker images (#14130) 2022-10-17 17:21:14 +00:00
dependabot[bot] c101fc6568
Bump click from 8.1.1 to 8.1.3 (#14201) 2022-10-17 17:54:11 +01:00
Erik Johnston 130668b66c Merge branch 'master' into develop 2022-10-17 17:04:40 +01:00
Patrick Cloke 4283bd1cf9
Support filtering the /messages API by relation type (MSC3874). (#14148)
Gated behind an experimental configuration flag.
2022-10-17 11:32:11 -04:00
dependabot[bot] 6b24235142
Bump psycopg2 from 2.9.3 to 2.9.4 (#14200) 2022-10-17 13:24:17 +00:00
dependabot[bot] 2fa1bf598d
Bump bcrypt from 3.2.0 to 4.0.1 (#14186) 2022-10-17 13:42:18 +01:00
David Robertson ccce8cdfc5
Use Pydantic when PUTting room aliases (#14179) 2022-10-17 13:39:12 +01:00
Nick Mills-Barrett 2c2c3f8b2c
Invalidate rooms for user caches when receiving membership events (#14155)
This should fix a race where the event notification comes in over
replication before the state replication, leaving a window during
which a sync may get an incorrect list of rooms for the user.
2022-10-17 13:27:51 +01:00
David Robertson cd01a1d3b5
Fix dead link to admin registration API (#14189)
* Fix dead link to admin registration API

* Changelog
2022-10-17 12:10:01 +01:00
Dirk Klimpel 1eb8dcf4c9
Remove not needed `replication` listener in docker compose example (#14107) 2022-10-17 12:00:09 +01:00
Erik Johnston 6b097a3e17 1.69.0 2022-10-17 11:31:15 +01:00
Eric Eastwood 40bb37eb27
Stop getting missing `prev_events` after we already know their signature is invalid (#13816)
While https://github.com/matrix-org/synapse/pull/13635 stops us from doing the slow thing after we've already done it once, this PR stops us from doing one of the slow things in the first place.

Related to
 - https://github.com/matrix-org/synapse/issues/13622
    - https://github.com/matrix-org/synapse/pull/13635
 - https://github.com/matrix-org/synapse/issues/13676

Part of https://github.com/matrix-org/synapse/issues/13356

Follow-up to https://github.com/matrix-org/synapse/pull/13815 which tracks event signature failures.

With this PR, we avoid the call to the costly `_get_state_ids_after_missing_prev_event` because the signature failure will count as an attempt before and we filter events based on the backoff before calling `_get_state_ids_after_missing_prev_event` now.

For example, this will save us 156s out of the 185s total that this `matrix.org` `/messages` request. If you want to see the full Jaeger trace of this, you can drag and drop this `trace.json` into your own Jaeger, https://gist.github.com/MadLittleMods/4b12d0d0afe88c2f65ffcc907306b761

To explain this exact scenario around `/messages` -> backfill, we call `/backfill` and first check the signatures of the 100 events. We see bad signature for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` and `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` (both member events). Then we process the 98 events remaining that have valid signatures but one of the events references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event`. So we have to do the whole `_get_state_ids_after_missing_prev_event` rigmarole which pulls in those same events which fail again because the signatures are still invalid.

 - `backfill`
    - `outgoing-federation-request` `/backfill`
    - `_check_sigs_and_hash_and_fetch`
       - `_check_sigs_and_hash_and_fetch_one` for each event received over backfill
          -  `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)`
          -  `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)`
   - `_process_pulled_events`
      - `_process_pulled_event` for each validated event
         -  Event `$Q0iMdqtz3IJYfZQU2Xk2WjB5NDF8Gg8cFSYYyKQgKJ0` references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event` which is missing so we try to get it
            - `_get_state_ids_after_missing_prev_event`
               - `outgoing-federation-request` `/state_ids`
               -  `get_pdu` for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` which fails the signature check again
               -  `get_pdu` for `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` which fails the signature check
2022-10-15 00:36:49 -05:00
dependabot[bot] 616dcc1d18
Bump types-bleach from 4.1.4 to 5.0.3 (#14188)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-14 23:57:55 +00:00
dependabot[bot] 8cd22674a1
Bump types-pillow from 9.2.2 to 9.2.2.1 (#14187)
* Bump types-pillow from 9.2.2 to 9.2.2.1

Bumps [types-pillow](https://github.com/python/typeshed) from 9.2.2 to 9.2.2.1.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-pillow
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2022-10-15 00:16:25 +01:00
realtyem c75836fe77
Strip whitespace from worker types in Dockerfile-workers (#14165) 2022-10-14 19:38:04 +00:00
Patrick Cloke bc2bd92b93 Merge remote-tracking branch 'origin/release-v1.69' into develop 2022-10-14 14:11:27 -04:00
Patrick Cloke d1bdeccb50
Accept threaded receipts for events related to the root event. (#14174)
The root node of a thread (and events related to it) are considered
"part of a thread" when validating receipts. This allows clients which
show the root node in both the main timeline and the threaded timeline
to easily send receipts in either.

Note that threaded notifications are not created for these events, these
events created notifications on the main timeline.
2022-10-14 18:05:25 +00:00
Erik Johnston b43be004b4 1.69.0rc4 2022-10-14 15:05:04 +01:00
Erik Johnston d241a1350d
Fix background update to use an index (#14181) 2022-10-14 13:46:23 +00:00