Commit Graph

4568 Commits

Author SHA1 Message Date
Richard van der Hoff 24b590de32
Remove code which updates `application_services_state.last_txn` (#12680)
This column is unused as of #12209, so let's stop writing to it.
2022-05-17 11:07:18 +01:00
David Robertson 1402159bb8
Merge branch 'master' into develop 2022-05-17 11:00:54 +01:00
Erik Johnston 32ef24fbd7
Add index to cache invalidations (#12747)
For workers that rarely write to the cache the `get_all_updated_caches`
query can become expensive if the worker falls behind when reading the
cache.
2022-05-17 09:34:59 +00:00
Erik Johnston fcf951d5dc
Track in memory events using weakrefs (#10533) 2022-05-17 10:34:27 +01:00
David Robertson 1fe202a1a3
Tidy up and type-hint the database engine modules (#12734)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-05-17 00:34:38 +01:00
Andrew Morgan 83be72d76c
Add `StreamKeyType` class and replace string literals with constants (#12567) 2022-05-16 15:35:31 +00:00
Erik Johnston 4ea546067d
Fix query performance for /sync (#12745) 2022-05-16 16:30:35 +01:00
David Robertson b4eb163434 Synapse 1.59.0rc2 (2022-05-16)
==============================
 
 Synapse 1.59 makes several changes that server administrators should be aware of:
 
 - Device name lookup over federation is now disabled by default. ([\#12616](https://github.com/matrix-org/synapse/issues/12616))
 - The `synapse.app.appservice` and `synapse.app.user_dir` worker application types are now deprecated. ([\#12452](https://github.com/matrix-org/synapse/issues/12452), [\#12654](https://github.com/matrix-org/synapse/issues/12654))
 
 See [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1590) for more details.
 
 Additionally, this release removes the non-standard `m.login.jwt` login type from Synapse. It can be replaced with `org.matrix.login.jwt` for identical behaviour. This is only used if `jwt_config.enabled` is set to `true` in the configuration. ([\#12597](https://github.com/matrix-org/synapse/issues/12597))
 
 Bugfixes
 --------
 
 - Fix a bug introduced in Synapse 1.58.0 where `/sync` would fail if the most recent event in a room was rejected. ([\#12729](https://github.com/matrix-org/synapse/issues/12729))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEE1508oLYUKainYFJakD7OEIo53t0FAmKCTJQACgkQkD7OEIo5
 3t1JFw/+NCKc2jBSXQHfcN6cSqfmBFd3sCLDkTiVtQy8k55OW4ioE+qvL7NlMWQW
 4Qon/DpCGV5KV3mJ3CdqaUqaA6B69L68JSUAkDgIlMfPNOsAPHvojp+Pf1uSPApc
 PXjhHhiV/BHRLjNmfoavbWB1XfvVfDXCg/9hCH7+EFktmsCAyyzJSb/oaFLKtP/l
 dUCivzAe2JcQcCSx/cO0zU4NqIoeSymK/ffjvHdEr+FHGR6vyT//7t/71Rm2tbZ6
 6j2utZ4VlT9gXylcQKP37Q2HKMP3pgocDwXWTLk2ZJU47oYc9S+iXGwP75fX900L
 bkmxzx2opKw6LCyvU3sMcw+GHpqfcOG1A6uJEGP6JzVj27xDEOto4EYWkhnSz5Cn
 LMHHSh1ZPNtq/aqm6SxlSdgs6uWpyxTAxeniBItSL80HgtOGCeFbddepKu/hcU1G
 V97AdP5+K4aXC36yVGibsFooBgcXRb+xdzrmQoeYHj4Ky3sybBVkBJ7uS4tp5TB5
 h+YbYSpWLDZy/ldbOL3NAqbjhg+l87QnKHAUhpdU1T48a672mE6hWSRq41N0HyP+
 ldEtgzfyjGFMOQ3lvzZ9HUweivgyBQZKxhXKHTB7IQQDtNAKR14YWx3syA76tJki
 RhOdcNwAmVB8L0BEGBVYWKLnw7qoU838hcFMZFBRAkNAY8xZjXk=
 =iA2S
 -----END PGP SIGNATURE-----
gpgsig -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEE1508oLYUKainYFJakD7OEIo53t0FAmKCV9EACgkQkD7OEIo5
 3t0sIQ//RR816yuZT2Ln4hFyHLxafO7wyzmSCLFMaMixJNPYrq3dwGTz92mjHUY/
 tV0XY3Yjg9YqzHps/eQy9hvhnZao0RZDxoXRPnXlDqMYiHsIe61UmYH0F1E6Nz1U
 xln5T/Sl+iIlfR3NskOxRaMk/PIbQYKHb0tQIDhkQo5KkuZZsrf0t/IL+9RO6Tmy
 DFYOpSs7L5U+NufeO2/xR4jCltLb1zunypjNIOBb0G3LJvyTtRSoFscaYdSFEFEJ
 v9lPfJwhUnKJ/NWiH0CgViboN5Z2Igk1jlPdb8o2p7ketD58w4gGMn67Eq0hotau
 esy8FA/sx5/V+pf8mYDeZNTdgBk03W2A5XxYB4JUsCQChtMaELqMZCL6Vk8hi1oi
 S8WnfFoc9EWAbOh/J4bBv00zzA54vGf8ra1qeirndKDyd5pkZnsMaJxohU9LhEJb
 OSTED1NNZuhuO+phwVx0CtnsSkf2Sz2xc8E5sm891s/dU0cRZmXgCt7rMFreEsD3
 IdzUd1oDQ47JhV8EMntRw1Y8/CDPrml5wZ9On18btxfTPTjUJNtLSaIkZ462/AgZ
 naBk1OlzaPV8QwerDqrxy0c891CVqrr3iJ62luFb40kTyy85Facdfw8OyrCtMmvL
 TEgXM5xaFxfydjX2Sfz3Y8QAAokEzLhVlj3sR93acSEDdzN+Gms=
 =JmDV
 -----END PGP SIGNATURE-----

Merge tag 'v1.59.0rc2' into develop

Synapse 1.59.0rc2 (2022-05-16)
==============================

Synapse 1.59 makes several changes that server administrators should be aware of:

- Device name lookup over federation is now disabled by default. ([\#12616](https://github.com/matrix-org/synapse/issues/12616))
- The `synapse.app.appservice` and `synapse.app.user_dir` worker application types are now deprecated. ([\#12452](https://github.com/matrix-org/synapse/issues/12452), [\#12654](https://github.com/matrix-org/synapse/issues/12654))

See [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1590) for more details.

Additionally, this release removes the non-standard `m.login.jwt` login type from Synapse. It can be replaced with `org.matrix.login.jwt` for identical behaviour. This is only used if `jwt_config.enabled` is set to `true` in the configuration. ([\#12597](https://github.com/matrix-org/synapse/issues/12597))

Bugfixes
--------

- Fix a bug introduced in Synapse 1.58.0 where `/sync` would fail if the most recent event in a room was rejected. ([\#12729](https://github.com/matrix-org/synapse/issues/12729))
2022-05-16 14:55:18 +01:00
Patrick Cloke 86a515ccbf
Consolidate logic for parsing relations. (#12693)
Parse the `m.relates_to` event content field (which describes relations)
in a single place, this is used during:

* Event persistence.
* Validation of the Client-Server API.
* Fetching bundled aggregations.
* Processing of push rules.

Each of these separately implement the logic and each made slightly
different assumptions about what was valid. Some had minor / potential
bugs.
2022-05-16 12:42:45 +00:00
Erik Johnston 8689230a55
Fix bug /sync returning 404 (#12729)
* Fix bug /sync returning 404

Fixes #12571
2022-05-16 12:06:56 +01:00
David Robertson aec69d2481
Another batch of type annotations (#12726) 2022-05-13 12:35:31 +01:00
David Robertson 17e1eb7749
Reduce the number of "untyped defs" (#12716) 2022-05-12 14:33:50 +00:00
Eric Eastwood 84facf769e
Fix `/messages` throwing a 500 when querying for non-existent room (#12683)
Fix https://github.com/matrix-org/synapse/issues/12678

Complement test added:  https://github.com/matrix-org/complement/pull/369

**Before:** 500 internal server error

**After:** According to the [spec](https://spec.matrix.org/latest/client-server-api/#get_matrixclientv3roomsroomidmessages), calling `/messages` against a non-existent `room_id` should throw a 403 forbidden (since you're not part of the room). This also matches the behavior before https://github.com/matrix-org/synapse/pull/12370 which regressed Synapse to the 500 behavior.
```json
{
    "errcode": "M_FORBIDDEN",
    "error": "User @test:my.synapse.server not in room !dne:my.synapse.server, and room previews are disabled"
}
```
2022-05-10 23:39:14 -05:00
Erik Johnston c72d26c1e1
Refactor `EventContext` (#12689)
Refactor how the `EventContext` class works, with the intention of reducing the amount of state we fetch from the DB during event processing.

The idea here is to get rid of the cached `current_state_ids` and `prev_state_ids` that live in the `EventContext`, and instead defer straight to the database (and its caching). 

One change that may have a noticeable effect is that we now no longer prefill the `get_current_state_ids` cache on a state change. However, that query is relatively light, since its just a case of reading a table from the DB (unlike fetching state at an event which is more heavyweight). For deployments with workers this cache isn't even used.


Part of #12684
2022-05-10 19:43:13 +00:00
Dirk Klimpel 989fa33096
Add some type hints to datastore. (#12477) 2022-05-10 14:07:48 -04:00
Richard van der Hoff 147f098fb4
Stop writing to `event_reference_hashes` (#12679)
This table is never read, since #11794. We stop writing to it; in future we can
drop it altogether.
2022-05-10 15:35:08 +01:00
Šimon Brandner ade3008821
Implement MSC3786: Add a default push rule to ignore m.room.server_acl events (#12601)
Fixes vector-im/element-web#20788
Implements matrix-org/matrix-spec-proposals#3786
2022-05-10 08:57:36 +01:00
David Robertson fa0eab9c8e
Use `ParamSpec` in a few places (#12667) 2022-05-09 10:27:39 +00:00
David Robertson 26c1ad71c5
Use `Concatenate` to annotate `do_execute` (#12666) 2022-05-09 10:28:38 +01:00
Patrick Cloke ddc8bba00f
Remove unused receipt datastore methods. (#12632)
The last usage was removed in 5a1dd297c3 (#8059).
2022-05-05 07:51:19 -04:00
Will Hunt 2d74a8c178
Add `mau_appservice_trial_days` config (#12619)
* Add mau_appservice_trial_days

* Add a test

* Tweaks

* changelog

* Ensure we sync after the delay

* Fix types

* Add config statement

* Fix test

* Reinstate logging that got removed

* Fix feature name
2022-05-04 19:33:26 +01:00
Šimon Brandner 116a4c8340
Implement changes to MSC2285 (hidden read receipts) (#12168)
* Changes hidden read receipts to be a separate receipt type
  (instead of a field on `m.read`).
* Updates the `/receipts` endpoint to accept `m.fully_read`.
2022-05-04 11:59:22 -04:00
Patrick Cloke 75dff3dc98
Include bundled aggregations for the latest event in a thread. (#12273)
The `latest_event` field of the bundled aggregations for `m.thread` relations
did not include bundled aggregations itself. This resulted in clients needing to
immediately request the event from the server (and thus making it useless that
the latest event itself was serialized instead of just including an event ID).
2022-05-04 08:38:18 -04:00
andrew do 01e625513a
remove constantly lib use and switch to enums. (#12624) 2022-05-04 11:26:11 +00:00
Richard van der Hoff 96e0cdbc5a
Add a consistency check on events read from the database (#12620)
I've seen a few errors which can only plausibly be explained by the calculated
event id for an event being different from the ID of the event in the
database. It should be cheap to check this, so let's do so and raise an
exception.
2022-05-03 21:27:52 +01:00
Erik Johnston ae7858f184
Fix race when persisting an event and deleting a room (#12594)
This works by taking a row level lock on the `rooms` table at the start of both transactions, ensuring that they don't run at the same time. In the event persistence transaction we also check that there is an entry still in the `rooms` table.

I can't figure out how to do this in SQLite. I was just going to lock the table, but it seems that we don't support that in SQLite either, so I'm *really* confused as to how we maintain integrity in SQLite when using `lock_table`....
2022-05-03 11:47:21 +01:00
Patrick Cloke 3ae56d125c
Improve the docstrings for the receipts store. (#12581) 2022-04-28 17:58:58 +00:00
Brendan Abolivier 5ef673de4f
Add a module API to allow modules to edit push rule actions (#12406)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-04-27 13:55:33 +00:00
David Robertson 6463244375
Remove unused `# type: ignore`s (#12531)
Over time we've begun to use newer versions of mypy, typeshed, stub
packages---and of course we've improved our own annotations. This makes
some type ignore comments no longer necessary. I have removed them.

There was one exception: a module that imports `select.epoll`. The
ignore is redundant on Linux, but I've kept it ignored for those of us
who work on the source tree using not-Linux. (#11771)

I'm more interested in the config line which enforces this. I want
unused ignores to be reported, because I think it's useful feedback when
annotating to know when you've fixed a problem you had to previously
ignore.

* Installing extras before typechecking

Lacking an easy way to install all extras generically, let's bite the bullet and
make install the hand-maintained `all` extra before typechecking.

Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to
the release/v1 branch.
2022-04-27 14:03:44 +01:00
Patrick Cloke 8a23bde823
Consistently use collections.abc.Mapping to check frozendict. (#12564) 2022-04-27 09:00:07 -04:00
Dirk Klimpel b76f1a4d5f
Add some type hints to datastore (#12485) 2022-04-27 13:05:00 +01:00
Erik Johnston f59e3f4c90
Mark remote device list updates as already handled (#12557) 2022-04-26 17:07:21 +01:00
Erik Johnston c48ab3734e
Fix sending opentracing contexts to remote servers (#12555) 2022-04-26 14:48:16 +00:00
Richard van der Hoff 17d99f758a
Optimise backfill calculation (#12522)
Try to avoid an OOM by checking fewer extremities.

Generally this is a big rewrite of _maybe_backfill, to try and fix some of the TODOs and other problems in it. It's best reviewed commit-by-commit.
2022-04-26 10:27:11 +01:00
Sean Quah 8a87b4435a
Handle cancellation in `EventsWorkerStore._get_events_from_cache_or_db` (#12529)
Multiple calls to `EventsWorkerStore._get_events_from_cache_or_db` can
reuse the same database fetch, which is initiated by the first call.
Ensure that cancelling the first call doesn't cancel the other calls
sharing the same database fetch.

Signed-off-by: Sean Quah <seanq@element.io>
2022-04-25 19:39:17 +01:00
Sean Quah a50fb411b3
Update `delay_cancellation` to accept any awaitable (#12468)
This will mainly be useful when dealing with module callbacks, which are
all typed as returning `Awaitable`s instead of coroutines or
`Deferred`s.

Signed-off-by: Sean Quah <seanq@element.io>
2022-04-22 18:20:06 +01:00
Richard van der Hoff f5668f0b4a
Await un-partial-stating after a partial-state join (#12399)
When we join a room via the faster-joins mechanism, we end up with "partial
state" at some points on the event DAG. Many parts of the codebase need to
wait for the full state to load. So, we implement a mechanism to keep track of
which events have partial state, and wait for them to be fully-populated.
2022-04-21 07:42:03 +01:00
Tulir Asokan 4bc8cb4669
Implement MSC2815: allow room moderators to view redacted event content (#12427)
Implements matrix-org/matrix-spec-proposals#2815

Signed-off-by: Tulir Asokan <tulir@maunium.net>
2022-04-20 12:57:39 +01:00
Erik Johnston c1482a352a
Fix returned count of delete extremities admin API (#12496) 2022-04-19 16:49:45 +01:00
Travis Ralston f8d3ee9570
Fix grammatical error in error message (#12483)
* Fix grammatical error in error message

* changelog
2022-04-18 12:41:55 -06:00
Erik Johnston 0b014eb25e
Only send out device list updates for our own users (#12465)
Broke in #12365
2022-04-14 13:05:31 +01:00
Nick Mills-Barrett e3a49f4784
Fix missing sync events during historical batch imports (#12319)
Discovered after much in-depth investigation in #12281.

Closes: #12281
Closes: #3305

Signed off by: Nick Mills-Barrett nick@beeper.com
2022-04-13 11:38:35 +01:00
Erik Johnston aa28110264
Process device list updates asynchronously (#12365) 2022-04-12 16:50:40 +01:00
Richard van der Hoff 320186319a
Resync state after partial-state join (#12394)
We work through all the events with partial state, updating the state at each
of them. Once it's done, we recalculate the state for the whole room, and then
mark the room as having complete state.
2022-04-12 13:23:43 +00:00
Patrick Cloke 86cf6a3a17
Remove references to unstable identifiers from MSC3440. (#12382)
Removes references to unstable thread relation, unstable
identifiers for filtering parameters, and the experimental
config flag.
2022-04-12 08:42:03 -04:00
Dirk Klimpel 1783156dbc
Add some type hints to datastore (#12423)
* Add some type hints to datastore

* newsfile

* change `Collection` to `List`

* refactor return type of `select_users_txn`

* correct type hint in `stream.py`

* Remove `Optional` in `select_users_txn`

* remove not needed return type in `__init__`

* Revert change in `get_stream_id_for_event_txn`

* Remove import from `Literal`
2022-04-12 11:54:00 +01:00
Patrick Cloke 772bad2562
Do not consider events by ignored users for bundled aggregations (#12235)
Consider the requester's ignored users when calculating the
bundled aggregations.

See #12285 / 4df10d3214
for corresponding changes for the `/relations` endpoint.
2022-04-11 10:09:57 -04:00
David Robertson 961ee75a9b
Disallow untyped defs in synapse._scripts (#12422)
Of note: 

* No untyped defs in `register_new_matrix_user`

This one might be contraversial. `request_registration` has three
dependency-injection arguments used for testing. I'm removing the
injection of the `requests` module and using `unitest.mock.patch` in the
test cases instead.

Doing `reveal_type(requests)` and `reveal_type(requests.get)` before the
change:

```
synapse/_scripts/register_new_matrix_user.py:45: note: Revealed type is "Any"
synapse/_scripts/register_new_matrix_user.py:46: note: Revealed type is "Any"
```

And after:

```
synapse/_scripts/register_new_matrix_user.py:44: note: Revealed type is "types.ModuleType"
synapse/_scripts/register_new_matrix_user.py:45: note: Revealed type is "def (url: Union[builtins.str, builtins.bytes], params: Union[Union[_typeshed.SupportsItems[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]], Tuple[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]], typing.Iterable[Tuple[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]]], builtins.str, builtins.bytes], None] =, data: Union[Any, None] =, headers: Union[Any, None] =, cookies: Union[Any, None] =, files: Union[Any, None] =, auth: Union[Any, None] =, timeout: Union[Any, None] =, allow_redirects: builtins.bool =, proxies: Union[Any, None] =, hooks: Union[Any, None] =, stream: Union[Any, None] =, verify: Union[Any, None] =, cert: Union[Any, None] =, json: Union[Any, None] =) -> requests.models.Response"
```

* Drive-by comment in `synapse.storage.types`

* No untyped defs in `synapse_port_db`

This was by far the most painful. I'm happy to break this up into
smaller pieces for review if it's not managable as-is.
2022-04-11 12:41:55 +01:00
reivilibre e630722f11
Optimise `_update_client_ips_batch_txn` to batch together database operations. (#12252)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-04-08 15:29:13 +01:00
Richard van der Hoff 6fe757d69e
Fix `synapse_event_persisted_position` metric (#12390)
Fixes a bug introduced in #11417 where we would only included backfilled events
in `synapse_event_persisted_position`
2022-04-06 13:52:39 +00:00
Richard van der Hoff ae01a7edd3
Update type annotations for compatiblity with prometheus_client 0.14 (#12389)
Principally, `prometheus_client.REGISTRY.register` now requires its argument to
extend `prometheus_client.Collector`.

Additionally, `Gauge.set` is now annotated so that passing `Optional[int]`
causes an error.
2022-04-06 12:59:04 +00:00
Eric Eastwood 793d03e2c5
Generate historic pagination token for `/messages` when no `?from` token provided (#12370) 2022-04-06 11:40:28 +01:00
Sean Quah 800ba87cc8
Refactor and convert `Linearizer` to async (#12357)
Refactor and convert `Linearizer` to async. This makes a `Linearizer`
cancellation bug easier to fix.

Also refactor to use an async context manager, which eliminates an
unlikely footgun where code that doesn't immediately use the context
manager could forget to release the lock.

Signed-off-by: Sean Quah <seanq@element.io>
2022-04-05 15:43:52 +01:00
Erik Johnston 66053b6bfb
Prefill more stream change caches. (#12372) 2022-04-05 14:26:41 +01:00
Erik Johnston b446c99ac9
Prefill the device_list_stream_cache (#12367)
* Prefill the device_list_stream_cache

* Newsfile

* Newsfile
2022-04-04 20:12:25 +01:00
Erik Johnston 5c9e39e619
Track device list updates per room. (#12321)
This is a first step in dealing with #7721.

The idea is basically that rather than calculating the full set of users a device list update needs to be sent to up front, we instead simply record the rooms the user was in at the time of the change. This will allow a few things:

1. we can defer calculating the set of remote servers that need to be poked about the change; and
2. during `/sync` and `/keys/changes` we can avoid also avoid calculating users who share rooms with other users, and instead just look at the rooms that have changed.

However, care needs to be taken to correctly handle server downgrades. As such this PR writes to both `device_lists_changes_in_room` and the `device_lists_outbound_pokes` table synchronously. In a future release we can then bump the database schema compat version to `69` and then we can assume that the new `device_lists_changes_in_room` exists and is handled.

There is a temporary option to disable writing to `device_lists_outbound_pokes` synchronously, allowing us to test the new code path does work (and by implication upgrading to a future release and downgrading to this one will work correctly).

Note: Ideally we'd do the calculation of room to servers on a worker (e.g. the background worker), but currently only master can write to the `device_list_outbound_pokes` table.
2022-04-04 15:25:20 +01:00
Nick Mills-Barrett 993d90f82b
Use a sequence to generate AS transaction IDs, drop `last_txn` AS state (#12209)
Switching to a sequence means there's no need to track `last_txn` on the
AS state table to generate new TXN IDs. This also means that there is
no longer contention between the AS scheduler and AS handler on updates
to the `application_services_state` table, which will prevent serialization
errors during the complete AS txn transaction.
2022-04-01 13:33:25 +00:00
reivilibre f871222880
Move `update_client_ip` background job from the main process to the background worker. (#12251) 2022-04-01 13:08:55 +01:00
Richard van der Hoff 319a805cd3
Raise an exception when getting state at an outlier (#12191)
It seems like calling `_get_state_group_for_events` for an event where the
state is unknown is an error. Accordingly, let's raise an exception rather than
silently returning an empty result.
2022-04-01 13:01:49 +01:00
Richard van der Hoff 9b43df1f7b
Optimise `_get_state_after_missing_prev_event`: use `/state` (#12040)
If we're missing most of the events in the room state, then we may as well call the /state endpoint, instead of individually requesting each and every event.
2022-04-01 12:53:42 +01:00
Patrick Cloke 11df4ec6c2
Add more type hints to the main state store. (#12267) 2022-03-31 13:38:09 -04:00
Patrick Cloke adbf975623
Remove an unnecessary class from the relations code. (#12338)
The PaginationChunk class attempted to bundle some properties
together, but really just caused callers to jump through hoops and
hid implementation details.
2022-03-31 07:13:49 -04:00
Patrick Cloke c31c1091d4
Remove the unused and unstable `/aggregations` endpoint. (#12293)
This endpoint was removed from MSC2675 before it was approved.
It is currently unspecified (even in any MSCs) and therefore subject to
removal. It is not implemented by any known clients.

This also changes the bundled aggregation format for `m.annotation`,
which previously included pagination tokens for the `/aggregations`
endpoint, which are no longer useful.
2022-03-30 11:45:32 -04:00
Andrew Morgan d8d0271977
Send device list updates to application services (MSC3202) - part 1 (#11881)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-03-30 14:39:27 +01:00
Andrew Morgan c8cbd66d3b
Start application service stream token tracking from 1 (#12193)
Co-authored-by: Erik Johnston <erik@matrix.org>
2022-03-30 11:10:12 +00:00
Brendan Abolivier 437a8ed9ef
Add a configuration to exclude rooms from sync response (#12310) 2022-03-30 09:43:04 +00:00
Sean Quah 8a519f8abc
Update `LoggingTransaction.call_after` and `call_on_exception` docstrings (#12315)
Document the behaviour of `LoggingTransaction.call_after` and
`LoggingTransaction.call_on_exception` when transactions are retried.

Signed-off-by: Sean Quah <seanq@element.io>
2022-03-29 12:31:05 +01:00
David Robertson a2b00a4486
Bump `black` and `click` versions (#12320) 2022-03-29 10:41:19 +00:00
Richard van der Hoff 287a9c1e20
Exclude outliers in `on_backfill_request` (#12314)
When we are processing a `/backfill` request from a remote server, exclude any
outliers from consideration early on. We can't return outliers anyway (since we
don't know the state at the outlier), and filtering them out earlier means that
we won't attempt to calulate the state for them.
2022-03-28 19:20:14 +01:00
Dirk Klimpel ac95167d2f
Add some type hints to datastore. (#12255) 2022-03-28 14:11:14 -04:00
reivilibre 89f11f8c6f
Improve type annotations for `execute_values`. (#12311) 2022-03-28 17:21:23 +01:00
Erik Johnston 7ca8ee67a5
Add cache for `get_membership_from_event_ids` (#12272)
This should speed up push rule calculations for rooms with large numbers of local users when the main push rule cache fails.

Co-authored-by: reivilibre <oliverw@matrix.org>
2022-03-25 14:58:56 +00:00
Shay e78d4f61fc
Refuse to start if DB has an unsafe locale (#12262) 2022-03-23 10:23:05 -07:00
David Robertson f4c5e5864c
Use psycopg2 type stubs (#12269) 2022-03-23 14:03:24 +00:00
Jonathan de Jong 516d092ff9
Rename shared_rooms to mutual_rooms (#12036)
Co-authored-by: reivilibre <olivier@librepush.net>
2022-03-23 11:19:20 +00:00
Patrick Cloke 8fe930c215
Move get_bundled_aggregations to relations handler. (#12237)
The get_bundled_aggregations code is fairly high-level and uses
a lot of store methods, we move it into the handler as that seems
like a better fit.
2022-03-18 17:49:32 +00:00
Patrick Cloke 80e0e1f35e
Only fetch thread participation for events with threads. (#12228)
We fetch the thread summary in two phases:

1. The summary that is shared by all users (count of messages and latest event).
2. Whether the requesting user has participated in the thread.

There's no use in attempting step 2 for events which did not return a summary
from step 1.
2022-03-18 17:15:45 +00:00
Dirk Klimpel c46065fa3d
Add some type hints to datastore (#12248)
* inherit `MonthlyActiveUsersStore` from `RegistrationWorkerStore`

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-03-18 15:24:18 +00:00
David Robertson 872dbb0181
Correct `check_username_for_spam` annotations and docs (#12246)
* Formally type the UserProfile in user searches
* export UserProfile in synapse.module_api
* Update docs

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-03-18 13:51:41 +00:00
Sean Quah 6121056740
Handle cancellation in `DatabasePool.runInteraction()` (#12199)
To handle cancellation, we ensure that `after_callback`s and
`exception_callback`s are always run, since the transaction will
complete on another thread regardless of cancellation.

We also wait until everything is done before releasing the
`CancelledError`, so that logging contexts won't get used after they
have been finished.

Signed-off-by: Sean Quah <seanq@element.io>
2022-03-16 15:07:41 +00:00
Patrick Cloke c486fa5fd9
Add some missing type hints to cache datastore. (#12216) 2022-03-16 10:37:04 -04:00
Patrick Cloke dda9b7fc4d
Use the ignored_users table to test event visibility & sync. (#12225)
Instead of fetching the raw account data and re-parsing it. The
ignored_users table is a denormalised version of the account data
for quick searching.
2022-03-15 14:06:05 -04:00
Sean Quah 8e5706d144
Fix broken background updates when using sqlite with `enable_search` off (#12215)
Signed-off-by: Sean Quah <seanq@element.io>
2022-03-14 17:52:58 +00:00
Shay ef3619e61d
Add config settings for background update parameters (#11980) 2022-03-11 10:46:45 -08:00
Patrick Cloke bc9dff1d95
Remove unnecessary pass statements. (#12206) 2022-03-11 07:06:21 -05:00
Patrick Cloke ea27528b5d
Support stable identifiers for MSC3440: Threading (#12151)
The unstable identifiers are still supported if the experimental configuration
flag is enabled. The unstable identifiers will be removed in a future release.
2022-03-10 15:36:13 +00:00
Patrick Cloke 88cd6f9378
Allow retrieving the relations of a redacted event. (#12130)
This is allowed per MSC2675, although the original implementation did
not allow for it and would return an empty chunk / not bundle aggregations.

The main thing to improve is that the various caches get cleared properly
when an event is redacted, and that edits must not leak if the original
event is redacted (as that would presumably leak something similar to
the original event content).
2022-03-10 09:03:59 -05:00
Patrick Cloke 690cb4f3b3
Allow for ignoring some arguments when caching. (#12189)
* `@cached` can now take an `uncached_args` which is an iterable of names to not use in the cache key.
* Requires `@cached`, @cachedList` and `@lru_cache` to use keyword arguments for clarity.
* Asserts that keyword-only arguments in cached functions are not accepted. (I tested this briefly and I don't believe this works properly.)
2022-03-09 18:07:41 +00:00
Patrick Cloke 032688854b
Remove some unused variables/parameters. (#12187) 2022-03-09 15:29:39 +00:00
Shay 26211fec24
Fix a bug in background updates wherein background updates are never run using the default batch size (#12157) 2022-03-07 09:44:33 -08:00
Patrick Cloke f63bedef07
Invalidate caches when an event with a relation is redacted. (#12121)
The caches for the target of the relation must be cleared
so that the bundled aggregations are re-calculated after
the redaction is processed.
2022-03-07 14:00:05 +00:00
Erik Johnston 0752ab7a36
Reduce to-device queries for /sync. (#12163) 2022-03-04 17:57:27 +00:00
Patrick Cloke cd1ae3d0b4
Remove backwards compatibility with RelationPaginationToken. (#12138) 2022-03-04 07:10:10 -05:00
Richard van der Hoff d56202b038
Fix type of `events` in `StateGroupStorage` and `StateHandler` (#12156)
We make multiple passes over this, so a regular iterable won't do.
2022-03-04 10:25:18 +00:00
reivilibre c7b2f1ccdc
Back out in-flight state caching changes. (#12126) 2022-03-02 10:37:04 +00:00
lukasdenk 8e56a1b73c
Make get_room_version use cached get_room_version_id. (#11808) 2022-03-02 10:35:34 +00:00
reivilibre c893632319
Order in-flight state group queries in biggest-first order (#11610)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-03-01 13:41:57 +00:00
Richard van der Hoff e2e1d90a5e
Faster joins: persist to database (#12012)
When we get a partial_state response from send_join, store information in the
database about it:
 * store a record about the room as a whole having partial state, and stash the
   list of member servers too.
 * flag the join event itself as having partial state
 * also, for any new events whose prev-events are partial-stated, note that
   they will *also* be partial-stated.

We don't yet make any attempt to interpret this data, so API calls (and a bunch
of other things) are just going to get incorrect data.
2022-03-01 12:49:54 +00:00
Sean Quah b43c3ef8e2
Ensure that `get_datastores().main` is typed (#12070)
Signed-off-by: Sean Quah <seanq@element.io>
2022-02-25 10:20:40 +00:00
Sean Quah f3fd8558cd
Minor typing fixes for `synapse/storage/persist_events.py` (#12069)
Signed-off-by: Sean Quah <seanq@element.io>
2022-02-25 10:19:49 +00:00
reivilibre 2cc5ea933d
Add support for MSC3202: sending one-time key counts and fallback key usage states to Application Services. (#11617)
Co-authored-by: Erik Johnston <erik@matrix.org>
2022-02-24 17:55:45 +00:00
Sean Quah 41cf4c2cf6
Fix non-strings in the `event_search` table (#12037)
Don't attempt to add non-string `value`s to `event_search` and add a
background update to clear out bad rows from `event_search` when
using sqlite.

Signed-off-by: Sean Quah <seanq@element.io>
2022-02-24 11:52:28 +00:00
Richard van der Hoff e24ff8ebe3
Remove `HomeServer.get_datastore()` (#12031)
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.

Part of #11733
2022-02-23 11:04:02 +00:00
reivilibre dcb6a37837
Cap the number of in-flight requests for state from a single group (#11608) 2022-02-22 14:24:31 +00:00
reivilibre 235d2916ce
Fix slow performance of `/logout` in some cases where refresh tokens are in use. The slowness existed since the initial implementation of refresh tokens. (#12056) 2022-02-22 13:29:04 +00:00
Richard van der Hoff 3070af4809
remote join processing: get create event from state, not auth_chain (#12039)
A follow-up to #12005, in which I apparently missed that there are a bunch of other places that assume the create event is in the auth chain.
2022-02-21 19:27:35 +00:00
Dirk Klimpel 7c82da27aa
Add type hints to `synapse/storage/databases/main` (#11984) 2022-02-21 16:03:06 +00:00
Patrick Cloke 444b04058b
Document why auth providers aren't validated in the admin API. (#12004)
Since it is reasonable to give a future or past auth provider,
which might not be in the current configuration.
2022-02-18 17:24:25 +00:00
reivilibre 284ea2025a
Track and deduplicate in-flight requests to `_get_state_for_groups`. (#10870)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-02-18 17:23:31 +00:00
reivilibre eb609c65d0
Fix bug in `StateFilter.return_expanded()` and add some tests. (#12016) 2022-02-18 14:54:31 +00:00
Eric Eastwood 5a6911598a
Fix 500 error with Postgres when looking backwards with the MSC3030 `/timestamp_to_event` endpoint (#12024) 2022-02-18 12:11:18 +00:00
Patrick Cloke 7a92d68441 Fix a typo in a comment. 2022-02-16 06:53:21 -05:00
Erik Johnston 2b5643b3af
Optimise calculating device_list changes in `/sync`. (#11974)
For users with large accounts it is inefficient to calculate the set of
users they share a room with (and takes a lot of space in the cache).
Instead we can look at users whose devices have changed since the last
sync and check if they share a room with the syncing user.
2022-02-15 15:01:00 +00:00
Erik Johnston dc9fe61050
Fix incorrect `get_rooms_for_user` for remote user (#11999)
When the server leaves a room the `get_rooms_for_user` cache is not
correctly invalidated for the remote users in the room. This means that
subsequent calls to `get_rooms_for_user` for the remote users would
incorrectly include the room (it shouldn't be included because the
server no longer knows anything about the room).
2022-02-15 14:26:28 +00:00
Patrick Cloke e44f91d678
Refactor search code to reduce function size. (#11991)
Splits the search code into a few logical functions instead of a single
unreadable function.

There are also a few additional changes for readability.

After refactoring it was clear to see there were some unused and
unnecessary variables, which were simplified.
2022-02-15 13:47:05 +00:00
Patrick Cloke 45f45404de
Fix incorrect thread summaries when the latest event is edited. (#11992)
If the latest event in a thread was edited than the original
event content was included in bundled aggregation for
threads instead of the edited event content.
2022-02-15 08:26:57 -05:00
Richard van der Hoff 63c46349c4
Implement MSC3706: partial state in `/send_join` response (#11967)
* Make `get_auth_chain_ids` return a Set

It has a set internally, and a set is often useful where it gets used, so let's
avoid converting to an intermediate list.

* Minor refactors in `on_send_join_request`

A little bit of non-functional groundwork

* Implement MSC3706: partial state in /send_join response
2022-02-12 10:44:16 +00:00
Patrick Cloke b65acead42
Fetch thread summaries for multiple events in a single query (#11752)
This should reduce database usage when fetching bundled aggregations
as the number of individual queries (and round trips to the database) are
reduced.
2022-02-11 09:50:14 -05:00
Erik Johnston 79fb64e417
Fix to-device being dropped in limited sync in SQLite. (#11966)
If ther are more than 100 to-device messages pending for a device
`/sync` will only return the first 100, however the next batch token was
incorrectly calculated and so all other pending messages would be
dropped.

This is due to `txn.rowcount` only returning the number of rows that
*changed*, rather than the number *selected* in SQLite.
2022-02-11 13:38:05 +00:00
Patrick Cloke df36945ff0
Support pagination tokens from /sync and /messages in the relations API. (#11952) 2022-02-10 10:52:48 -05:00
Patrick Cloke 8c94b3abe9
Experimental support to include bundled aggregations in search results (MSC3666) (#11837) 2022-02-08 09:21:20 -05:00
Patrick Cloke 8b309adb43
Fetch edits for multiple events in a single query. (#11660)
This should reduce database usage when fetching bundled aggregations
as the number of individual queries (and round trips to the database) are
reduced.
2022-02-08 07:43:30 -05:00
Andrew Morgan 3655585e85
Add a docstring to `add_device_change_to_streams` and fix some nearby types (#11912) 2022-02-08 10:52:22 +00:00
Eric Eastwood fef2e792be
Fix historical messages backfilling in random order on remote homeservers (MSC2716) (#11114)
Fix https://github.com/matrix-org/synapse/issues/11091
Fix https://github.com/matrix-org/synapse/issues/10764 (side-stepping the issue because we no longer have to deal with `fake_prev_event_id`)

 1. Made the `/backfill` response return messages in `(depth, stream_ordering)` order (previously only sorted by `depth`)
    - Technically, it shouldn't really matter how `/backfill` returns things but I'm just trying to make the `stream_ordering` a little more consistent from the origin to the remote homeservers in order to get the order of messages from `/messages` consistent ([sorted by `(topological_ordering, stream_ordering)`](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)).
    - Even now that we return backfilled messages in order, it still doesn't guarantee the same `stream_ordering` (and more importantly the [`/messages` order](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)) on the other server. For example, if a room has a bunch of history imported and someone visits a permalink to a historical message back in time, their homeserver will skip over the historical messages in between and insert the permalink as the next message in the `stream_order` and totally throw off the sort.
       - This will be even more the case when we add the [MSC3030 jump to date API endpoint](https://github.com/matrix-org/matrix-doc/pull/3030) so the static archives can navigate and jump to a certain date.
       - We're solving this in the future by switching to [online topological ordering](https://github.com/matrix-org/gomatrixserverlib/issues/187) and [chunking](https://github.com/matrix-org/synapse/issues/3785) which by its nature will apply retroactively to fix any inconsistencies introduced by people permalinking
 2. As we're navigating `prev_events` to return in `/backfill`, we order by `depth` first (newest -> oldest) and now also tie-break based on the `stream_ordering` (newest -> oldest). This is technically important because MSC2716 inserts a bunch of historical messages at the same `depth` so it's best to be prescriptive about which ones we should process first. In reality, I think the code already looped over the historical messages as expected because the database is already in order.
 3. Making the historical state chain and historical event chain float on their own by having no `prev_events` instead of a fake `prev_event` which caused backfill to get clogged with an unresolvable event. Fixes https://github.com/matrix-org/synapse/issues/11091 and https://github.com/matrix-org/synapse/issues/10764
 4. We no longer find connected insertion events by finding a potential `prev_event` connection to the current event we're iterating over. We now solely rely on marker events which when processed, add the insertion event as an extremity and the federating homeserver can ask about it when time calls.
    - Related discussion, https://github.com/matrix-org/synapse/pull/11114#discussion_r741514793


Before | After
--- | ---
![](https://user-images.githubusercontent.com/558581/139218681-b465c862-5c49-4702-a59e-466733b0cf45.png) | ![](https://user-images.githubusercontent.com/558581/146453159-a1609e0a-8324-439d-ae44-e4bce43ac6d1.png)



#### Why aren't we sorting topologically when receiving backfill events?

> The main reason we're going to opt to not sort topologically when receiving backfill events is because it's probably best to do whatever is easiest to make it just work. People will probably have opinions once they look at [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) which could change whatever implementation anyway.
> 
> As mentioned, ideally we would do this but code necessary to make the fake edges but it gets confusing and gives an impression of “just whyyyy” (feels icky). This problem also dissolves with online topological ordering.
>
> -- https://github.com/matrix-org/synapse/pull/11114#discussion_r741517138

See https://github.com/matrix-org/synapse/pull/11114#discussion_r739610091 for the technical difficulties
2022-02-07 15:54:13 -06:00
Patrick Cloke a8da046907
Invalidate the get_users_in_room{_with_profile} caches only when necessary. (#11878)
The get_users_in_room and get_users_in_room_with_profiles
are now only invalidated when the  membership of a room changes,
instead of during any state change in the room.
2022-02-02 12:24:07 -05:00
Patrick Cloke acda9f07c8
Revert experimental push rules from #7997. (#11884)
Manually reverts the merge from cdbb8e6d6e.
2022-02-02 09:49:31 -05:00
reivilibre af795173be
Add a background database update to purge account data for deactivated users. (#11655) 2022-02-02 11:37:18 +00:00
Andrew Morgan 64ec45fc1b
Send to-device messages to application services (#11215)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-02-01 14:13:38 +00:00
Patrick Cloke 02755c3188
Remove the obsolete MSC1849 configuration flag. (#11843)
MSC1849 was replaced by MSC2675, which was merged.
The configuration flag, which defaulted to true, is no
longer useful.
2022-01-31 10:13:32 -05:00
Brendan Abolivier 6d482ba259
Pass `isolation_level` to `runWithConnection` (#11847)
This was missed in https://github.com/matrix-org/synapse/pull/11799
2022-01-27 17:45:39 +00:00
Richard van der Hoff 57e4786e90
Create singletons for `StateFilter.{all,none}()` (#11836)
No point recreating these for each call, since they are frozen
2022-01-27 10:54:27 +00:00
Vaishnav Nair cef0d5d90a
Include `prev_content` field in AS events (#11798)
* Include 'prev_content' field in AS events

Signed-off-by: Vaishnav Nair <nairvaishnav007@icloud.com>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-01-26 14:48:27 +00:00
Patrick Cloke 2897fb6b4f
Improvements to bundling aggregations. (#11815)
This is some odds and ends found during the review of #11791
and while continuing to work in this code:

* Return attrs classes instead of dictionaries from some methods
  to improve type safety.
* Call `get_bundled_aggregations` fewer times.
* Adds a missing assertion in the tests.
* Do not return empty bundled aggregations for an event (preferring
  to not include the bundle at all, as the docstring states).
2022-01-26 08:27:04 -05:00
Dirk Klimpel 6a72c910f1
Add admin API to get a list of federated rooms (#11658) 2022-01-25 16:11:40 +00:00
Nick Barrett b59d285f7c
Db txn set isolation level (#11799)
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-01-25 15:14:46 +01:00
Richard van der Hoff fc8598bc87
Minor updates, and docs, for schema delta files (#11823)
* Make functions in python deltas optional

It's annoying to always have to write stubs for these.

* Documentation for delta files

* changelog
2022-01-25 14:11:13 +00:00
Andrew Morgan c3040dd5cc Synapse 1.51.0rc2 (2022-01-24)
==============================
 
 Bugfixes
 --------
 
 - Fix a bug introduced in Synapse 1.40.0 that caused Synapse to fail to process incoming federation traffic after handling a large amount of events in a v1 room. ([\#11806](https://github.com/matrix-org/synapse/issues/11806))
 -----BEGIN PGP SIGNATURE-----
 
 iQJHBAABCAAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmHum5QTHGFuZHJld0Bh
 bW9yZ2FuLnh5egAKCRCIhIgNLv5f9G6vD/9Dw6V0687vzahnU6aYWaecRSO1sbww
 EtcCiXOh0r8HXPAwEIXJYomSTTbPl0eAwP8T1the1WZVQArvRW2VvMzDoBheO8bt
 dCm4CTKFCGUsI/GrlFDPtjAEd6kruAETmpHZn7bAFSqtIpissD6FjUwg/ND/NJfM
 zVM/bMgM0+js6eD9J5/k16E1ZWj7Lbp+/fKN+qTeQrXzIeT13WQZ4Nz1o/cqe/21
 o/coI3FmYq8CzKpfA6qZMDd/OtYpYqwwr7otSmW+6qHeZI/yqeoxlpzfgSZGpRUe
 mtXqbQllZQeqrbm8oK96GNmKcThy6awTwJPoD46b1AOkmFdf/lGSqO0lnTQRRPqR
 hyPPdrx6Lt2t7DDbVVyUElzkqLPhVJtrItPDC8669sWvmSgsPQdUqIsRmhF+8aJe
 ffjRvKbGRICnNZkT+qGf1HwuMHajZyAIHAS/kyFDKUZCvau3VQ1wlyOqVQ1hWr7+
 3k3CobjSx4y1bYKXncAK6hWH6lE8M319jaTnVfYXocDLWRonyFf7o286Q+c8WBGF
 tY0FzvUPb5S2kktC4WLwcqtcTWK1cu5MI6GfD69EqL7iifJRVyKDeUoD7tiUvgzN
 O2wBl2soJIsAU+8y16WQu7p+k2nZIomCCAySnW3C8mlxvv7CKQu0Url8J7IH8uZE
 ZVN2MMe7H+ZHJQ==
 =Fpsa
 -----END PGP SIGNATURE-----

Merge tag 'v1.51.0rc2' into develop

Synapse 1.51.0rc2 (2022-01-24)
==============================

Bugfixes
--------

- Fix a bug introduced in Synapse 1.40.0 that caused Synapse to fail to process incoming federation traffic after handling a large amount of events in a v1 room. ([\#11806](https://github.com/matrix-org/synapse/issues/11806))
2022-01-24 13:55:03 +00:00
reivilibre df54c8485a
Remove account data (including client config, push rules and ignored users) upon user deactivation. (#11621)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-01-24 13:37:00 +00:00
Andrew Morgan dc671d3ea7
Fix logic for dropping old events in fed queue (#11806)
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
Co-authored-by: Richard van der Hoff <richard@matrix.org>
2022-01-24 12:20:01 +00:00
Shay 9006ee36d1
Drop support for and remove references to EOL Python 3.6 (#11683)
* remove reference in comments to python3.6

* upgrade tox python env in script

* bump python version in example for completeness

* upgrade python version requirement in setup doc

* upgrade necessary python version in __init__.py

* upgrade python version in setup.py

* newsfragment

* drops refs to bionic and replace with focal

* bump refs to postgres 9.6 to 10

* fix hanging ci

* try installing tzdata first

* revert change made in b979f336

* ignore new random mypy error while debugging other error

* fix lint error for temporary workaround

* revert change to install list

* try passing env var

* export debian frontend var?

* move line and add comment

* bump pillow dependency

* bump lxml depenency

* install libjpeg-dev for pillow

* bump automat version to one compatible with py3.8

* add libwebp for pillow

* bump twisted trunk python version

* change suffix of newsfragment

* remove redundant python 3.7 checks

* lint
2022-01-21 14:23:26 -08:00
Richard van der Hoff 2aa37a4250
Add `state_key` and `rejection_reason` to `events` (#11792)
... and start populating them for new events
2022-01-21 12:21:28 +00:00
Richard van der Hoff 9f2016e96e
Drop unused table `public_room_list_stream`. (#11795)
This is a follow-up to #10565.
2022-01-21 09:19:56 +00:00
Richard van der Hoff 2277275485
Stop reading from `event_reference_hashes` (#11794)
Preparation for dropping this table altogether. Part of #6574.
2022-01-21 09:18:10 +00:00
reivilibre 4c2096599c
Make the `get_global_account_data_by_type_for_user` cache be a tree-cache whose key is prefixed with the user ID (#11788) 2022-01-21 08:38:36 +00:00
reivilibre e83520cc42
Make `get_account_data_for_room_and_type` a tree cache (#11789) 2022-01-21 08:01:37 +00:00
David Robertson f160fe18e3
Debug for device lists updates (#11760)
Debug for #8631.

I'm having a hard time tracking down what's going wrong in that issue.
In the reported example, I could see server A sending federation traffic
to server B and all was well. Yet B reports out-of-sync device updates
from A.

I couldn't see what was _in_ the events being sent from A to B. So I
have added some crude logging to track

- when we have updates to send to a remote HS
- the edus we actually accumulate to send
- when a federation transaction includes a device list update edu
- when such an EDU is received

This is a bit of a sledgehammer.
2022-01-20 13:38:44 +00:00
Richard van der Hoff 5572e6cc4b
Comments and typing for `_update_outliers_txn` (#11776)
A couple of surprises for me here, so thought I'd document them
2022-01-19 19:45:36 +00:00
Patrick Cloke 68acb0a29d
Include whether the requesting user has participated in a thread. (#11577)
Per updates to MSC3440.

This is implement as a separate method since it needs to be cached
on a per-user basis, instead of a per-thread basis.
2022-01-18 11:38:57 -05:00
Richard van der Hoff 251b5567ec
Remove `log_function` and its uses (#11761)
I've never found this terribly useful. I think it was added in the early days
of Synapse, without much thought as to what would actually be useful to log,
and has just been cargo-culted ever since.

Rather, it tends to clutter up debug logs with useless information.
2022-01-18 13:06:04 +00:00
Daniel Sonck 6b241f5286
Make pagination of rooms in admin api stable (#11737)
Always add state.room_id after the configurable ORDER BY. Otherwise,
for any sort, certain pages can contain results from
other pages. (Especially when sorting by creator, since there may
be many rooms by the same creator)

* Document different order direction of numerical fields

"joined_members", "joined_local_members", "version" and "state_events"
are ordered in descending direction by default (dir=f). Added a note
in tests to explain the differences in ordering.

Signed-off-by: Daniël Sonck <daniel@sonck.nl>
2022-01-17 11:42:51 +00:00