1eea73b413
Fix rate limit metrics registering twice and misreporting ( #13649 )
...
* Fix rate limit metrics registering twice and misreporting
Fix https://github.com/matrix-org/synapse/issues/13641
* Fix lints
* Add changelog
* Document `metrics_name=None`.
2022-08-30 12:08:29 +01:00
682dfcfc0d
Fix that user cannot `/forget` rooms after the last member has left ( #13546 )
2022-08-30 09:58:38 +00:00
51d732db3b
Optimize how we calculate `likely_domains` during backfill ( #13575 )
...
Optimize how we calculate `likely_domains` during backfill because I've seen this take 17s in production just to `get_current_state` which is used to `get_domains_from_state` (see case [*2. Loading tons of events* in the `/messages` investigation issue](https://github.com/matrix-org/synapse/issues/13356 )).
There are 3 ways we currently calculate hosts that are in the room:
1. `get_current_state` -> `get_domains_from_state`
- Used in `backfill` to calculate `likely_domains` and `/timestamp_to_event` because it was cargo-culted from `backfill`
- This one is being eliminated in favor of `get_current_hosts_in_room` in this PR 🕳
1. `get_current_hosts_in_room`
- Used for other federation things like sending read receipts and typing indicators
1. `get_hosts_in_room_at_events`
- Used when pushing out events over federation to other servers in the `_process_event_queue_loop`
Fix https://github.com/matrix-org/synapse/issues/13626
Part of https://github.com/matrix-org/synapse/issues/13356
Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.2tvwz3yhcafh )
### Query performance
#### Before
The query from `get_current_state` sucks just because we have to get all 80k events. And we see almost the exact same performance locally trying to get all of these events (16s vs 17s):
```
synapse=# SELECT type, state_key, event_id FROM current_state_events WHERE room_id = '!OGEhHVWSdvArJzumhm:matrix.org';
Time: 16035.612 ms (00:16.036)
synapse=# SELECT type, state_key, event_id FROM current_state_events WHERE room_id = '!OGEhHVWSdvArJzumhm:matrix.org';
Time: 4243.237 ms (00:04.243)
```
But what about `get_current_hosts_in_room`: When there is 8M rows in the `current_state_events` table, the previous query in `get_current_hosts_in_room` took 13s from complete freshness (when the events were first added). But takes 930ms after a Postgres restart or 390ms if running back to back to back.
```sh
$ psql synapse
synapse=# \timing on
synapse=# SELECT COUNT(DISTINCT substring(state_key FROM '@[^:]*:(.*)$'))
FROM current_state_events
WHERE
type = 'm.room.member'
AND membership = 'join'
AND room_id = '!OGEhHVWSdvArJzumhm:matrix.org';
count
-------
4130
(1 row)
Time: 13181.598 ms (00:13.182)
synapse=# SELECT COUNT(*) from current_state_events where room_id = '!OGEhHVWSdvArJzumhm:matrix.org';
count
-------
80814
synapse=# SELECT COUNT(*) from current_state_events;
count
---------
8162847
synapse=# SELECT pg_size_pretty( pg_total_relation_size('current_state_events') );
pg_size_pretty
----------------
4702 MB
```
#### After
I'm not sure how long it takes from complete freshness as I only really get that opportunity once (maybe restarting computer but that's cumbersome) and it's not really relevant to normal operating times. Maybe you get closer to the fresh times the more access variability there is so that Postgres caches aren't as exact. Update: The longest I've seen this run for is 6.4s and 4.5s after a computer restart.
After a Postgres restart, it takes 330ms and running back to back takes 260ms.
```sh
$ psql synapse
synapse=# \timing on
Timing is on.
synapse=# SELECT
substring(c.state_key FROM '@[^:]*:(.*)$') as host
FROM current_state_events c
/* Get the depth of the event from the events table */
INNER JOIN events AS e USING (event_id)
WHERE
c.type = 'm.room.member'
AND c.membership = 'join'
AND c.room_id = '!OGEhHVWSdvArJzumhm:matrix.org'
GROUP BY host
ORDER BY min(e.depth) ASC;
Time: 333.800 ms
```
#### Going further
To improve things further we could add a `limit` parameter to `get_current_hosts_in_room`. Realistically, we don't need 4k domains to choose from because there is no way we're going to query that many before we a) probably get an answer or b) we give up.
Another thing we can do is optimize the query to use a index skip scan:
- https://wiki.postgresql.org/wiki/Loose_indexscan
- Index Skip Scan, https://commitfest.postgresql.org/37/1741/
- https://www.timescale.com/blog/how-we-made-distinct-queries-up-to-8000x-faster-on-postgresql/
2022-08-30 01:38:14 -05:00
4f6de33f41
Print complement failure results last ( #13639 )
...
Since github always scrolls to the bottom of any test output, let's put the
failed tests last and hide any successful packages.
2022-08-28 20:05:30 +00:00
c4e29b6908
Improve documentation around user registration ( #13640 )
...
Update a bunch of the documentation for user registration, add some cross
links, etc.
2022-08-26 13:29:31 +00:00
5e5c8150d7
Generate missing configuration files at startup ( #13615 )
...
If things like the signing key file are missing, let's just try to generate
them on startup.
Again, this is useful for k8s-like deployments where we just want to generate
keys on the first run.
2022-08-26 11:26:06 +00:00
998e211836
Update debhelper ( #13594 )
...
* Update debian packaging to debhelper version 12
Don't call dh_installinit anymore, because it has been deprecated, and use
dh_installsystemd instead of dh_systemd_enable for the same reason.
Signed-off-by: Jörg Behrmann <behrmann@physik.fu-berlin.de>
* Drop preinst script
It was used for reasons of interactions of dh_systemd_start and dh_installinit,
which have both be deprecated
Signed-off-by: Jörg Behrmann <behrmann@physik.fu-berlin.de>
* Drop /etc/default file
It was no longer being installed.
* Remove debian/compat file
This is managed by the control file nowadays
2022-08-26 08:10:54 +00:00
967d7bad6c
Move the execution of the retention purge_jobs to the main worker ( #13632 )
...
Fixes #9927
Signed-off-by: Brad Murray brad@beeper.com
2022-08-26 08:38:10 +01:00
978666a088
Debian packaging: explicitly allocate a group for the system user ( #13593 )
...
Otherwise the files of the synapse user are readable by the nobody user, which
is unsafe.
Signed-off-by: Jörg Behrmann <behrmann@physik.fu-berlin.de>
2022-08-25 16:56:55 +00:00
d092e6f32a
Support `registration_shared_secret` in a file ( #13614 )
...
A new `registration_shared_secret_path` option. This is kinda handy for k8s deployments and things.
2022-08-25 16:27:46 +00:00
a2ce614447
register_new_matrix_user: read server url from config ( #13616 )
...
Fixes https://github.com/matrix-org/synapse/issues/3672 :
`https://localhost:8448 ` is virtually never right.
2022-08-25 15:29:08 +01:00
a282446502
Update automation for incoming issues ( #13629 )
...
GitHub appears to be deprecating addProjectNextItem by not allowing it to be used alongside projectV2 to get the project ID, so switching to using addProjectV2ItemById instead.
2022-08-25 12:09:23 +01:00
0bf180cbb4
Comment about a better future where we can get the state diff between two events ( #13586 )
...
Split off from https://github.com/matrix-org/synapse/pull/13561
Part of https://github.com/matrix-org/synapse/issues/13356
Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.2tvwz3yhcafh )
2022-08-24 18:59:27 -05:00
c406d50d2d
Rename `event_map` to `unpersisted_events` ( #13603 )
2022-08-24 21:06:31 +01:00
1a209efdb2
Update `get_users_in_room` mis-use to get hosts with dedicated `get_current_hosts_in_room` ( #13605 )
...
See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755
2022-08-24 14:15:37 -05:00
d58615c82c
Directly lookup local membership instead of getting all members in a room first (`get_users_in_room` mis-use) ( #13608 )
...
See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755
2022-08-24 14:13:12 -05:00
b93bd95e8a
When loading current ids, sort by `stream_id` to avoid incorrect overwrite and avoid errors caused by sorting alphabetical instance name which can be `null` ( #13585 )
...
When loading current ids, sort by stream ID so that we don't want to overwrite the `current_position` of an instance to a lower stream ID than we're actually at ([discussion](https://github.com/matrix-org/synapse/pull/13585#discussion_r951795379 )). Previously, it sorted alphabetically by instance name which can be `null` and throw errors but more importantly, accomplishes nothing.
Fixes the following startup error which is why I started looking into this area:
```
$ poetry run synapse_homeserver --config-path homeserver.yaml
****************************************************************
Error during initialisation:
'<' not supported between instances of 'NoneType' and 'str'
There may be more information in the logs.
****************************************************************
```
Somehow my database ended up looking like the following, notice the `instance_name` is `null` in the db, and we can't sort `NoneType` things. Another question is why do we see the `instance_name` as `null` sometimes instead of `master` in monolith mode?
```
$ psql synapse
synapse=# SELECT * FROM stream_positions;
stream_name | instance_name | stream_id
-----------------+---------------+-----------
account_data | master | 1242
events | master | 1787
to_device | master | 58
presence_stream | master | 485638
receipts | master | 341
backfill | master | -139106
(6 rows)
synapse=# SELECT instance_name, stream_id FROM receipts_linearized;
instance_name | stream_id
---------------+-----------
| 211
| 3
| 4
| 212
| 213
| 224
| 228
| 164
| 313
| 253
| 38
| 321
| 324
| 189
| 192
| 193
| 194
| 195
| 197
| 198
| 275
| 79
| 339
| 340
| 82
| 341
| 84
| 85
| 91
| 119
```
2022-08-24 12:53:46 -05:00
c807b814ae
Use dedicated `get_local_users_in_room` to find local users when calculating `join_authorised_via_users_server` of a `/make_join` request ( #13606 )
...
Use dedicated `get_local_users_in_room` to find local users when calculating `join_authorised_via_users_server` ("the authorising user for joining a restricted room") of a `/make_join` request.
Found while working on https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755 but it's not related.
2022-08-24 11:14:28 -05:00
371db86a86
First draft of triage_labelled action ( #13612 )
2022-08-24 13:59:33 +01:00
be4250c7a8
Add experimental configuration option to allow disabling legacy Prometheus metric names. ( #13540 )
...
Co-authored-by: David Robertson <davidr@element.io>
2022-08-24 11:35:54 +00:00
2e2040c93e
Add GitHub automation for new issues ( #13610 )
...
Set up automation to move newly opened issues in GitHub to the issue triage board.
2022-08-24 12:10:32 +01:00
b687010f89
Rewrite get push actions queries ( #13597 )
2022-08-24 10:12:51 +01:00
ba882c0357
Faster Room Joins: fix `/make_knock` blocking indefinitely when the room in question is a partial-stated room. ( #13583 )
...
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-08-24 09:09:59 +00:00
7af07f9716
Instrument `_check_sigs_and_hash_and_fetch` to trace time spent in child concurrent calls ( #13588 )
...
Instrument `_check_sigs_and_hash_and_fetch` to trace time spent in child concurrent calls because I've see `_check_sigs_and_hash_and_fetch` take [10.41s to process 100 events](https://github.com/matrix-org/synapse/issues/13587 )
Fix https://github.com/matrix-org/synapse/issues/13587
Part of https://github.com/matrix-org/synapse/issues/13356
2022-08-23 21:53:37 -05:00
a25a37002c
Write about the chain cover a little. ( #13602 )
...
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-08-23 17:41:55 +00:00
f7ddfe17a3
Speed up `@cachedList` ( #13591 )
...
This speeds things up by ~2x.
The vast majority of the time is now spent in `LruCache` moving things around the linked lists.
We do this via two things:
1. Don't create a deferred per-key during bulk set operations in `DeferredCache`. Instead, only create them if a subsequent caller asks for the key.
2. Add a bulk lookup API to `DeferredCache` rather than use a loop.
2022-08-23 14:53:27 +00:00
05c9c7363b
Fix regression caused by #13573 ( #13600 )
...
Broke in #13573 .
2022-08-23 14:14:05 +00:00
bdfff9c36e
Update openid.md ( #13568 )
...
Linking the help article may prevent confusion regarding the creation of the necessary rule using auth0.
2022-08-23 14:34:10 +01:00
ca3d19b05f
Synapse 1.66.0rc1 (2022-08-23)
...
==============================
This release removes the ability for homeservers to delegate email ownership
verification and password reset confirmation to identity servers. This removal
was originally planned for Synapse 1.64, but was later deferred until now.
See the [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660 ) for more details.
Features
--------
- Improve validation of request bodies for the following client-server API endpoints: [`/account/password`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpassword ), [`/account/password/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpasswordemailrequesttoken ), [`/account/deactivate`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountdeactivate ) and [`/account/3pid/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidemailrequesttoken ). ([\#13188](https://github.com/matrix-org/synapse/issues/13188 ), [\#13563](https://github.com/matrix-org/synapse/issues/13563 ))
- Add forgotten status to [Room Details Admin API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#room-details-api ). ([\#13503](https://github.com/matrix-org/synapse/issues/13503 ))
- Add an experimental implementation for [MSC3852 (Expose user agents on `Device`)](https://github.com/matrix-org/matrix-spec-proposals/pull/3852 ). ([\#13549](https://github.com/matrix-org/synapse/issues/13549 ))
- Add `org.matrix.msc2716v4` experimental room version with updated content fields. Part of [MSC2716 (Importing history)](https://github.com/matrix-org/matrix-spec-proposals/pull/2716 ). ([\#13551](https://github.com/matrix-org/synapse/issues/13551 ))
- Add support for compression to federation responses. ([\#13537](https://github.com/matrix-org/synapse/issues/13537 ))
- Improve performance of sending messages in rooms with thousands of local users. ([\#13522](https://github.com/matrix-org/synapse/issues/13522 ), [\#13547](https://github.com/matrix-org/synapse/issues/13547 ))
Bugfixes
--------
- Faster room joins: make `/joined_members` block whilst the room is partial stated. ([\#13514](https://github.com/matrix-org/synapse/issues/13514 ))
- Fix a bug introduced in Synapse 1.21.0 where the [`/event_reports` Admin API](https://matrix-org.github.io/synapse/develop/admin_api/event_reports.html ) could return a total count which was larger than the number of results you can actually query for. ([\#13525](https://github.com/matrix-org/synapse/issues/13525 ))
- Fix a bug introduced in Synapse 1.52.0 where sending server notices fails if `max_avatar_size` or `allowed_avatar_mimetypes` is set and not `system_mxid_avatar_url`. ([\#13566](https://github.com/matrix-org/synapse/issues/13566 ))
- Fix a bug where the `opentracing.force_tracing_for_users` config option would not apply to [`/sendToDevice`](https://spec.matrix.org/v1.3/client-server-api/#put_matrixclientv3sendtodeviceeventtypetxnid ) and [`/keys/upload`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3keysupload ) requests. ([\#13574](https://github.com/matrix-org/synapse/issues/13574 ))
Improved Documentation
----------------------
- Add `openssl` example for generating registration HMAC digest. ([\#13472](https://github.com/matrix-org/synapse/issues/13472 ))
- Tidy up Synapse's README. ([\#13491](https://github.com/matrix-org/synapse/issues/13491 ))
- Document that event purging related to the `redaction_retention_period` config option is executed only every 5 minutes. ([\#13492](https://github.com/matrix-org/synapse/issues/13492 ))
- Add a warning to retention documentation regarding the possibility of database corruption. ([\#13497](https://github.com/matrix-org/synapse/issues/13497 ))
- Document that the `DOCKER_BUILDKIT=1` flag is needed to build the docker image. ([\#13515](https://github.com/matrix-org/synapse/issues/13515 ))
- Add missing links in `user_consent` section of configuration manual. ([\#13536](https://github.com/matrix-org/synapse/issues/13536 ))
- Fix the doc and some warnings that were referring to the nonexistent `custom_templates_directory` setting (instead of `custom_template_directory`). ([\#13538](https://github.com/matrix-org/synapse/issues/13538 ))
Deprecations and Removals
-------------------------
- Remove the ability for homeservers to delegate email ownership verification
and password reset confirmation to identity servers. See [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660 ) for more details.
Internal Changes
----------------
- Update the rejected state of events during de-partial-stating. ([\#13459](https://github.com/matrix-org/synapse/issues/13459 ))
- Avoid blocking lazy-loading `/sync`s during partial joins due to remote memberships. Pull remote memberships from auth events instead of the room state. ([\#13477](https://github.com/matrix-org/synapse/issues/13477 ))
- Refuse to start when faster joins is enabled on a deployment with workers, since worker configurations are not currently supported. ([\#13531](https://github.com/matrix-org/synapse/issues/13531 ))
- Allow use of both `@trace` and `@tag_args` stacked on the same function. ([\#13453](https://github.com/matrix-org/synapse/issues/13453 ))
- Instrument the federation/backfill part of `/messages` for understandable traces in Jaeger. ([\#13489](https://github.com/matrix-org/synapse/issues/13489 ))
- Instrument `FederationStateIdsServlet` (`/state_ids`) for understandable traces in Jaeger. ([\#13499](https://github.com/matrix-org/synapse/issues/13499 ), [\#13554](https://github.com/matrix-org/synapse/issues/13554 ))
- Track HTTP response times over 10 seconds from `/messages` (`synapse_room_message_list_rest_servlet_response_time_seconds`). ([\#13533](https://github.com/matrix-org/synapse/issues/13533 ))
- Add metrics to track how the rate limiter is affecting requests (sleep/reject). ([\#13534](https://github.com/matrix-org/synapse/issues/13534 ), [\#13541](https://github.com/matrix-org/synapse/issues/13541 ))
- Add metrics to time how long it takes us to do backfill processing (`synapse_federation_backfill_processing_before_time_seconds`, `synapse_federation_backfill_processing_after_time_seconds`). ([\#13535](https://github.com/matrix-org/synapse/issues/13535 ), [\#13584](https://github.com/matrix-org/synapse/issues/13584 ))
- Add metrics to track rate limiter queue timing (`synapse_rate_limit_queue_wait_time_seconds`). ([\#13544](https://github.com/matrix-org/synapse/issues/13544 ))
- Update metrics to track `/messages` response time by room size. ([\#13545](https://github.com/matrix-org/synapse/issues/13545 ))
- Refactor methods in `synapse.api.auth.Auth` to use `Requester` objects everywhere instead of user IDs. ([\#13024](https://github.com/matrix-org/synapse/issues/13024 ))
- Clean-up tests for notifications. ([\#13471](https://github.com/matrix-org/synapse/issues/13471 ))
- Add some miscellaneous comments to document sync, especially around `compute_state_delta`. ([\#13474](https://github.com/matrix-org/synapse/issues/13474 ))
- Use literals in place of `HTTPStatus` constants in tests. ([\#13479](https://github.com/matrix-org/synapse/issues/13479 ), [\#13488](https://github.com/matrix-org/synapse/issues/13488 ))
- Add comments about how event push actions are rotated. ([\#13485](https://github.com/matrix-org/synapse/issues/13485 ))
- Modify HTML template content to better support mobile devices' screen sizes. ([\#13493](https://github.com/matrix-org/synapse/issues/13493 ))
- Add a linter script which will reject non-strict types in Pydantic models. ([\#13502](https://github.com/matrix-org/synapse/issues/13502 ))
- Reduce the number of tests using legacy TCP replication. ([\#13543](https://github.com/matrix-org/synapse/issues/13543 ))
- Allow specifying additional request fields when using the `HomeServerTestCase.login` helper method. ([\#13549](https://github.com/matrix-org/synapse/issues/13549 ))
- Make `HomeServerTestCase` load any configured homeserver modules automatically. ([\#13558](https://github.com/matrix-org/synapse/issues/13558 ))
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEE1508oLYUKainYFJakD7OEIo53t0FAmMExBwACgkQkD7OEIo5
3t0sGA/9FLZLdTWeROLUenbInvz3POqi6DgqlE0f8IDdp3zHjjuDJIE4HZwnPhhj
c3H5FcUnTPhQoVZ8Ybno3J3KN8jfpe4tFhnhlM1x9hOi1fujfqCnFd2qWvlzkYXX
J4oN6Dz0BbhqdhqnS2tcNacZJOielJ47UtNNBZkzprpzjL3qqRUK3Hm/gGK8GtGN
BLon7JK0pKG1GFMVaoIWsr+j7880QD8DGXTjoZrjo/yGdp+zJldwJTbN2ymNdMw1
3Caq7eCk+m6jzCUJxI7tTJVioXvf7+TScIsozAXuQtBy9W1RctvVMMoR9cIlN+JJ
dj5GKhKBv8uJpJU6dJnlYpjNC5+5hPHr1gfnkoYbcyFF6PybJrHx9jfhEIEXmI1+
j0DlJJdFnX+eeq4deFd+LPU5hYaF8SkSVRK3G+/NmKI3YYTReD6ootZzV5O4Z9P0
NUwLK/OSMZrsFUfuwysUttfHq0uHGGs7mytlRgeqb/1RKRC80nxVJu3qv/xF6kKj
KF98Yr2IDgp/MFhBpGAO3ylNgK8aQ3YxED4qW+kfnDeHPkma5mTdAtJ1bpvKcec+
3awpFGWcmYVQPflwp9J3HFJHqWkj32LMfuSgDffLs8mWuj87fP7eAdjzv6M4zeAM
4qh+cmvNC7ArfQ+ngz3g+xVD1lHi84s+fMgo7L+ySGfgDqTLhT4=
=1F3P
-----END PGP SIGNATURE-----
gpgsig -----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEE1508oLYUKainYFJakD7OEIo53t0FAmMEz10ACgkQkD7OEIo5
3t02GhAApioIpEedKaeSATI42N8rWhfuni3vaPFAXwmOD6BvBkZlP0gcg+vUoBjP
I0VIkGWZ3MKIhCSJkjwPJh1vHcpE09KYqIYRKO5vSbqHZKHYUJ+ThjcCR4IkOBPD
3Dc3IRgQaM2Nh8eMcyRY7eyxo+2y+VN2G+SXr1B+A5/4N58ZrxfV97MtLYJ1a6AH
0iIknOxb9FPArYs4ORbk6x2rvRO2QPGh88uKwJ8uHXvHlUbJ3fmHLVvRa1m0RKAW
ojkzj/Dysa+4gasMIvtzPePtIQnMyovaQLul+bGXzE0js4lqzSOIP12x1zNpLuuq
LSiyD3gMgGeogumhCAU6b23xF6jBaGtgtKKpaYuK8pqkuBTRNN8kQ6LB+7g3WQ2L
QWVos+/+wKUEHoVVD/q3u7o4CrKYqv4zE9Pj6kw5e1USOoy7803ub2ccjY1UgG3M
mZ7c48WNQQ61o9oH2+9+Ct6bab3R/Lo1MbS3zIKN8uktB4CdrhVAz3kYgV5Bl8bo
PsldxUccOTTeS5qmUGvm2/szVKjdqyCWRx9AY9eUH9Kc4u/GzmNq0UVOZF8yxM0q
WEo7GFgjM3k9IIC3K/fSmSC1iBjPSHpYvqO+c8oSNnjq2iucZeIhiEB07e89+KSa
Wiz9DDWP8Z7W6D9wRAeXyPJFB6ZdSSBgP1I90z6lQjSMzy46Kv4=
=pJ2V
-----END PGP SIGNATURE-----
Merge tag 'v1.66.0rc1' into develop
Synapse 1.66.0rc1 (2022-08-23)
==============================
This release removes the ability for homeservers to delegate email ownership
verification and password reset confirmation to identity servers. This removal
was originally planned for Synapse 1.64, but was later deferred until now.
See the [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660 ) for more details.
Features
--------
- Improve validation of request bodies for the following client-server API endpoints: [`/account/password`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpassword ), [`/account/password/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpasswordemailrequesttoken ), [`/account/deactivate`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountdeactivate ) and [`/account/3pid/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidemailrequesttoken ). ([\#13188](https://github.com/matrix-org/synapse/issues/13188 ), [\#13563](https://github.com/matrix-org/synapse/issues/13563 ))
- Add forgotten status to [Room Details Admin API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#room-details-api ). ([\#13503](https://github.com/matrix-org/synapse/issues/13503 ))
- Add an experimental implementation for [MSC3852 (Expose user agents on `Device`)](https://github.com/matrix-org/matrix-spec-proposals/pull/3852 ). ([\#13549](https://github.com/matrix-org/synapse/issues/13549 ))
- Add `org.matrix.msc2716v4` experimental room version with updated content fields. Part of [MSC2716 (Importing history)](https://github.com/matrix-org/matrix-spec-proposals/pull/2716 ). ([\#13551](https://github.com/matrix-org/synapse/issues/13551 ))
- Add support for compression to federation responses. ([\#13537](https://github.com/matrix-org/synapse/issues/13537 ))
- Improve performance of sending messages in rooms with thousands of local users. ([\#13522](https://github.com/matrix-org/synapse/issues/13522 ), [\#13547](https://github.com/matrix-org/synapse/issues/13547 ))
Bugfixes
--------
- Faster room joins: make `/joined_members` block whilst the room is partial stated. ([\#13514](https://github.com/matrix-org/synapse/issues/13514 ))
- Fix a bug introduced in Synapse 1.21.0 where the [`/event_reports` Admin API](https://matrix-org.github.io/synapse/develop/admin_api/event_reports.html ) could return a total count which was larger than the number of results you can actually query for. ([\#13525](https://github.com/matrix-org/synapse/issues/13525 ))
- Fix a bug introduced in Synapse 1.52.0 where sending server notices fails if `max_avatar_size` or `allowed_avatar_mimetypes` is set and not `system_mxid_avatar_url`. ([\#13566](https://github.com/matrix-org/synapse/issues/13566 ))
- Fix a bug where the `opentracing.force_tracing_for_users` config option would not apply to [`/sendToDevice`](https://spec.matrix.org/v1.3/client-server-api/#put_matrixclientv3sendtodeviceeventtypetxnid ) and [`/keys/upload`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3keysupload ) requests. ([\#13574](https://github.com/matrix-org/synapse/issues/13574 ))
Improved Documentation
----------------------
- Add `openssl` example for generating registration HMAC digest. ([\#13472](https://github.com/matrix-org/synapse/issues/13472 ))
- Tidy up Synapse's README. ([\#13491](https://github.com/matrix-org/synapse/issues/13491 ))
- Document that event purging related to the `redaction_retention_period` config option is executed only every 5 minutes. ([\#13492](https://github.com/matrix-org/synapse/issues/13492 ))
- Add a warning to retention documentation regarding the possibility of database corruption. ([\#13497](https://github.com/matrix-org/synapse/issues/13497 ))
- Document that the `DOCKER_BUILDKIT=1` flag is needed to build the docker image. ([\#13515](https://github.com/matrix-org/synapse/issues/13515 ))
- Add missing links in `user_consent` section of configuration manual. ([\#13536](https://github.com/matrix-org/synapse/issues/13536 ))
- Fix the doc and some warnings that were referring to the nonexistent `custom_templates_directory` setting (instead of `custom_template_directory`). ([\#13538](https://github.com/matrix-org/synapse/issues/13538 ))
Deprecations and Removals
-------------------------
- Remove the ability for homeservers to delegate email ownership verification
and password reset confirmation to identity servers. See [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660 ) for more details.
Internal Changes
----------------
- Update the rejected state of events during de-partial-stating. ([\#13459](https://github.com/matrix-org/synapse/issues/13459 ))
- Avoid blocking lazy-loading `/sync`s during partial joins due to remote memberships. Pull remote memberships from auth events instead of the room state. ([\#13477](https://github.com/matrix-org/synapse/issues/13477 ))
- Refuse to start when faster joins is enabled on a deployment with workers, since worker configurations are not currently supported. ([\#13531](https://github.com/matrix-org/synapse/issues/13531 ))
- Allow use of both `@trace` and `@tag_args` stacked on the same function. ([\#13453](https://github.com/matrix-org/synapse/issues/13453 ))
- Instrument the federation/backfill part of `/messages` for understandable traces in Jaeger. ([\#13489](https://github.com/matrix-org/synapse/issues/13489 ))
- Instrument `FederationStateIdsServlet` (`/state_ids`) for understandable traces in Jaeger. ([\#13499](https://github.com/matrix-org/synapse/issues/13499 ), [\#13554](https://github.com/matrix-org/synapse/issues/13554 ))
- Track HTTP response times over 10 seconds from `/messages` (`synapse_room_message_list_rest_servlet_response_time_seconds`). ([\#13533](https://github.com/matrix-org/synapse/issues/13533 ))
- Add metrics to track how the rate limiter is affecting requests (sleep/reject). ([\#13534](https://github.com/matrix-org/synapse/issues/13534 ), [\#13541](https://github.com/matrix-org/synapse/issues/13541 ))
- Add metrics to time how long it takes us to do backfill processing (`synapse_federation_backfill_processing_before_time_seconds`, `synapse_federation_backfill_processing_after_time_seconds`). ([\#13535](https://github.com/matrix-org/synapse/issues/13535 ), [\#13584](https://github.com/matrix-org/synapse/issues/13584 ))
- Add metrics to track rate limiter queue timing (`synapse_rate_limit_queue_wait_time_seconds`). ([\#13544](https://github.com/matrix-org/synapse/issues/13544 ))
- Update metrics to track `/messages` response time by room size. ([\#13545](https://github.com/matrix-org/synapse/issues/13545 ))
- Refactor methods in `synapse.api.auth.Auth` to use `Requester` objects everywhere instead of user IDs. ([\#13024](https://github.com/matrix-org/synapse/issues/13024 ))
- Clean-up tests for notifications. ([\#13471](https://github.com/matrix-org/synapse/issues/13471 ))
- Add some miscellaneous comments to document sync, especially around `compute_state_delta`. ([\#13474](https://github.com/matrix-org/synapse/issues/13474 ))
- Use literals in place of `HTTPStatus` constants in tests. ([\#13479](https://github.com/matrix-org/synapse/issues/13479 ), [\#13488](https://github.com/matrix-org/synapse/issues/13488 ))
- Add comments about how event push actions are rotated. ([\#13485](https://github.com/matrix-org/synapse/issues/13485 ))
- Modify HTML template content to better support mobile devices' screen sizes. ([\#13493](https://github.com/matrix-org/synapse/issues/13493 ))
- Add a linter script which will reject non-strict types in Pydantic models. ([\#13502](https://github.com/matrix-org/synapse/issues/13502 ))
- Reduce the number of tests using legacy TCP replication. ([\#13543](https://github.com/matrix-org/synapse/issues/13543 ))
- Allow specifying additional request fields when using the `HomeServerTestCase.login` helper method. ([\#13549](https://github.com/matrix-org/synapse/issues/13549 ))
- Make `HomeServerTestCase` load any configured homeserver modules automatically. ([\#13558](https://github.com/matrix-org/synapse/issues/13558 ))
2022-08-23 14:00:09 +01:00
aec87a0f93
Speed up fetching large numbers of push rules ( #13592 )
2022-08-23 13:15:43 +01:00
ea85a2bf6c
Remove manually-added changelog
2022-08-23 12:40:35 +01:00
956e015413
Drop support for delegating email validation, round 2 ( #13596 )
2022-08-23 11:40:00 +00:00
5e7847dc92
Cache user IDs instead of profile objects ( #13573 )
...
The profile objects are never used and increase cache size significantly.
2022-08-23 09:49:59 +00:00
79281f517d
Update changelog
2022-08-23 10:22:47 +01:00
f8b9abdcdb
Adjust changelog
2022-08-23 10:10:20 +01:00
d6f5699737
Describe changes to admin API in 1.66
...
Cross-ref #13525
2022-08-23 10:09:45 +01:00
f0b23927fc
1.66.0rc1
2022-08-23 09:49:51 +01:00
37f329c9ad
Fix that sending server notices fail if avatar is `None` ( #13566 )
...
Indroduced in #11846 .
2022-08-23 09:48:35 +01:00
9385c41ba4
Fix Prometheus metrics being negative (mixed up start/end) ( #13584 )
...
Fix:
- https://github.com/matrix-org/synapse/pull/13535#discussion_r949582508
- https://github.com/matrix-org/synapse/pull/13533#discussion_r949577244
2022-08-23 08:47:30 +01:00
3dd175b628
`synapse.api.auth.Auth` cleanup: make permission-related methods use `Requester` instead of the `UserID` ( #13024 )
...
Part of #13019
This changes all the permission-related methods to rely on the Requester instead of the UserID. This is a first step towards enabling scoped access tokens at some point, since I expect the Requester to have scope-related informations in it.
It also changes methods which figure out the user/device/appservice out of the access token to return a Requester instead of something else. This avoids having store-related objects in the methods signatures.
2022-08-22 14:17:59 +01:00
94375f7a91
Remove redundant opentracing spans for `/sendToDevice` and `/keys/upload` ( #13574 )
2022-08-22 10:03:11 +01:00
06df5d4250
MSC2716v4 room version - remove namespace from MSC2716 event content fields ( #13551 )
...
Complement PR: https://github.com/matrix-org/complement/pull/450
As suggested in
https://github.com/matrix-org/matrix-spec-proposals/pull/2716#discussion_r941444525
2022-08-19 15:37:01 -05:00
f9f03426de
Implement MSC3852: Expose `last_seen_user_agent` to users for their own devices; also expose to Admin API ( #13549 )
2022-08-19 16:17:10 +00:00
40e3e68cd7
Register homeserver modules when creating test homeserver ( #13558 )
2022-08-19 16:52:20 +01:00
f3fba4914d
Reduce the number of tests using TCP replication. ( #13543 )
...
Uses Redis replication in additional test cases (instead of
TCP replication). A small step towards dropping TCP replication.
2022-08-19 08:25:24 -04:00
3a245f6cfe
Fix validation problem that occurs when a user tries to deactivate their account or change their password. ( #13563 )
2022-08-19 11:03:29 +00:00
2c42673a9b
Add metrics to track `/messages` response time by room size ( #13545 )
...
Follow-up to https://github.com/matrix-org/synapse/pull/13533
Part of https://github.com/matrix-org/synapse/issues/13356
2022-08-18 14:15:53 -05:00
b251cff819
Fix incorrect juggling of logging contexts in `_PerHostRatelimiter` ( #13554 )
...
Signed-off-by: Sean Quah <seanq@matrix.org>
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-08-18 16:26:26 +01:00
d64653d062
Track number of hosts affected by the rate limiter ( #13541 )
...
Track number of hosts affected by the rate limiter so we can differentiate one really noisy homeserver from a general ratelimit tuning problem across the federation.
Follow-up to https://github.com/matrix-org/synapse/pull/13534
Part of https://github.com/matrix-org/synapse/issues/13356
2022-08-18 10:05:07 -05:00
22ea51faf9
Add support for compression to federation responses ( #13537 )
...
Closes #13415 .
Signed-off-by: Ayush Anand <iamayushanand@gmail.com>
2022-08-18 15:14:47 +01:00
Eric Eastwood