Commit Graph

20518 Commits

Author SHA1 Message Date
Etienne Dysli Metref b0eb64ff7b
Remove mention of OIDC certification from Dex (#11470)
Dex isn't yet [1,2] a certified OpenID Provider implementation. As of today, it's not on the list maintained by the OpenID Foundation. [3]

[1] https://github.com/dexidp/dex/issues/42
[2] https://github.com/dexidp/dex/issues/262
[3] https://openid.net/certification/
2021-12-01 09:40:51 +00:00
Shay f1795463bf
Add a note about huge pages to our Postgres doc (#11467)
* Add note to postgres doc about hugepages

* Newsfragment
2021-11-30 19:05:20 -08:00
Shay 70cbb1a5e3
Don't start Synapse master process if `worker_app` is set (#11416)
* Add check to catch syanpse master process starting when workers are configured

* add test to verify that starting master process with worker config raises error

* newsfragment

* specify config.worker.worker_app in check

* update test

* report specific config option that triggered the error

Co-authored-by: reivilibre <oliverw@matrix.org>

* clarify error message

Co-authored-by: reivilibre <oliverw@matrix.org>

Co-authored-by: reivilibre <oliverw@matrix.org>
2021-11-30 10:12:18 -08:00
Maximilian Bosch 42bf020463
Expose worker & homeserver as entrypoints in `setup.py` (#11449)
Co-authored-by: reivilibre <oliverw@matrix.org>
2021-11-30 16:52:45 +00:00
Patrick Cloke 379f2650cf
Bundle relations of relations into the `/relations` result. (#11284)
Per updates to MSC2675 which now states that bundled
aggregations should be included from the `/relations`
endpoint.
2021-11-30 11:33:33 -05:00
Sean Quah 7ff22d6da4
Fix `LruCache` corruption bug with a `size_callback` that can return 0 (#11454)
When all entries in an `LruCache` have a size of 0 according to the
provided `size_callback`, and `drop_from_cache` is called on a cache
node, the node would be unlinked from the LRU linked list but remain in
the cache dictionary. An assertion would be later be tripped due to the
inconsistency.

Avoid unintentionally calling `__len__` and use a strict `is None`
check instead when unwrapping the weak reference.
2021-11-30 16:28:02 +00:00
Sean Quah 5a0b652d36
Eliminate a few `Any`s in `LruCache` type hints (#11453) 2021-11-30 15:39:07 +00:00
Dirk Klimpel 432a174bc1
Remove unnecessary `json.dumps` from `tests.rest.admin` (#11461)
The tests helpers automatically convert dictionaries to
JSON payloads, no need to do it manually for each
test.
2021-11-30 14:51:04 +00:00
Brendan Abolivier b14f8a1baf Merge branch 'master' into develop 2021-11-30 14:28:30 +00:00
Patrick Cloke 28f5252c1f
Add missing copyright header. (#11460) 2021-11-30 13:23:53 +00:00
Richard van der Hoff f13a8d1c69
synctl stop: wait for processes to exit (#11459)
If you're trying to shut down Synapse, it's rather handy if it *actually* shuts
down before you move on.
2021-11-30 11:51:12 +00:00
Marcus a9481223d1
Improved push typing (#11409)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2021-11-30 11:49:20 +00:00
Brendan Abolivier e713855dca
Merge trust_identity_server_for_password_resets PRs 2021-11-30 11:48:06 +00:00
Brendan Abolivier f663426804
Move notices up 2021-11-30 11:26:18 +00:00
Brendan Abolivier 3d831415cc
Fixup changelog 2021-11-30 11:25:11 +00:00
Brendan Abolivier 4bdad80de1 1.48.0 2021-11-30 11:24:21 +00:00
Dirk Klimpel 35b1900f00
Convert status codes to `HTTPStatus` in `tests.rest.admin` (#11455) 2021-11-30 09:53:54 +00:00
Dirk Klimpel e8ae94a223
Convert status codes to `HTTPStatus` in `synapse.rest.admin` (#11452) 2021-11-29 22:19:45 +00:00
Eric Eastwood fb58611d21
Refactor `backfilled` into specific behavior function arguments (`_persist_events_and_state_updates`) (#11417)
Part of https://github.com/matrix-org/synapse/issues/11300

Call stack:

 - `_persist_events_and_state_updates` (added `use_negative_stream_ordering`)
    - `_persist_events_txn`
       - `_update_room_depths_txn` (added `update_room_forward_stream_ordering`)
       - `_update_metadata_tables_txn`
          - `_store_room_members_txn` (added `inhibit_local_membership_updates`)

Using keyword-only arguments (`*`) to reduce the mistakes from `backfilled` being left as a positional argument somewhere and being interpreted wrong by our new arguments.
2021-11-29 16:01:54 -06:00
Patrick Cloke a4521ce0a8
Support the stable /hierarchy endpoint from MSC2946 (#11329)
This also makes additional updates where the implementation
had drifted from the approved MSC.

Unstable endpoints will be removed at a later data.
2021-11-29 14:32:20 -05:00
Erik Johnston d08ef6f155
Make background updates controllable via a plugin (#11306)
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-11-29 17:57:06 +01:00
Patrick Cloke 9d1971a5c4
Return the stable `event` field from `/send_join` per MSC3083. (#11413)
This does not remove the unstable field and still parses both.
Handling of the unstable field will need to be removed in the
future.
2021-11-29 15:43:20 +00:00
Sean Quah 7564b8e118
Update the media repository documentation (#11415) 2021-11-29 15:37:56 +00:00
reivilibre a82b90ab32
Add type annotations to some of the configuration surrounding refresh tokens. (#11428) 2021-11-29 13:34:14 +00:00
Richard van der Hoff 9cd13c5f63
Fix perspectives requests for multiple keys for the same server (#11440)
If we tried to request multiple keys for the same server, we would end up
dropping some of those requests.
2021-11-29 13:15:36 +00:00
Tulir Asokan 7b4e228e41
Fix using MSC2716 batch sending with event persistence workers (#11220)
Signed-off-by: Tulir Asokan <tulir@beeper.com>
2021-11-29 13:13:23 +00:00
David Robertson dc0a3cd596
disallow-untyped-defs for the module_api (#11029) 2021-11-29 11:28:12 +00:00
David Robertson aa457b625e
Fix changelog filename in #11441 2021-11-29 11:15:28 +00:00
David Robertson 776ad3e5e9
Add a test case for the SendJoinParser (#11441)
This would have caught the bug #11438 introduced in #11217 and fixed in #11439.
2021-11-29 11:11:46 +00:00
Daniel Molkentin e5c5e213ea
Bump ijson dependency to 3.1 (#11438) (#11439)
Since e81fa92648, Synapse depends on
the use_float flag which has been introduced in ijson 3.1 and
is not available in 3.0. This is known to cause runtime errors
with send_join.

Signed-off-by: Daniel Molkentin <danimo@infra.run>

Co-authored-by: Daniel Molkentin <danimo@infra.run>
2021-11-26 20:51:22 +00:00
reivilibre 1b6691dce4
Update MSC2918 refresh token support to confirm with the latest revision: accept the `refresh_tokens` parameter in the request body rather than in the URL parameters. (#11430) 2021-11-26 19:06:16 +00:00
Sean Quah ffd858aa68
Add type hints to `synapse/storage/databases/main/events_worker.py` (#11411)
Also refactor the stream ID trackers/generators a bit and try to
document them better.
2021-11-26 18:41:31 +00:00
reivilibre 1d8b80b334
Support expiry of refresh tokens and expiry of the overall session when refresh tokens are in use. (#11425) 2021-11-26 14:27:14 +00:00
Michael Kaye e2c300e7e4
Create healthcheck script for synapse-workers container (#11429)
The intent is to iterate through all the worker ports and only
report healthy when all are healthy, starting with the main process.
2021-11-26 14:05:20 +00:00
Sean Quah c675a18071
Track ongoing event fetches correctly (again) (#11376)
The previous fix for the ongoing event fetches counter
(8eec25a1d9) was both insufficient and
incorrect.

When the database is unreachable, `_do_fetch` never gets run and so
`_event_fetch_ongoing` is never decremented.

The previous fix also moved the `_event_fetch_ongoing` decrement outside
of the `_event_fetch_lock` which allowed race conditions to corrupt the
counter.
2021-11-26 13:47:24 +00:00
Brendan Abolivier c54c9df286
Fix docker hub name 2021-11-25 16:22:54 +00:00
Brendan Abolivier d4dcc0524f
Incorporate review from synapse-dev 2021-11-25 16:21:00 +00:00
Sean Quah 7862f821de
Annotate string constants in `synapse.api.constants` with `Final` (#11356)
This change makes mypy complain if the constants are ever reassigned,
and, more usefully, makes mypy type them as `Literal`s instead of `str`s,
allowing code of the following form to pass mypy:
```py
def do_something(membership: Literal["join", "leave"], ...): ...

do_something(Membership.JOIN, ...)
```
2021-11-25 16:14:23 +00:00
Brendan Abolivier b757b68454
Fixup changelog 2021-11-25 16:07:23 +00:00
Brendan Abolivier 946c102ac9 1.48.0rc1 2021-11-25 15:57:04 +00:00
Brendan Abolivier 0d88c4f903
Improve performance of `remove_{hidden,deleted}_devices_from_device_inbox` (#11421)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-11-25 15:14:54 +00:00
Brendan Abolivier 7f9841bdec
Lower minumum batch size to 1 for background updates (#11422)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-11-24 19:21:44 +00:00
reivilibre f25c75d376
Rename unstable `access_token_lifetime` configuration option to `refreshable_access_token_lifetime` to make it clear it only concerns refreshable access tokens. (#11388) 2021-11-23 17:01:34 +00:00
Patrick Cloke 55669bd3de
Add missing type hints to config base classes (#11377) 2021-11-23 15:21:19 +00:00
Shay 7cebaf9644
Remove code invalidated by deprecated config flag 'trust_identity_servers_for_password_resets' (#11395)
* remove background update code related to deprecated config flag

* changelog entry

* update changelog

* Delete 11394.removal

Duplicate, wrong number

* add no-op background update and change newfragment so it will be consolidated with associated work

* remove unused code

* Remove code associated with deprecated flag from legacy docker dynamic config file

Co-authored-by: reivilibre <oliverw@matrix.org>
2021-11-23 06:46:40 -08:00
Sean Quah 454c3d7694 Merge branch 'master' into develop 2021-11-23 13:06:56 +00:00
Sean Quah fcb9441791 Synapse 1.47.1 (2021-11-23)
===========================
 
 This release fixes a security issue in the media store, affecting all prior releases of Synapse. Server administrators are encouraged to update Synapse as soon as possible. We are not aware of these vulnerabilities being exploited in the wild.
 
 Server administrators who are unable to update Synapse may use the workarounds described in the linked GitHub Security Advisory below.
 
 Security advisory
 -----------------
 
 The following issue is fixed in 1.47.1.
 
 - **[GHSA-3hfw-x7gx-437c](https://github.com/matrix-org/synapse/security/advisories/GHSA-3hfw-x7gx-437c) / [CVE-2021-41281](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-41281): Path traversal when downloading remote media.**
 
   Synapse instances with the media repository enabled can be tricked into downloading a file from a remote server into an arbitrary directory, potentially outside the media store directory.
 
   The last two directories and file name of the path are chosen randomly by Synapse and cannot be controlled by an attacker, which limits the impact.
 
   Homeservers with the media repository disabled are unaffected. Homeservers configured with a federation whitelist are also unaffected.
 
   Fixed by [91f2bd090](https://github.com/matrix-org/synapse/commit/91f2bd090).
 -----BEGIN PGP SIGNATURE-----
 
 iQGzBAABCgAdFiEEWMTnW8Z8khaaf90R+84KzgcyGG8FAmGc1oUACgkQ+84Kzgcy
 GG9KbQv/e2KDS5Dfp0pxbVBDP368EaG7zh98/LiIPVzBHSj39BLDTaqCFSXYVlqd
 bQKKcYVt/iJFAcPnUPKV0qsqhrxbnaU1V0oeI8h5IaVUHIE5m8HVTMss5lvbLyIq
 flWbgYedoSL+vIFNFbsZqIGgFls0b0VDJJQCpl3gmkJgtruw/iFHaTnRjReooFpH
 ktK/Kt7iiGpgj2R9KsuZfmrUXJuCG4wiOjzkq/lYBwfMT5I0H8+wxG208YOvpcmb
 yVIPVCb7N2ol/6nrfFcH+vpMIuUjJnNn2XFVSEKyEBdh8oO19tVditMM3o1M/vSG
 A56D8lqcwhorr6utev1hFW/fxCTNkschxGSktAd9FN79b5OupKG1OlRI6bQJmuKa
 KecspFC/pxsB/tMRkp2XOGuIPuFXB2w9Uf4ZM3g2I1LaJ4omr/vX0f+1CDG+DhGd
 /xVif9Pw5AqRXdZQIBK2ZQq9DISdjcHhWgSrmQn5Hpp9uDrc4DR21RAXyApOd+vF
 cwP9WTnF
 =B6uZ
 -----END PGP SIGNATURE-----

Merge tag 'v1.47.1'

Synapse 1.47.1 (2021-11-23)
===========================

This release fixes a security issue in the media store, affecting all prior releases of Synapse. Server administrators are encouraged to update Synapse as soon as possible. We are not aware of these vulnerabilities being exploited in the wild.

Server administrators who are unable to update Synapse may use the workarounds described in the linked GitHub Security Advisory below.

Security advisory
-----------------

The following issue is fixed in 1.47.1.

- **[GHSA-3hfw-x7gx-437c](https://github.com/matrix-org/synapse/security/advisories/GHSA-3hfw-x7gx-437c) / [CVE-2021-41281](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-41281): Path traversal when downloading remote media.**

  Synapse instances with the media repository enabled can be tricked into downloading a file from a remote server into an arbitrary directory, potentially outside the media store directory.

  The last two directories and file name of the path are chosen randomly by Synapse and cannot be controlled by an attacker, which limits the impact.

  Homeservers with the media repository disabled are unaffected. Homeservers configured with a federation whitelist are also unaffected.

  Fixed by [91f2bd090](https://github.com/matrix-org/synapse/commit/91f2bd090).
2021-11-23 12:39:09 +00:00
Patrick Cloke 6a5dd485bd
Refactor the code to inject bundled relations during serialization. (#11408) 2021-11-23 06:43:56 -05:00
Kostas 1035663833
Add config for customizing the claim used for JWT logins. (#11361)
Allows specifying a different claim (from the default "sub") to use
when calculating the localpart of the Matrix ID used during the
JWT login.
2021-11-22 13:01:03 -05:00
Patrick Cloke 3d893b8cf2
Store arbitrary relations from events. (#11391)
Instead of only known relation types. This also reworks the background
update for thread relations to crawl events and search for any relation
type, not just threaded relations.
2021-11-22 12:01:47 -05:00