Commit Graph

1700 Commits

Author SHA1 Message Date
Olivier Wilkinson (reivilibre) e059094119 Synapse 1.42.0rc1 (2021-09-01)
==============================
 
 Server administrators are reminded to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420).
 
 Features
 --------
 
 - Add support for [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231): Token authenticated registration. Users can be required to submit a token during registration to authenticate themselves. Contributed by Callum Brown. ([\#10142](https://github.com/matrix-org/synapse/issues/10142))
 - Add support for [MSC3283](https://github.com/matrix-org/matrix-doc/pull/3283): Expose `enable_set_displayname` in capabilities. ([\#10452](https://github.com/matrix-org/synapse/issues/10452))
 - Port the `PresenceRouter` module interface to the new generic interface. ([\#10524](https://github.com/matrix-org/synapse/issues/10524))
 - Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946). ([\#10613](https://github.com/matrix-org/synapse/issues/10613), [\#10725](https://github.com/matrix-org/synapse/issues/10725))
 
 Bugfixes
 --------
 
 - Validate new `m.room.power_levels` events. Contributed by @aaronraimist. ([\#10232](https://github.com/matrix-org/synapse/issues/10232))
 - Display an error on User-Interactive Authentication fallback pages when authentication fails. Contributed by Callum Brown. ([\#10561](https://github.com/matrix-org/synapse/issues/10561))
 - Remove pushers when deleting an e-mail address from an account. Pushers for old unlinked emails will also be deleted. ([\#10581](https://github.com/matrix-org/synapse/issues/10581), [\#10734](https://github.com/matrix-org/synapse/issues/10734))
 - Reject Client-Server `/keys/query` requests which provide `device_ids` incorrectly. ([\#10593](https://github.com/matrix-org/synapse/issues/10593))
 - Rooms with unsupported room versions are no longer returned via `/sync`. ([\#10644](https://github.com/matrix-org/synapse/issues/10644))
 - Enforce the maximum length for per-room display names and avatar URLs. ([\#10654](https://github.com/matrix-org/synapse/issues/10654))
 - Fix a bug which caused the `synapse_user_logins_total` Prometheus metric not to be correctly initialised on restart. ([\#10677](https://github.com/matrix-org/synapse/issues/10677))
 - Improve `ServerNoticeServlet` to avoid duplicate requests and add unit tests. ([\#10679](https://github.com/matrix-org/synapse/issues/10679))
 - Fix long-standing issue which caused an error when a thumbnail is requested and there are multiple thumbnails with the same quality rating. ([\#10684](https://github.com/matrix-org/synapse/issues/10684))
 - Fix a regression introduced in v1.41.0 which affected the performance of concurrent fetches of large sets of events, in extreme cases causing the process to hang. ([\#10703](https://github.com/matrix-org/synapse/issues/10703))
 - Fix a regression introduced in Synapse 1.41 which broke email transmission on Systems using older versions of the Twisted library. ([\#10713](https://github.com/matrix-org/synapse/issues/10713))
 
 Improved Documentation
 ----------------------
 
 - Add documentation on how to connect Django with Synapse using OpenID Connect and django-oauth-toolkit. Contributed by @HugoDelval. ([\#10192](https://github.com/matrix-org/synapse/issues/10192))
 - Advertise https://matrix-org.github.io/synapse documentation in the `README` and `CONTRIBUTING` files. ([\#10595](https://github.com/matrix-org/synapse/issues/10595))
 - Fix some of the titles not rendering in the OpenID Connect documentation. ([\#10639](https://github.com/matrix-org/synapse/issues/10639))
 - Minor clarifications to the documentation for reverse proxies. ([\#10708](https://github.com/matrix-org/synapse/issues/10708))
 - Remove table of contents from the top of installation and contributing documentation pages. ([\#10711](https://github.com/matrix-org/synapse/issues/10711))
 
 Deprecations and Removals
 -------------------------
 
 - Remove deprecated Shutdown Room and Purge Room Admin API. ([\#8830](https://github.com/matrix-org/synapse/issues/8830))
 
 Internal Changes
 ----------------
 
 - Improve type hints for the proxy agent and SRV resolver modules. Contributed by @dklimpel. ([\#10608](https://github.com/matrix-org/synapse/issues/10608))
 - Clean up some of the federation event authentication code for clarity. ([\#10614](https://github.com/matrix-org/synapse/issues/10614), [\#10615](https://github.com/matrix-org/synapse/issues/10615), [\#10624](https://github.com/matrix-org/synapse/issues/10624), [\#10640](https://github.com/matrix-org/synapse/issues/10640))
 - Add a comment asking developers to leave a reason when bumping the database schema version. ([\#10621](https://github.com/matrix-org/synapse/issues/10621))
 - Remove not needed database updates in modify user admin API. ([\#10627](https://github.com/matrix-org/synapse/issues/10627))
 - Convert room member storage tuples to `attrs` classes. ([\#10629](https://github.com/matrix-org/synapse/issues/10629), [\#10642](https://github.com/matrix-org/synapse/issues/10642))
 - Use auto-attribs for the attrs classes used in sync. ([\#10630](https://github.com/matrix-org/synapse/issues/10630))
 - Make `backfill` and `get_missing_events` use the same codepath. ([\#10645](https://github.com/matrix-org/synapse/issues/10645))
 - Improve the performance of the `/hierarchy` API (from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946)) by caching responses received over federation. ([\#10647](https://github.com/matrix-org/synapse/issues/10647))
 - Run a nightly CI build against Twisted trunk. ([\#10651](https://github.com/matrix-org/synapse/issues/10651), [\#10672](https://github.com/matrix-org/synapse/issues/10672))
 - Do not print out stack traces for network errors when fetching data over federation. ([\#10662](https://github.com/matrix-org/synapse/issues/10662))
 - Simplify tests for device admin rest API. ([\#10664](https://github.com/matrix-org/synapse/issues/10664))
 - Add missing type hints to REST servlets. ([\#10665](https://github.com/matrix-org/synapse/issues/10665), [\#10666](https://github.com/matrix-org/synapse/issues/10666), [\#10674](https://github.com/matrix-org/synapse/issues/10674))
 - Flatten the `tests.synapse.rests` package by moving the contents of `v1` and `v2_alpha` into the parent. ([\#10667](https://github.com/matrix-org/synapse/issues/10667))
 - Update `complement.sh` to rebuild the base Docker image when run with workers. ([\#10686](https://github.com/matrix-org/synapse/issues/10686))
 - Split the event-processing methods in `FederationHandler` into a separate `FederationEventHandler`. ([\#10692](https://github.com/matrix-org/synapse/issues/10692))
 - Remove unused `compare_digest` function. ([\#10706](https://github.com/matrix-org/synapse/issues/10706))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCgAdFiEE8SRSDO7gYkSP4chELS76LzL74EcFAmEvd3cACgkQLS76LzL7
 4EcF0hAAvOjDeH2hn0eI4YX03/BTCesJ5pCTbEJeC9Hsui3SbWamyYarH0Ae0gM6
 BP9x5/XkCdsZhKT2OJedtZQUQ0cUws5FdJP87NnE1m7KBW9djbTxmkue3JKaHNTT
 ajvEoLe8UjOFbjvyMc+188uWVgtKQ6KLcetwllTpYY3FJ3+yk9CQP1ydYzwfHcHe
 hXW3b20WoRdAmH0B27mj1oQzCeXclbj1kCthcBcxH5RytA8bBc1itBtpAP2FFaeS
 Ml6paoXSs0wZaXXL5T1MhQtcVOOG5Yh7EPsemdyfqkwhIk9QtV3nmGfC7K5iEeq0
 DldTgBNE0wLO+ycdgEZ2oCvitCLT4WDoKkVqKPA91wpuvCMeK5lnDdqbFUSN3hfd
 l8Oz+ydhjsCx7+pBBopnsyGrPIHNaN/vySYGF3lPa3ANKS2J0IVlAaJ59bZqzg2C
 vzdyMBCnn0fhnXMkADPLnIYqtebvC09JlPBRGo5BKmwmlM8Ea1dLQ+Lcy5vXalcw
 +GaYw3U805c03EvsWQxWchjwq6IwMeP06YpK/1ruMjlrCWYJp3V7f/4hXjlTdYm5
 OEWc/kHOnlLfH2cA4aS0sV5WsnEuftotQIzBXuXWw5hSMdFmH7eAviPg0vXgfeA/
 i0FOkOgEYR8O3RMh4Cip8+h44t8SPNjYUFJK9ip3DmNateeUCH0=
 =XxMc
 -----END PGP SIGNATURE-----

Merge tag 'v1.42.0rc1' into develop

Synapse 1.42.0rc1 (2021-09-01)
==============================

Server administrators are reminded to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420).

Features
--------

- Add support for [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231): Token authenticated registration. Users can be required to submit a token during registration to authenticate themselves. Contributed by Callum Brown. ([\#10142](https://github.com/matrix-org/synapse/issues/10142))
- Add support for [MSC3283](https://github.com/matrix-org/matrix-doc/pull/3283): Expose `enable_set_displayname` in capabilities. ([\#10452](https://github.com/matrix-org/synapse/issues/10452))
- Port the `PresenceRouter` module interface to the new generic interface. ([\#10524](https://github.com/matrix-org/synapse/issues/10524))
- Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946). ([\#10613](https://github.com/matrix-org/synapse/issues/10613), [\#10725](https://github.com/matrix-org/synapse/issues/10725))

Bugfixes
--------

- Validate new `m.room.power_levels` events. Contributed by @aaronraimist. ([\#10232](https://github.com/matrix-org/synapse/issues/10232))
- Display an error on User-Interactive Authentication fallback pages when authentication fails. Contributed by Callum Brown. ([\#10561](https://github.com/matrix-org/synapse/issues/10561))
- Remove pushers when deleting an e-mail address from an account. Pushers for old unlinked emails will also be deleted. ([\#10581](https://github.com/matrix-org/synapse/issues/10581), [\#10734](https://github.com/matrix-org/synapse/issues/10734))
- Reject Client-Server `/keys/query` requests which provide `device_ids` incorrectly. ([\#10593](https://github.com/matrix-org/synapse/issues/10593))
- Rooms with unsupported room versions are no longer returned via `/sync`. ([\#10644](https://github.com/matrix-org/synapse/issues/10644))
- Enforce the maximum length for per-room display names and avatar URLs. ([\#10654](https://github.com/matrix-org/synapse/issues/10654))
- Fix a bug which caused the `synapse_user_logins_total` Prometheus metric not to be correctly initialised on restart. ([\#10677](https://github.com/matrix-org/synapse/issues/10677))
- Improve `ServerNoticeServlet` to avoid duplicate requests and add unit tests. ([\#10679](https://github.com/matrix-org/synapse/issues/10679))
- Fix long-standing issue which caused an error when a thumbnail is requested and there are multiple thumbnails with the same quality rating. ([\#10684](https://github.com/matrix-org/synapse/issues/10684))
- Fix a regression introduced in v1.41.0 which affected the performance of concurrent fetches of large sets of events, in extreme cases causing the process to hang. ([\#10703](https://github.com/matrix-org/synapse/issues/10703))
- Fix a regression introduced in Synapse 1.41 which broke email transmission on Systems using older versions of the Twisted library. ([\#10713](https://github.com/matrix-org/synapse/issues/10713))

Improved Documentation
----------------------

- Add documentation on how to connect Django with Synapse using OpenID Connect and django-oauth-toolkit. Contributed by @HugoDelval. ([\#10192](https://github.com/matrix-org/synapse/issues/10192))
- Advertise https://matrix-org.github.io/synapse documentation in the `README` and `CONTRIBUTING` files. ([\#10595](https://github.com/matrix-org/synapse/issues/10595))
- Fix some of the titles not rendering in the OpenID Connect documentation. ([\#10639](https://github.com/matrix-org/synapse/issues/10639))
- Minor clarifications to the documentation for reverse proxies. ([\#10708](https://github.com/matrix-org/synapse/issues/10708))
- Remove table of contents from the top of installation and contributing documentation pages. ([\#10711](https://github.com/matrix-org/synapse/issues/10711))

Deprecations and Removals
-------------------------

- Remove deprecated Shutdown Room and Purge Room Admin API. ([\#8830](https://github.com/matrix-org/synapse/issues/8830))

Internal Changes
----------------

- Improve type hints for the proxy agent and SRV resolver modules. Contributed by @dklimpel. ([\#10608](https://github.com/matrix-org/synapse/issues/10608))
- Clean up some of the federation event authentication code for clarity. ([\#10614](https://github.com/matrix-org/synapse/issues/10614), [\#10615](https://github.com/matrix-org/synapse/issues/10615), [\#10624](https://github.com/matrix-org/synapse/issues/10624), [\#10640](https://github.com/matrix-org/synapse/issues/10640))
- Add a comment asking developers to leave a reason when bumping the database schema version. ([\#10621](https://github.com/matrix-org/synapse/issues/10621))
- Remove not needed database updates in modify user admin API. ([\#10627](https://github.com/matrix-org/synapse/issues/10627))
- Convert room member storage tuples to `attrs` classes. ([\#10629](https://github.com/matrix-org/synapse/issues/10629), [\#10642](https://github.com/matrix-org/synapse/issues/10642))
- Use auto-attribs for the attrs classes used in sync. ([\#10630](https://github.com/matrix-org/synapse/issues/10630))
- Make `backfill` and `get_missing_events` use the same codepath. ([\#10645](https://github.com/matrix-org/synapse/issues/10645))
- Improve the performance of the `/hierarchy` API (from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946)) by caching responses received over federation. ([\#10647](https://github.com/matrix-org/synapse/issues/10647))
- Run a nightly CI build against Twisted trunk. ([\#10651](https://github.com/matrix-org/synapse/issues/10651), [\#10672](https://github.com/matrix-org/synapse/issues/10672))
- Do not print out stack traces for network errors when fetching data over federation. ([\#10662](https://github.com/matrix-org/synapse/issues/10662))
- Simplify tests for device admin rest API. ([\#10664](https://github.com/matrix-org/synapse/issues/10664))
- Add missing type hints to REST servlets. ([\#10665](https://github.com/matrix-org/synapse/issues/10665), [\#10666](https://github.com/matrix-org/synapse/issues/10666), [\#10674](https://github.com/matrix-org/synapse/issues/10674))
- Flatten the `tests.synapse.rests` package by moving the contents of `v1` and `v2_alpha` into the parent. ([\#10667](https://github.com/matrix-org/synapse/issues/10667))
- Update `complement.sh` to rebuild the base Docker image when run with workers. ([\#10686](https://github.com/matrix-org/synapse/issues/10686))
- Split the event-processing methods in `FederationHandler` into a separate `FederationEventHandler`. ([\#10692](https://github.com/matrix-org/synapse/issues/10692))
- Remove unused `compare_digest` function. ([\#10706](https://github.com/matrix-org/synapse/issues/10706))
2021-09-01 14:58:14 +01:00
Olivier Wilkinson (reivilibre) 6b2aca473a 1.42.0rc1 2021-09-01 11:47:24 +01:00
Patrick Cloke e2481dbe93
Allow configuration of the oEmbed URLs. (#10714)
This adds configuration options (under an `oembed` section) to
configure which URLs are matched to use oEmbed for URL
previews.
2021-08-31 18:37:07 -04:00
David Robertson 46ff99ef95
Advertise matrix-org.github.io/synapse docs (#10595)
Point to the book where possible, and use hyperlinks to github to refer to files not included in the book.

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-08-31 11:29:27 +01:00
Azrenbeth 54aa7047eb
Removed page summaries from the top of installation and contributing doc pages (#10711)
- Removed page summaries from CONTRIBUTING and installation pages as 
this information was already in the table of contents on the right hand side
- Fixed some broken links in CONTRIBUTING
- Added margin-right tag for when table of contents is being shown
(otherwise the text in the page sometimes overlaps with it)
2021-08-27 14:19:17 +00:00
Richard van der Hoff 051ddac53b
Clarifications to reverse_proxy.md (#10708)
* Update reverse_proxy.md

* Create 10708.doc
2021-08-27 12:54:21 +01:00
Azrenbeth ad17fbd20e
Remove pushers when deleting 3pid from account (#10581)
When a user deletes an email from their account it will
now also remove all pushers for that email and that user
(even if these pushers were created by a different client)
2021-08-26 13:53:57 +01:00
Hugo DELVAL 86415f162d
doc: add django-oauth-toolkit to oidc doc (#10192)
Signed-off-by: Hugo Delval <hugo.delval@gmail.com>
2021-08-23 17:12:36 +00:00
Andrew Morgan 3e83f97154
Fix the titles in the OIDC documentation (#10639)
* Fix the titles in the OIDC documentation

Having them as links broke the table-of-contents rendering in mdbook.
Plus there's no reason for only some of the provider titles to be links.

* Changelog

* Add link to google idp docs
2021-08-23 14:58:31 +01:00
Callum Brown 947dbbdfd1
Implement MSC3231: Token authenticated registration (#10142)
Signed-off-by: Callum Brown <callum@calcuode.com>

This is part of my GSoC project implementing [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231).
2021-08-21 22:14:43 +01:00
Dirk Klimpel 220f901229
Remove not needed database updates in modify user admin API (#10627) 2021-08-19 10:25:05 +01:00
Erik Johnston 78a70a2e0b Merge branch 'release-v1.41' into develop 2021-08-18 17:02:47 +01:00
Erik Johnston b9c35586a4 Update docs/upgrade.md with new version 2021-08-18 16:59:36 +01:00
Andrew Morgan 3692f7fd33
Mount /_synapse/admin/v1/users/{userId}/media admin API on media workers only (#10628)
Co-authored-by: Patrick Cloke <patrickc@matrix.org>
2021-08-18 13:25:12 +01:00
Callum Brown 6e613a10d0
Display an error page during failure of fallback UIA. (#10561) 2021-08-18 08:13:35 -04:00
Dirk Klimpel 430241a1e9
Remove deprecated Shutdown Room and Purge Room Admin API (#8830) 2021-08-17 21:19:13 +01:00
Andrew Morgan 703e3a9e85
Allow /createRoom to be run on workers (#10564)
Fixes https://github.com/matrix-org/synapse/issues/7867
2021-08-17 14:33:16 +01:00
Azrenbeth 1a9f531c79
Port the PresenceRouter module interface to the new generic interface (#10524)
Port the PresenceRouter module interface to the new generic interface introduced in v1.37.0
2021-08-17 13:22:45 +00:00
Brendan Abolivier 5639759980
Centralise the custom template directory (#10596)
Several configuration sections are using separate settings for custom template directories, which can be confusing. This PR adds a new top-level configuration for a custom template directory which is then used for every module. The only exception is the consent templates, since the consent template directory require a specific hierarchy, so it's probably better that it stays separate from everything else.
2021-08-17 14:45:24 +02:00
reivilibre 5f7b1e1f27
Make `PeriodicallyFlushingMemoryHandler` the default logging handler. (#10518) 2021-08-17 13:13:11 +01:00
Dirk Klimpel 3bcd525b46
Allow to edit `external_ids` by Edit User admin API (#10598)
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-08-17 11:56:11 +01:00
Will Hunt a933c2c7d8
Add an admin API to check if a username is available (#10578)
This adds a new API GET /_synapse/admin/v1/username_available?username=foo to check if a username is available. It is the counterpart to https://matrix.org/docs/spec/client_server/r0.6.0#get-matrix-client-r0-register-available, except that it works even if registration is disabled.
2021-08-17 10:52:38 +01:00
reivilibre 19e51b14d2
Manhole: wrap coroutines in `defer.ensureDeferred` automatically (#10602) 2021-08-16 18:11:48 +01:00
Dirk Klimpel 915b37e5ef
Admin API to delete media for a specific user (#10558) 2021-08-11 19:29:59 +00:00
Dirk Klimpel 339c3918e1
support federation queries through http connect proxy (#10475)
Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-08-11 15:34:59 +01:00
Drew Short 189c055eb6
Moved homeserver documentation above reverse proxy examples (#10551)
Signed-off-by: Drew Short <warrick@sothr.com>
2021-08-09 15:12:53 +00:00
Richard van der Hoff 74d7336686
Add a setting to disable TLS for sending email (#10546)
This is mostly useful in case the server offers TLS, but doesn't present a valid certificate.
2021-08-06 10:13:34 +00:00
Richard van der Hoff 0cb4274dbf Merge remote-tracking branch 'origin/release-v1.40' into develop 2021-08-05 19:01:03 +01:00
Dirk Klimpel 4578531002
fix broken links in `upgrade.md` (#10543)
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-08-05 19:00:44 +01:00
Dirk Klimpel 834cdc3606
Add documentation for configuring a forward proxy. (#10443) 2021-08-05 07:20:05 -04:00
Kento Okamoto 72935b7c50
Add warnings to ip_range_blacklist usage with proxies (#10129)
Per issue #9812 using `url_preview_ip_range_blacklist` with a proxy via `HTTPS_PROXY` or `HTTP_PROXY` environment variables has some inconsistent bahavior than mentioned. This PR changes the following:

- Changes the Sample Config file to include a note mentioning that `url_preview_ip_range_blacklist` and `ip_range_blacklist` is ignored when using a proxy
- Changes some logic in synapse/config/repository.py to send a warning when both `*ip_range_blacklist` configs and a proxy environment variable are set and but no longer throws an error.

Signed-off-by: Kento Okamoto <kentokamoto@protonmail.com>
2021-08-03 18:13:34 +00:00
Richard van der Hoff 4b10880da3
Make sync response cache time configurable. (#10513) 2021-08-03 14:45:04 +01:00
reivilibre f4ac934afe
Revert use of PeriodicallyFlushingMemoryHandler by default (#10515) 2021-08-03 11:30:39 +01:00
Eric Eastwood 2bae2c632f
Add developer documentation to explain room DAG concepts like `outliers` and `state_groups` (#10464) 2021-08-03 10:08:57 +00:00
Toni Spets ba5287f5e8
Allow setting transaction limit for db connections (#10440)
Setting the value will help PostgreSQL free up memory by recycling
the connections in the connection pool.

Signed-off-by: Toni Spets <toni.spets@iki.fi>
2021-08-02 13:24:43 +00:00
reivilibre e16eab29d6
Add a PeriodicallyFlushingMemoryHandler to prevent logging silence (#10407)
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2021-07-27 14:32:05 +01:00
Dirk Klimpel 89c4ca81bb
Add `creation_ts` to list users admin API (#10448)
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-07-22 16:05:16 +02:00
Eric Eastwood d518b05a86
Move dev/ docs to development/ (#10453) 2021-07-22 12:58:24 +02:00
Patrick Cloke 5b68816de9
Fix the hierarchy of OpenID providers in the docs. (#10445) 2021-07-21 13:48:06 -04:00
Brendan Abolivier a743bf4694
Port the ThirdPartyEventRules module interface to the new generic interface (#10386)
Port the third-party event rules interface to the generic module interface introduced in v1.37.0
2021-07-20 12:39:46 +02:00
Brendan Abolivier 36dc15412d
Add a module type for account validity (#9884)
This adds an API for third-party plugin modules to implement account validity, so they can provide this feature instead of Synapse. The module implementing the current behaviour for this feature can be found at https://github.com/matrix-org/synapse-email-account-validity.

To allow for a smooth transition between the current feature and the new module, hooks have been added to the existing account validity endpoints to allow their behaviours to be overridden by a module.
2021-07-16 18:11:53 +02:00
Brendan Abolivier 9f497024aa Merge branch 'master' into develop 2021-07-15 14:54:45 +01:00
Brendan Abolivier 3fffb71254
Make deprecation notice of the spam checker doc more obvious (#10395) 2021-07-15 15:54:22 +02:00
Luke Walsh 23a90a6a5c
Updating install prerequisites for newer macOS & ARM Macs. (#9971) 2021-07-15 13:18:58 +01:00
Andrew Morgan c141455049
Docs: Use something other than the document name to describe a page (#10399)
Our documentation has a history of using a document's name as a way to link to it, such as "See [workers.md]() for details". This makes sense when you're traversing a directory of files, but less sense when the files are abstracted away - as they are on the documentation website.

This PR changes the links to various documentation pages to something that fits better into the surrounding sentence, as you would when making any hyperlink on the web.
2021-07-15 12:47:55 +01:00
Brendan Abolivier 5ecad4e7a5
Update the logcontext doc (#10353)
By referring to awaitables instead of deferreds.
2021-07-15 11:38:05 +01:00
Moritz Dietz 0ae95b3847
doc: Add delegation example to the caddy reverse proxy section (#10368) 2021-07-14 18:50:30 +02:00
Richard van der Hoff 28ffff73c1
Instructions on installing RC debs (#10396) 2021-07-14 17:12:01 +01:00
Dirk Klimpel e938f69697
Fix some links in `docs` and `contrib` (#10370) 2021-07-13 11:55:48 +01:00
Erik Johnston 251cfc4e09 Synapse 1.38.0rc2 (2021-07-09)
==============================
 
 Bugfixes
 --------
 
 - Fix bug where inbound federation in a room could be delayed due to not correctly dropping a lock. Introduced in v1.37.1. ([\#10336](https://github.com/matrix-org/synapse/issues/10336))
 
 Improved Documentation
 ----------------------
 
 - Update links to documentation in the sample config. Contributed by @dklimpel. ([\#10287](https://github.com/matrix-org/synapse/issues/10287))
 - Fix broken links in [INSTALL.md](INSTALL.md). Contributed by @dklimpel. ([\#10331](https://github.com/matrix-org/synapse/issues/10331))
 -----BEGIN PGP SIGNATURE-----
 
 iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAmDoH+4QHGVyaWtAbWF0
 cml4Lm9yZwAKCRClQuTtGw+sCXxYCACneuRvkdvYqiH+PhPe8tXqhhJIifH1LecY
 FlJqp4OJPR2VFzio1btsgpRPQyLBLHZkJ9pgWsXAETbYOO+hSeOc4nIHsyqlSJhe
 v01sCUE4sle3DBrw15fG4XpercsiM3TFMyR9pV9laq9nIn8j+CY5K6W5t12/mYGy
 asHS0IKilCMhJlFwgE3eBr6P6fywi0JoIrr8EpfIs4eC2qDFpUlsrAQSkbE1JvdP
 O4BGZJKVysg3a6WYSWdJytqLYe942k8qUF4B4h4VmQi0xbuKSsTLiK/cFC8ohRMv
 E+O5O/KgwqwE/XOcukbsjlHxuiiFZTq6154PwLxXUpNnsMNn2/ph
 =6iBw
 -----END PGP SIGNATURE-----

Merge tag 'v1.38.0rc2' into develop

Synapse 1.38.0rc2 (2021-07-09)
==============================

Bugfixes
--------

- Fix bug where inbound federation in a room could be delayed due to not correctly dropping a lock. Introduced in v1.37.1. ([\#10336](https://github.com/matrix-org/synapse/issues/10336))

Improved Documentation
----------------------

- Update links to documentation in the sample config. Contributed by @dklimpel. ([\#10287](https://github.com/matrix-org/synapse/issues/10287))
- Fix broken links in [INSTALL.md](INSTALL.md). Contributed by @dklimpel. ([\#10331](https://github.com/matrix-org/synapse/issues/10331))
2021-07-09 11:26:17 +01:00
Brendan Abolivier 33ae301fee
Fix formatting in the logcontext doc (#10337) 2021-07-08 18:16:30 +02:00
Cristina f6767abc05
Remove functionality associated with unused historical stats tables (#9721)
Fixes #9602
2021-07-08 16:57:13 +01:00
Dirk Klimpel 974261cd81
Fix broken links in INSTALL.md (#10331)
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-07-08 15:46:13 +01:00
Richard van der Hoff 189652b2fe
Fix a broken link in the admin api docs (#10322)
* Fix a broken link in the admin api docs

* Rename 10321.doc to 10321.docs

* Rename 10321.docs to 10322.doc
2021-07-07 12:54:57 +01:00
Richard van der Hoff 240b3ce253 Merge remote-tracking branch 'origin/release-v1.38' into develop
merge @dklimpel's fixes to the sample config
2021-07-07 12:37:10 +01:00
Dirk Klimpel 56fd5fa8e1
Update links to documentation in sample config (#10287)
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-07-07 12:35:45 +01:00
Dirk Klimpel 2d044667cf
Simplify structure of room admin API docs (#10313) 2021-07-07 12:18:36 +01:00
Richard van der Hoff d7a94a7dcc
Add upgrade notes about disk space for events migration (#10314) 2021-07-06 11:00:05 +01:00
Erik Johnston 7a5873277e
Add support for evicting cache entries based on last access time. (#10205) 2021-07-05 16:32:12 +01:00
Brendan Abolivier 10671da05b
Fix bad link in modules documentation (#10302)
Fix link in modules doc to point at instructions on registering a callback instead of ones on registering a web resource.
2021-07-02 13:20:43 +02:00
Dirk Klimpel 6c02cca95f
Add SSO `external_ids` to Query User Account admin API (#10261)
Related to #10251
2021-07-01 11:26:24 +02:00
Andrew Morgan 04c8f308f4
Fix the homeserver config example in presence router docs (#10288)
The presence router docs include some sample homeserver config. At some point we changed the name of the [config option](859dc05b36/docs/sample_config.yaml (L104-L113)), but forgot to update the docs.

I've also added `presence.enabled: true` to the example, as that's the new way to enable presence (the `presence_enabled` option has been deprecated).
2021-06-30 23:43:58 +01:00
Andrew Morgan 7f25d73859
Convert UPGRADE.rst to markdown (#10166)
This PR:

* Converts UPGRADE.rst to markdown and moves the contents into the `docs/` directory.
* Updates the contents of UPGRADE.rst to point to the website instead.
* Updates links around the codebase that point to UPGRADE.rst.

`pandoc` + some manual editing was used to convert from RST to md.
2021-06-23 16:57:57 +01:00
jkanefendt 756fd513df
Implement config option `sso.update_profile_information` (#10108)
Implemented config option sso.update_profile_information to keep user's display name in sync with the SSO displayname.

Signed-off-by: Johannes Kanefendt <johannes.kanefendt@krzn.de>
2021-06-21 23:48:57 +01:00
Brendan Abolivier e9f2ad8603
Describe callbacks signatures as async in new modules doc (#10206) 2021-06-18 16:55:53 +02:00
Brendan Abolivier 1b3e398bea
Standardise the module interface (#10062)
This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.
2021-06-18 12:15:52 +01:00
Brendan Abolivier 08c8469322
Remove support for ACME v1 (#10194)
Fixes #9778

ACME v1 has been fully decommissioned for existing installs on June 1st 2021(see https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/27), so we can now safely remove it from Synapse.
2021-06-17 18:56:48 +01:00
Patrick Cloke 76f9c701c3
Always require users to re-authenticate for dangerous operations. (#10184)
Dangerous actions means deactivating an account, modifying an account
password, or adding a 3PID.

Other actions (deleting devices, uploading keys) can re-use the same UI
auth session if ui_auth.session_timeout is configured.
2021-06-16 11:07:28 -04:00
Michael Kaye b8b282aa32
A guide to the request log lines format. (#8436)
This doc is short but a useful guide to what the request log lines mean.

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
Co-authored-by: Erik Johnston <erik@matrix.org>
Co-authored-by: Daniele Sluijters <daenney@users.noreply.github.com>
2021-06-16 13:31:55 +01:00
Dirk Klimpel 0adc2882c1
Fix broken links in documentation (#10180)
* Fix broken links in documentation

* newsfile
2021-06-16 13:15:52 +01:00
Richard van der Hoff 13577aa55e
Notes on boolean columns in database schemas (#10164) 2021-06-11 17:13:56 +01:00
Richard van der Hoff c1b9922498
Support for database schema version ranges (#9933)
This is essentially an implementation of the proposal made at https://hackmd.io/@richvdh/BJYXQMQHO, though the details have ended up looking slightly different.
2021-06-11 14:45:53 +01:00
Aaron Raimist e6245e6d48
Mention that you need to configure max upload size in reverse proxy as well (#10122)
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-06-10 11:40:24 +01:00
Dan Callahan 7dc14730d9
Name release branches just after major.minor (#10013)
With the prior format, 1.33.0 / 1.33.1 / 1.33.2 got separate branches:

    release-v1.33.0
    release-v1.33.1
    release-v1.33.2

Under the new model, all three would share a common branch:

    release-v1.33

As before, RCs and actual releases exist as tags on these branches.

This better reflects our support model, e.g., that the "1.33" series had
a formal release followed by two patches / updates.

Signed-off-by: Dan Callahan <danc@element.io>
2021-06-08 11:44:50 +01:00
Rohan Sharma beb251e3ee
Make link in docs use HTTPS (#10130)
Fixes #10121

Signed-off-by: Rohan Sharma <rhnsharma5113@gmail.com>
2021-06-07 16:35:02 +01:00
Andrew Morgan fd9856e4a9
Compile and render Synapse's docs into a browsable, mobile-friendly and searchable website (#10086) 2021-06-03 17:20:40 +01:00
Andrew Morgan 73636cab69
Convert admin api docs to markdown (#10089)
So that they render nicely in mdbook (see #10086), and so that we no longer have a mix of structured text languages in our documentation (excluding files outside of `docs/`).
2021-06-03 14:06:03 +01:00
Dirk Klimpel 0284d2a297
Add new admin APIs to remove media by media ID from quarantine. (#10044)
Related to: #6681, #5956, #10040

Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-06-02 18:50:35 +01:00
Erik Johnston 5eed6348ce
Move some more endpoints off master (#10084) 2021-05-27 22:45:43 +01:00
Callum Brown 8fb9af570f
Make reason and score optional for report_event (#10077)
Implements MSC2414: https://github.com/matrix-org/matrix-doc/pull/2414
See #8551 

Signed-off-by: Callum Brown <callum@calcuode.com>
2021-05-27 18:42:23 +01:00
Patrick Cloke f42e4c4eb9
Remove the experimental spaces enabled flag. (#10063)
In lieu of just always enabling the unstable spaces endpoint and
unstable room version.
2021-05-26 14:35:16 -04:00
Dirk Klimpel 65e6c64d83
Add an admin API for unprotecting local media from quarantine (#10040)
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-05-26 11:19:47 +01:00
Aaron Raimist 3e1beb75e6
Update CAPTCHA documentation to mention turning off verify origin feature (#10046)
* Update CAPTCHA documentation to mention turning off verify origin

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-05-26 10:55:30 +01:00
Jerin J Titus 057ce7b754
Remove tls_fingerprints option (#9280)
Signed-off-by: Jerin J Titus <72017981+jerinjtitus@users.noreply.github.com>
2021-05-24 17:43:30 +01:00
Patrick Cloke 316f89e87f
Enable experimental spaces by default. (#10011)
The previous spaces_enabled flag now defaults to true and
is exposed in the sample config.
2021-05-24 08:57:14 -04:00
Dirk Klimpel 387c297489
Add missing entry to the table of contents of room admin API (#10043) 2021-05-24 07:37:30 -04:00
Andrew Morgan d983ced596 Merge branch 'master' of github.com:matrix-org/synapse into develop 2021-05-20 14:39:46 +01:00
Javier Junquera Sánchez 141b073c7b
Update user_directory.md (#10016)
Signed-off-by: Javier Junquera Sánchez <javier@junquera.io>
2021-05-20 14:24:19 +01:00
Savyasachee Jha 5bba1b4905
Hardened systemd unit files (#9803)
Signed-off-by: Savyasachee Jha savya.jha@hawkradius.com
2021-05-19 11:44:16 +01:00
Andrew Morgan 4d6e5a5e99
Use a database table to hold the users that should have full presence sent to them, instead of something in-memory (#9823) 2021-05-18 14:13:45 +01:00
Richard van der Hoff 6660912226
Update postgres docs (#9989) 2021-05-14 13:14:48 +01:00
Richard van der Hoff c14f99be46
Support enabling opentracing by user (#9978)
Add a config option which allows enabling opentracing by user id, eg for
debugging requests made by a test user.
2021-05-14 10:51:08 +01:00
Andrew Morgan 976216959b
Update minimum supported version in postgres.md (#9988) 2021-05-14 09:21:00 +01:00
Patrick Cloke d19bccdbec
Update SSO mapping providers documentation about unique IDs. (#9980) 2021-05-13 14:37:20 -04:00
Patrick Cloke 28c6841102
Send the `m.room.create` stripped event with invites (support MSC1772). (#9966)
MSC1772 specifies the m.room.create event should be sent as part
of the invite_state. This was done optionally behind an experimental
flag, but is now done by default due to MSC1772 being approved.
2021-05-11 10:58:58 -04:00
Aaron Raimist dc6366a9bd
Add config option to hide device names over federation (#9945)
Now that cross signing exists there is much less of a need for other people to look at devices and verify them individually. This PR adds a config option to allow you to prevent device display names from being shared with other servers.

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-05-11 14:03:23 +01:00
Erik Johnston 1fb9a2d0bf
Limit how often GC happens by time. (#9902)
Synapse can be quite memory intensive, and unless care is taken to tune
the GC thresholds it can end up thrashing, causing noticable performance
problems for large servers. We fix this by limiting how often we GC a
given generation, regardless of current counts/thresholds.

This does not help with the reverse problem where the thresholds are set
too high, but that should only happen in situations where they've been
manually configured.

Adds a `gc_min_seconds_between` config option to override the defaults.

Fixes #9890.
2021-05-05 16:53:45 +01:00
Andrew Morgan b85821aca2
Add port parameter to the sample config for psycopg2 args (#9911)
Adds the `port` option with the default value to the sample config file.
2021-05-04 13:28:59 +01:00
ThibF 0085dc5abc
Delete room endpoint (#9889)
Support the delete of a room through DELETE request and mark
previous request as deprecated through documentation.

Signed-off-by: Thibault Ferrante <thibault.ferrante@pm.me>
2021-04-29 10:31:45 +01:00
Patrick Cloke 683d6f75af
Rename handler and config modules which end in handler/config. (#9816) 2021-04-20 14:55:20 -04:00
Andrew Morgan 71f0623de9
Port "Allow users to click account renewal links multiple times without hitting an 'Invalid Token' page #74" from synapse-dinsic (#9832)
This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
2021-04-19 19:16:34 +01:00
Dirk Klimpel 1fc97ee876
Add an admin API to manage ratelimit for a specific user (#9648) 2021-04-13 10:26:37 +01:00
Andrew Morgan 04819239ba
Add a Synapse Module for configuring presence update routing (#9491)
At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though.

This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around.

A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync. 

The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being:

* Sending state for a specific set or all known users to a defined set of local and remote users.
* The ability to trigger an initial sync for specific users, so they receive all current state.
2021-04-06 14:38:30 +01:00
Erik Johnston 5688a74cf3 Merge branch 'master' into develop 2021-04-06 13:29:29 +01:00
Erik Johnston a888cbdd31
Add deprecation policy doc (#9723)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-04-06 13:07:36 +01:00
Dirk Klimpel bb0fe02a52
Add `order_by` to list user admin API (#9691) 2021-04-01 11:28:53 +01:00
Denis Kasak 5ff8eb97c6
Make sample config allowed_local_3pids regex stricter. (#9719)
The regex should be terminated so that subdomain matches of another
domain are not accepted. Just ensuring that someone doesn't shoot
themselves in the foot by copying our example.

Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
2021-03-31 12:27:20 +00:00
Richard van der Hoff f02663c4dd
Replace `room_invite_state_types` with `room_prejoin_state` (#9700)
`room_invite_state_types` was inconvenient as a configuration setting, because
anyone that ever set it would not receive any new types that were added to the
defaults. Here, we deprecate the old setting, and replace it with a couple of
new settings under `room_prejoin_state`.
2021-03-30 12:12:44 +01:00
Richard van der Hoff 4bbd535450
Update the OIDC sample config (#9695)
I've reiterated the advice about using `oidc` to migrate, since I've seen a few
people caught by this.

I've also removed a couple of the examples as they are duplicating the OIDC
documentation, and I think they might be leading people astray.
2021-03-29 15:40:11 +01:00
Paul Tötterman d7d4232a2d
Preserve host in example apache config (#9696)
Fixes redirect loop

Signed-off-by: Paul Tötterman <paul.totterman@iki.fi>
2021-03-26 10:38:31 +00:00
Ben Banfield-Zanin 4655d2221e
docs: fallback/web endpoint does not appear to be mounted on workers (#9679) 2021-03-24 11:43:04 +00:00
Erik Johnston e550ab17ad
Increase default join burst ratelimiting (#9674)
It's legitimate behaviour to try and join a bunch of rooms at once.
2021-03-23 14:52:20 +00:00
Hubbe dd5e5dc1d6
Add SSO attribute requirements for OIDC providers (#9609)
Allows limiting who can login using OIDC via the claims
made from the IdP.
2021-03-16 11:46:07 -04:00
Andrew Morgan 847ecdd8fa
Pass SSO IdP information to spam checker's registration function (#9626)
Fixes https://github.com/matrix-org/synapse/issues/9572

When a SSO user logs in for the first time, we create a local Matrix user for them. This goes through the register_user flow, which ends up triggering the spam checker. Spam checker modules don't currently have any way to differentiate between a user trying to sign up initially, versus an SSO user (whom has presumably already been approved elsewhere) trying to log in for the first time.

This PR passes `auth_provider_id` as an argument to the `check_registration_for_spam` function. This argument will contain an ID of an SSO provider (`"saml"`, `"cas"`, etc.) if one was used, else `None`.
2021-03-16 12:41:41 +00:00
Richard van der Hoff dd69110d95
Add support for stable MSC2858 API (#9617)
The stable format uses different brand identifiers, so we need to support two
identifiers for each IdP.
2021-03-16 11:21:26 +00:00
Richard van der Hoff 5b5bc188cf
Clean up config settings for stats (#9604)
... and complain if people try to turn it off.
2021-03-16 10:57:54 +00:00
Jason Robinson 17cd48fe51
Fix spam checker modules documentation example (#9580)
Mention that parse_config must exist and note the
check_media_file_for_spam method.
2021-03-10 10:42:51 -05:00
Andrew Morgan e9df3f496b
Link to the List user's media admin API from media Admin API docs (#9571)
Earlier [I was convinced](https://github.com/matrix-org/synapse/issues/9565) that we didn't have an Admin API for listing media uploaded by a user. Foolishly I was looking under the Media Admin API documentation, instead of the User Admin API documentation.

I thought it'd be helpful to link to the latter so others don't hit the same dead end :)
2021-03-09 15:15:52 +00:00
Richard van der Hoff eaada74075
JWT OIDC secrets for Sign in with Apple (#9549)
Apple had to be special. They want a client secret which is generated from an EC key.

Fixes #9220. Also fixes #9212 while I'm here.
2021-03-09 15:03:37 +00:00
Erik Johnston 576c91c7c1 Fixup sample config
After 0764d0c6e5
2021-03-09 11:40:45 +00:00
Leo Bärring 0fc4eb103a
Update reverse proxy to add OpenBSD relayd example configuration. (#9508)
Update reverse proxy to add OpenBSD relayd example configuration.

Signed-off-by: Leo Bärring <leo.barring@protonmail.com>
2021-03-06 11:49:19 +00:00
Richard van der Hoff a5daae2a5f
Update nginx reverse-proxy docs (#9512)
Turns out nginx overwrites the Host header by default.
2021-03-03 11:08:11 +00:00
Richard van der Hoff d8e95e5452
Add support for X-Forwarded-Proto (#9472)
rewrite XForwardedForRequest to set `isSecure()` based on
`X-Forwarded-Proto`. Also implement `getClientAddress()` while we're here.
2021-02-24 18:11:33 +00:00
Andrew Morgan 00bf80cb8e Fix typo in spam checker documentation 2021-02-24 17:51:52 +00:00
Erik Johnston 66f4949e7f
Fix deleting pushers when using sharded pushers. (#9465) 2021-02-22 21:14:42 +00:00
Dirk Klimpel 71c9f8de6d
Add an `order_by` field to list users' media admin API. (#8978) 2021-02-22 14:38:51 -05:00
Richard van der Hoff 70ea9593ff
example systemd config: propagate reloads to units (#9463)
It should be possible to reload `synapse.target` to have the reload propagate
to all the synapse units.
2021-02-22 18:43:17 +00:00
Andrew Morgan e22b71810e
Clean up the user directory sample config section (#9385)
The user directory sample config section was a little messy, and didn't adhere to our [recommended config format guidelines](https://github.com/matrix-org/synapse/blob/develop/docs/code_style.md#configuration-file-format).

This PR cleans that up a bit.
2021-02-22 11:44:31 +00:00
Andrew Morgan 13e9029f44
Add a config option to prioritise local users in user directory search results (#9383)
This PR adds a homeserver config option, `user_directory.prefer_local_users`, that when enabled will show local users higher in user directory search results than remote users. This option is off by default.

Note that turning this on doesn't necessarily mean that remote users will always be put below local users, but they should be assuming all other ranking factors (search query match, profile information present etc) are identical.

This is useful for, say, University networks that are openly federating, but want to prioritise local students and staff in the user directory over other random users.
2021-02-19 11:02:03 +00:00
AndrewFerr 9bc74743d5
Add configs to make profile data more private (#9203)
Add off-by-default configuration settings to:
- disable putting an invitee's profile info in invite events
- disable profile lookup via federation

Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
2021-02-19 09:50:41 +00:00
Rishabh Arya e17553e185
Parse ui_auth.session_timeout as a duration (instead of treating it as ms) (#9426) 2021-02-18 09:18:14 -05:00
David Vo bb2577f6b7
Add http2 to the nginx example config (#9390) 2021-02-18 08:46:16 -05:00
Richard van der Hoff 626afd7e89 Revert "Update workers.md"
This reverts commit a8878960c0.
2021-02-18 11:56:25 +00:00
Dirk Klimpel c8d9383cfb
Add the shadow-banning status to the display user admin API. (#9400) 2021-02-17 15:19:23 -05:00
Richard van der Hoff 53f1c4da81
Update workers.md 2021-02-17 17:14:23 +00:00
Richard van der Hoff a8878960c0
Update workers.md
tiny typo in sso paths
2021-02-17 17:11:24 +00:00
Eric Eastwood 0a00b7ff14
Update black, and run auto formatting over the codebase (#9381)
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Marcus 5636e597c3
Fix OIDC gitiea redirect URL. (#9404)
Fixes a "conflict" from 846b9d3df0
and d1f13c7485.
2021-02-16 14:06:55 -05:00
Erik Johnston ddfdf94506
Document that pusher instances are shardable (#9407) 2021-02-16 13:27:49 +00:00
Andrew Morgan ff40c8099d Fix sample config
Just a small change missed in 7950aa8a27.
2021-02-12 22:18:40 +00:00
Patrick Cloke 2c9b4a5f16 Synapse 1.27.0rc2 (2021-02-11)
==============================
 
 Features
 --------
 
 - Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297))
 
 Bugfixes
 --------
 
 - Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302))
 - Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313))
 
 Improved Documentation
 ----------------------
 
 - Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAlX4AACgkQM/xY9qcR
 MEjxhg/+JzOGDmgxy4Vm9oU84htkkJsflU755ykxhwbIjVy2j+07pMgtWMmebXIS
 /tWjinwRB9OgKIK+j5RWdYzz/GRTkLCbafTLhhNOWsnaCO7nnAfZFozLOMd9g2qN
 309cS8efZkxGCISGSRm1QQKjQDx6HHx7nZVAwpqb778Q9TMrBYRAQeCk9yws7FfL
 GsP5YrBI42n84YhvflWA8J8QRYkeJJaggVWzE0XvwdI6raY1BmxMKcDvyl9iKpGb
 /QXsdketD3eJxdOlU92O5ZxRXmL00bdyAFdw2+J3Y+wjoGBnC5njKSFG3j8Z2UcP
 xIdB6w/zyGoPLKd4s7tkDI3axE7qrwFetA4NytannWGEHZ9q5tgOd5PA1kTeKYvn
 ao2os4pKMjWQOHiWLskqZVXgmoW5Kb2zgyZU/vhFAz8ImuidFULOjpEytAWqtt1B
 iuL/fRlM3z0BpwQNd3NBsblYTpZri8gdTp7ULJYtaKpT2MeG0sg3swJAptUjw9L4
 awbQuBFZ8o/E/0xwiGieSxbR6b6Nz5WiCyLaeMz6b3Y4YdomttOAZ6pL6E9y8ygU
 2wKYleSmg+tbaBKKZR5CSCwXOduSYPhWw7oqsfEfvT0NCQLuItfGNFZqxRzsf73H
 lCV0wwxNZBv8kRyMs6KivMvSI6SKinAjXS8AjzHE402ozdwOCv0=
 =NSyp
 -----END PGP SIGNATURE-----

Merge tag 'v1.27.0rc2' into develop

Synapse 1.27.0rc2 (2021-02-11)
==============================

Features
--------

- Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297))

Bugfixes
--------

- Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302))
- Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313))

Improved Documentation
----------------------

- Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
2021-02-11 11:56:03 -05:00
Patrick Cloke e40d88cff3
Backout changes for automatically calculating the public baseurl. (#9313)
This breaks some people's configurations (if their Client-Server API
is not accessed via port 443).
2021-02-11 11:16:54 -05:00
Thomas Mortagne 5f716fa777
Add XWiki OIDC provider example. (#9324) 2021-02-09 11:54:52 -05:00
David Teller b0b2cac057
Merge pull request #9150 from Yoric/develop-context
New API /_synapse/admin/rooms/{roomId}/context/{eventId}
2021-02-08 15:53:44 +01:00
Erik Johnston adc96d4236 Merge branch 'erikj/media_spam_checker' into develop 2021-02-04 17:01:59 +00:00
Erik Johnston 7e8083eb48 Add check_media_file_for_spam spam checker hook 2021-02-04 17:01:30 +00:00
dykstranet 982d9eb211
Correct `matrix-synapse.service` reference in TURN howto docs. (#9308) 2021-02-04 11:22:44 -05:00
Patrick Cloke 2ab6e67ab7
Fix escaping of braces in OIDC sample config. (#9317)
This fixes the Jinja2 templates for the mapping provider.
2021-02-04 09:06:20 -05:00
Richard van der Hoff 3f534d3fdf Merge branch 'social_login_hotfixes' into develop 2021-02-03 20:34:27 +00:00
Patrick Cloke afa18f1baa
Clarify documentation about escaping URLs in templates. (#9310) 2021-02-03 14:51:38 -05:00
dykstranet e4cdecb310
config: Add detail to auto_join_rooms comment (#9291)
config: Add detail to auto_join_rooms comment

Signed-off-by: Gary Dykstra <gary@dykstranet.com>
2021-02-03 15:21:30 +00:00
Patrick Cloke 4ca054a4ea
Convert blacklisted IPv4 addresses to compatible IPv6 addresses. (#9240)
Also add a few more IP ranges to the default blacklist.
2021-02-03 07:13:46 -05:00
Travis Ralston b60bb28bbc
Add an admin API to get the current room state (#9168)
This could arguably replace the existing admin API for `/members`, however that is out of scope of this change.

This sort of endpoint is ideal for moderation use cases as well as other applications, such as needing to retrieve various bits of information about a room to perform a task (like syncing power levels between two places). This endpoint exposes nothing more than an admin would be able to access with a `select *` query on their database.
2021-02-02 11:16:29 +00:00
Richard van der Hoff 8f75bf1df7
Put SAML callback URI under /_synapse/client. (#9289) 2021-02-02 09:43:50 +00:00
Richard van der Hoff 846b9d3df0
Put OIDC callback URI under /_synapse/client. (#9288) 2021-02-01 22:56:01 +00:00
Oliver Hanikel d1f13c7485
Add an OpenID example config for Gitea. (#9134) 2021-02-01 16:21:09 -05:00
Richard van der Hoff 5963426b95 Merge branch 'social_login' into develop 2021-02-01 18:46:12 +00:00
Richard van der Hoff c543bf87ec
Collect terms consent from the user during SSO registration (#9276) 2021-02-01 18:37:41 +00:00
Richard van der Hoff e5d70c8a82
Improve styling and wording of SSO UIA templates (#9286)
fixes #9171
2021-02-01 18:36:04 +00:00
Richard van der Hoff 18ab35284a Merge branch 'social_login' into develop 2021-02-01 17:28:37 +00:00
Richard van der Hoff 4167494c90
Replace username picker with a template (#9275)
There's some prelimiary work here to pull out the construction of a jinja environment to a separate function.

I wanted to load the template at display time rather than load time, so that it's easy to update on the fly. Honestly, I think we should do this with all our templates: the risk of ending up with malformed templates is far outweighed by the improved turnaround time for an admin trying to update them.
2021-02-01 15:52:50 +00:00
Richard van der Hoff 8aed29dc61
Improve styling and wording of SSO redirect confirm template (#9272) 2021-02-01 15:50:56 +00:00
Richard van der Hoff 9c715a5f19
Fix SSO on workers (#9271)
Fixes #8966.

* Factor out build_synapse_client_resource_tree

Start a function which will mount resources common to all workers.

* Move sso init into build_synapse_client_resource_tree

... so that we don't have to do it for each worker

* Fix SSO-login-via-a-worker

Expose the SSO login endpoints on workers, like the documentation says.

* Update workers config for new endpoints

Add documentation for endpoints recently added (#8942, #9017, #9262)

* remove submit_token from workers endpoints list

this *doesn't* work on workers (yet).

* changelog

* Add a comment about the odd path for SAML2Resource
2021-02-01 15:47:59 +00:00
Erik Johnston f2c1560eca
Ratelimit invites by room and target user (#9258) 2021-01-29 16:38:29 +00:00
Richard van der Hoff 0d81a6fa3e Merge branch 'social_login' into develop 2021-01-28 22:08:11 +00:00
Erik Johnston 4b73488e81
Ratelimit 3PID /requestToken API (#9238) 2021-01-28 17:39:21 +00:00
David Teller fe52dae6bd FIXUP: Documenting /_synapse/admin/v1/rooms/<room_id>/context/<event_id> 2021-01-28 12:30:21 +01:00
Richard van der Hoff 34efb4c604
Add notes on integrating with Facebook for SSO login. (#9244) 2021-01-27 22:57:16 +00:00
Richard van der Hoff a083aea396
Add 'brand' field to MSC2858 response (#9242)
We've decided to add a 'brand' field to help clients decide how to style the
buttons.

Also, fix up the allowed characters for idp_id, while I'm in the area.
2021-01-27 21:31:45 +00:00
Richard van der Hoff 869667760f
Support for scraping email addresses from OIDC providers (#9245) 2021-01-27 21:28:59 +00:00
Erik Johnston 93b61589b0
Add a note to changelog about redis usage (#9227) 2021-01-27 14:06:27 +00:00
Patrick Cloke 26837d5dbe
Do not require the CAS service URL setting (use public_baseurl instead). (#9199)
The current configuration is handled for backwards compatibility,
but is considered deprecated.
2021-01-26 10:49:25 -05:00
Jason Robinson cee4010f94 Merge branch 'develop' into jaywink/admin-forward-extremities
# Conflicts:
#	synapse/rest/admin/__init__.py
2021-01-26 10:15:32 +02:00
Patrick Cloke 4a55d267ee
Add an admin API for shadow-banning users. (#9209)
This expands the current shadow-banning feature to be usable via
the admin API and adds documentation for it.

A shadow-banned users receives successful responses to their
client-server API requests, but the events are not propagated into rooms.

Shadow-banning a user should be used as a tool of last resort and may lead
to confusing or broken behaviour for the client.
2021-01-25 14:49:39 -05:00
Jason Robinson 8965b6cfec Merge branch 'develop' into jaywink/admin-forward-extremities 2021-01-23 21:41:35 +02:00
Jason Robinson 930ba00971 Add depth and received_ts to forward_extremities admin API response
Also add a warning on the admin API documentation.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
2021-01-23 21:34:32 +02:00
Richard van der Hoff b5120f09f1 Merge remote-tracking branch 'origin/release-v1.26.0' into develop 2021-01-21 13:17:07 +00:00
Richard van der Hoff 7447f19702
Prefix idp_id with "oidc-" (#9189)
... to avoid clashes with other SSO mechanisms
2021-01-21 12:25:02 +00:00
Patrick Cloke f81d02d75b Synapse 1.26.0rc1 (2021-01-20)
==============================
 
 This release brings a new schema version for Synapse and rolling back to a previous
 verious is not trivial.  Please review [UPGRADE.rst](UPGRADE.rst) for more details
 on these changes  and for general upgrade guidance.
 
 Features
 --------
 
 - Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015), [\#9017](https://github.com/matrix-org/synapse/issues/9017), [\#9036](https://github.com/matrix-org/synapse/issues/9036), [\#9067](https://github.com/matrix-org/synapse/issues/9067), [\#9081](https://github.com/matrix-org/synapse/issues/9081), [\#9082](https://github.com/matrix-org/synapse/issues/9082), [\#9105](https://github.com/matrix-org/synapse/issues/9105), [\#9107](https://github.com/matrix-org/synapse/issues/9107), [\#9109](https://github.com/matrix-org/synapse/issues/9109), [\#9110](https://github.com/matrix-org/synapse/issues/9110), [\#9127](https://github.com/matrix-org/synapse/issues/9127), [\#9153](https://github.com/matrix-org/synapse/issues/9153), [\#9154](https://github.com/matrix-org/synapse/issues/9154), [\#9177](https://github.com/matrix-org/synapse/issues/9177))
 - During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091))
 - Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159))
 - Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024))
 - Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984))
 - Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086))
 - Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932))
 - Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948))
 - Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042), [\#9043](https://github.com/matrix-org/synapse/issues/9043), [\#9044](https://github.com/matrix-org/synapse/issues/9044), [\#9130](https://github.com/matrix-org/synapse/issues/9130))
 - Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068))
 - Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092))
 - Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104), [\#9166](https://github.com/matrix-org/synapse/issues/9166))
 
 Bugfixes
 --------
 
 - Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023))
 - Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028))
 - Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051))
 - Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053))
 - Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054))
 - Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059))
 - Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070))
 - Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071))
 - Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114), [\#9116](https://github.com/matrix-org/synapse/issues/9116))
 - Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117))
 - Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128))
 - Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108))
 - Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145))
 - Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161))
 
 Improved Documentation
 ----------------------
 
 - Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997))
 - Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035))
 - Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040))
 - Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057))
 - Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151))
 
 Deprecations and Removals
 -------------------------
 
 - Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039))
 
 Internal Changes
 ----------------
 
 - Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868), [\#9029](https://github.com/matrix-org/synapse/issues/9029), [\#9115](https://github.com/matrix-org/synapse/issues/9115), [\#9118](https://github.com/matrix-org/synapse/issues/9118), [\#9124](https://github.com/matrix-org/synapse/issues/9124))
 - Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939))
 - Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016))
 - Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018))
 - Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025))
 - Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030))
 - Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031))
 - Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033))
 - Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038))
 - Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041))
 - Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055))
 - Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058))
 - Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063))
 - Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069))
 - Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080))
 - Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093))
 - Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098))
 - Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106))
 - Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112))
 - Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125))
 - Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144))
 - Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146))
 - Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAIVFkACgkQM/xY9qcR
 MEhLfQ//Z4xnP5Icx732k2xpdR+JaXX6heNVFm/UuLNd6wq+uXjO/Dgc9IKQ5LXD
 kfG+/OnNGDr/eUk4mQ5s/ccBvaTrOid2IIemJd4lUBy/Q5LvPvfKIp08QFr72WJT
 7Y4Ma54nNSSMUAdESgj/aFAHjATbfHhxPOZ5OdQHGLAuJ/OUwR2ksasVNCuINbgh
 8vrxrzjeYn1Zl3UTBrsdCcat7AFqYjxK/Y0i6JxfSgRwDZjUbW4M2C4OGWZJPUrU
 2YTPHangxd+22+HObE2KDEzaHDdCR/Kj4dhCiJBynZ5iucuHSS+mgf39RtbjqWML
 01cmXXUV+FH7rnCilmF9rYdmHZ6L9BJxtrQDQU4dB106BX9G7hKR6rYHc9Z6tjOh
 3tg0cKWYMXOsTNnvRqiFNh2q0yBKS228HRAwyGosW/6NiYj43ArtYP6knX7dQyBX
 0mYNkfry2dyX6Kj5el0i9MOTHYQxfc8apsBm2M3OTNYukgQKA0NhbAquibaAxEEW
 2qIqbSp4CUlGPvM7+u+ZGTu0hEbVKKNjqMfXhuY8A4/BWiV2mSqKcEirR5cE/LqY
 mq5Ac7vbO4Uh+1xiRw/G9ITi1dqAjYIzhBawlhdUPOh+aINWTUikYYzjUSmm4PbN
 H9BTPLA9iAdVRfuWJ9um2G0DdS8Qpx/aIRh3MScXVly6cGCC6Do=
 =kgqn
 -----END PGP SIGNATURE-----

Merge tag 'v1.26.0rc1' into develop

Synapse 1.26.0rc1 (2021-01-20)
==============================

This release brings a new schema version for Synapse and rolling back to a previous
verious is not trivial.  Please review [UPGRADE.rst](UPGRADE.rst) for more details
on these changes  and for general upgrade guidance.

Features
--------

- Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015), [\#9017](https://github.com/matrix-org/synapse/issues/9017), [\#9036](https://github.com/matrix-org/synapse/issues/9036), [\#9067](https://github.com/matrix-org/synapse/issues/9067), [\#9081](https://github.com/matrix-org/synapse/issues/9081), [\#9082](https://github.com/matrix-org/synapse/issues/9082), [\#9105](https://github.com/matrix-org/synapse/issues/9105), [\#9107](https://github.com/matrix-org/synapse/issues/9107), [\#9109](https://github.com/matrix-org/synapse/issues/9109), [\#9110](https://github.com/matrix-org/synapse/issues/9110), [\#9127](https://github.com/matrix-org/synapse/issues/9127), [\#9153](https://github.com/matrix-org/synapse/issues/9153), [\#9154](https://github.com/matrix-org/synapse/issues/9154), [\#9177](https://github.com/matrix-org/synapse/issues/9177))
- During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091))
- Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159))
- Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024))
- Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984))
- Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086))
- Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932))
- Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948))
- Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042), [\#9043](https://github.com/matrix-org/synapse/issues/9043), [\#9044](https://github.com/matrix-org/synapse/issues/9044), [\#9130](https://github.com/matrix-org/synapse/issues/9130))
- Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068))
- Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092))
- Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104), [\#9166](https://github.com/matrix-org/synapse/issues/9166))

Bugfixes
--------

- Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023))
- Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028))
- Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051))
- Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053))
- Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054))
- Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059))
- Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070))
- Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071))
- Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114), [\#9116](https://github.com/matrix-org/synapse/issues/9116))
- Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117))
- Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128))
- Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108))
- Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145))
- Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161))

Improved Documentation
----------------------

- Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997))
- Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035))
- Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040))
- Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057))
- Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151))

Deprecations and Removals
-------------------------

- Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039))

Internal Changes
----------------

- Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868), [\#9029](https://github.com/matrix-org/synapse/issues/9029), [\#9115](https://github.com/matrix-org/synapse/issues/9115), [\#9118](https://github.com/matrix-org/synapse/issues/9118), [\#9124](https://github.com/matrix-org/synapse/issues/9124))
- Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939))
- Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016))
- Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018))
- Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025))
- Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030))
- Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031))
- Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033))
- Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038))
- Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041))
- Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055))
- Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058))
- Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063))
- Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069))
- Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080))
- Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093))
- Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098))
- Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106))
- Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112))
- Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125))
- Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144))
- Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146))
- Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157))
2021-01-20 11:27:39 -05:00
Richard van der Hoff 0cd2938bc8
Support icons for Identity Providers (#9154) 2021-01-20 08:15:14 -05:00
rht a5b9c87ac6
docs: Add link to Matrix VoIP tester for turn-howto (#9135)
Signed-off-by: rht <rhtbot@protonmail.com>
2021-01-20 12:41:57 +00:00
Richard van der Hoff fa50e4bf4d
Give `public_baseurl` a default value (#9159) 2021-01-20 12:30:41 +00:00
Patrick Cloke de45bf5b5b
Quote pip install with brackets to avoid shell interpretation. (#9151) 2021-01-18 11:12:20 -05:00
Matthew Hodgson 883d4e6f2b link to the scalability blog post from workers.md 2021-01-18 00:27:27 +00:00
Richard van der Hoff 9de6b94117
Land support for multiple OIDC providers (#9110)
This is the final step for supporting multiple OIDC providers concurrently.

First of all, we reorganise the config so that you can specify a list of OIDC providers, instead of a single one. Before:

    oidc_config:
       enabled: true
       issuer: "https://oidc_provider"
       # etc

After:

    oidc_providers:
     - idp_id: prov1
       issuer: "https://oidc_provider"

     - idp_id: prov2
       issuer: "https://another_oidc_provider"

The old format is still grandfathered in.

With that done, it's then simply a matter of having OidcHandler instantiate a new OidcProvider for each configured provider.
2021-01-15 16:55:29 +00:00
Patrick Cloke 3e4cdfe5d9
Add an admin API endpoint to protect media. (#9086)
Protecting media stops it from being quarantined when
e.g. all media in a room is quarantined. This is useful
for sticker packs and other media that is uploaded by
server administrators, but used by many people.
2021-01-15 11:18:09 -05:00
Richard van der Hoff 5310808d3b Give the user a better error when they present bad SSO creds
If a user tries to do UI Auth via SSO, but uses the wrong account on the SSO
IdP, try to give them a better error.

Previously, the UIA would claim to be successful, but then the operation in
question would simply fail with "auth fail". Instead, serve up an error page
which explains the failure.
2021-01-13 20:22:41 +00:00
Patrick Cloke d1eb1b96e8
Register the /devices endpoint on workers. (#9092) 2021-01-13 12:35:40 -05:00
Dirk Klimpel 7a2e9b549d
Remove user's avatar URL and displayname when deactivated. (#8932)
This only applies if the user's data is to be erased.
2021-01-12 16:30:15 -05:00
Jason Robinson da16d06301 Address pr feedback
* docs updates
* prettify SQL
* add missing copyright
* cursor_to_dict
* update touched files copyright years

Signed-off-by: Jason Robinson <jasonr@matrix.org>
2021-01-11 23:43:58 +02:00
Jason Robinson 0b77329fe2
Clarify rooms.md
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-01-11 23:05:36 +02:00
David Teller b161528fcc
Also support remote users on the joined_rooms admin API. (#8948)
For remote users, only the rooms which the server knows about are returned.
Local users have all of their joined rooms returned.
2021-01-11 14:32:17 -05:00
Erik Johnston 1315a2e8be
Use a chain cover index to efficiently calculate auth chain difference (#8868) 2021-01-11 16:09:22 +00:00
Jason Robinson e2c16edc78 Add changelog and admin API docs
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2021-01-09 22:58:29 +02:00
Christopher Rücker bce0c91d9a
Keycloak mapping_provider example (#9037) (#9057)
This PR adds the missing user_mapping_provider section in oidc.md

Signed-off-by: Christopher Rücker chris-ruecker@protonmail.com
2021-01-08 18:29:30 +00:00
Emelie 9066c2fd7f
Fix typo in docs/systemd-with-workers/README.md (#9035)
Signed-off-by: Emelie em@nao.sh
2021-01-07 15:31:01 +00:00
Richard van der Hoff 111b673fc1
Add initial support for a "pick your IdP" page (#9017)
During login, if there are multiple IdPs enabled, offer the user a choice of
IdPs.
2021-01-05 11:25:28 +00:00
Jerin J Titus cfcf5541b4
Update the value of group_creation_prefix in sample config. (#8992)
Removes the trailing slash with causes issues with matrix.to/Element.
2020-12-29 09:30:48 -05:00
Patrick Cloke 68bb26da69
Allow redacting events on workers (#8994)
Adds the redacts endpoint to workers that have the client listener.
2020-12-29 07:40:12 -05:00
Patrick Cloke 4218473f9e
Refactor the CAS handler in prep for using the abstracted SSO code. (#8958)
This makes the CAS handler look more like the SAML/OIDC handlers:

* Render errors to users instead of throwing JSON errors.
* Internal reorganization.
2020-12-18 13:09:45 -05:00
Patrick Cloke 56e00ca85e
Send the location of the web client to the IS when inviting via 3PIDs. (#8930)
Adds a new setting `email.invite_client_location` which, if defined, is
passed to the identity server during invites.
2020-12-18 11:01:57 -05:00
Erik Johnston d781a81e69
Allow server admin to get admin bit in rooms where local user is an admin (#8756)
This adds an admin API that allows a server admin to get power in a room if a local user has power in a room. Will also invite the user if they're not in the room and its a private room. Can specify another user (rather than the admin user) to be granted power.

Co-authored-by: Matthew Hodgson <matthew@matrix.org>
2020-12-18 15:37:19 +00:00
Richard van der Hoff 28877fade9
Implement a username picker for synapse (#8942)
The final part (for now) of my work to implement a username picker in synapse itself. The idea is that we allow
`UsernameMappingProvider`s to return `localpart=None`, in which case, rather than redirecting the browser
back to the client, we redirect to a username-picker resource, which allows the user to enter a username.
We *then* complete the SSO flow (including doing the client permission checks).

The static resources for the username picker itself (in 
https://github.com/matrix-org/synapse/tree/rav/username_picker/synapse/res/username_picker)
are essentially lifted wholesale from
https://github.com/matrix-org/matrix-synapse-saml-mozilla/tree/master/matrix_synapse_saml_mozilla/res. 
As the comment says, we might want to think about making them customisable, but that can be a follow-up. 

Fixes #8876.
2020-12-18 14:19:46 +00:00
Patrick Cloke 5d4c330ed9
Allow re-using a UI auth validation for a period of time (#8970) 2020-12-18 07:33:57 -05:00
Dirk Klimpel 06006058d7
Make search statement in List Room and User Admin API case-insensitive (#8931) 2020-12-17 10:43:37 +00:00
Patrick Cloke 44b7d4c6d6
Fix the sample config location for the ip_range_whitelist setting. (#8954)
Move it from the federation section to the server section to match
ip_range_blacklist.
2020-12-16 14:40:47 -05:00
David Teller f14428b25c
Allow spam-checker modules to be provide async methods. (#8890)
Spam checker modules can now provide async methods. This is implemented
in a backwards-compatible manner.
2020-12-11 14:05:15 -05:00
Dirk Klimpel 0a34cdfc66
Add number of local devices to Room Details Admin API (#8886) 2020-12-11 10:42:47 +00:00
Dirk Klimpel a5f7aff5e5
Deprecate Shutdown Room and Purge Room Admin API (#8829)
Deprecate both APIs in favour of the Delete Room API.

Related: #8663 and #8810
2020-12-10 11:42:48 +00:00
Patrick Cloke 344ab0b53a
Default to blacklisting reserved IP ranges and add a whitelist. (#8870)
This defaults `ip_range_blacklist` to reserved IP ranges and also adds an
`ip_range_whitelist` setting to override it.
2020-12-09 13:56:06 -05:00
Dirk Klimpel 43bf3c5178
Combine related media admin API docs (#8839)
Related: #8810
Also a few small improvements.

Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-12-09 16:19:57 +00:00
Richard van der Hoff 025fa06fc7
Clarify config template comments (#8891) 2020-12-08 14:03:08 +00:00
Patrick Cloke 96358cb424
Add authentication to replication endpoints. (#8853)
Authentication is done by checking a shared secret provided
in the Synapse configuration file.
2020-12-04 10:56:28 -05:00
Patrick Cloke 112f6bd49e Synapse 1.24.0rc2 (2020-12-04)
==============================
 
 Bugfixes
 --------
 
 - Fix a regression in v1.24.0rc1 which failed to allow SAML mapping providers which were unable to redirect users to an additional page. ([\#8878](https://github.com/matrix-org/synapse/issues/8878))
 
 Internal Changes
 ----------------
 
 - Add support for the `prometheus_client` newer than 0.9.0. Contributed by Jordan Bancino. ([\#8875](https://github.com/matrix-org/synapse/issues/8875))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAl/KQ6MACgkQM/xY9qcR
 MEjqOg/9Fd4Ow0QdnK3O6J0bmXP6+g2GfjJCAKlYlF/nwX1ez9jZVed00D2h9/hD
 YCUJCEv+XZpbv5F4usiYwoLGithbueo7AKEOQJQMEuQBHU1/E6TJ9iiKR3lTBNNw
 ccuyMMqqEatYxoa4DIO/lrwm2mhwRNdMPt0r1DPlML13bb/TVbeXXykbOCyZSKm1
 YuXpxYDngRPL9o46I77l7/KNTSpYYeZi2qwp2orl6GBfa0KSvR2Oi6uohKYzuoHC
 +XLvHSFMpHAZkaZsIMxU5hRVH6jtkb/D37CABuKQsdlvHcooGK4Sdt6aoUyhPtoh
 6YwajSazWKEJ/9YO5JV22qkWFPi9T6s/lPZfjOSu/euwMw7DDQJ/8t8Sm6N7sJDS
 A/5FctOAd1jd9BiAaqkHB2zLHZSomIHQpJJfDyE3UgAGQQ6fm2Wg7SkLByDx/MYC
 4D74dBuVBPENM2VxNoyjHbpVSeQi8t0RgWb44/PUAOtpwm3f3eVRuS8zY6uR2bZr
 YZUbJylqKhRW6dBdZH+EzVolRGvBgLkc27IzRCQ3kGUEo2Wem05s6MuKMvLxoIvN
 NHLWNClUWITT8FgkfFP3c6CK40dwIGJzYryI+71qPW/R7snf2fazuoHlH1pAmFh0
 E0/nBMTw1deixOYXbG1ah63AhJ1NjdTfimDsKmWZLqVc6g7g4d4=
 =is1Y
 -----END PGP SIGNATURE-----

Merge tag 'v1.24.0rc2' into develop

Synapse 1.24.0rc2 (2020-12-04)
==============================

Bugfixes
--------

- Fix a regression in v1.24.0rc1 which failed to allow SAML mapping providers which were unable to redirect users to an additional page. ([\#8878](https://github.com/matrix-org/synapse/issues/8878))

Internal Changes
----------------

- Add support for the `prometheus_client` newer than 0.9.0. Contributed by Jordan Bancino. ([\#8875](https://github.com/matrix-org/synapse/issues/8875))
2020-12-04 09:14:31 -05:00
Patrick Cloke 22c6c19f91
Fix a regression that mapping providers should be able to redirect users. (#8878)
This was broken in #8801.
2020-12-04 08:25:15 -05:00
Richard van der Hoff 6e4f71c057
Fix a buglet in the SAML username mapping provider doc (#8873)
the constructor is called with a `module_api`.
2020-12-04 10:14:15 +00:00
Patrick Cloke 30fba62108
Apply an IP range blacklist to push and key revocation requests. (#8821)
Replaces the `federation_ip_range_blacklist` configuration setting with an
`ip_range_blacklist` setting with wider scope. It now applies to:

* Federation
* Identity servers
* Push notifications
* Checking key validitity for third-party invite events

The old `federation_ip_range_blacklist` setting is still honored if present, but
with reduced scope (it only applies to federation and identity servers).
2020-12-02 11:09:24 -05:00
Andrew Morgan d1be293f00 Fix typo in password_auth_providers doc
A word got removed accidentally in 83434df381.
2020-12-01 10:34:52 +00:00
Andrew Morgan 17fa58bdd1
Add a config option to change whether unread push notification counts are per-message or per-room (#8820)
This PR adds a new config option to the `push` section of the homeserver config, `group_unread_count_by_room`. By default Synapse will group push notifications by room (so if you have 1000 unread messages, if they lie in 55 rooms, you'll see an unread count on your phone of 55).

However, it is also useful to be able to send out the true count of unread messages if desired. If `group_unread_count_by_room` is set to `false`, then with the above example, one would see an unread count of 1000 (email anyone?).
2020-11-30 18:43:54 +00:00
Richard van der Hoff a090b86209
Add `force_purge` option to delete-room admin api. (#8843) 2020-11-30 16:48:12 +00:00
Dirk Klimpel 14f81a6d24
Improve documentation how to configure prometheus for workers (#8822) 2020-11-26 10:42:55 +00:00
Dirk Klimpel 3f0ff53158
Remove deprecated `/_matrix/client/*/admin` endpoints (#8785)
These are now only available via `/_synapse/admin/v1`.
2020-11-25 16:26:11 -05:00
Andrew Morgan 2b110dda2a
Fix the formatting of push config section (#8818)
This PR updates the push config's formatting to better align with our [code style guidelines](https://github.com/matrix-org/synapse/blob/develop/docs/code_style.md#configuration-file-format).
2020-11-25 21:02:53 +00:00
Patrick Cloke 4fd222ad70
Support trying multiple localparts for OpenID Connect. (#8801)
Abstracts the SAML and OpenID Connect code which attempts to regenerate
the localpart of a matrix ID if it is already in use.
2020-11-25 10:04:22 -05:00
Dirk Klimpel b08dc7effe
Clarify documentation of the admin list media API (#8795)
Clarify that the list media API only shows media from unencrypted events.
2020-11-24 09:04:51 -05:00
Richard van der Hoff e3d7806704
Update turn-howto (#8779)
Some hopefully-useful notes on setting up a turnserver.
2020-11-24 12:52:22 +00:00
Patrick Cloke 79bfe966e0
Improve error checking for OIDC/SAML mapping providers (#8774)
Checks that the localpart returned by mapping providers for SAML and
OIDC are valid before registering new users.

Extends the OIDC tests for existing users and invalid data.
2020-11-19 14:25:17 -05:00
Ben Banfield-Zanin 53a6f5ddf0
SAML: Allow specifying the IdP entityid to use. (#8630)
If the SAML metadata includes multiple IdPs it is necessary to
specify which IdP to redirect users to for authentication.
2020-11-19 09:57:13 -05:00
Marcus Schopen d356588339
SAML: Document allowing a clock/time difference from IdP (#8731)
Updates the sample configuration with the pysaml2 configuration for
accepting clock skew/drift between the homeserver and IdP.
2020-11-18 07:36:28 -05:00
chagai95 e487d9fabc
a comma too much (#8771)
Signed-off-by: Chagai Friedlander chagai95@gmail.com
2020-11-17 14:13:56 +00:00
Erik Johnston f737368a26
Add admin API for logging in as a user (#8617) 2020-11-17 10:51:25 +00:00
Adrian Wannenmacher f1de4bb58b
Clarify the usecase for an msisdn delegate (#8734)
Signed-off-by: Adrian Wannenmacher <tfld@tfld.dev>
2020-11-14 23:09:36 +00:00
Dirk Klimpel 023f791143
Migrate documentation `docs/admin_api/event_reports` to markdown (#8742)
Related to #8714. `event_reports.rst` was introduced in Synapse 1.21.0.
2020-11-13 13:57:55 +00:00
Marcus Schopen 68fc0dcb5a
SAML: add <mdui:UIInfo> element examples (#8718)
add some mdui:UIInfo element examples for saml2_config in homeserver.yaml
2020-11-13 12:07:50 +00:00
Marcus Schopen c059413001
Notes on SSO logins and media_repository worker (#8701)
If SSO login is used (e.g. SAML) in a multi worker setup, it should be mentioned that currently all SAML logins must run on the same worker, see https://github.com/matrix-org/synapse/issues/7530

Also, if you are using different ports (for example 443 and 8448) in a reverse proxy for client and federation, the path `/_matrix/media` on the client and federation port must point to the listener of the `media_repository` worker, otherwise you'll get a 404 on the federation port for the path `/_matrix/media`, if a remote server is trying to get the media object on federation port, see https://github.com/matrix-org/synapse/issues/8695
2020-11-06 14:33:07 +00:00
Dirk Klimpel c3119d1536
Add an admin API for users' media statistics (#8700)
Add `GET /_synapse/admin/v1/statistics/users/media` to get statisics about local media usage by users.
Related to #6094
It is the first API for statistics.
Goal is to avoid/reduce usage of sql queries like [Wiki analyzing Synapse](https://github.com/matrix-org/synapse/wiki/SQL-for-analyzing-Synapse-PostgreSQL-database-stats)

Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-11-05 18:59:12 +00:00
Dirk Klimpel e4676bd877
Add `displayname` to Shared-Secret Registration for admins (#8722)
Add `displayname` to Shared-Secret Registration for admins to `POST /_synapse/admin/v1/register`
2020-11-05 13:55:45 +00:00
Dirk Klimpel 4fda58ddd2
Remove the "draft" status of the Room Details Admin API (#8702)
Fixes #8550
2020-11-03 12:48:25 +00:00
Erik Johnston 4b09b7438e
Document how to set up multiple event persisters (#8706) 2020-11-03 10:27:11 +00:00
Matthew Hodgson d04c2d19b3 grammar 2020-11-02 21:22:36 +00:00
Matthew Hodgson 11fd90a2b7 typo 2020-11-02 13:33:56 +00:00
Andrew Morgan 26b46796ea Fix typos in systemd-with-workers doc 2020-11-02 12:56:16 +00:00
Andrew Morgan 305545682d Fix typo in workers doc 2020-11-02 12:36:18 +00:00
Patrick Cloke 8f1aefa694
Improve the sample config for SSO (OIDC, SAML, and CAS). (#8635) 2020-10-30 10:01:59 -04:00
Patrick Cloke 00b24aa545
Support generating structured logs in addition to standard logs. (#8607)
This modifies the configuration of structured logging to be usable from
the standard Python logging configuration.

This also separates the formatting of logs from the transport allowing
JSON logs to files or standard logs to sockets.
2020-10-29 07:27:37 -04:00
Dirk Klimpel 2239813278
Add an admin APIs to allow server admins to list users' pushers (#8610)
Add an admin API `GET /_synapse/admin/v1/users/<user_id>/pushers` like https://matrix.org/docs/spec/client_server/latest#get-matrix-client-r0-pushers
2020-10-28 15:02:42 +00:00
Michael Kaye f49c2093b5
Cross-link documentation to the prometheus recording rules. (#8667) 2020-10-27 15:29:50 -04:00
Dirk Klimpel 9b7c28283a
Add admin API to list users' local media (#8647)
Add admin API `GET /_synapse/admin/v1/users/<user_id>/media` to get information of users' uploaded files.
2020-10-27 14:12:31 +00:00
Dirk Klimpel 66e6801c3e
Split admin API for reported events into a detail and a list view (#8539)
Split admin API for reported events in detail und list view.
API was introduced with #8217 in synapse v.1.21.0.

It makes the list (`GET /_synapse/admin/v1/event_reports`) less complex and provides a better overview.
The details can be queried with: `GET /_synapse/admin/v1/event_reports/<report_id>`.
It is similar to room and users API.

It is a kind of regression in `GET /_synapse/admin/v1/event_reports`.  `event_json` was removed. But the api was introduced one version before and it is an admin API (not under spec).

Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-10-26 18:16:37 +00:00
Peter Krantz 6c9ab61df5
Added basic instructions for Azure AD to OpenId documentation (#8582)
Signed-off-by: Peter Krantz peter.krantz@gmail.com
2020-10-26 17:49:55 +00:00
Dirk Klimpel 49d72dea2a
Add an admin api to delete local media. (#8519)
Related to: #6459, #3479

Add `DELETE /_synapse/admin/v1/media/<server_name>/<media_id>` to delete
a single file from server.
2020-10-26 17:02:28 +00:00
Andrew Morgan f6a3859a73
Fix filepath of Dex example config (#8657) 2020-10-26 16:53:11 +00:00
Andrew Morgan 73d8209694
Correct the package name in OpenID Connect install instructions (#8634)
The OpenID Connect install instructions suggested installing `synapse[oidc]`, but our PyPI package is called `matrix-synapse`.
2020-10-26 14:45:33 +00:00
Dirk Klimpel 913f8a06e4
Add field `total` to device list in admin API (#8644) 2020-10-26 14:07:51 +00:00
Patrick Cloke 34a5696f93
Fix typos and spelling errors. (#8639) 2020-10-23 12:38:40 -04:00
Andrew Morgan 4fb7a68a65 Correct the package name in authlib install instructions 2020-10-22 18:25:58 +01:00
Christopher May-Townsend 1cf4a68108
Add note to manhole.md about bind_address when using with docker (#8526)
Signed-off-by: Christopher May-Townsend <chris@maytownsend.co.uk>
2020-10-14 15:28:59 +01:00
Brendan Abolivier 9e66f3761c
Update documentation on retention policies limits (#8529)
* Update documentation on retention policies limits

Document the changes from https://github.com/matrix-org/synapse/pull/8104
2020-10-14 15:00:49 +01:00
Erik Johnston 8de3703d21
Make event persisters periodically announce position over replication. (#8499)
Currently background proccesses stream the events stream use the "minimum persisted position" (i.e. `get_current_token()`) rather than the vector clock style tokens. This is broadly fine as it doesn't matter if the background processes lag a small amount. However, in extreme cases (i.e. SyTests) where we only write to one event persister the background processes will never make progress.

This PR changes it so that the `MultiWriterIDGenerator` keeps the current position of a given instance as up to date as possible (i.e using the latest token it sees if its not in the process of persisting anything), and then periodically announces that over replication. This then allows the "minimum persisted position" to advance, albeit with a small lag.
2020-10-12 15:51:41 +01:00
Mateusz Przybyłowicz ca2db5dd0c
Increase default max_upload_size from 10M to 50M (#8502)
Signed-off-by: Mateusz Przybyłowicz <uamfhq@gmail.com>
2020-10-09 16:58:23 +01:00
Richard van der Hoff 4f0637346a
Combine `SpamCheckerApi` with the more generic `ModuleApi`. (#8464)
Lots of different module apis is not easy to maintain.

Rather than adding yet another ModuleApi(hs, hs.get_auth_handler()) incantation, first add an hs.get_module_api() method and use it where possible.
2020-10-07 12:03:26 +01:00
Andrew Morgan 01f82bfe32
Remove docs/sphinx and related references (#8480)
https://github.com/matrix-org/synapse/tree/develop/docs/sphinx doesn't seem to really be utilised or changed recently since the initial commit. I like the idea of exportable documentation of the codebase, but at the moment after running through the build instructions the generated website wasn't very useful...
2020-10-07 11:45:31 +01:00
Richard van der Hoff 785437dc0d
Update default room version to 6 (#8461)
Per https://github.com/matrix-org/matrix-doc/pull/2788
2020-10-05 21:40:51 +01:00
Patrick Cloke f64c6aae68
Update manhole documentation for async/await. (#8462) 2020-10-05 09:40:19 -04:00
Patrick Cloke 62894673e6
Allow background tasks to be run on a separate worker. (#8369) 2020-10-02 08:23:15 -04:00
BBBSnowball 05ee048f2c
Add config option for always using "userinfo endpoint" for OIDC (#7658)
This allows for connecting to certain IdPs, e.g. GitLab.
2020-10-01 13:54:35 -04:00
Richard van der Hoff c1ef579b63
Add prometheus metrics to track federation delays (#8430)
Add a pair of federation metrics to track the delays in sending PDUs to/from 
particular servers.
2020-10-01 11:09:12 +01:00
Patrick Cloke 8b40843392
Allow additional SSO properties to be passed to the client (#8413) 2020-09-30 13:02:43 -04:00
Aaron Raimist 8238b55e08
Update description of server_name config option (#8415) 2020-09-29 13:50:25 -04:00
Erik Johnston bd380d942f
Add checks for postgres sequence consistency (#8402) 2020-09-28 18:00:30 +01:00
Tdxdxoz abd04b6af0
Allow existing users to login via OpenID Connect. (#8345)
Co-authored-by: Benjamin Koch <bbbsnowball@gmail.com>

This adds configuration flags that will match a user to pre-existing users
when logging in via OpenID Connect. This is useful when switching to
an existing SSO system.
2020-09-25 07:01:45 -04:00
Julian Fietkau a4e63e5a47
Add note to reverse_proxy.md about disabling Apache's mod_security2 (#8375)
This change adds a note and a few lines of configuration settings for Apache users to disable ModSecurity for Synapse's virtual hosts. With ModSecurity enabled and running with its default settings, Matrix clients are unable to send chat messages through the Synapse installation. With this change, ModSecurity can be disabled only for the Synapse virtual hosts.
2020-09-23 11:14:08 +01:00
Dirk Klimpel 4da01f9c61
Admin API for reported events (#8217)
Add an admin API to read entries of table `event_reports`. API: `GET /_synapse/admin/v1/event_reports`
2020-09-22 18:15:04 +01:00
Dirk Klimpel d688b4bafc
Admin API for querying rooms where a user is a member (#8306)
Add a new admin API `GET /_synapse/admin/v1/users/<user_id>/joined_rooms` to
list all rooms where a user is a member.
2020-09-18 15:26:36 +01:00
Tulir Asokan b82d68c0bd
Add the topic and avatar to the room details admin API (#8305) 2020-09-14 10:07:04 -04:00
Patrick Cloke 6605470bfb
Improve SAML error messages (#8248) 2020-09-14 09:05:36 -04:00
Patrick Cloke a9dbe98ef9 Synapse 1.20.0rc3 (2020-09-11)
==============================
 
 Bugfixes
 --------
 
 - Fix a bug introduced in v1.20.0rc1 where the wrong exception was raised when invalid JSON data is encountered. ([\#8291](https://github.com/matrix-org/synapse/issues/8291))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAl9bbA8ACgkQM/xY9qcR
 MEjjJg//ZGcvPLr8y3F5JXptcjWA3AFja3DgztBA3uNFoDrwVhR9m3+F4rL/Fbat
 gnaccWsKIXOjZw5WOThQEUIfYMKmHEsaDl7xtlUPC4ylmSV+31ypu1KiRSxND2kT
 TmOupvqw+4/E4RajpYX6WT3e2oQUpIcBAKsUyZL3ZXECR/yNYrA6w2w7i4wrv3cU
 QDzBFrCcr5aJql7VUc88BlaZUG6xom2/kkPtjmO6imPnPGBHLN22uPU7zQ4n7Vgi
 Nrg45v5WHVCx57WoXqAEap1zdKgoRCna1x0NkqYh0OAXq1l6aVwlf/Pdynt91f2x
 yyaMYjMskjyatHlzONMV4kz0w03dUrGJXiAx2ldEDd32SKBUxLJ/CYtNEdoXZ4Mm
 o0OJfDbaSJwlqK7FhdZJE3kCSkUvlmVasDErXfQRDVHy5sx9Ufr4P0kkaZyXDys1
 UAWFcCJwy9dgFR2679PoXk1s/gHf10wuk6FOs8ESbcJJOvCljxWrOIqKNzSUL3HQ
 Hj5V1zHYXB6HtgiI7tbKxSn/d2uajHI9b8LVOkyV9RKsb1DGGRgNQQ5Kz4Zud7dC
 vuQ2jVw7JIo41W3QIXvcL1KdQeje2nwYwuwNREZaEDZJsGLjxaJ6dNyBVgkmYYFP
 gxhplSOSsFjQRGEsBrc2utWxDnLVGPHEAt7iHVSqjwVUyARvxjo=
 =e2T1
 -----END PGP SIGNATURE-----

Merge tag 'v1.20.0rc3' into develop

Synapse 1.20.0rc3 (2020-09-11)
==============================

Bugfixes
--------

- Fix a bug introduced in v1.20.0rc1 where the wrong exception was raised when invalid JSON data is encountered. ([\#8291](https://github.com/matrix-org/synapse/issues/8291))
2020-09-11 08:30:36 -04:00
Andrew Morgan 95d869c357
Add /_synapse/client to the reverse proxy docs (#8227)
This PR adds a information about forwarding `/_synapse/client` endpoints through your reverse proxy. The first of these endpoints are introduced in https://github.com/matrix-org/synapse/pull/8004.
2020-09-10 13:26:34 +01:00
Andrew Morgan 192e98111d
Remove shared rooms info from upgrade/workers doc as it's still experimental (#8290) 2020-09-10 13:08:08 +01:00
Andrew Morgan a3a90ee031
Show a confirmation page during user password reset (#8004)
This PR adds a confirmation step to resetting your user password between clicking the link in your email and your password actually being reset.

This is to better align our password reset flow with the industry standard of requiring a confirmation from the user after email validation.
2020-09-10 11:45:12 +01:00
Andrew Morgan 094896a69d
Add a config option for validating 'next_link' parameters against a domain whitelist (#8275)
This is a config option ported over from DINUM's Sydent: https://github.com/matrix-org/sydent/pull/285

They've switched to validating 3PIDs via Synapse rather than Sydent, and would like to retain this functionality.

This original purpose for this change is phishing prevention. This solution could also potentially be replaced by a similar one to https://github.com/matrix-org/synapse/pull/8004, but across all `*/submit_token` endpoint.

This option may still be useful to enterprise even with that safeguard in place though, if they want to be absolutely sure that their employees don't follow links to other domains.
2020-09-08 16:03:09 +01:00
Richard van der Hoff ad28030c12
Systemd docs: configure workers to start after main process. (#8276) 2020-09-08 10:57:43 +01:00
Will Hunt b257c788c0
Add /user/{user_id}/shared_rooms/ api (#7785)
* Add shared_rooms api

* Add changelog

* Add .

* Wrap response in {"rooms": }

* linting

* Add unstable_features key

* Remove options from isort that aren't part of 5.x

`-y` and `-rc` are now default behaviour and no longer exist.

`dont-skip` is no longer required

https://timothycrosley.github.io/isort/CHANGELOG/#500-penny-july-4-2020

* Update imports to make isort happy

* Add changelog

* Update tox.ini file with correct invocation

* fix linting again for isort

* Vendor prefix unstable API

* Fix to match spec

* import Codes

* import Codes

* Use FORBIDDEN

* Update changelog.d/7785.feature

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

* Implement get_shared_rooms_for_users

* a comma

* trailing whitespace

* Handle the easy feedback

* Switch to using runInteraction

* Add tests

* Feedback

* Seperate unstable endpoint from v2

* Add upgrade node

* a line

* Fix style by adding a blank line at EOF.

* Update synapse/storage/databases/main/user_directory.py

Co-authored-by: Tulir Asokan <tulir@maunium.net>

* Update synapse/storage/databases/main/user_directory.py

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

* Update UPGRADE.rst

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

* Fix UPGRADE/CHANGELOG unstable paths

unstable unstable unstable

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: Tulir Asokan <tulir@maunium.net>

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Tulir Asokan <tulir@maunium.net>
2020-09-02 13:18:40 +01:00
Brendan Abolivier b5133dd97f
Explain better what GDPR-erased means (#8189)
Fixes https://github.com/matrix-org/synapse/issues/8185
2020-09-01 16:31:59 +01:00
Andrew Morgan 74bf8d4d06
Wording fixes to 'name' user admin api filter (#8163)
Some fixes to wording I noticed after merging #7377.
2020-08-25 15:03:24 +01:00
Andrew Morgan 79ac619403 Fix missing double-backtick in RST document 2020-08-25 14:24:06 +01:00
Manuel Stahl 97962ad17b
Search in columns 'name' and 'displayname' in the admin users endpoint (#7377)
* Search in columns 'name' and 'displayname' in the admin users endpoint

Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
2020-08-25 14:18:14 +01:00
Brendan Abolivier 420484a334
Allow capping a room's retention policy (#8104) 2020-08-24 18:21:04 +01:00
Ryan Cole cfeb37f039
Updated docs: Added note about missing 308 redirect support. (#8120)
* Updated docs: Added note about missing 308 redirect support.

* Added changelog
2020-08-19 12:26:50 +01:00
Patrick Cloke acfb7c3b5d
Add a link to the matrix-synapse-rest-password-provider. (#8111) 2020-08-18 09:54:35 -04:00
Andrew Morgan e04e465b4d
Use the default templates when a custom template file cannot be found (#8037)
Fixes https://github.com/matrix-org/synapse/issues/6583
2020-08-17 17:05:00 +01:00
Erik Johnston 0304ad0c3d Move setting of Filter into code.
We do this to prevent foot guns. The default config uses a MemoryFilter,
but users are free to change to logging to files directly. If they do
then they have to ensure to set the `filters: [context]` on the right
handler, otherwise records get written with the wrong context.

Instead we move the logic to happen when we generate a record, which is
when we *log* rather than *handle*.

(It's possible to add filters to loggers in the config, however they
don't apply to descendant loggers and so they have to be manually set on
*every* logger used in the code base)
2020-08-11 18:10:46 +01:00
Erik Johnston db131b6b22
Change the default log config to reduce disk I/O and storage (#8040)
* Change default log config to buffer by default.

This batches up writes to the filesystem, which is more efficient for
disk I/O. This means that it can take some time for logs to get written
to disk. Note that ERROR logs (and above) immediately flush the buffer.

This only effects new installs, as we only write the log config if
started with `--generate-config` (in the same way we do for generating
signing keys).

* Default to keeping last 4 days of logs.

This hopefully reduces the amount of logs kept for new servers. Keeping
the last 1GB of logs is likely overkill for new servers, but equally may
not be enough for busy ones.

Instead, we keep the last four days worth of logs, enough so that admins
can investigate any problems that happened over e.g. a long weekend.
2020-08-11 18:09:46 +01:00
Richard van der Hoff 0cb169900e
Implement login blocking based on SAML attributes (#8052)
Hopefully this mostly speaks for itself. I also did a bit of cleaning up of the
error handling.

Fixes #8047
2020-08-11 16:08:10 +01:00
Travis Ralston 1048ed2afa
Clarify that undoing a shutdown might not be possible (#8010) 2020-08-07 17:16:24 +01:00
Erik Johnston 7620912d84
Add health check endpoint (#8048) 2020-08-07 14:21:24 +01:00
Erik Johnston 079bc3c8e3
Fixup worker doc (again) (#8000) 2020-08-06 10:35:59 +01:00
Erik Johnston a7bdf98d01
Rename database classes to make some sense (#8033) 2020-08-05 21:38:57 +01:00
Erik Johnston faba873d4b Merge branch 'develop' of github.com:matrix-org/synapse into erikj/add_rate_limiting_to_joins 2020-07-31 15:07:01 +01:00
Erik Johnston 18de00adb4 Add ratelimiting on joins 2020-07-31 15:06:56 +01:00
Travis Ralston e2a4ba6f9b
Add docs for undoing room shutdowns (#7998)
These docs were tested successfully in production by a customer, so it's probably fine.
2020-07-31 04:41:44 +01:00
Stuart Mumford 6d4b790021
Update workers docs (#7990) 2020-07-30 17:30:11 +01:00
Erik Johnston 606805bf06
Fix typo in docs/workers.md (#7992) 2020-07-30 16:28:36 +01:00
Olivier Wilkinson (reivilibre) 3aa36b782c Merge branch 'master' into develop 2020-07-30 15:18:36 +01:00
Erik Johnston 2c1b9d6763
Update worker docs with recent enhancements (#7969) 2020-07-29 23:22:13 +01:00
Aaron Raimist 2184f61fae
Various improvements to the docs (#7899) 2020-07-29 10:35:44 -04:00
Dirk Klimpel e866e3b896
Add an option to disable purge in delete room admin API (#7964)
Add option ```purge``` to ```POST /_synapse/admin/v1/rooms/<room_id>/delete```
Fixes: #3761

Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-07-28 20:08:23 +01:00
lugino-emeritus 3857de2194
Option to allow server admins to join complex rooms (#7902)
Fixes #7901.

Signed-off-by: Niklas Tittjung <nik_t.01@web.de>
2020-07-28 13:41:44 +01:00
Erik Johnston aaf9ce72a0
Fix typo in metrics docs (#7966) 2020-07-28 10:03:18 +01:00
Patrick Cloke 83434df381
Update the auth providers to be async. (#7935) 2020-07-23 15:45:39 -04:00
Brendan Abolivier 55f2617f8c
Update the dates for ACME v1 EOL
As per https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430
2020-07-22 16:18:40 +01:00
Adrian 64d2280299
Fix a typo in the sample config. (#7890) 2020-07-20 13:42:52 -04:00
Andrew Morgan 5ecf98f59e
Change sample config's postgres user to synapse_user (#7889)
The [postgres setup docs](https://github.com/matrix-org/synapse/blob/develop/docs/postgres.md#set-up-database) recommend setting up your database with user `synapse_user`.

However, uncommenting the postgres defaults in the sample config leave you with user `synapse`.

This PR switches the sample config to recommend `synapse_user`. Took a me a second to figure this out, so assume this will beneficial to others.
2020-07-20 18:29:25 +01:00
Patrick Cloke 852930add7
Add a default limit (of 100) to get/sync operations. (#7858) 2020-07-17 07:59:23 -04:00
Michael Albert fff483ea96
Add admin endpoint to get members in a room. (#7842) 2020-07-16 16:43:23 -04:00
Luke Faraone b0f031f92a
Combine nginx federation server blocks (#7823)
I'm pretty sure there's no technical reason these have to be distinct server blocks, so collapse into one and go with the more terse location block.

Signed-off-by: Luke W Faraone <luke@faraone.cc>
2020-07-16 16:01:45 +01:00
Patrick Cloke 8c7d0f163d
Allow accounts to be re-activated from the admin APIs. (#7847) 2020-07-15 11:00:21 -04:00
Patrick Cloke 111e70d75c
Return the proper 403 Forbidden error during errors with JWT logins. (#7844) 2020-07-15 07:10:21 -04:00
Brendan Abolivier 85223106f3
Allow email subjects to be customised through Synapse's configuration (#7846) 2020-07-14 19:10:42 +01:00
Dirk Klimpel 491f0dab1b
Add delete room admin endpoint (#7613)
The Delete Room admin API allows server admins to remove rooms from server
and block these rooms.
`DELETE /_synapse/admin/v1/rooms/<room_id>`
It is a combination and improvement of "[Shutdown room](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/shutdown_room.md)" and "[Purge room](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/purge_room.md)" API.

Fixes: #6425 

It also fixes a bug in [synapse/storage/data_stores/main/room.py](synapse/storage/data_stores/main/room.py) in ` get_room_with_stats`.
It should return `None` if the room is unknown. But it returns an `IndexError`.
901b1fa561/synapse/storage/data_stores/main/room.py (L99-L105)

Related to:
- #5575
- https://github.com/Awesome-Technologies/synapse-admin/issues/17

Signed-off-by: Dirk Klimpel dirk@klimpel.org
2020-07-14 12:36:23 +01:00
Patrick Cloke 77d2c05410
Add the option to validate the `iss` and `aud` claims for JWT logins. (#7827) 2020-07-14 07:16:43 -04:00
Erik Johnston f299441cc6
Add ability to shard the federation sender (#7798) 2020-07-10 18:26:36 +01:00
Nicolai Søborg 96bb01d8ec
Change Caddy links (old is deprecated) (#7789)
* Change Caddy links

Current links points to Caddy v1 which is deprecated.

Signed-off-by: Nicolai Søborg <git@xn--sb-lka.org>
2020-07-08 10:09:16 +01:00
Patrick Cloke 2a266f4511
Add documentation for JWT login type and improve sample config. (#7776) 2020-07-06 08:31:51 -04:00
Patrick Cloke 71cccf1593
Additional configuration options for auto-join rooms (#7763) 2020-06-30 15:41:36 -04:00
Erik Johnston b44bdd7f7b
Support running multiple media repos. (#7706)
This requires a new config option to specify which media repo should be
responsible for running background jobs to e.g. clear out expired URL
preview caches.
2020-06-17 14:13:30 +01:00
Richard van der Hoff e452973fd2
fix broken link in sample config (#7712) 2020-06-16 19:50:16 +01:00
hungrymonkey 5c5516f80e
Add instructions for authing with Keycloak via OpenID (#7659) 2020-06-16 11:28:21 -04:00
Patrick Cloke b9df7f70bb
Increase the default SAML session expirary time to 15 minutes. (#7664) 2020-06-11 07:55:45 -04:00
wondratsch c746889bb0
fix typo in sample_config.yaml (#7652)
Just a simple typo fix.

Signed-off-by: wondratsch 28294257+wondratsch@users.noreply.github.com
2020-06-11 11:51:10 +01:00
Andrew Morgan fcd6961441
Add option to enable encryption by default for new rooms (#7639)
Fixes https://github.com/matrix-org/synapse/issues/2431

Adds config option `encryption_enabled_by_default_for_room_type`, which determines whether encryption should be enabled with the default encryption algorithm in private or public rooms upon creation. Whether the room is private or public is decided based upon the room creation preset that is used.

Part of this PR is also pulling out all of the individual instances of `m.megolm.v1.aes-sha2` into a constant variable to eliminate typos ala https://github.com/matrix-org/synapse/pull/7637

Based on #7637
2020-06-10 17:44:34 +01:00
Travis Ralston 09099313e6
Add an option to disable autojoin for guest accounts (#6637)
Fixes https://github.com/matrix-org/synapse/issues/3177
2020-06-05 18:18:15 +01:00
Richard van der Hoff 1bc00fd76d
Clarifications to the admin api documentation (#7647)
* Clarify how to authenticate
* path params are not the same thing as query params
* Fix documentation for `/_synapse/admin/v2/users/<user_id>`
2020-06-05 17:31:05 +01:00
Dirk Klimpel 2970ce8367
Add device management to admin API (#7481)
- Admin is able to
  - change displaynames
  - delete devices
  - list devices
  - get device informations

Fixes #7330
2020-06-05 13:07:22 +01:00
Richard van der Hoff 11de843626
Cleanups to the OpenID Connect integration (#7628)
docs, default configs, comments. Nothing very significant.
2020-06-03 21:13:17 +01:00
Richard van der Hoff 1bbc9e2df6
Clean up exception handling in SAML2ResponseResource (#7614)
* Expose `return_html_error`, and allow it to take a Jinja2 template instead of a raw string

* Clean up exception handling in SAML2ResponseResource

  * use the existing code in `return_html_error` instead of re-implementing it
    (giving it a jinja2 template rather than inventing a new form of template)

  * do the exception-catching in the REST layer rather than in the handler
    layer, to make sure we catch all exceptions.
2020-06-03 10:41:12 +01:00
Christopher Cooper c4a820b32a
allow emails to be passed through SAML (#7385)
Signed-off-by: Christopher Cooper <cooperc@ocf.berkeley.edu>
2020-05-27 17:40:08 +01:00
Jason Robinson 4be968d05d
Fix sample config docs error (#7581)
'client_auth_method' commented out value was erronously 'client_auth_basic',
when code and docstring says it should be 'client_secret_basic'.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
2020-05-27 13:52:18 +01:00
Erik Johnston d7d8a2e7ee Fix up comments 2020-05-27 13:34:46 +01:00
Erik Johnston 4ba55559ac
Fix specifying cache factors via env vars with * in name. (#7580)
This mostly applise to `*stateGroupCache*` and co.

Broke in #6391.
2020-05-27 13:17:01 +01:00
Richard van der Hoff 66a564c859
Fix some DETECTED VIOLATIONS in the config file (#7550)
consistency ftw
2020-05-22 10:11:50 +01:00
David Vo d74cdc1a42
Ensure worker config exists in systemd service (#7528) 2020-05-21 13:47:23 +01:00
Richard van der Hoff 4fa74c7606
Minor clarifications to the TURN docs (#7533) 2020-05-20 11:04:34 +01:00
Paul Tötterman ab3e19d814
Improve API doc readability (#7527) 2020-05-19 11:20:23 +01:00
Richard van der Hoff 24d9151a08
Formatting for reverse-proxy docs (#7514)
also a small clarification to nginx
2020-05-15 15:13:39 +01:00
Jeff Peeler 572b444dab
Add Caddy 2 example (#7463)
The specific headers that are passed using this new configuration format
are Host and X-Forwarded-For, which should be all that's required.

Note that for production another matcher should be added in the first
section to properly handle the base_url lookup:
reverse_proxy /.well-known/matrix/* http://localhost:8008

Signed-off-by: Jeff Peeler <jpeeler@gmail.com>
2020-05-15 14:36:01 +01:00
Richard van der Hoff ec0b72bc4e Merge branch 'master' into develop 2020-05-14 18:12:00 +01:00
Richard van der Hoff 66d03639dc
Notes on using git (#7496)
* general updates to CONTRIBUTING.md
* notes on updating your PR
* Notes on squash-merging or otherwise
* document git branching model
2020-05-14 18:03:10 +01:00
Patrick Cloke 51fb0fc2e5
Update documentation about SSO mapping providers (#7458) 2020-05-12 10:51:07 -04:00
Amber Brown 7cb8b4bc67
Allow configuration of Synapse's cache without using synctl or environment variables (#6391) 2020-05-11 18:45:23 +01:00
Andrew Morgan 5cf758cdd6 Merge branch 'release-v1.13.0' into develop
* release-v1.13.0:
  Don't UPGRADE database rows
  RST indenting
  Put rollback instructions in upgrade notes
  Fix changelog typo
  Oh yeah, RST
  Absolute URL it is then
  Fix upgrade notes link
  Provide summary of upgrade issues in changelog. Fix )
  Move next version notes from changelog to upgrade notes
  Changelog fixes
  1.13.0rc1
  Documentation on setting up redis (#7446)
  Rework UI Auth session validation for registration (#7455)
  Fix errors from malformed log line (#7454)
  Drop support for redis.dbid (#7450)
2020-05-11 16:46:33 +01:00
Neil Johnson 85155654c5
Documentation on setting up redis (#7446) 2020-05-11 13:21:15 +01:00
Andrew Morgan 67feea8044
Extend spam checker to allow for multiple modules (#7435) 2020-05-08 19:25:48 +01:00
Quentin Gliech 616af44137
Implement OpenID Connect-based login (#7256) 2020-05-08 08:30:40 -04:00
Manuel Stahl a4a5ec4096
Add room details admin endpoint (#7317) 2020-05-07 15:33:07 -04:00
Brendan Abolivier 5bb26b7c4f Merge branch 'release-v1.13.0' into develop 2020-05-07 17:31:19 +02:00
Erik Johnston d7983b63a6
Support any process writing to cache invalidation stream. (#7436) 2020-05-07 13:51:08 +01:00
Brendan Abolivier d9b8d27494
Add a configuration setting for the dummy event threshold (#7422)
Add dummy_events_threshold which allows configuring the number of forward extremities a room needs for Synapse to send forward extremities in it.
2020-05-07 10:35:23 +01:00
Richard van der Hoff 207b1737ee
Update reverse_proxy.md
a couple of cleanups
2020-05-05 11:29:29 +01:00
Erik Johnston 37f6823f5b
Add instance name to RDATA/POSITION commands (#7364)
This is primarily for allowing us to send those commands from workers, but for now simply allows us to ignore echoed RDATA/POSITION commands that we sent (we get echoes of sent commands when using redis). Currently we log a WARNING on the master process every time we receive an echoed RDATA.
2020-04-29 16:23:08 +01:00
Andrew Morgan c58ae367d8
Clean up admin api docs (#7361) 2020-04-28 20:06:03 +01:00
Manuel Stahl 04dd7d182d
Return total number of users and profile attributes in admin users endpoint (#6881)
Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
2020-04-28 18:19:36 +01:00
Brendan Abolivier 036fab5d8a
Document monitoring workers (#7357)
It doesn't seem to be documented anywhere and means that you suddenly start losing metrics without any obvious reason when you go from monolith to workers (e.g. #7312).
2020-04-27 21:36:47 +02:00
lub aa2492907f
Add some explanation to application_services.md (#7091)
Signed-off-by: Simon Körner <git@lubiland.de>
2020-04-27 15:03:09 +01:00
Patrick Cloke 7bfe0902ce
Add documentation to the sample config about the templates for SSO. (#7343) 2020-04-24 15:03:49 -04:00
Patrick Cloke 204664d1ad Synapse v1.12.4
Features:
 
 * Always send users their own device updates. (#7160)
 * Add support for handling GET requests for account_data on a worker. (#7311)
 
 Bugfixes:
 
 * Fix a bug that prevented cross-signing with users on worker-mode synapses. (#7255)
 * Do not treat display names as globs in push rules. (#7271)
 * Fix a bug with cross-signing devices belonging to remote users who did not share a
   room with any user on the local homeserver. (#7289)
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAl6hs9EACgkQM/xY9qcR
 MEjq2w/9EC6m4Bx/2bDlohUy3GKfe9BUpZnMDGSgQR1v+J+OIA4yZ1s16hmY95fr
 rGjNaZlcUbLUFj9GfaLOOZjvuXQ0KslO2ojP1lcSaKCfx9WK1CH0DtCNVB0lAYQv
 sfu2C2I91XetkiMHo5GqYBP1d6IK3OTI7YRz72sHOQTVPFcSB1/wCUPuan7P90HE
 gBdfxtl7Xl1TU0tfvJoeRLVarktVnE4LiXREfm1eGpRECPclewu7sbx5p1ESN5mg
 /RBIh0FP4rOL4cP4Qt5qiCo68woJmBc4CwiV6pvcOyUMtMGqmwO9K/wJxqY+pxnh
 JKBNqv90BCSxD55Nywslx9yA52kshuRdcdqHuOVYOHo4Dcwu+ecyySpmzuZ2EJPc
 hDEkf3dnpfaVoMn3uesbwGNvHEr69QXxR8yE1XZvlAdKzvy4Po/0qRAZuD+NG9qf
 YjN94bDbkASlFQgPn3wF7R+4RA/HdOf/1Ns6YKt2dFRikHb18WCBPWxHcSwscrgE
 W8xma8rVTlqfRBuAcZ6y4K3KSHFX6dxLUNxqUngyAhjTInSPuqt2EzlAdqRpnJBS
 t+hhj+zOh2ixxbz4ZrGdMx0jPF2XARr38x3F/LIjhHLcgOlHdCl4WWkxi9BSXOTd
 cscvQ8GJBH9c0ANy7PGBntXlohQiW3eVVqblWRbV95TpHVoimME=
 =Hc/7
 -----END PGP SIGNATURE-----

Merge tag 'v1.12.4'

Synapse v1.12.4

Features:

* Always send users their own device updates. (#7160)
* Add support for handling GET requests for account_data on a worker. (#7311)

Bugfixes:

* Fix a bug that prevented cross-signing with users on worker-mode synapses. (#7255)
* Do not treat display names as globs in push rules. (#7271)
* Fix a bug with cross-signing devices belonging to remote users who did not share a
  room with any user on the local homeserver. (#7289)
2020-04-23 12:03:33 -04:00
Brendan Abolivier 2e3b9a0fcb
Revert "Revert "Merge pull request #7315 from matrix-org/babolivier/request_token""
This reverts commit 1adf6a5587.
2020-04-23 11:23:53 +02:00
Brendan Abolivier fb825759e3 Merge branch 'master' into develop 2020-04-23 11:23:33 +02:00
Brendan Abolivier 1adf6a5587
Revert "Merge pull request #7315 from matrix-org/babolivier/request_token"
This reverts commit 6f4319368b, reversing
changes made to 0d775fcc2d.
2020-04-23 11:23:10 +02:00
Brendan Abolivier 6f4319368b
Merge pull request #7315 from matrix-org/babolivier/request_token
Config option to inhibit 3PID errors on /requestToken
2020-04-23 10:38:57 +02:00
Richard van der Hoff 71a1abb8a1
Stop the master relaying USER_SYNC for other workers (#7318)
Long story short: if we're handling presence on the current worker, we shouldn't be sending USER_SYNC commands over replication.

In an attempt to figure out what is going on here, I ended up refactoring some bits of the presencehandler code, so the first 4 commits here are non-functional refactors to move this code slightly closer to sanity. (There's still plenty to do here :/). Suggest reviewing individual commits.

Fixes (I hope) #7257.
2020-04-22 22:39:04 +01:00
Brendan Abolivier 69ad7cc13b
Config option to inhibit 3PID errors on /requestToken
Adds a request_token_inhibit_errors configuration flag (disabled by
default) which, if enabled, change the behaviour of all /requestToken
endpoints so that they return a 200 and a fake sid if the 3PID was/was
not found associated with an account (depending on the endpoint),
instead of an error.

Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2020-04-22 23:38:42 +02:00
Dirk Klimpel 6b6685db9f
Extend room admin api with additional attributes (#7225) 2020-04-22 13:38:41 +01:00
Richard van der Hoff 2aa5bf13c8 Merge branch 'release-v1.12.4' into develop 2020-04-22 13:09:23 +01:00
Richard van der Hoff 974c0d726a
Support GET account_data requests on a worker (#7311) 2020-04-21 10:46:30 +01:00
Lars Franke 13917232d5
Fix indention in generated config file (#7300)
Also adjust sample_config.yaml

Signed-off-by: Lars Franke <frcl@mailbox.org>
2020-04-20 16:51:27 +01:00
nataraj-hates-MS-for-stealing-github 0d775fcc2d
Improve example TURN configuration in documentation (#7284) 2020-04-17 08:04:23 -04:00
Tristan Lins c07fca9e2f
Clarify the comments for media_storage_providers options (#7272) 2020-04-17 07:09:33 -04:00
Andrew Morgan a48138784e
Allow specifying the value of Accept-Language header for URL previews (#7265) 2020-04-15 13:35:29 +01:00
Ryan Hovland 4a0dadafbe
Add setting to nginx configuration to allow larger file uploads (#7251) 2020-04-13 17:23:36 +01:00
Andrew Morgan a026bdaab7
Add matrix-synapse-shared-secret-auth as an example password provider (#7248) 2020-04-09 12:49:05 +01:00
Richard van der Hoff cae4121484
Make systemd-with-workers doc official (#7234)
Simplify and update this documentation, and make it part of the core dist.
2020-04-08 11:59:26 +01:00
Andrew Morgan 29b7e22b93
Add documentation to password_providers config option (#7238) 2020-04-08 00:46:50 +01:00
Martin Milata b0db928c63
Extend web_client_location to handle absolute URLs (#7006)
Log warning when filesystem path is used.

Signed-off-by: Martin Milata <martin@martinmilata.cz>
2020-04-03 11:57:34 -04:00
Richard van der Hoff 0122ef1037 Revert "Merge pull request #7153 from matrix-org/babolivier/sso_whitelist_login_fallback"
This was incorrectly merged to master.

This reverts commit 319c41f573, reversing
changes made to 229eb81498.
2020-04-03 11:17:39 +01:00
siroccal 250f87d0de
Update postgres.md (#7119) 2020-04-01 12:44:51 +01:00
Jostein Kjønigsen 2e826cd80c
Improve TURN documentation. (#7167) 2020-03-31 15:50:48 +01:00
Andrew Morgan d9f29f8dae
Fix a small typo in the `metrics_flags` config option. (#7171) 2020-03-30 17:38:21 +01:00
Erik Johnston 4f21c33be3
Remove usage of "conn_id" for presence. (#7128)
* Remove `conn_id` usage for UserSyncCommand.

Each tcp replication connection is assigned a "conn_id", which is used
to give an ID to a remotely connected worker. In a redis world, there
will no longer be a one to one mapping between connection and instance,
so instead we need to replace such usages with an ID generated by the
remote instances and included in the replicaiton commands.

This really only effects UserSyncCommand.

* Add CLEAR_USER_SYNCS command that is sent on shutdown.

This should help with the case where a synchrotron gets restarted
gracefully, rather than rely on 5 minute timeout.
2020-03-30 16:37:24 +01:00
Patrick Cloke c5f89fba55
Add developer documentation for running a local CAS server (#7147) 2020-03-30 07:28:42 -04:00
Richard van der Hoff b7da598a61 Always whitelist the login fallback for SSO (#7153)
That fallback sets the redirect URL to itself (so it can process the login
token then return gracefully to the client). This would make it pointless to
ask the user for confirmation, since the URL the confirmation page would be
showing wouldn't be the client's.
2020-03-27 20:24:52 +00:00
Dirk Klimpel fb69690761
Admin API to join users to a room. (#7051) 2020-03-27 19:16:43 +00:00
Dirk Klimpel 8327eb9280
Add options to prevent users from changing their profile. (#7096) 2020-03-27 19:15:23 +00:00
Brendan Abolivier 319c41f573
Merge pull request #7153 from matrix-org/babolivier/sso_whitelist_login_fallback
Always whitelist the login fallback for SSO
2020-03-27 15:34:41 +01:00
Brendan Abolivier 63aea691a7
Update the wording of the config comment 2020-03-27 15:09:12 +01:00
Brendan Abolivier 7083147961
Regenerate sample config 2020-03-26 19:01:54 +01:00
Dirk Klimpel e8e2ddb60a
Allow server admins to define and enforce a password policy (MSC2000). (#7118) 2020-03-26 16:51:13 +00:00
Aaron Raimist 6ca5e56fd1
Remove unused captcha_bypass_secret option (#7137)
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2020-03-25 17:49:34 +00:00
Erik Johnston 4cff617df1
Move catchup of replication streams to worker. (#7024)
This changes the replication protocol so that the server does not send down `RDATA` for rows that happened before the client connected. Instead, the server will send a `POSITION` and clients then query the database (or master out of band) to get up to date.
2020-03-25 14:54:01 +00:00
Richard van der Hoff 39230d2171
Clean up some LoggingContext stuff (#7120)
* Pull Sentinel out of LoggingContext

... and drop a few unnecessary references to it

* Factor out LoggingContext.current_context

move `current_context` and `set_context` out to top-level functions.

Mostly this means that I can more easily trace what's actually referring to
LoggingContext, but I think it's generally neater.

* move copy-to-parent into `stop`

this really just makes `start` and `stop` more symetric. It also means that it
behaves correctly if you manually `set_log_context` rather than using the
context manager.

* Replace `LoggingContext.alive` with `finished`

Turn `alive` into `finished` and make it a bit better defined.
2020-03-24 14:45:33 +00:00
Richard van der Hoff 5126cb1253 Merge branch 'master' into develop 2020-03-23 13:54:29 +00:00
Richard van der Hoff 229eb81498 Synapse 1.12.0 (2020-03-23)
===========================
 
 No significant changes since 1.12.0rc1.
 
 Debian packages and Docker images are rebuilt using the latest versions of
 dependency libraries, including Twisted 20.3.0. **Please see security advisory
 below**.
 
 Security advisory
 -----------------
 
 Synapse may be vulnerable to request-smuggling attacks when it is used with a
 reverse-proxy. The vulnerabilties are fixed in Twisted 20.3.0, and are
 described in
 [CVE-2020-10108](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10108)
 and
 [CVE-2020-10109](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10109).
 For a good introduction to this class of request-smuggling attacks, see
 https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn.
 
 We are not aware of these vulnerabilities being exploited in the wild, and
 do not believe that they are exploitable with current versions of any reverse
 proxies. Nevertheless, we recommend that all Synapse administrators ensure that
 they have the latest versions of the Twisted library to ensure that their
 installation remains secure.
 
 * Administrators using the [`matrix.org` Docker
   image](https://hub.docker.com/r/matrixdotorg/synapse/) or the [Debian/Ubuntu
   packages from
   `matrix.org`](https://github.com/matrix-org/synapse/blob/master/INSTALL.md#matrixorg-packages)
   should ensure that they have version 1.12.0 installed: these images include
   Twisted 20.3.0.
 * Administrators who have [installed Synapse from
   source](https://github.com/matrix-org/synapse/blob/master/INSTALL.md#installing-from-source)
   should upgrade Twisted within their virtualenv by running:
   ```sh
   <path_to_virtualenv>/bin/pip install 'Twisted>=20.3.0'
   ```
 * Administrators who have installed Synapse from distribution packages should
   consult the information from their distributions.
 
 The `matrix.org` Synapse instance was not vulnerable to these vulnerabilities.
 
 Advance notice of change to the default `git` branch for Synapse
 ----------------------------------------------------------------
 
 Currently, the default `git` branch for Synapse is `master`, which tracks the
 latest release.
 
 After the release of Synapse 1.13.0, we intend to change this default to
 `develop`, which is the development tip. This is more consistent with common
 practice and modern `git` usage.
 
 Although we try to keep `develop` in a stable state, there may be occasions
 where regressions creep in. Developers and distributors who have scripts which
 run builds using the default branch of `Synapse` should therefore consider
 pinning their scripts to `master`.
 
 Synapse 1.12.0rc1 (2020-03-19)
 ==============================
 
 Features
 --------
 
 - Changes related to room alias management ([MSC2432](https://github.com/matrix-org/matrix-doc/pull/2432)):
   - Publishing/removing a room from the room directory now requires the user to have a power level capable of modifying the canonical alias, instead of the room aliases. ([\#6965](https://github.com/matrix-org/synapse/issues/6965))
   - Validate the `alt_aliases` property of canonical alias events. ([\#6971](https://github.com/matrix-org/synapse/issues/6971))
   - Users with a power level sufficient to modify the canonical alias of a room can now delete room aliases. ([\#6986](https://github.com/matrix-org/synapse/issues/6986))
   - Implement updated authorization rules and redaction rules for aliases events, from [MSC2261](https://github.com/matrix-org/matrix-doc/pull/2261) and [MSC2432](https://github.com/matrix-org/matrix-doc/pull/2432). ([\#7037](https://github.com/matrix-org/synapse/issues/7037))
   - Stop sending m.room.aliases events during room creation and upgrade. ([\#6941](https://github.com/matrix-org/synapse/issues/6941))
   - Synapse no longer uses room alias events to calculate room names for push notifications. ([\#6966](https://github.com/matrix-org/synapse/issues/6966))
   - The room list endpoint no longer returns a list of aliases. ([\#6970](https://github.com/matrix-org/synapse/issues/6970))
   - Remove special handling of aliases events from [MSC2260](https://github.com/matrix-org/matrix-doc/pull/2260) added in v1.10.0rc1. ([\#7034](https://github.com/matrix-org/synapse/issues/7034))
 - Expose the `synctl`, `hash_password` and `generate_config` commands in the snapcraft package. Contributed by @devec0. ([\#6315](https://github.com/matrix-org/synapse/issues/6315))
 - Check that server_name is correctly set before running database updates. ([\#6982](https://github.com/matrix-org/synapse/issues/6982))
 - Break down monthly active users by `appservice_id` and emit via Prometheus. ([\#7030](https://github.com/matrix-org/synapse/issues/7030))
 - Render a configurable and comprehensible error page if something goes wrong during the SAML2 authentication process. ([\#7058](https://github.com/matrix-org/synapse/issues/7058), [\#7067](https://github.com/matrix-org/synapse/issues/7067))
 - Add an optional parameter to control whether other sessions are logged out when a user's password is modified. ([\#7085](https://github.com/matrix-org/synapse/issues/7085))
 - Add prometheus metrics for the number of active pushers. ([\#7103](https://github.com/matrix-org/synapse/issues/7103), [\#7106](https://github.com/matrix-org/synapse/issues/7106))
 - Improve performance when making HTTPS requests to sygnal, sydent, etc, by sharing the SSL context object between connections. ([\#7094](https://github.com/matrix-org/synapse/issues/7094))
 
 Bugfixes
 --------
 
 - When a user's profile is updated via the admin API, also generate a displayname/avatar update for that user in each room. ([\#6572](https://github.com/matrix-org/synapse/issues/6572))
 - Fix a couple of bugs in email configuration handling. ([\#6962](https://github.com/matrix-org/synapse/issues/6962))
 - Fix an issue affecting worker-based deployments where replication would stop working, necessitating a full restart, after joining a large room. ([\#6967](https://github.com/matrix-org/synapse/issues/6967))
 - Fix `duplicate key` error which was logged when rejoining a room over federation. ([\#6968](https://github.com/matrix-org/synapse/issues/6968))
 - Prevent user from setting 'deactivated' to anything other than a bool on the v2 PUT /users Admin API. ([\#6990](https://github.com/matrix-org/synapse/issues/6990))
 - Fix py35-old CI by using native tox package. ([\#7018](https://github.com/matrix-org/synapse/issues/7018))
 - Fix a bug causing `org.matrix.dummy_event` to be included in responses from `/sync`. ([\#7035](https://github.com/matrix-org/synapse/issues/7035))
 - Fix a bug that renders UTF-8 text files incorrectly when loaded from media. Contributed by @TheStranjer. ([\#7044](https://github.com/matrix-org/synapse/issues/7044))
 - Fix a bug that would cause Synapse to respond with an error about event visibility if a client tried to request the state of a room at a given token. ([\#7066](https://github.com/matrix-org/synapse/issues/7066))
 - Repair a data-corruption issue which was introduced in Synapse 1.10, and fixed in Synapse 1.11, and which could cause `/sync` to return with 404 errors about missing events and unknown rooms. ([\#7070](https://github.com/matrix-org/synapse/issues/7070))
 - Fix a bug causing account validity renewal emails to be sent even if the feature is turned off in some cases. ([\#7074](https://github.com/matrix-org/synapse/issues/7074))
 
 Improved Documentation
 ----------------------
 
 - Updated CentOS8 install instructions. Contributed by Richard Kellner. ([\#6925](https://github.com/matrix-org/synapse/issues/6925))
 - Fix `POSTGRES_INITDB_ARGS` in the `contrib/docker/docker-compose.yml` example docker-compose configuration. ([\#6984](https://github.com/matrix-org/synapse/issues/6984))
 - Change date in [INSTALL.md](./INSTALL.md#tls-certificates) for last date of getting TLS certificates to November 2019. ([\#7015](https://github.com/matrix-org/synapse/issues/7015))
 - Document that the fallback auth endpoints must be routed to the same worker node as the register endpoints. ([\#7048](https://github.com/matrix-org/synapse/issues/7048))
 
 Deprecations and Removals
 -------------------------
 
 - Remove the unused query_auth federation endpoint per [MSC2451](https://github.com/matrix-org/matrix-doc/pull/2451). ([\#7026](https://github.com/matrix-org/synapse/issues/7026))
 
 Internal Changes
 ----------------
 
 - Add type hints to `logging/context.py`. ([\#6309](https://github.com/matrix-org/synapse/issues/6309))
 - Add some clarifications to `README.md` in the database schema directory. ([\#6615](https://github.com/matrix-org/synapse/issues/6615))
 - Refactoring work in preparation for changing the event redaction algorithm. ([\#6874](https://github.com/matrix-org/synapse/issues/6874), [\#6875](https://github.com/matrix-org/synapse/issues/6875), [\#6983](https://github.com/matrix-org/synapse/issues/6983), [\#7003](https://github.com/matrix-org/synapse/issues/7003))
 - Improve performance of v2 state resolution for large rooms. ([\#6952](https://github.com/matrix-org/synapse/issues/6952), [\#7095](https://github.com/matrix-org/synapse/issues/7095))
 - Reduce time spent doing GC, by freezing objects on startup. ([\#6953](https://github.com/matrix-org/synapse/issues/6953))
 - Minor perfermance fixes to `get_auth_chain_ids`. ([\#6954](https://github.com/matrix-org/synapse/issues/6954))
 - Don't record remote cross-signing keys in the `devices` table. ([\#6956](https://github.com/matrix-org/synapse/issues/6956))
 - Use flake8-comprehensions to enforce good hygiene of list/set/dict comprehensions. ([\#6957](https://github.com/matrix-org/synapse/issues/6957))
 - Merge worker apps together. ([\#6964](https://github.com/matrix-org/synapse/issues/6964), [\#7002](https://github.com/matrix-org/synapse/issues/7002), [\#7055](https://github.com/matrix-org/synapse/issues/7055), [\#7104](https://github.com/matrix-org/synapse/issues/7104))
 - Remove redundant `store_room` call from `FederationHandler._process_received_pdu`. ([\#6979](https://github.com/matrix-org/synapse/issues/6979))
 - Update warning for incorrect database collation/ctype to include link to documentation. ([\#6985](https://github.com/matrix-org/synapse/issues/6985))
 - Add some type annotations to the database storage classes. ([\#6987](https://github.com/matrix-org/synapse/issues/6987))
 - Port `synapse.handlers.presence` to async/await. ([\#6991](https://github.com/matrix-org/synapse/issues/6991), [\#7019](https://github.com/matrix-org/synapse/issues/7019))
 - Add some type annotations to the federation base & client classes. ([\#6995](https://github.com/matrix-org/synapse/issues/6995))
 - Port `synapse.rest.keys` to async/await. ([\#7020](https://github.com/matrix-org/synapse/issues/7020))
 - Add a type check to `is_verified` when processing room keys. ([\#7045](https://github.com/matrix-org/synapse/issues/7045))
 - Add type annotations and comments to the auth handler. ([\#7063](https://github.com/matrix-org/synapse/issues/7063))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEv27Axt/F4vrTL/8QOSor00I9eP8FAl54vN0ACgkQOSor00I9
 eP/LOgf/U22VypPu2Cl4vofxigeeUL+ph0gEQDCsF7i3EPi9ObeTi4aUggC95dBs
 MY4JQENmVrk1NhVWQpC08mjehsr4xUzJuwnPPIeGQ6X5U/2BS4YUasxOt9A+iLUz
 8IxTzUgx4T+CTfibZvn6xdE/tZK/b2N3BoOikesutD2aQGC5Fm6w2HReoY4Qrdgw
 AwHlsIV22PYgEn0RL5y6DJ2NUU9SdeSmPjKNe+R1rHDlTpvH7LSyINhOFkYDgRPY
 xmlH+Ek5+7vLi3AlWg6pA001mMWGADlC4T84URcf1fQv6hXT1iM+A9CqC57jGlfT
 nHDphCtz0Uk9kmgT4To+hclLDWecYw==
 =iSEk
 -----END PGP SIGNATURE-----

Merge tag 'v1.12.0'

Synapse 1.12.0 (2020-03-23)
===========================

No significant changes since 1.12.0rc1.

Debian packages and Docker images are rebuilt using the latest versions of
dependency libraries, including Twisted 20.3.0. **Please see security advisory
below**.

Security advisory
-----------------

Synapse may be vulnerable to request-smuggling attacks when it is used with a
reverse-proxy. The vulnerabilties are fixed in Twisted 20.3.0, and are
described in
[CVE-2020-10108](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10108)
and
[CVE-2020-10109](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10109).
For a good introduction to this class of request-smuggling attacks, see
https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn.

We are not aware of these vulnerabilities being exploited in the wild, and
do not believe that they are exploitable with current versions of any reverse
proxies. Nevertheless, we recommend that all Synapse administrators ensure that
they have the latest versions of the Twisted library to ensure that their
installation remains secure.

* Administrators using the [`matrix.org` Docker
  image](https://hub.docker.com/r/matrixdotorg/synapse/) or the [Debian/Ubuntu
  packages from
  `matrix.org`](https://github.com/matrix-org/synapse/blob/master/INSTALL.md#matrixorg-packages)
  should ensure that they have version 1.12.0 installed: these images include
  Twisted 20.3.0.
* Administrators who have [installed Synapse from
  source](https://github.com/matrix-org/synapse/blob/master/INSTALL.md#installing-from-source)
  should upgrade Twisted within their virtualenv by running:
  ```sh
  <path_to_virtualenv>/bin/pip install 'Twisted>=20.3.0'
  ```
* Administrators who have installed Synapse from distribution packages should
  consult the information from their distributions.

The `matrix.org` Synapse instance was not vulnerable to these vulnerabilities.

Advance notice of change to the default `git` branch for Synapse
----------------------------------------------------------------

Currently, the default `git` branch for Synapse is `master`, which tracks the
latest release.

After the release of Synapse 1.13.0, we intend to change this default to
`develop`, which is the development tip. This is more consistent with common
practice and modern `git` usage.

Although we try to keep `develop` in a stable state, there may be occasions
where regressions creep in. Developers and distributors who have scripts which
run builds using the default branch of `Synapse` should therefore consider
pinning their scripts to `master`.

Synapse 1.12.0rc1 (2020-03-19)
==============================

Features
--------

- Changes related to room alias management ([MSC2432](https://github.com/matrix-org/matrix-doc/pull/2432)):
  - Publishing/removing a room from the room directory now requires the user to have a power level capable of modifying the canonical alias, instead of the room aliases. ([\#6965](https://github.com/matrix-org/synapse/issues/6965))
  - Validate the `alt_aliases` property of canonical alias events. ([\#6971](https://github.com/matrix-org/synapse/issues/6971))
  - Users with a power level sufficient to modify the canonical alias of a room can now delete room aliases. ([\#6986](https://github.com/matrix-org/synapse/issues/6986))
  - Implement updated authorization rules and redaction rules for aliases events, from [MSC2261](https://github.com/matrix-org/matrix-doc/pull/2261) and [MSC2432](https://github.com/matrix-org/matrix-doc/pull/2432). ([\#7037](https://github.com/matrix-org/synapse/issues/7037))
  - Stop sending m.room.aliases events during room creation and upgrade. ([\#6941](https://github.com/matrix-org/synapse/issues/6941))
  - Synapse no longer uses room alias events to calculate room names for push notifications. ([\#6966](https://github.com/matrix-org/synapse/issues/6966))
  - The room list endpoint no longer returns a list of aliases. ([\#6970](https://github.com/matrix-org/synapse/issues/6970))
  - Remove special handling of aliases events from [MSC2260](https://github.com/matrix-org/matrix-doc/pull/2260) added in v1.10.0rc1. ([\#7034](https://github.com/matrix-org/synapse/issues/7034))
- Expose the `synctl`, `hash_password` and `generate_config` commands in the snapcraft package. Contributed by @devec0. ([\#6315](https://github.com/matrix-org/synapse/issues/6315))
- Check that server_name is correctly set before running database updates. ([\#6982](https://github.com/matrix-org/synapse/issues/6982))
- Break down monthly active users by `appservice_id` and emit via Prometheus. ([\#7030](https://github.com/matrix-org/synapse/issues/7030))
- Render a configurable and comprehensible error page if something goes wrong during the SAML2 authentication process. ([\#7058](https://github.com/matrix-org/synapse/issues/7058), [\#7067](https://github.com/matrix-org/synapse/issues/7067))
- Add an optional parameter to control whether other sessions are logged out when a user's password is modified. ([\#7085](https://github.com/matrix-org/synapse/issues/7085))
- Add prometheus metrics for the number of active pushers. ([\#7103](https://github.com/matrix-org/synapse/issues/7103), [\#7106](https://github.com/matrix-org/synapse/issues/7106))
- Improve performance when making HTTPS requests to sygnal, sydent, etc, by sharing the SSL context object between connections. ([\#7094](https://github.com/matrix-org/synapse/issues/7094))

Bugfixes
--------

- When a user's profile is updated via the admin API, also generate a displayname/avatar update for that user in each room. ([\#6572](https://github.com/matrix-org/synapse/issues/6572))
- Fix a couple of bugs in email configuration handling. ([\#6962](https://github.com/matrix-org/synapse/issues/6962))
- Fix an issue affecting worker-based deployments where replication would stop working, necessitating a full restart, after joining a large room. ([\#6967](https://github.com/matrix-org/synapse/issues/6967))
- Fix `duplicate key` error which was logged when rejoining a room over federation. ([\#6968](https://github.com/matrix-org/synapse/issues/6968))
- Prevent user from setting 'deactivated' to anything other than a bool on the v2 PUT /users Admin API. ([\#6990](https://github.com/matrix-org/synapse/issues/6990))
- Fix py35-old CI by using native tox package. ([\#7018](https://github.com/matrix-org/synapse/issues/7018))
- Fix a bug causing `org.matrix.dummy_event` to be included in responses from `/sync`. ([\#7035](https://github.com/matrix-org/synapse/issues/7035))
- Fix a bug that renders UTF-8 text files incorrectly when loaded from media. Contributed by @TheStranjer. ([\#7044](https://github.com/matrix-org/synapse/issues/7044))
- Fix a bug that would cause Synapse to respond with an error about event visibility if a client tried to request the state of a room at a given token. ([\#7066](https://github.com/matrix-org/synapse/issues/7066))
- Repair a data-corruption issue which was introduced in Synapse 1.10, and fixed in Synapse 1.11, and which could cause `/sync` to return with 404 errors about missing events and unknown rooms. ([\#7070](https://github.com/matrix-org/synapse/issues/7070))
- Fix a bug causing account validity renewal emails to be sent even if the feature is turned off in some cases. ([\#7074](https://github.com/matrix-org/synapse/issues/7074))

Improved Documentation
----------------------

- Updated CentOS8 install instructions. Contributed by Richard Kellner. ([\#6925](https://github.com/matrix-org/synapse/issues/6925))
- Fix `POSTGRES_INITDB_ARGS` in the `contrib/docker/docker-compose.yml` example docker-compose configuration. ([\#6984](https://github.com/matrix-org/synapse/issues/6984))
- Change date in [INSTALL.md](./INSTALL.md#tls-certificates) for last date of getting TLS certificates to November 2019. ([\#7015](https://github.com/matrix-org/synapse/issues/7015))
- Document that the fallback auth endpoints must be routed to the same worker node as the register endpoints. ([\#7048](https://github.com/matrix-org/synapse/issues/7048))

Deprecations and Removals
-------------------------

- Remove the unused query_auth federation endpoint per [MSC2451](https://github.com/matrix-org/matrix-doc/pull/2451). ([\#7026](https://github.com/matrix-org/synapse/issues/7026))

Internal Changes
----------------

- Add type hints to `logging/context.py`. ([\#6309](https://github.com/matrix-org/synapse/issues/6309))
- Add some clarifications to `README.md` in the database schema directory. ([\#6615](https://github.com/matrix-org/synapse/issues/6615))
- Refactoring work in preparation for changing the event redaction algorithm. ([\#6874](https://github.com/matrix-org/synapse/issues/6874), [\#6875](https://github.com/matrix-org/synapse/issues/6875), [\#6983](https://github.com/matrix-org/synapse/issues/6983), [\#7003](https://github.com/matrix-org/synapse/issues/7003))
- Improve performance of v2 state resolution for large rooms. ([\#6952](https://github.com/matrix-org/synapse/issues/6952), [\#7095](https://github.com/matrix-org/synapse/issues/7095))
- Reduce time spent doing GC, by freezing objects on startup. ([\#6953](https://github.com/matrix-org/synapse/issues/6953))
- Minor perfermance fixes to `get_auth_chain_ids`. ([\#6954](https://github.com/matrix-org/synapse/issues/6954))
- Don't record remote cross-signing keys in the `devices` table. ([\#6956](https://github.com/matrix-org/synapse/issues/6956))
- Use flake8-comprehensions to enforce good hygiene of list/set/dict comprehensions. ([\#6957](https://github.com/matrix-org/synapse/issues/6957))
- Merge worker apps together. ([\#6964](https://github.com/matrix-org/synapse/issues/6964), [\#7002](https://github.com/matrix-org/synapse/issues/7002), [\#7055](https://github.com/matrix-org/synapse/issues/7055), [\#7104](https://github.com/matrix-org/synapse/issues/7104))
- Remove redundant `store_room` call from `FederationHandler._process_received_pdu`. ([\#6979](https://github.com/matrix-org/synapse/issues/6979))
- Update warning for incorrect database collation/ctype to include link to documentation. ([\#6985](https://github.com/matrix-org/synapse/issues/6985))
- Add some type annotations to the database storage classes. ([\#6987](https://github.com/matrix-org/synapse/issues/6987))
- Port `synapse.handlers.presence` to async/await. ([\#6991](https://github.com/matrix-org/synapse/issues/6991), [\#7019](https://github.com/matrix-org/synapse/issues/7019))
- Add some type annotations to the federation base & client classes. ([\#6995](https://github.com/matrix-org/synapse/issues/6995))
- Port `synapse.rest.keys` to async/await. ([\#7020](https://github.com/matrix-org/synapse/issues/7020))
- Add a type check to `is_verified` when processing room keys. ([\#7045](https://github.com/matrix-org/synapse/issues/7045))
- Add type annotations and comments to the auth handler. ([\#7063](https://github.com/matrix-org/synapse/issues/7063))
2020-03-23 13:54:17 +00:00
Richard van der Hoff c165c1233b
Improve database configuration docs (#6988)
Attempts to clarify the sample config for databases, and add some stuff about
tcp keepalives to `postgres.md`.
2020-03-20 15:24:22 +00:00
Patrick Cloke 88b41986db
Add an option to the set password API to choose whether to logout other devices. (#7085) 2020-03-18 07:50:00 -04:00
Richard van der Hoff 4ce50519cd
Update postgres.md
fix broken link
2020-03-17 18:08:43 +00:00
Richard van der Hoff 6a35046363 Revert "Add options to disable setting profile info for prevent changes. (#7053)"
This reverts commit 54dd28621b, reversing
changes made to 6640460d05.
2020-03-17 11:25:01 +00:00
Brendan Abolivier f9e98176bf
Put the file in the templates directory 2020-03-11 20:31:42 +00:00
Brendan Abolivier 900bca9707
Update wording and config 2020-03-11 19:40:30 +00:00
Brendan Abolivier 54dd28621b
Add options to disable setting profile info for prevent changes. (#7053) 2020-03-10 22:23:01 +00:00
Dirk Klimpel 751d51dd12
Update sample_config.yaml 2020-03-10 21:41:25 +01:00
Brendan Abolivier 51c094c4ac
Update sample config 2020-03-10 14:00:29 +00:00
dklimpel 885134529f updates after review 2020-03-09 22:09:29 +01:00
Patrick Cloke 66315d862f
Update routing of fallback auth in the worker docs. (#7048) 2020-03-09 07:19:24 -04:00
dklimpel 99bbe177b6 add disable_3pid_changes 2020-03-08 21:58:12 +01:00
dklimpel 20545a2199 lint2 2020-03-08 15:28:00 +01:00
dklimpel fb078f921b changelog 2020-03-08 15:19:07 +01:00
Brendan Abolivier 43f874055d
Merge branch 'master' into develop 2020-03-03 15:20:49 +00:00
Richard van der Hoff b68041df3d Add a whitelist for the SSO confirmation step. 2020-03-02 17:05:09 +00:00
Brendan Abolivier b2bd54a2e3 Add a confirmation step to the SSO login flow 2020-03-02 16:36:32 +00:00
Richard van der Hoff 4c2ed3f20e
Fix minor issues with email config (#6962)
* Give `notif_template_html`, `notif_template_text` default values (fixes #6960)
 * Don't complain if `smtp_host` and `smtp_port` are unset, since they have sensible defaults (fixes #6961)
 * Set the example for `enable_notifs` to `True`, for consistency and because it's more useful
 * Raise errors as ConfigError rather than RuntimeError for nicer formatting
2020-02-24 15:18:38 +00:00
Patrick Cloke 509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
Brendan Abolivier bbe39f808c
Merge pull request #6940 from matrix-org/babolivier/federate.md
Clean up and update federation docs
2020-02-19 10:58:59 +00:00
Brendan Abolivier 71d65407e7
Incorporate review 2020-02-19 10:03:19 +00:00
Brendan Abolivier 61b457e3ec
Incorporate review 2020-02-18 17:20:03 +00:00
Brendan Abolivier a0d2f9d089
Phrasing 2020-02-18 16:16:49 +00:00
Brendan Abolivier d484126bf7
Merge pull request #6907 from matrix-org/babolivier/acme-config
Add mention and warning about ACME v1 deprecation to the TLS config
2020-02-18 16:11:31 +00:00
Erik Johnston 818def8248
Fix worker docs to point `/publicised_groups` API correctly. (#6938) 2020-02-18 15:27:45 +00:00
Brendan Abolivier d009535639
Add mention of SRV records as an advanced topic 2020-02-18 14:07:41 +00:00
Brendan Abolivier ba7a523854
Argh trailing spaces 2020-02-18 13:57:15 +00:00
Brendan Abolivier e837be5b5c
Fix links in the reverse proxy doc 2020-02-18 13:53:58 +00:00
Brendan Abolivier 3c67eee6dc
Make federate.md more of a sumary of the steps to follow to set up replication 2020-02-18 13:51:03 +00:00
Brendan Abolivier 8ee0d74516
Split the delegating documentation out of federate.md and trim it down 2020-02-18 12:05:45 +00:00
Richard van der Hoff 97a42bbc3a
Add a warning about indentation to generated config (#6920)
Fixes #6916.
2020-02-14 16:22:30 +00:00
Patrick Cloke 49f877d32e
Filter the results of user directory searching via the spam checker (#6888)
Add a method to the spam checker to filter the user directory results.
2020-02-14 07:17:54 -05:00
Brendan Abolivier 5820ed905f
Add mention and warning about ACME v1 deprecation to the Synapse config 2020-02-13 14:20:08 +00:00
Patrick Cloke 361de49c90
Add documentation for the spam checker module (#6906)
Add documentation for the spam checker.
2020-02-13 07:40:57 -05:00
Brendan Abolivier 862669d6cc
Update docs/ACME.md 2020-02-13 11:29:08 +00:00
Brendan Abolivier 459d089af7
Mention that using Synapse to serve certificates requires restarts 2020-02-12 21:05:30 +00:00
Brendan Abolivier e45a7c0939
Remove duplicated info about certbot et al 2020-02-12 20:14:59 +00:00
Brendan Abolivier f092029d2d
Update ACME.md to mention ACME v1 deprecation 2020-02-12 20:14:16 +00:00
Brendan Abolivier 6cd34da8b1
Merge pull request #6891 from matrix-org/babolivier/retention-doc-amend
Spell out that the last event sent to a room won't be deleted by a purge
2020-02-12 20:12:20 +00:00
Brendan Abolivier 08e050c3fd
Rephrase 2020-02-12 15:39:40 +00:00
Brendan Abolivier 47acbc519f Merge branch 'master' into develop 2020-02-12 13:24:09 +00:00
Brendan Abolivier d9239b5257 Synapse 1.10.0 (2020-02-12)
===========================
 
 **WARNING to client developers**: As of this release Synapse validates `client_secret` parameters in the Client-Server API as per the spec. See [\#6766](https://github.com/matrix-org/synapse/issues/6766) for details.
 
 Updates to the Docker image
 ---------------------------
 
 - Update the docker images to Alpine Linux 3.11. ([\#6897](https://github.com/matrix-org/synapse/issues/6897))
 
 Synapse 1.10.0rc5 (2020-02-11)
 ==============================
 
 Bugfixes
 --------
 
 - Fix the filtering introduced in 1.10.0rc3 to also apply to the state blocks returned by `/sync`. ([\#6884](https://github.com/matrix-org/synapse/issues/6884))
 
 Synapse 1.10.0rc4 (2020-02-11)
 ==============================
 
 This release candidate was built incorrectly and is superceded by 1.10.0rc5.
 
 Synapse 1.10.0rc3 (2020-02-10)
 ==============================
 
 Features
 --------
 
 - Filter out `m.room.aliases` from the CS API to mitigate abuse while a better solution is specced. ([\#6878](https://github.com/matrix-org/synapse/issues/6878))
 
 Internal Changes
 ----------------
 
 - Fix continuous integration failures with old versions of `pip`, which were introduced by a release of the `zipp` library. ([\#6880](https://github.com/matrix-org/synapse/issues/6880))
 
 Synapse 1.10.0rc2 (2020-02-06)
 ==============================
 
 Bugfixes
 --------
 
 - Fix an issue with cross-signing where device signatures were not sent to remote servers. ([\#6844](https://github.com/matrix-org/synapse/issues/6844))
 - Fix to the unknown remote device detection which was introduced in 1.10.rc1. ([\#6848](https://github.com/matrix-org/synapse/issues/6848))
 
 Internal Changes
 ----------------
 
 - Detect unexpected sender keys on remote encrypted events and resync device lists. ([\#6850](https://github.com/matrix-org/synapse/issues/6850))
 
 Synapse 1.10.0rc1 (2020-01-31)
 ==============================
 
 Features
 --------
 
 - Add experimental support for updated authorization rules for aliases events, from [MSC2260](https://github.com/matrix-org/matrix-doc/pull/2260). ([\#6787](https://github.com/matrix-org/synapse/issues/6787), [\#6790](https://github.com/matrix-org/synapse/issues/6790), [\#6794](https://github.com/matrix-org/synapse/issues/6794))
 
 Bugfixes
 --------
 
 - Warn if postgres database has a non-C locale, as that can cause issues when upgrading locales (e.g. due to upgrading OS). ([\#6734](https://github.com/matrix-org/synapse/issues/6734))
 - Minor fixes to `PUT /_synapse/admin/v2/users` admin api. ([\#6761](https://github.com/matrix-org/synapse/issues/6761))
 - Validate `client_secret` parameter using the regex provided by the Client-Server API, temporarily allowing `:` characters for older clients. The `:` character will be removed in a future release. ([\#6767](https://github.com/matrix-org/synapse/issues/6767))
 - Fix persisting redaction events that have been redacted (or otherwise don't have a redacts key). ([\#6771](https://github.com/matrix-org/synapse/issues/6771))
 - Fix outbound federation request metrics. ([\#6795](https://github.com/matrix-org/synapse/issues/6795))
 - Fix bug where querying a remote user's device keys that weren't cached resulted in only returning a single device. ([\#6796](https://github.com/matrix-org/synapse/issues/6796))
 - Fix race in federation sender worker that delayed sending of device updates. ([\#6799](https://github.com/matrix-org/synapse/issues/6799), [\#6800](https://github.com/matrix-org/synapse/issues/6800))
 - Fix bug where Synapse didn't invalidate cache of remote users' devices when Synapse left a room. ([\#6801](https://github.com/matrix-org/synapse/issues/6801))
 - Fix waking up other workers when remote server is detected to have come back online. ([\#6811](https://github.com/matrix-org/synapse/issues/6811))
 
 Improved Documentation
 ----------------------
 
 - Clarify documentation related to `user_dir` and `federation_reader` workers. ([\#6775](https://github.com/matrix-org/synapse/issues/6775))
 
 Internal Changes
 ----------------
 
 - Record room versions in the `rooms` table. ([\#6729](https://github.com/matrix-org/synapse/issues/6729), [\#6788](https://github.com/matrix-org/synapse/issues/6788), [\#6810](https://github.com/matrix-org/synapse/issues/6810))
 - Propagate cache invalidates from workers to other workers. ([\#6748](https://github.com/matrix-org/synapse/issues/6748))
 - Remove some unnecessary admin handler abstraction methods. ([\#6751](https://github.com/matrix-org/synapse/issues/6751))
 - Add some debugging for media storage providers. ([\#6757](https://github.com/matrix-org/synapse/issues/6757))
 - Detect unknown remote devices and mark cache as stale. ([\#6776](https://github.com/matrix-org/synapse/issues/6776), [\#6819](https://github.com/matrix-org/synapse/issues/6819))
 - Attempt to resync remote users' devices when detected as stale. ([\#6786](https://github.com/matrix-org/synapse/issues/6786))
 - Delete current state from the database when server leaves a room. ([\#6792](https://github.com/matrix-org/synapse/issues/6792))
 - When a client asks for a remote user's device keys check if the local cache for that user has been marked as potentially stale. ([\#6797](https://github.com/matrix-org/synapse/issues/6797))
 - Add background update to clean out left rooms from current state. ([\#6802](https://github.com/matrix-org/synapse/issues/6802), [\#6816](https://github.com/matrix-org/synapse/issues/6816))
 - Refactoring work in preparation for changing the event redaction algorithm. ([\#6803](https://github.com/matrix-org/synapse/issues/6803), [\#6805](https://github.com/matrix-org/synapse/issues/6805), [\#6806](https://github.com/matrix-org/synapse/issues/6806), [\#6807](https://github.com/matrix-org/synapse/issues/6807), [\#6820](https://github.com/matrix-org/synapse/issues/6820))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAl5D780ACgkQHgFcFF8Z
 Fs32MxAApvBbTGMhC5E2SLt8Yh4DSfDmg8yCPLMRVTdJuUQyUVgKYFSl9LqzqCGe
 oLpzwbXXDpN1FYGuO3daFBSN3RaU1XCQPMayekTaF6Wvq9N9rNW1tAi6cQTjUf20
 bTAxZljXAIkfXOMbahlFIqAtL+ItTbECKPW4zbndIbC4TiWw/bJPFHeSuoQbHbsI
 TOgPjM4q2aydHFeTDL2Wqqm99pobPvnEji5sBzHOK+IyIaN0HZI7BZAsm38SDSEA
 HQTW342RNl7Vm0t2KPdhBq/43r4ENKSUbZjCctFZA+iqbV2mmE96zd2pLOIh9OLP
 7KoCDKlplQjInJjN3MQvPzn/cxHnZOmt3AwhQjfhOMcJWcwz00ouWp38oszgZi1c
 8y1htf1/Tr26veYxF7umlPHT0h8LNj4Wi7ceQjoIDPu4X70X6vXOk5sJmeJ+KqI5
 PJ4yUgw0erWX0I7jybavIvruyA9ZqCwk5FIIyG54IBIZLfughi/fcce49gfvJTa6
 Rdjvp0g28qxynkRKq9hVOBSX/TRXTedHe8tHBvSzoDRScvI2aDWQYjJkmGzzLF6L
 +eWiZ2ICJ2fs7XYIxPhEr/vSpmc/1R+k6vZIPeBTAs0muH8WoK2en6KZPmJ7C25L
 2FHdA+1S1Ybqsvyue5S5Sl6yGcvpzGzc1R4mHoB0jZki2v3Cwcs=
 =j25Z
 -----END PGP SIGNATURE-----

Merge tag 'v1.10.0'

Synapse 1.10.0 (2020-02-12)
===========================

**WARNING to client developers**: As of this release Synapse validates `client_secret` parameters in the Client-Server API as per the spec. See [\#6766](https://github.com/matrix-org/synapse/issues/6766) for details.

Updates to the Docker image
---------------------------

- Update the docker images to Alpine Linux 3.11. ([\#6897](https://github.com/matrix-org/synapse/issues/6897))

Synapse 1.10.0rc5 (2020-02-11)
==============================

Bugfixes
--------

- Fix the filtering introduced in 1.10.0rc3 to also apply to the state blocks returned by `/sync`. ([\#6884](https://github.com/matrix-org/synapse/issues/6884))

Synapse 1.10.0rc4 (2020-02-11)
==============================

This release candidate was built incorrectly and is superceded by 1.10.0rc5.

Synapse 1.10.0rc3 (2020-02-10)
==============================

Features
--------

- Filter out `m.room.aliases` from the CS API to mitigate abuse while a better solution is specced. ([\#6878](https://github.com/matrix-org/synapse/issues/6878))

Internal Changes
----------------

- Fix continuous integration failures with old versions of `pip`, which were introduced by a release of the `zipp` library. ([\#6880](https://github.com/matrix-org/synapse/issues/6880))

Synapse 1.10.0rc2 (2020-02-06)
==============================

Bugfixes
--------

- Fix an issue with cross-signing where device signatures were not sent to remote servers. ([\#6844](https://github.com/matrix-org/synapse/issues/6844))
- Fix to the unknown remote device detection which was introduced in 1.10.rc1. ([\#6848](https://github.com/matrix-org/synapse/issues/6848))

Internal Changes
----------------

- Detect unexpected sender keys on remote encrypted events and resync device lists. ([\#6850](https://github.com/matrix-org/synapse/issues/6850))

Synapse 1.10.0rc1 (2020-01-31)
==============================

Features
--------

- Add experimental support for updated authorization rules for aliases events, from [MSC2260](https://github.com/matrix-org/matrix-doc/pull/2260). ([\#6787](https://github.com/matrix-org/synapse/issues/6787), [\#6790](https://github.com/matrix-org/synapse/issues/6790), [\#6794](https://github.com/matrix-org/synapse/issues/6794))

Bugfixes
--------

- Warn if postgres database has a non-C locale, as that can cause issues when upgrading locales (e.g. due to upgrading OS). ([\#6734](https://github.com/matrix-org/synapse/issues/6734))
- Minor fixes to `PUT /_synapse/admin/v2/users` admin api. ([\#6761](https://github.com/matrix-org/synapse/issues/6761))
- Validate `client_secret` parameter using the regex provided by the Client-Server API, temporarily allowing `:` characters for older clients. The `:` character will be removed in a future release. ([\#6767](https://github.com/matrix-org/synapse/issues/6767))
- Fix persisting redaction events that have been redacted (or otherwise don't have a redacts key). ([\#6771](https://github.com/matrix-org/synapse/issues/6771))
- Fix outbound federation request metrics. ([\#6795](https://github.com/matrix-org/synapse/issues/6795))
- Fix bug where querying a remote user's device keys that weren't cached resulted in only returning a single device. ([\#6796](https://github.com/matrix-org/synapse/issues/6796))
- Fix race in federation sender worker that delayed sending of device updates. ([\#6799](https://github.com/matrix-org/synapse/issues/6799), [\#6800](https://github.com/matrix-org/synapse/issues/6800))
- Fix bug where Synapse didn't invalidate cache of remote users' devices when Synapse left a room. ([\#6801](https://github.com/matrix-org/synapse/issues/6801))
- Fix waking up other workers when remote server is detected to have come back online. ([\#6811](https://github.com/matrix-org/synapse/issues/6811))

Improved Documentation
----------------------

- Clarify documentation related to `user_dir` and `federation_reader` workers. ([\#6775](https://github.com/matrix-org/synapse/issues/6775))

Internal Changes
----------------

- Record room versions in the `rooms` table. ([\#6729](https://github.com/matrix-org/synapse/issues/6729), [\#6788](https://github.com/matrix-org/synapse/issues/6788), [\#6810](https://github.com/matrix-org/synapse/issues/6810))
- Propagate cache invalidates from workers to other workers. ([\#6748](https://github.com/matrix-org/synapse/issues/6748))
- Remove some unnecessary admin handler abstraction methods. ([\#6751](https://github.com/matrix-org/synapse/issues/6751))
- Add some debugging for media storage providers. ([\#6757](https://github.com/matrix-org/synapse/issues/6757))
- Detect unknown remote devices and mark cache as stale. ([\#6776](https://github.com/matrix-org/synapse/issues/6776), [\#6819](https://github.com/matrix-org/synapse/issues/6819))
- Attempt to resync remote users' devices when detected as stale. ([\#6786](https://github.com/matrix-org/synapse/issues/6786))
- Delete current state from the database when server leaves a room. ([\#6792](https://github.com/matrix-org/synapse/issues/6792))
- When a client asks for a remote user's device keys check if the local cache for that user has been marked as potentially stale. ([\#6797](https://github.com/matrix-org/synapse/issues/6797))
- Add background update to clean out left rooms from current state. ([\#6802](https://github.com/matrix-org/synapse/issues/6802), [\#6816](https://github.com/matrix-org/synapse/issues/6816))
- Refactoring work in preparation for changing the event redaction algorithm. ([\#6803](https://github.com/matrix-org/synapse/issues/6803), [\#6805](https://github.com/matrix-org/synapse/issues/6805), [\#6806](https://github.com/matrix-org/synapse/issues/6806), [\#6807](https://github.com/matrix-org/synapse/issues/6807), [\#6820](https://github.com/matrix-org/synapse/issues/6820))
2020-02-12 13:23:22 +00:00
Brendan Abolivier 6b21986e4e
Also spell it out in the purge history API doc 2020-02-11 17:56:04 +00:00
Brendan Abolivier a443d2a25d
Spell out that Synapse never purges the last event sent in a room 2020-02-11 17:37:09 +00:00
Erik Johnston 21db35f77e
Add support for putting fed user query API on workers (#6873) 2020-02-07 15:45:39 +00:00
Erik Johnston de2d267375
Allow moving group read APIs to workers (#6866) 2020-02-07 11:14:19 +00:00
Dirk Klimpel 56ca93ef59
Admin api to add an email address (#6789) 2020-02-07 10:29:36 +00:00
Robin Vleij f0561fcffd
Update documentation (#6859)
Update documentation to reflect the correct format of user_id (fully qualified).
2020-02-05 21:27:38 +00:00
Erik Johnston 02b44db922
Warn if postgres database has non-C locale. (#6734)
As using non-C locale can cause issues on upgrading OS.
2020-01-28 13:44:21 +00:00
Jason Robinson 1fe5001369 Fix federation_reader listeners doc as per PR review
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2020-01-27 10:21:25 +02:00
Jason Robinson aa6ad288f1 Clarifications to the workers documentation
* Add note that user_dir requires disabling user dir
  updates from the main synapse process.
* Add note that federation_reader should have
  the federation listener resource.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
2020-01-24 11:08:50 +02:00
Andrew Morgan d31f5f4d89
Update admin room docs with correct endpoints (#6770) 2020-01-23 11:37:26 +00:00
Andrew Morgan 90a28fb475
Admin API to list, filter and sort rooms (#6720) 2020-01-22 13:36:43 +00:00
Erik Johnston 5d7a6ad223
Allow streaming cache invalidate all to workers. (#6749) 2020-01-22 10:37:00 +00:00
Erik Johnston a8a50f5b57
Wake up transaction queue when remote server comes back online (#6706)
This will be used to retry outbound transactions to a remote server if
we think it might have come back up.
2020-01-17 10:27:19 +00:00
Richard van der Hoff 5ce0b17e38
Clarify the `account_validity` and `email` sections of the sample configuration. (#6685)
Generally try to make this more comprehensible, and make it match the
conventions.

I've removed the documentation for all the settings which allow you to change
the names of the template files, because I can't really see why they are
useful.
2020-01-17 10:04:15 +00:00
Andrew Morgan 1177d3f3a3
Quarantine media by ID or user ID (#6681) 2020-01-13 18:10:43 +00:00
Richard van der Hoff 47f4f493f0
Document more supported endpoints for workers (#6698) 2020-01-13 15:32:02 +00:00
Manuel Stahl d2906fe666 Allow admin users to create or modify users without a shared secret (#6495)
Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
2020-01-09 13:31:00 +00:00
Brendan Abolivier 3889fcd9d7
Fix typo in message retention policies doc 2020-01-08 13:27:29 +00:00
Brendan Abolivier cff1cb8685
Merge pull request #6624 from matrix-org/babolivier/retention_doc
Add complete documentation of the message retention policies support
2020-01-08 11:24:47 +00:00
Brendan Abolivier 2b6b7f482a
Merge pull request #6621 from matrix-org/babolivier/purge_job_config_typo
Fix a typo in the purge jobs configuration example
2020-01-07 16:17:40 +01:00
Brendan Abolivier 3675fb9bc6
Fix reference 2020-01-07 15:15:16 +00:00
Brendan Abolivier 7ba98a2874
Incorporate review 2020-01-07 15:14:33 +00:00
Brendan Abolivier 4be582d7c8 Merge branch 'develop' into babolivier/retention_doc 2020-01-07 15:07:19 +00:00
Brendan Abolivier 01fbd95736
Apply suggestions from code review
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2020-01-07 15:59:38 +01:00
Brendan Abolivier 391fb47791
Reword 2020-01-07 14:54:32 +00:00
Brendan Abolivier 3a86477162
Change the example from 5min to 12h
Have a purge job running every 5min is probably not something we want to advise admins to do as a sort-of default.
2020-01-07 14:53:07 +00:00
Richard van der Hoff 08815566bc
Automate generation of the sample and debian log configs (#6627) 2020-01-03 17:14:00 +00:00
Richard van der Hoff 98247c4a0e
Remove unused, undocumented "content repo" resource (#6628)
This looks like it got half-killed back in #888.

Fixes #6567.
2020-01-03 17:10:52 +00:00
Brendan Abolivier b7dec300b7
Fix vacuum instructions for sqlite 2020-01-03 13:51:59 +01:00
Brendan Abolivier 9279a2c4e4
Add a complete documentation of the message retention policies support 2020-01-03 13:45:03 +01:00
Brendan Abolivier dd2954f78d
Update sample config 2020-01-03 12:58:12 +01:00
ewaf1 0495097a7f Added the section 'Configuration' in /docs/turn-howto.md (#6614)
put the 2nd part of the "source installation"-section into a new section, because it also applies to Debian packages
2020-01-02 10:41:30 +00:00
Aaron Raimist 32779b59fa Reword sections of federate.md that explained delegation at time of Synapse 1.0 transition (#6601)
* Remove sections of federate.md explaining delegation at time of Synapse 1.0 transition

Signed-off-by: Aaron Raimist <aaron@raim.ist>

* Add changelog

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2020-01-02 10:28:20 +00:00
Richard van der Hoff dc96943d51 Merge branch 'master' into develop 2019-12-31 11:01:06 +00:00
Richard van der Hoff f03c877b32 sample log config
TODO: automate generation of this
2019-12-24 17:39:58 +00:00
Richard van der Hoff b95b762560
Add an export_signing_key script (#6546)
I want to do some key rotation, and it is silly that we don't have a way to do
this.
2019-12-19 11:11:14 +00:00
Will Hunt bfb95654c9 Add option to allow profile queries without sharing a room (#6523) 2019-12-16 16:11:55 +00:00
Andrew Morgan 0b90fc6ed2
Document Shutdown Room admin API (#6541) 2019-12-13 15:28:48 +00:00
Erik Johnston f46e05d053 Merge branch 'master' into develop 2019-12-13 10:55:47 +00:00
Erik Johnston bee1982d17 Synapse 1.7.0 (2019-12-13)
==========================
 
 This release changes the default settings so that only local authenticated users can query the server's room directory. See the [upgrade notes](UPGRADE.rst#upgrading-to-v170) for details.
 
 Support for SQLite versions before 3.11 is now deprecated. A future release will refuse to start if used with an SQLite version before 3.11.
 
 Administrators are reminded that SQLite should not be used for production instances. Instructions for migrating to Postgres are available [here](docs/postgres.md). A future release of synapse will, by default, disable federation for servers using SQLite.
 
 No significant changes since 1.7.0rc2.
 
 Synapse 1.7.0rc2 (2019-12-11)
 =============================
 
 Bugfixes
 --------
 
 - Fix incorrect error message for invalid requests when setting user's avatar URL. ([\#6497](https://github.com/matrix-org/synapse/issues/6497))
 - Fix support for SQLite 3.7. ([\#6499](https://github.com/matrix-org/synapse/issues/6499))
 - Fix regression where sending email push would not work when using a pusher worker. ([\#6507](https://github.com/matrix-org/synapse/issues/6507), [\#6509](https://github.com/matrix-org/synapse/issues/6509))
 
 Synapse 1.7.0rc1 (2019-12-09)
 =============================
 
 Features
 --------
 
 - Implement per-room message retention policies. ([\#5815](https://github.com/matrix-org/synapse/issues/5815), [\#6436](https://github.com/matrix-org/synapse/issues/6436))
 - Add etag and count fields to key backup endpoints to help clients guess if there are new keys. ([\#5858](https://github.com/matrix-org/synapse/issues/5858))
 - Add `/admin/v2/users` endpoint with pagination. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5925](https://github.com/matrix-org/synapse/issues/5925))
 - Require User-Interactive Authentication for `/account/3pid/add`, meaning the user's password will be required to add a third-party ID to their account. ([\#6119](https://github.com/matrix-org/synapse/issues/6119))
 - Implement the `/_matrix/federation/unstable/net.atleastfornow/state/<context>` API as drafted in MSC2314. ([\#6176](https://github.com/matrix-org/synapse/issues/6176))
 - Configure privacy-preserving settings by default for the room directory. ([\#6355](https://github.com/matrix-org/synapse/issues/6355))
 - Add ephemeral messages support by partially implementing [MSC2228](https://github.com/matrix-org/matrix-doc/pull/2228). ([\#6409](https://github.com/matrix-org/synapse/issues/6409))
 - Add support for [MSC 2367](https://github.com/matrix-org/matrix-doc/pull/2367), which allows specifying a reason on all membership events. ([\#6434](https://github.com/matrix-org/synapse/issues/6434))
 
 Bugfixes
 --------
 
 - Transfer non-standard power levels on room upgrade. ([\#6237](https://github.com/matrix-org/synapse/issues/6237))
 - Fix error from the Pillow library when uploading RGBA images. ([\#6241](https://github.com/matrix-org/synapse/issues/6241))
 - Correctly apply the event filter to the `state`, `events_before` and `events_after` fields in the response to `/context` requests. ([\#6329](https://github.com/matrix-org/synapse/issues/6329))
 - Fix caching devices for remote users when using workers, so that we don't attempt to refetch (and potentially fail) each time a user requests devices. ([\#6332](https://github.com/matrix-org/synapse/issues/6332))
 - Prevent account data syncs getting lost across TCP replication. ([\#6333](https://github.com/matrix-org/synapse/issues/6333))
 - Fix bug: TypeError in `register_user()` while using LDAP auth module. ([\#6406](https://github.com/matrix-org/synapse/issues/6406))
 - Fix an intermittent exception when handling read-receipts. ([\#6408](https://github.com/matrix-org/synapse/issues/6408))
 - Fix broken guest registration when there are existing blocks of numeric user IDs. ([\#6420](https://github.com/matrix-org/synapse/issues/6420))
 - Fix startup error when http proxy is defined. ([\#6421](https://github.com/matrix-org/synapse/issues/6421))
 - Fix error when using synapse_port_db on a vanilla synapse db. ([\#6449](https://github.com/matrix-org/synapse/issues/6449))
 - Fix uploading multiple cross signing signatures for the same user. ([\#6451](https://github.com/matrix-org/synapse/issues/6451))
 - Fix bug which lead to exceptions being thrown in a loop when a cross-signed device is deleted. ([\#6462](https://github.com/matrix-org/synapse/issues/6462))
 - Fix `synapse_port_db` not exiting with a 0 code if something went wrong during the port process. ([\#6470](https://github.com/matrix-org/synapse/issues/6470))
 - Improve sanity-checking when receiving events over federation. ([\#6472](https://github.com/matrix-org/synapse/issues/6472))
 - Fix inaccurate per-block Prometheus metrics. ([\#6491](https://github.com/matrix-org/synapse/issues/6491))
 - Fix small performance regression for sending invites. ([\#6493](https://github.com/matrix-org/synapse/issues/6493))
 - Back out cross-signing code added in Synapse 1.5.0, which caused a performance regression. ([\#6494](https://github.com/matrix-org/synapse/issues/6494))
 
 Improved Documentation
 ----------------------
 
 - Update documentation and variables in user contributed systemd reference file. ([\#6369](https://github.com/matrix-org/synapse/issues/6369), [\#6490](https://github.com/matrix-org/synapse/issues/6490))
 - Fix link in the user directory documentation. ([\#6388](https://github.com/matrix-org/synapse/issues/6388))
 - Add build instructions to the docker readme. ([\#6390](https://github.com/matrix-org/synapse/issues/6390))
 - Switch Ubuntu package install recommendation to use python3 packages in INSTALL.md. ([\#6443](https://github.com/matrix-org/synapse/issues/6443))
 - Write some docs for the quarantine_media api. ([\#6458](https://github.com/matrix-org/synapse/issues/6458))
 - Convert CONTRIBUTING.rst to markdown (among other small fixes). ([\#6461](https://github.com/matrix-org/synapse/issues/6461))
 
 Deprecations and Removals
 -------------------------
 
 - Remove admin/v1/users_paginate endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5925](https://github.com/matrix-org/synapse/issues/5925))
 - Remove fallback for federation with old servers which lack the /federation/v1/state_ids API. ([\#6488](https://github.com/matrix-org/synapse/issues/6488))
 
 Internal Changes
 ----------------
 
 - Add benchmarks for structured logging and improve output performance. ([\#6266](https://github.com/matrix-org/synapse/issues/6266))
 - Improve the performance of outputting structured logging. ([\#6322](https://github.com/matrix-org/synapse/issues/6322))
 - Refactor some code in the event authentication path for clarity. ([\#6343](https://github.com/matrix-org/synapse/issues/6343), [\#6468](https://github.com/matrix-org/synapse/issues/6468), [\#6480](https://github.com/matrix-org/synapse/issues/6480))
 - Clean up some unnecessary quotation marks around the codebase. ([\#6362](https://github.com/matrix-org/synapse/issues/6362))
 - Complain on startup instead of 500'ing during runtime when `public_baseurl` isn't set when necessary. ([\#6379](https://github.com/matrix-org/synapse/issues/6379))
 - Add a test scenario to make sure room history purges don't break `/messages` in the future. ([\#6392](https://github.com/matrix-org/synapse/issues/6392))
 - Clarifications for the email configuration settings. ([\#6423](https://github.com/matrix-org/synapse/issues/6423))
 - Add more tests to the blacklist when running in worker mode. ([\#6429](https://github.com/matrix-org/synapse/issues/6429))
 - Refactor data store layer to support multiple databases in the future. ([\#6454](https://github.com/matrix-org/synapse/issues/6454), [\#6464](https://github.com/matrix-org/synapse/issues/6464), [\#6469](https://github.com/matrix-org/synapse/issues/6469), [\#6487](https://github.com/matrix-org/synapse/issues/6487))
 - Port synapse.rest.client.v1 to async/await. ([\#6482](https://github.com/matrix-org/synapse/issues/6482))
 - Port synapse.rest.client.v2_alpha to async/await. ([\#6483](https://github.com/matrix-org/synapse/issues/6483))
 - Port SyncHandler to async/await. ([\#6484](https://github.com/matrix-org/synapse/issues/6484))
 -----BEGIN PGP SIGNATURE-----
 
 iQJEBAABCgAuFiEEumuwyPtYLL2OMhYdOtoG7cdT0R4FAl3zbaUQHGVyaWtAbWF0
 cml4Lm9yZwAKCRA62gbtx1PRHqxED/9Ldjx5pxRm99Rkfhpv6qaw79GaQsAERW+i
 rPzdC0u59PRDLyfB2MRGU+q4oTTmBN3APPuW79mhY4CRisKfUvw6tuaJM5avvRsF
 vlY6m4ObewPXvK9dumlKD4ORqks45Sv0VGLHoZH4MLvWu/3Z+ycUn0qf+KNy0JrV
 TNNnNfsEeo3FXrkLU/12TiEBqNTVCU2vi/nZaHoGlHtZGVoykyFk4z2i9hLKENH4
 ScwaN4zeFRTIWYMiEb+kI/6QQLTASvar5FstwXPpM2F+aZhrMc5M7YsEe16jx+bQ
 jOM1L67hMetcWWmE9QrgP030clIDg30xhNbVxFSD4WoGgEPFkf/zE91v0CF5ZMRJ
 u83hlhe8tHoO8DOfVl2DNw3x6Evx4Vb5P8bvszNwKAyVDanvkSe0GvYm84B0Tat6
 2UCyPR5mbU3fyiwpouaNYun8MppyMRQ/fCnNElvZwwoj9EQQcJ8QOnMK0pFv1AXV
 n4wwXqRb+WvvbhnKPxTsiTK0DPBbIiGqo1TIONSbhPF+wySR3YV80HCWofd+qlRT
 V+GbZihk4SCcmamgqYLeQYneOrIu6UWGP348t3scCz/RCUp7hmxaJhJ3GAC3hOmU
 alN6tDJn7Lo0CwY2frX8xPS6S31sq9IZyDhHkaL0chsEa1rgWJMOK3FRp/b+c8yU
 b3CEJObIjw==
 =RQVG
 -----END PGP SIGNATURE-----

Merge tag 'v1.7.0'

Synapse 1.7.0 (2019-12-13)
==========================

This release changes the default settings so that only local authenticated users can query the server's room directory. See the [upgrade notes](UPGRADE.rst#upgrading-to-v170) for details.

Support for SQLite versions before 3.11 is now deprecated. A future release will refuse to start if used with an SQLite version before 3.11.

Administrators are reminded that SQLite should not be used for production instances. Instructions for migrating to Postgres are available [here](docs/postgres.md). A future release of synapse will, by default, disable federation for servers using SQLite.

No significant changes since 1.7.0rc2.

Synapse 1.7.0rc2 (2019-12-11)
=============================

Bugfixes
--------

- Fix incorrect error message for invalid requests when setting user's avatar URL. ([\#6497](https://github.com/matrix-org/synapse/issues/6497))
- Fix support for SQLite 3.7. ([\#6499](https://github.com/matrix-org/synapse/issues/6499))
- Fix regression where sending email push would not work when using a pusher worker. ([\#6507](https://github.com/matrix-org/synapse/issues/6507), [\#6509](https://github.com/matrix-org/synapse/issues/6509))

Synapse 1.7.0rc1 (2019-12-09)
=============================

Features
--------

- Implement per-room message retention policies. ([\#5815](https://github.com/matrix-org/synapse/issues/5815), [\#6436](https://github.com/matrix-org/synapse/issues/6436))
- Add etag and count fields to key backup endpoints to help clients guess if there are new keys. ([\#5858](https://github.com/matrix-org/synapse/issues/5858))
- Add `/admin/v2/users` endpoint with pagination. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5925](https://github.com/matrix-org/synapse/issues/5925))
- Require User-Interactive Authentication for `/account/3pid/add`, meaning the user's password will be required to add a third-party ID to their account. ([\#6119](https://github.com/matrix-org/synapse/issues/6119))
- Implement the `/_matrix/federation/unstable/net.atleastfornow/state/<context>` API as drafted in MSC2314. ([\#6176](https://github.com/matrix-org/synapse/issues/6176))
- Configure privacy-preserving settings by default for the room directory. ([\#6355](https://github.com/matrix-org/synapse/issues/6355))
- Add ephemeral messages support by partially implementing [MSC2228](https://github.com/matrix-org/matrix-doc/pull/2228). ([\#6409](https://github.com/matrix-org/synapse/issues/6409))
- Add support for [MSC 2367](https://github.com/matrix-org/matrix-doc/pull/2367), which allows specifying a reason on all membership events. ([\#6434](https://github.com/matrix-org/synapse/issues/6434))

Bugfixes
--------

- Transfer non-standard power levels on room upgrade. ([\#6237](https://github.com/matrix-org/synapse/issues/6237))
- Fix error from the Pillow library when uploading RGBA images. ([\#6241](https://github.com/matrix-org/synapse/issues/6241))
- Correctly apply the event filter to the `state`, `events_before` and `events_after` fields in the response to `/context` requests. ([\#6329](https://github.com/matrix-org/synapse/issues/6329))
- Fix caching devices for remote users when using workers, so that we don't attempt to refetch (and potentially fail) each time a user requests devices. ([\#6332](https://github.com/matrix-org/synapse/issues/6332))
- Prevent account data syncs getting lost across TCP replication. ([\#6333](https://github.com/matrix-org/synapse/issues/6333))
- Fix bug: TypeError in `register_user()` while using LDAP auth module. ([\#6406](https://github.com/matrix-org/synapse/issues/6406))
- Fix an intermittent exception when handling read-receipts. ([\#6408](https://github.com/matrix-org/synapse/issues/6408))
- Fix broken guest registration when there are existing blocks of numeric user IDs. ([\#6420](https://github.com/matrix-org/synapse/issues/6420))
- Fix startup error when http proxy is defined. ([\#6421](https://github.com/matrix-org/synapse/issues/6421))
- Fix error when using synapse_port_db on a vanilla synapse db. ([\#6449](https://github.com/matrix-org/synapse/issues/6449))
- Fix uploading multiple cross signing signatures for the same user. ([\#6451](https://github.com/matrix-org/synapse/issues/6451))
- Fix bug which lead to exceptions being thrown in a loop when a cross-signed device is deleted. ([\#6462](https://github.com/matrix-org/synapse/issues/6462))
- Fix `synapse_port_db` not exiting with a 0 code if something went wrong during the port process. ([\#6470](https://github.com/matrix-org/synapse/issues/6470))
- Improve sanity-checking when receiving events over federation. ([\#6472](https://github.com/matrix-org/synapse/issues/6472))
- Fix inaccurate per-block Prometheus metrics. ([\#6491](https://github.com/matrix-org/synapse/issues/6491))
- Fix small performance regression for sending invites. ([\#6493](https://github.com/matrix-org/synapse/issues/6493))
- Back out cross-signing code added in Synapse 1.5.0, which caused a performance regression. ([\#6494](https://github.com/matrix-org/synapse/issues/6494))

Improved Documentation
----------------------

- Update documentation and variables in user contributed systemd reference file. ([\#6369](https://github.com/matrix-org/synapse/issues/6369), [\#6490](https://github.com/matrix-org/synapse/issues/6490))
- Fix link in the user directory documentation. ([\#6388](https://github.com/matrix-org/synapse/issues/6388))
- Add build instructions to the docker readme. ([\#6390](https://github.com/matrix-org/synapse/issues/6390))
- Switch Ubuntu package install recommendation to use python3 packages in INSTALL.md. ([\#6443](https://github.com/matrix-org/synapse/issues/6443))
- Write some docs for the quarantine_media api. ([\#6458](https://github.com/matrix-org/synapse/issues/6458))
- Convert CONTRIBUTING.rst to markdown (among other small fixes). ([\#6461](https://github.com/matrix-org/synapse/issues/6461))

Deprecations and Removals
-------------------------

- Remove admin/v1/users_paginate endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5925](https://github.com/matrix-org/synapse/issues/5925))
- Remove fallback for federation with old servers which lack the /federation/v1/state_ids API. ([\#6488](https://github.com/matrix-org/synapse/issues/6488))

Internal Changes
----------------

- Add benchmarks for structured logging and improve output performance. ([\#6266](https://github.com/matrix-org/synapse/issues/6266))
- Improve the performance of outputting structured logging. ([\#6322](https://github.com/matrix-org/synapse/issues/6322))
- Refactor some code in the event authentication path for clarity. ([\#6343](https://github.com/matrix-org/synapse/issues/6343), [\#6468](https://github.com/matrix-org/synapse/issues/6468), [\#6480](https://github.com/matrix-org/synapse/issues/6480))
- Clean up some unnecessary quotation marks around the codebase. ([\#6362](https://github.com/matrix-org/synapse/issues/6362))
- Complain on startup instead of 500'ing during runtime when `public_baseurl` isn't set when necessary. ([\#6379](https://github.com/matrix-org/synapse/issues/6379))
- Add a test scenario to make sure room history purges don't break `/messages` in the future. ([\#6392](https://github.com/matrix-org/synapse/issues/6392))
- Clarifications for the email configuration settings. ([\#6423](https://github.com/matrix-org/synapse/issues/6423))
- Add more tests to the blacklist when running in worker mode. ([\#6429](https://github.com/matrix-org/synapse/issues/6429))
- Refactor data store layer to support multiple databases in the future. ([\#6454](https://github.com/matrix-org/synapse/issues/6454), [\#6464](https://github.com/matrix-org/synapse/issues/6464), [\#6469](https://github.com/matrix-org/synapse/issues/6469), [\#6487](https://github.com/matrix-org/synapse/issues/6487))
- Port synapse.rest.client.v1 to async/await. ([\#6482](https://github.com/matrix-org/synapse/issues/6482))
- Port synapse.rest.client.v2_alpha to async/await. ([\#6483](https://github.com/matrix-org/synapse/issues/6483))
- Port SyncHandler to async/await. ([\#6484](https://github.com/matrix-org/synapse/issues/6484))
2019-12-13 10:55:33 +00:00
Mark Nowiasz 58fdcbdfe7 Update workers.md to make media_repository work (again) (#6519) 2019-12-11 16:23:38 +00:00
Andrew Morgan 4947de5a14
Allow SAML username provider plugins (#6411) 2019-12-10 17:30:16 +00:00
Manuel Stahl 649b6bc088 Replace /admin/v1/users_paginate endpoint with /admin/v2/users (#5925) 2019-12-05 18:12:23 +00:00
Neil Johnson cb0aeb147e
privacy by default for room dir (#6355)
Ensure that the the default settings for the room directory are that the it is hidden from public view by default.
2019-12-04 09:46:16 +00:00
Richard van der Hoff 620f98b65b
write some docs for the quarantine_media api (#6458) 2019-12-03 18:20:39 +00:00
Richard van der Hoff c48ea98007
Clarifications for the email configuration settings. (#6423)
Cf #6422
2019-11-28 09:29:18 +00:00
Brendan Abolivier 9e937c28ee Merge branch 'develop' into babolivier/message_retention 2019-11-26 17:53:57 +00:00
Aaron Raimist 24cc31ee96 Fix link to user_dir_populate.sql in the user directory docs (#6388) 2019-11-21 17:38:14 +00:00
Andrew Morgan 473acedcdd Merge branch 'develop' of github.com:matrix-org/synapse into anoa/homeserver_copy
* 'develop' of github.com:matrix-org/synapse:
  Blacklist PurgeRoomTestCase (#6361)
  Set room version default to 5
2019-11-14 10:26:27 +00:00
Brendan Abolivier a42567e4a8
Merge pull request #6220 from matrix-org/neilj/set_room_version_default_to_5
Set room version default to 5
2019-11-14 10:21:00 +00:00
Andrew Morgan e1648dc576 sample config 2019-11-12 13:15:59 +00:00
Brendan Abolivier 09957ce0e4
Implement per-room message retention policies 2019-11-04 17:09:22 +00:00
Richard van der Hoff cc6243b4c0
document the REPLICATE command a bit better (#6305)
since I found myself wonder how it works
2019-11-04 12:40:18 +00:00
Yash Jipkate 9677613e9c Modify doc to update Google ReCaptcha terms (#6257) 2019-10-30 12:30:20 +00:00
Andrew Morgan 46c12918ad
Fix typo in domain name in account_threepid_delegates config option (#6273) 2019-10-30 11:07:42 +00:00
Neil Johnson 2794b79052 Option to suppress resource exceeded alerting (#6173)
The expected use case is to suppress MAU limiting on small instances
2019-10-24 11:48:46 +01:00
Andrew Morgan 409c62b27b
Add config linting script that checks for bool casing (#6203)
Add a linting script that enforces all boolean values in the default config be lowercase.

This has annoyed me for a while so I decided to fix it.
2019-10-23 13:22:54 +01:00
Adrien Luxey 0327a00a37 Update postgres.md (#6234)
Added database owner authentication with `sudo` when `su` does not work
2019-10-22 13:48:02 +02:00
Neil Johnson 82c8799ec7 Set room version default to 5 2019-10-19 09:06:15 +01:00
Valérian Rousset be9b55e0d2 cas: support setting display name (#6114)
Now, the CAS server can return an attribute stating what's the desired displayname, instead of using the username directly.
2019-10-11 12:33:12 +01:00
Brendan Abolivier baf12bc02a Merge branch 'master' into develop 2019-10-02 14:41:02 +01:00
Erik Johnston 3423633d50 Fix 'redaction_retention_period' sampel config to match guidelines 2019-09-26 16:43:52 +01:00
Neil Johnson 8b8f8c7b3c Explicitly log when a homeserver does not have a trusted key server configured (#6090) 2019-09-26 12:57:01 +01:00
Erik Johnston 4fb3c129aa Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips_2 2019-09-25 17:53:13 +01:00
Erik Johnston 39b50ad42a Review comments 2019-09-25 17:22:33 +01:00
Erik Johnston 242017db8b Prune rows in user_ips older than configured period
Defaults to pruning everything older than 28d.
2019-09-24 15:53:17 +01:00
Richard van der Hoff ed8b92f0d2 Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work 2019-09-24 12:57:32 +01:00
Andrew Morgan 50776261e1 Add submit_url response parameter to msisdn /requestToken (#6079)
Second part of solving #6076
Fixes #6076

We return a submit_url parameter on calls to POST */msisdn/requestToken so that clients know where to submit token information to.
2019-09-23 21:21:03 +01:00
Richard van der Hoff 78e8ec368e
Merge pull request #6064 from matrix-org/rav/saml_config_cleanup
Make the sample saml config closer to our standards
2019-09-23 20:36:51 +01:00
Andrew Morgan e08ea43463 Use the federation blacklist for requests to untrusted Identity Servers (#6000)
Uses a SimpleHttpClient instance equipped with the federation_ip_range_blacklist list for requests to identity servers provided by user input. Does not use a blacklist when contacting identity servers specified by account_threepid_delegates. The homeserver trusts the latter and we don't want to prevent homeserver admins from specifying delegates that are on internal IP addresses.

Fixes #5935
2019-09-23 20:23:20 +01:00
Andrew Morgan df3401a71d
Allow HS to send emails when adding an email to the HS (#6042) 2019-09-20 15:21:30 +01:00
Richard van der Hoff b65327ff66 Merge branch 'develop' into rav/saml_mapping_work 2019-09-19 18:13:31 +01:00
Richard van der Hoff b789c7eb03 Merge branch 'develop' into rav/saml_config_cleanup 2019-09-19 15:05:31 +01:00
Jorik Schellekens 38fd1f8e3f Fix typo in account_threepid_delegates config (#6028) 2019-09-18 22:30:44 +01:00
Richard van der Hoff 7100b5cc9d fix sample config
this was apparently broken by #6040.
2019-09-18 10:16:00 +01:00
dstipp 379d2a8c39 (#5849) Convert rst to markdown (#6040)
Converting some of the rst documentation to markdown.  Attempted to
preserve whitespace and line breaks to minimize cosmetic change.
2019-09-17 12:55:29 +01:00
Richard van der Hoff a8ac40445c Record mappings from saml users in an external table
We want to assign unique mxids to saml users based on an incrementing
suffix. For that to work, we need to record the allocated mxid in a separate
table.
2019-09-13 16:01:46 +01:00
Richard van der Hoff 785cbd3999 Make the sample saml config closer to our standards
It' still not great, thanks to the nested dictionaries, but it's better.
2019-09-13 12:07:03 +01:00
Travis Ralston c755955f33 Add developer docs for using SAML without a server (#6032) 2019-09-13 08:58:18 +01:00
axel simon 1c7df13e7b add explanations on how to actually include an access_token (#6031) 2019-09-13 08:50:17 +01:00
Sorunome dd2e5b0038 add report_stats_endpoint config option (#6012)
This PR adds the optional `report_stats_endpoint` to configure where stats are reported to, if enabled.
2019-09-12 11:24:57 +01:00
Erik Johnston 66ace43546 Update sample config 2019-09-11 14:50:40 +01:00
Erik Johnston 8df88b5ff3 Update sample config 2019-09-11 10:58:26 +01:00
Erik Johnston 470dc621ae
Merge pull request #5934 from matrix-org/erikj/censor_redactions
Censor redactions in DB after a month
2019-09-09 15:29:39 +01:00
Erik Johnston 8b9ade8c78 Default to censoring redactions after seven days 2019-09-09 13:55:28 +01:00
Amber Brown 55d5b3af88
Servers-known-about statistic (#5981) 2019-09-07 01:45:51 +10:00
Andrew Morgan 0c0b82b6d1
Allow Synapse to send registration emails + choose Synapse or an external server to handle 3pid validation (#5987)
This is a combination of a few different PRs, finally all being merged into `develop`:

* #5875 
* #5876 
* #5868 (This one added the `/versions` flag but the flag itself was actually [backed out](891afb57cb (diff-e591d42d30690ffb79f63bb726200891)) in #5969. What's left is just giving /versions access to the config file, which could be useful in the future)
* #5835 
* #5969 
* #5940

Clients should not actually use the new registration functionality until https://github.com/matrix-org/synapse/pull/5972 is merged.

UPGRADE.rst, changelog entries and config file changes should all be reviewed closely before this PR is merged.
2019-09-06 11:35:28 +01:00
Erik Johnston 591d82f06b Merge branch 'develop' of github.com:matrix-org/synapse into erikj/censor_redactions 2019-09-05 17:27:46 +01:00
Erik Johnston 3ff0422d2d Make redaction retention period configurable 2019-09-05 17:16:45 +01:00
Erik Johnston 6e834e94fc
Fix and refactor room and user stats (#5971)
Previously the stats were not being correctly populated.
2019-09-04 13:04:27 +01:00
Jorik Schellekens 92c1550f4a Add a link to python's logging config schema (#5926) 2019-08-28 19:08:32 +01:00
Jorik Schellekens 6d97843793
Config templating (#5900)
Template config files

* Imagine a system composed entirely of x, y, z etc and the basic operations..

Wait George, why XOR? Why not just neq?

George: Eh, I didn't think of that..

Co-Authored-By: Erik Johnston <erik@matrix.org>
2019-08-28 13:12:22 +01:00
Amber Brown 7dc398586c
Implement a structured logging output system. (#5680) 2019-08-28 21:18:53 +10:00
reivilibre a3f0635686
Merge pull request #5914 from matrix-org/rei/admin_getadmin
Add GET method to admin API /users/@user:dom/admin
2019-08-28 09:44:22 +01:00
Olivier Wilkinson (reivilibre) 1b959b6977 Document GET method for retrieving admin bit of user in admin API
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-27 13:19:19 +01:00
Erik Johnston f5b50d0871
Merge pull request #5895 from matrix-org/erikj/notary_key
Add config option to sign remote key query responses with a separate key.
2019-08-27 11:51:37 +01:00
reivilibre 1a7e6eb633
Add Admin API capability to set adminship of a user (#5878)
Admin API: Set adminship of a user
2019-08-27 10:14:00 +01:00
Erik Johnston 7af5a63063 Fixup review comments 2019-08-23 15:36:28 +01:00
Jorik Schellekens 8767b63a82
Propagate opentracing contexts through EDUs (#5852)
Propagate opentracing contexts through EDUs
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-08-22 18:21:10 +01:00
Richard van der Hoff 119aa31b10
Servlet to purge old rooms (#5845) 2019-08-22 10:42:59 +01:00
Erik Johnston 5906be8589 Add config option for keys to use to sign keys
This allows servers to separate keys that are used to sign remote keys
when acting as a notary server.
2019-08-21 10:44:58 +01:00
Amber Brown 0b6fbb28a8
Don't load the media repo when configured to use an external media repo (#5754) 2019-08-13 21:49:28 +10:00
Brendan Abolivier f25f638c35
Lint 2019-08-01 12:19:08 +02:00
Brendan Abolivier 3ff3dfe5a3
Sample config 2019-08-01 12:08:25 +02:00
Richard van der Hoff 8c97f6414c
Remove non-functional 'expire_access_token' setting (#5782)
The `expire_access_token` didn't do what it sounded like it should do. What it
actually did was make Synapse enforce the 'time' caveat on macaroons used as
access tokens, but since our access token macaroons never contained such a
caveat, it was always a no-op.

(The code to add 'time' caveats was removed back in v0.18.5, in #1656)
2019-07-30 08:25:02 +01:00
Amber Brown 865077f1d1
Room Complexity Client Implementation (#5783) 2019-07-30 02:47:27 +10:00
Jorik Schellekens 3641784e8c
Make Jaeger fully configurable (#5694)
* Allow Jaeger to be configured

* Update sample config
2019-07-23 15:46:04 +01:00
Amber Brown 4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00
Jorik Schellekens 0fd171770a Merge branch 'release-v1.2.0' into develop 2019-07-22 11:18:50 +01:00
Jorik Schellekens 826e6ec3bd
Opentracing Documentation (#5703)
* Opentracing survival guide

* Update decorator names in doc

* Doc cleanup

These are all alterations as a result of comments in #5703, it
includes mostly typos and clarifications. The most interesting
changes are:

- Split developer and user docs into two sections
- Add a high level description of OpenTracing

* newsfile

* Move contributer specific info to docstring.

* Sample config.

* Trailing whitespace.

* Update 5703.misc

* Apply suggestions from code review

Mostly just rewording parts of the docs for clarity.

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-07-22 11:15:21 +01:00
Richard van der Hoff b73ce4ba81
Update the coding style doc (#5719)
A few fixes and removal of duplicated stuff, but mostly a bunch of the words on the config file.
2019-07-19 11:55:14 +01:00
Richard van der Hoff 82345bc09a
Clean up opentracing configuration options (#5712)
Clean up config settings and dead code.

This is mostly about cleaning up the config format, to bring it into line with our conventions. In particular:
 * There should be a blank line after `## Section ##' headings
 * There should be a blank line between each config setting
 * There should be a `#`-only line between a comment and the setting it describes
 * We don't really do the `#  #` style commenting-out of whole sections if we can help it
 * rename `tracer_enabled` to `enabled`

While we're here, do more config parsing upfront, which makes it easier to use
later on.

Also removes redundant code from LogContextScopeManager.

Also changes the changelog fragment to a `feature` - it's exciting!
2019-07-18 15:06:54 +01:00
Amber Brown 7ad1d76356
Support Prometheus_client 0.4.0+ (#5636) 2019-07-18 23:57:15 +10:00
Richard van der Hoff 1def298119
Improve `Depends` specs in debian package. (#5675)
This is basically a contrived way of adding a `Recommends` on `libpq5`, to fix #5653.

The way this is supposed to happen in debhelper is to run
`dh_shlibdeps`, which in turn runs `dpkg-shlibdeps`, which spits things out
into `debian/<package>.substvars` whence they can later be included by
`control`.

Previously, we had disabled `dh_shlibdeps`, mostly because `dpkg-shlibdeps`
gets confused about PIL's interdependent objects, but that's not really the
right thing to do and there is another way to work around that.

Since we don't always use postgres, we don't necessarily want a hard Depends on
libpq5, so I've actually ended up adding an explicit invocation of
`dpkg-shlibdeps` for `psycopg2`.

I've also updated the build-depends list for the package, which was missing a
couple of entries.
2019-07-17 17:47:07 +01:00
Richard van der Hoff 5f158ec039
Implement access token expiry (#5660)
Record how long an access token is valid for, and raise a soft-logout once it
expires.
2019-07-12 17:26:02 +01:00
Ulrik Günther d445b3ae57 Update reverse_proxy.rst (#5397)
Updates reverse_proxy.rst with information about nginx' URI normalisation.
2019-07-12 11:46:18 +01:00
Lrizika 39e9839a04 Improved docs on setting up Postgresql (#5661)
Added that synapse_user needs a database to access before it can auth
Noted you'll need to enable password auth, linked to pg_hba.conf docs
2019-07-11 14:31:36 +01:00
Jorik Schellekens 38a6d3eea7
Add basic opentracing support (#5544)
* Configure and initialise tracer

Includes config options for the tracer and sets up JaegerClient.

* Scope manager using LogContexts

We piggy-back our tracer scopes by using log context.
The current log context gives us the current scope. If new scope is
created we create a stack of scopes in the context.

* jaeger is a dependency now

* Carrier inject and extraction for Twisted Headers

* Trace federation requests on the way in and out.

The span is created in _started_processing and closed in
_finished_processing because we need a meaningful log context.

* Create logcontext for new scope.

Instead of having a stack of scopes in a logcontext we create a new
context for a new scope if the current logcontext already has a scope.

* Remove scope from logcontext if logcontext is top level

* Disable tracer if not configured

* typo

* Remove dependence on jaeger internals

* bools

* Set service name

* :Explicitely state that the tracer is disabled

* Black is the new black

* Newsfile

* Code style

* Use the new config setup.

* Generate config.

* Copyright

* Rename config to opentracing

* Remove user whitelisting

* Empty whitelist by default

* User ConfigError instead of RuntimeError

* Use isinstance

* Use tag constants for opentracing.

* Remove debug comment and no need to explicitely record error

* Two errors a "s(c)entry"

* Docstrings!

* Remove debugging brainslip

* Homeserver Whitlisting

* Better opentracing config comment

* linting

* Inclue worker name in service_name

* Make opentracing an optional dependency

* Neater config retreival

* Clean up dummy tags

* Instantiate tracing as object instead of global class

* Inlcude opentracing as a homeserver member.

* Thread opentracing to the request level

* Reference opetnracing through hs

* Instantiate dummy opentracin g for tests.

* About to revert, just keeping the unfinished changes just in case

* Revert back to global state, commit number:

9ce4a3d9067bf9889b86c360c05ac88618b85c4f

* Use class level methods in tracerutils

* Start and stop requests spans in a place where we
have access to the authenticated entity

* Seen it, isort it

* Make sure to close the active span.

* I'm getting black and blue from this.

* Logger formatting

Co-Authored-By: Erik Johnston <erik@matrix.org>

* Outdated comment

* Import opentracing at the top

* Return a contextmanager

* Start tracing client requests from the servlet

* Return noop context manager if not tracing

* Explicitely say that these are federation requests

* Include servlet name in client requests

* Use context manager

* Move opentracing to logging/

* Seen it, isort it again!

* Ignore twisted return exceptions on context exit

* Escape the scope

* Scopes should be entered to make them useful.

* Nicer decorator names

* Just one init, init?

* Don't need to close something that isn't open

* Docs make you smarter
2019-07-11 10:36:03 +01:00
Amber Brown 463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ (#5606) 2019-07-04 00:07:04 +10:00
Richard van der Hoff b4fd86a9b4 Merge branch 'develop' into rav/saml2_client 2019-07-01 14:21:03 +01:00
Amber Brown be3b901ccd
Update the TLS cipher string and provide configurability for TLS on outgoing federation (#5550) 2019-06-28 18:19:09 +10:00
Daniel Hoffend 9646a593ac Added possibilty to disable local password authentication (#5092)
Signed-off-by: Daniel Hoffend <dh@dotlan.net>
2019-06-27 18:37:29 +01:00
Andrew Morgan c548dbc4b1
Make it clearer that the template dir is relative to synapse's root dir (#5543)
Helps address #5444
2019-06-27 18:20:17 +01:00
Richard van der Hoff 47fa836abb
Merge pull request #5313 from twrist/patch-1
Update HAProxy example rules
2019-06-27 00:53:48 +01:00
Richard van der Hoff dde4118341 update sample config 2019-06-27 00:41:04 +01:00
Richard van der Hoff a0acfcc73e update sample config 2019-06-26 23:56:28 +01:00
Andrew Morgan 3eb8c7b0eb Merge branch 'master' into develop
* master:
  Fix broken link in MSC1711 FAQ
  Update changelog to better expain password reset change (#5545)
2019-06-25 18:08:56 +01:00
Richard van der Hoff c8cb186260
Fix broken link in MSC1711 FAQ 2019-06-25 12:27:56 +01:00
Andrew Morgan 28604ab03d
Add info about black to code_style.rst (#5537)
Fixes #5533

Adds information about how to install and run black on the codebase.
2019-06-24 17:48:05 +01:00
Richard van der Hoff 4ac7ef4b67
Merge pull request #5524 from matrix-org/rav/new_cmdline_options
Add --data-dir and --open-private-ports options.
2019-06-24 17:25:57 +01:00
Brendan Abolivier deb4fe6ef3
Merge pull request #5534 from matrix-org/babolivier/federation-publicrooms
Split public rooms directory auth config in two
2019-06-24 16:08:02 +01:00
Brendan Abolivier bfe84e051e Split public rooms directory auth config in two 2019-06-24 15:42:31 +01:00
Richard van der Hoff 3f8a252dd8 Add "--open-private-ports" cmdline option
This is helpful when generating a config file for running synapse under docker.
2019-06-24 14:15:34 +01:00
Richard van der Hoff edea4bb5be Allow configuration of the path used for ACME account keys.
Because sticking it in the same place as the config isn't necessarily the right
thing to do.
2019-06-24 13:51:22 +01:00
Richard van der Hoff 6cda36777b Drop support for cpu_affinity (#5525)
This has no useful purpose on python3, and is generally a source of confusion.
2019-06-22 11:01:55 +10:00
Erik Johnston 60b912cf0d
Update docs/workers.rst
E_TOO_MANY_NEGATIVES

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-06-21 11:54:03 +01:00
Erik Johnston f3ab533374 Support pagination API in client_reader worker 2019-06-21 10:43:12 +01:00
Amber Brown 32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Amber Brown eba7caf09f
Remove Postgres 9.4 support (#5448) 2019-06-18 00:59:00 +10:00
Brendan Abolivier f12e1f029c
Merge pull request #5440 from matrix-org/babolivier/third_party_event_rules
Allow server admins to define implementations of extra rules for allowing or denying incoming events
2019-06-14 19:37:59 +01:00
Brendan Abolivier f874b16b2e Add plugin APIs for implementations of custom event rules. 2019-06-14 18:16:03 +01:00
Neil Johnson 426218323b
Neilj/improve federation docs (#5419)
Add FAQ questions to federate.md. Add a health warning making it clear that the 1711 upgrade FAQ is now out of date.
2019-06-11 12:17:43 +01:00
Neil Johnson a11865016e
Set default room version to v4. (#5379)
Set default room version to v4.
2019-06-06 20:13:47 +01:00
Andrew Morgan 3719680ee4
Add ability to perform password reset via email without trusting the identity server (#5377)
Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option.

This PR is a culmination of 3 smaller PRs which have each been separately reviewed:

* #5308
* #5345
* #5368
2019-06-06 17:34:07 +01:00
Richard van der Hoff 9fbb20a531
Stop hardcoding trust of old matrix.org key (#5374)
There are a few changes going on here:

* We make checking the signature on a key server response optional: if no
  verify_keys are specified, we trust to TLS to validate the connection.

* We change the default config so that it does not require responses to be
  signed by the old key.

* We replace the old 'perspectives' config with 'trusted_key_servers', which
  is also formatted slightly differently.

* We emit a warning to the logs every time we trust a key server response
  signed by the old key.
2019-06-06 17:33:11 +01:00
Neil Johnson 833c406b9b
Neilj/1.0 upgrade notes (#5371)
1.0 upgrade/install notes
2019-06-06 17:23:02 +01:00
Richard van der Hoff 7603a706eb Merge branch 'rav/fix_custom_ca' into rav/enable_tls_verification 2019-06-05 16:32:35 +01:00
Richard van der Hoff b4f1cd31f4 Update sample config 2019-06-05 15:30:10 +01:00
Richard van der Hoff 95ab2eb4a1
Fix notes about well-known and acme (#5357)
fixes #4951
2019-06-05 15:12:33 +01:00
Neil Johnson 26713515de
Neilj/mau tracking config explainer (#5284)
Improve documentation of monthly active user blocking and mau_trial_days
2019-06-05 13:16:23 +01:00
Ike Johnson 145f57897d
Update HAProxy example rules
These new rules allow a user to instead route only matrix traffic, allowing them to run matrix on the domain without affecting their existing websites
2019-06-02 23:10:27 +08:00
Erik Johnston 58cce39f3a
Merge pull request #5276 from matrix-org/babolivier/account_validity_job_delta
Allow configuring a range for the account validity startup job
2019-05-31 12:11:56 +01:00
Brendan Abolivier e975b15101 Sample config 2019-05-31 11:14:21 +01:00
Brendan Abolivier 6bfc5ad3a1 Sample config 2019-05-31 09:56:57 +01:00
Travis Ralston 3e1af5109c Clarify that the admin change password endpoint logs them out (#5303) 2019-05-31 09:45:46 +01:00
Erik Johnston 8541db741a
Merge pull request #5283 from aaronraimist/captcha-docs
Specify the type of reCAPTCHA key to use (#5013)
2019-05-29 19:02:27 +01:00
Amber Brown 0729ef01f8 regenerate sample config 2019-05-29 16:41:25 +10:00
Aaron Raimist f795595e95
Specify the type of reCAPTCHA key to use (#5013)
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2019-05-28 22:04:27 -05:00
Aaron Raimist 9b6f72663e
Fix docs on resetting the user directory (#5036)
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2019-05-28 20:54:01 -05:00
Brendan Abolivier 4aba561c65
Config and changelog 2019-05-28 16:55:10 +01:00
Richard van der Hoff dba9152d15
Add missing blank line in config (#5249) 2019-05-24 14:12:38 +01:00
Andrew Morgan 6368150a74
Add config option for setting homeserver's default room version (#5223)
Replaces DEFAULT_ROOM_VERSION constant with a method that first checks the config, then returns a hardcoded value if the option is not present.

That hardcoded value is now located in the server.py config file.
2019-05-23 15:00:20 +01:00
Amber Brown 4a30e4acb4
Room Statistics (#4338) 2019-05-21 11:36:50 -05:00
Brendan Abolivier 6a5a70edf0
Merge pull request #5204 from matrix-org/babolivier/account_validity_expiration_date
Add startup background job for account validity
2019-05-21 14:55:15 +01:00
Brendan Abolivier 384122efa8
Doc 2019-05-21 14:39:36 +01:00
Richard van der Hoff da5ef0bb42 Merge remote-tracking branch 'origin/master' into develop 2019-05-17 12:39:48 +01:00
Richard van der Hoff 7ce1f97a13
Stop telling people to install the optional dependencies. (#5197)
* Stop telling people to install the optional dependencies.

They're optional.

Also update the postgres docs a bit for clarity(?)
2019-05-17 12:38:03 +01:00
Brendan Abolivier a5fe16c5a7
Changelog + sample config 2019-05-16 15:11:37 +01:00
Amber Brown f1e5b41388
Make all the rate limiting options more consistent (#5181) 2019-05-15 12:06:04 -05:00
Andrew Morgan 5a4b328f52 Add ability to blacklist ip ranges for federation traffic (#5043) 2019-05-13 19:05:06 +01:00
Gergely Polonkai cd3f30014a Make Prometheus snippet less confusing on the metrics collection doc (#4288)
Signed-off-by: Gergely Polonkai <gergely@polonkai.eu>
2019-05-10 09:15:08 +01:00
colonelkrud d9a02d1201 Add AllowEncodedSlashes to apache (#5068)
* Add AllowEncodedSlashes to apache

Add `AllowEncodedSlashes On` to apache config to support encoding for v3 rooms. "The AllowEncodedSlashes setting is not inherited by virtual hosts, and virtual hosts are used in many default Apache configurations, such as the one in Ubuntu. The workaround is to add the AllowEncodedSlashes setting inside a <VirtualHost> container (/etc/apache2/sites-available/default in Ubuntu)." Source: https://stackoverflow.com/questions/4390436/need-to-allow-encoded-slashes-on-apache

* change allowencodedslashes to nodecode
2019-05-09 23:27:04 +01:00
Matthew Hodgson c0e0740bef add options to require an access_token to GET /profile and /publicRooms on CS API (#5083)
This commit adds two config options:

* `restrict_public_rooms_to_local_users`

Requires auth to fetch the public rooms directory through the CS API and disables fetching it through the federation API.

* `require_auth_for_profile_requests`

When set to `true`, requires that requests to `/profile` over the CS API are authenticated, and only returns the user's profile if the requester shares a room with the profile's owner, as per MSC1301.

MSC1301 also specifies a behaviour for federation (only returning the profile if the server asking for it shares a room with the profile's owner), but that's currently really non-trivial to do in a not too expensive way. Next step is writing down a MSC that allows a HS to specify which user sent the profile query. In this implementation, Synapse won't send a profile query over federation if it doesn't believe it already shares a room with the profile's owner, though.

Groups have been intentionally omitted from this commit.
2019-05-08 18:26:56 +01:00
Richard van der Hoff 59e2d2694d
Remove the requirement to authenticate for /admin/server_version. (#5122)
This endpoint isn't much use for its intended purpose if you first need to get
yourself an admin's auth token.

I've restricted it to the `/_synapse/admin` path to make it a bit easier to
lock down for those concerned about exposing this information. I don't imagine
anyone is using it in anger currently.
2019-05-07 09:29:30 +01:00
Travis Ralston 3fdff14207 Fix spelling in server notices admin API docs (#5142) 2019-05-06 22:15:02 +01:00
Richard van der Hoff 4804206dbe Fix sample config
... after it got broken in 1565ebec2c.
2019-05-06 22:13:35 +01:00
Richard van der Hoff 836d3adcce Merge branch 'master' into develop 2019-05-03 19:25:01 +01:00
Richard van der Hoff 1565ebec2c more config comment updates 2019-05-03 15:50:59 +01:00
Richard van der Hoff 1a7104fde3 Blacklist 0.0.0.0 and :: by default for URL previews 2019-05-03 15:35:49 +01:00
Richard van der Hoff 12f9d51e82
Add admin api for sending server_notices (#5121) 2019-05-02 11:59:16 +01:00
Brendan Abolivier c193b39134
Merge pull request #5124 from matrix-org/babolivier/aliases
Add some limitations to alias creation
2019-05-02 11:22:40 +01:00
Brendan Abolivier 84196cb231 Add some limitations to alias creation 2019-05-02 11:05:11 +01:00
Richard van der Hoff cc4bd762df Fix sample config 2019-05-01 16:48:23 +01:00
Richard van der Hoff 8e9ca83537 Move admin API to a new prefix 2019-05-01 15:44:30 +01:00
Brendan Abolivier c1799b0f85
Merge pull request #5116 from matrix-org/babolivier/account_expiration
Fix path in account validity admin route's doc
2019-05-01 11:59:56 +01:00
Brendan Abolivier 031919dafb Fix whole path for admin route 2019-05-01 11:38:27 +01:00
Brendan Abolivier d8e357b7cf Fix typo in account validity admin route 2019-05-01 11:34:22 +01:00
Andrew Morgan 6824ddd93d Config option for verifying federation certificates (MSC 1711) (#4967) 2019-04-25 14:22:49 +01:00
Erik Johnston ca90336a69 Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/account_expiration 2019-04-17 19:44:40 +01:00
Brendan Abolivier eaf41a943b Add management endpoints for account validity 2019-04-17 19:34:45 +01:00
Brendan Abolivier 91934025b9
Merge pull request #5047 from matrix-org/babolivier/account_expiration
Send out emails with links to extend an account's validity period
2019-04-17 14:57:39 +01:00
Brendan Abolivier 20f0617e87 Send out emails with links to extend an account's validity period 2019-04-17 14:42:20 +01:00
Erik Johnston 6e27a8620f
Merge pull request #5063 from matrix-org/erikj/move_endpoints
Move some rest endpoints to client reader
2019-04-15 18:55:01 +01:00
Erik Johnston ec638a1602 Only handle GET requests for /push_rules 2019-04-15 18:51:48 +01:00
Erik Johnston d5adf297e6 Move some rest endpoints to client reader 2019-04-15 17:21:03 +01:00
Brendan Abolivier bfc8fdf1fc
Merge pull request #5027 from matrix-org/babolivier/account_expiration
Add time-based account expiration
2019-04-09 17:02:41 +01:00
Brendan Abolivier 747aa9f8ca Add account expiration feature 2019-04-09 16:46:04 +01:00
Neil Johnson b25e387c0d
add context to phonehome stats (#5020)
add context to phonehome stats
2019-04-08 15:47:39 +01:00
Brendan Abolivier 8e85493b0c
Add config option to block users from looking up 3PIDs (#5010) 2019-04-04 17:25:47 +01:00
Erik Johnston c192bf8970 Add admin API for group deletion 2019-04-03 16:29:52 +01:00
Andrew Morgan bbd244c7b2
Support 3PID login in password providers (#4931)
Adds a new method, check_3pid_auth, which gives password providers
the chance to allow authentication with third-party identifiers such
as email or msisdn.
2019-03-26 17:48:30 +00:00
Richard van der Hoff 7105057cf2
Fix nginx example in ACME doc. (#4923) 2019-03-25 09:59:36 +00:00
Colin W ab4e4c6c2f Update Apache Setup To Remove Location Syntax (#4870)
This one should close #4841. Many thanks to @dev4223 for bringing it up and finding a solution.

Signed-off-by: Colin White
2019-03-21 14:05:56 +00:00
Erik Johnston 09f991a63d
Merge pull request #4896 from matrix-org/erikj/disable_room_directory
Add option to disable search room lists
2019-03-21 10:16:54 +00:00
Erik Johnston 263f2c9ce1
Merge pull request #4895 from matrix-org/erikj/disable_user_search
Add option to disable searching in the user dir
2019-03-20 16:47:15 +00:00
Richard van der Hoff a902d13180
Batch up outgoing read-receipts to reduce federation traffic. (#4890)
Rate-limit outgoing read-receipts as per #4730.
2019-03-20 16:02:25 +00:00
Erik Johnston cd8c5b91ad Fix up sample config 2019-03-20 14:35:41 +00:00
Erik Johnston 926f29ea6d Fix up config comments 2019-03-20 14:24:53 +00:00
Erik Johnston 213c98c00a Add option to disable search room lists
This disables both local and remote room list searching.
2019-03-19 17:10:52 +00:00
Erik Johnston 855bf4658d Update sample config 2019-03-19 16:47:04 +00:00
Erik Johnston b616a8717b Add note on tuning postgres 2019-03-19 16:05:32 +00:00
Richard van der Hoff d2a537ea60 Merge remote-tracking branch 'origin/master' into develop 2019-03-19 10:37:50 +00:00
Michael Kaye 9482a84c0a Repoint docs for federation (#4881) 2019-03-19 10:37:18 +00:00
Richard van der Hoff fd463b4f5d
Comment out most options in the generated config. (#4863)
Make it so that most options in the config are optional, and commented out in
the generated config.

The reasons this is a good thing are as follows:

* If we decide that we should change the default for an option, we can do so,
  and only those admins that have deliberately chosen to override that option
  will be stuck on the old setting.

* It moves us towards a point where we can get rid of the super-surprising
  feature of synapse where the default settings for the config come from the
  generated yaml.

* It makes setting up a test config for unit testing an order of magnitude
  easier (see forthcoming PR).

* It makes the generated config more consistent, and hopefully easier for users
  to understand.
2019-03-19 10:06:40 +00:00
Brendan Abolivier 651ad8bc96 Add ratelimiting on failed login attempts (#4865) 2019-03-18 12:57:20 +00:00
Brendan Abolivier 899e523d6d
Add ratelimiting on login (#4821)
Add two ratelimiters on login (per-IP address and per-userID).
2019-03-15 17:46:16 +00:00
Richard van der Hoff 9ffadcdbad
fix some typos in federate.md 2019-03-15 09:43:24 +00:00
Andrew Morgan 7998ca3a66
Document using a certificate with a full chain (#4849) 2019-03-13 15:26:29 +00:00
Neil Johnson 332b60ec68 Merge branch 'master' of github.com:matrix-org/synapse into develop 2019-03-12 17:15:21 +00:00
Neil Johnson 83193a9362
fix orphaned sentence 2019-03-12 16:57:17 +00:00
Neil Johnson 8b692bf7c2
Neilj/improved delegation doc 2 (#4832)
Improved federation configuration docs.  Specifically detailing  .well-known and SRV based delegation methods. 

Inspiration Valentin Lab <valentin.lab@kalysto.org> for https://github.com/matrix-org/synapse/pull/4781
2019-03-12 14:23:28 +00:00
Aaron Raimist 8ea1b41a0e Clarify what registration_shared_secret allows for (#2885) (#4844)
* Clarify what registration_shared_secret allows for (#2885)

Signed-off-by: Aaron Raimist <aaron@raim.ist>

* Add changelog

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2019-03-11 18:21:52 +00:00
Matthew Hodgson 8f4b9f5210 Reword the sample config header to be less scary (#4801) 2019-03-07 07:09:01 +00:00
Brendan Abolivier c23e8c3333 Update sample config 2019-03-05 18:03:48 +00:00
Erik Johnston 16c8b4ecbd
Merge pull request #4772 from jbweston/jbweston/server-version-api
Add 'server_version' endpoint to admin API
2019-03-05 16:31:00 +00:00
Brendan Abolivier a4c3a361b7
Add rate-limiting on registration (#4735)
* Rate-limiting for registration

* Add unit test for registration rate limiting

* Add config parameters for rate limiting on auth endpoints

* Doc

* Fix doc of rate limiting function

Co-Authored-By: babolivier <contact@brendanabolivier.com>

* Incorporate review

* Fix config parsing

* Fix linting errors

* Set default config for auth rate limiting

* Fix tests

* Add changelog

* Advance reactor instead of mocked clock

* Move parameters to registration specific config and give them more sensible default values

* Remove unused config options

* Don't mock the rate limiter un MAU tests

* Rename _register_with_store into register_with_store

* Make CI happy

* Remove unused import

* Update sample config

* Fix ratelimiting test for py2

* Add non-guest test
2019-03-05 14:25:33 +00:00
Erik Johnston c3c542bb4a
Merge pull request #4796 from matrix-org/erikj/factor_out_e2e_keys
Allow /keys/{changes,query} API to run on worker
2019-03-05 09:06:25 +00:00
Erik Johnston bfa7d46a10 Allow /keys/{changes,query} API to run on worker 2019-03-04 18:30:01 +00:00
Richard van der Hoff 8e28bc5eee
Include a default configuration file in the 'docs' directory. (#4791) 2019-03-04 17:14:58 +00:00
Seebi aba5eeabd5 Fix v4v6 option in HAProxy example config (#4790)
The v4v6 option only has a usage one ipv6 socket: https://serverfault.com/q/747895

Signed-off-by: Flakebi <flakebi@t-online.de>
2019-03-04 13:19:41 +00:00
Joseph Weston 144cbfd650
add API documentation
Signed-off-by: Joseph Weston <joseph@weston.cloud>
2019-03-02 03:07:04 +01:00
Erik Johnston 76550c58d2
Merge pull request #4759 from matrix-org/erikj/3pid_client_reader
Move /account/3pid to client_reader
2019-02-27 16:11:21 +00:00
Erik Johnston 54f9ce11a7 Move /account/3pid to client_reader 2019-02-27 14:26:08 +00:00
Erik Johnston 4cff9376f7 Move server key queries to federation reader 2019-02-27 13:43:53 +00:00
Erik Johnston 7590e9fa28
Merge pull request #4749 from matrix-org/erikj/replication_connection_backoff
Fix tightloop over connecting to replication server
2019-02-27 11:00:59 +00:00
Paul Tötterman 4bc7483518 Fix apache reverse proxy example (#4742)
So that it actually works. See https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass

Signed-off-by: Paul Tötterman <paul.totterman@iki.fi>
2019-02-26 18:01:45 +00:00
Erik Johnston 313987187e Fix tightloop over connecting to replication server
If the client failed to process incoming commands during the initial set
up of the replication connection it would immediately disconnect and
reconnect, resulting in a tightloop.

This can happen, for example, when subscribing to a stream that has a
row that is too long in the backlog.

The fix here is to not consider the connection successfully set up until
the client has succesfully subscribed and caught up with the streams.
This ensures that the retry logic timers aren't reset until then,
meaning that if an error does happen during start up the client will
continue backing off before retrying again.
2019-02-26 15:05:41 +00:00
Richard van der Hoff fcd6f01dc7
Minor tweaks to acme docs (#4689) 2019-02-22 10:56:42 +00:00
Benoît S 16e0680498 Added HAProxy example (#4660)
* Added HAProxy example

Proposal of an example with HAProxy. Asked by #4541.

Signed-off-by: Benoît S. (“Benpro”) <gitlab@benpro.fr>

* Following suggestions of @richvdh
2019-02-21 17:44:10 +00:00
Erik Johnston c003450057
Merge pull request #4671 from matrix-org/erikj/state_cache_invalidation
Batch cache invalidation over replication
2019-02-19 13:14:30 +00:00
Erik Johnston 62175a20e5 Docs 2019-02-19 11:38:40 +00:00
Erik Johnston bc8fa1509d Documentation 2019-02-19 11:24:59 +00:00
Erik Johnston 128902d60a Update worker docs 2019-02-18 17:21:51 +00:00
Erik Johnston 8b9ae6d3a6 Update docs 2019-02-18 15:26:13 +00:00
Richard van der Hoff 00cf679bf2 Synapse 0.99.1 (2019-02-14)
===========================
 
 Features
 --------
 
 - Include m.room.encryption on invites by default ([\#3902](https://github.com/matrix-org/synapse/issues/3902))
 - Federation OpenID listener resource can now be activated even if federation is disabled ([\#4420](https://github.com/matrix-org/synapse/issues/4420))
 - Synapse's ACME support will now correctly reprovision a certificate that approaches its expiry while Synapse is running. ([\#4522](https://github.com/matrix-org/synapse/issues/4522))
 - Add ability to update backup versions ([\#4580](https://github.com/matrix-org/synapse/issues/4580))
 - Allow the "unavailable" presence status for /sync.
   This change makes Synapse compliant with r0.4.0 of the Client-Server specification. ([\#4592](https://github.com/matrix-org/synapse/issues/4592))
 - There is no longer any need to specify `no_tls`: it is inferred from the absence of TLS listeners ([\#4613](https://github.com/matrix-org/synapse/issues/4613), [\#4615](https://github.com/matrix-org/synapse/issues/4615), [\#4617](https://github.com/matrix-org/synapse/issues/4617), [\#4636](https://github.com/matrix-org/synapse/issues/4636))
 - The default configuration no longer requires TLS certificates. ([\#4614](https://github.com/matrix-org/synapse/issues/4614))
 
 Bugfixes
 --------
 
 - Copy over room federation ability on room upgrade. ([\#4530](https://github.com/matrix-org/synapse/issues/4530))
 - Fix noisy "twisted.internet.task.TaskStopped" errors in logs ([\#4546](https://github.com/matrix-org/synapse/issues/4546))
 - Synapse is now tolerant of the `tls_fingerprints` option being None or not specified. ([\#4589](https://github.com/matrix-org/synapse/issues/4589))
 - Fix 'no unique or exclusion constraint' error ([\#4591](https://github.com/matrix-org/synapse/issues/4591))
 - Transfer Server ACLs on room upgrade. ([\#4608](https://github.com/matrix-org/synapse/issues/4608))
 - Fix failure to start when not TLS certificate was given even if TLS was disabled. ([\#4618](https://github.com/matrix-org/synapse/issues/4618))
 - Fix self-signed cert notice from generate-config. ([\#4625](https://github.com/matrix-org/synapse/issues/4625))
 - Fix performance of `user_ips` table deduplication background update ([\#4626](https://github.com/matrix-org/synapse/issues/4626), [\#4627](https://github.com/matrix-org/synapse/issues/4627))
 
 Internal Changes
 ----------------
 
 - Change the user directory state query to use a filtered call to the db instead of a generic one. ([\#4462](https://github.com/matrix-org/synapse/issues/4462))
 - Reject federation transactions if they include more than 50 PDUs or 100 EDUs. ([\#4513](https://github.com/matrix-org/synapse/issues/4513))
 - Reduce duplication of ``synapse.app`` code. ([\#4567](https://github.com/matrix-org/synapse/issues/4567))
 - Fix docker upload job to push -py2 images. ([\#4576](https://github.com/matrix-org/synapse/issues/4576))
 - Add port configuration information to ACME instructions. ([\#4578](https://github.com/matrix-org/synapse/issues/4578))
 - Update MSC1711 FAQ to calrify .well-known usage ([\#4584](https://github.com/matrix-org/synapse/issues/4584))
 - Clean up default listener configuration ([\#4586](https://github.com/matrix-org/synapse/issues/4586))
 - Clarifications for reverse proxy docs ([\#4607](https://github.com/matrix-org/synapse/issues/4607))
 - Move ClientTLSOptionsFactory init out of `refresh_certificates` ([\#4611](https://github.com/matrix-org/synapse/issues/4611))
 - Fail cleanly if listener config lacks a 'port' ([\#4616](https://github.com/matrix-org/synapse/issues/4616))
 - Remove redundant entries from docker config ([\#4619](https://github.com/matrix-org/synapse/issues/4619))
 - README updates ([\#4621](https://github.com/matrix-org/synapse/issues/4621))
 -----BEGIN PGP SIGNATURE-----
 
 iQFHBAABCgAxFiEEQlNDQm4FMsm53u1sih+T1XW16NUFAlxlemgTHHJpY2hhcmRA
 bWF0cml4Lm9yZwAKCRCKH5PVdbXo1eKYCACR9TcOvMver/YyD2qP+dY6Lt24f8zG
 zYYzHGAHin+p204q8Pp6o0XLe4UuLDuhAyNVPZyj1wzwHYdubRvdah1uFwPdxmCY
 tGbJG5p37ykSEfEwcxdXEjYfPqflOwQL5aCeXyCCLWSdVVFKkWCXGgw8F6WPkgrI
 QwWKTfsM3wCnfa8ryKAXHxcmX2G1JncZ0ouUZTVNz5vokBsA19IaLvfJ5Rv3Kk59
 eXsBB/yE+9Dat4A439AGfVDQDKiGYvuhppJmUdYRMqxulzakd8diyZqBDAHZafqt
 QdjxnDx2e0OtSxI3RSevABnDnNyJ4NsUEtrny1Lh/MV72T9K3yEbHuwH
 =UCD1
 -----END PGP SIGNATURE-----

Merge tag 'v0.99.1'

Synapse 0.99.1 (2019-02-14)
===========================

Features
--------

- Include m.room.encryption on invites by default ([\#3902](https://github.com/matrix-org/synapse/issues/3902))
- Federation OpenID listener resource can now be activated even if federation is disabled ([\#4420](https://github.com/matrix-org/synapse/issues/4420))
- Synapse's ACME support will now correctly reprovision a certificate that approaches its expiry while Synapse is running. ([\#4522](https://github.com/matrix-org/synapse/issues/4522))
- Add ability to update backup versions ([\#4580](https://github.com/matrix-org/synapse/issues/4580))
- Allow the "unavailable" presence status for /sync.
  This change makes Synapse compliant with r0.4.0 of the Client-Server specification. ([\#4592](https://github.com/matrix-org/synapse/issues/4592))
- There is no longer any need to specify `no_tls`: it is inferred from the absence of TLS listeners ([\#4613](https://github.com/matrix-org/synapse/issues/4613), [\#4615](https://github.com/matrix-org/synapse/issues/4615), [\#4617](https://github.com/matrix-org/synapse/issues/4617), [\#4636](https://github.com/matrix-org/synapse/issues/4636))
- The default configuration no longer requires TLS certificates. ([\#4614](https://github.com/matrix-org/synapse/issues/4614))

Bugfixes
--------

- Copy over room federation ability on room upgrade. ([\#4530](https://github.com/matrix-org/synapse/issues/4530))
- Fix noisy "twisted.internet.task.TaskStopped" errors in logs ([\#4546](https://github.com/matrix-org/synapse/issues/4546))
- Synapse is now tolerant of the `tls_fingerprints` option being None or not specified. ([\#4589](https://github.com/matrix-org/synapse/issues/4589))
- Fix 'no unique or exclusion constraint' error ([\#4591](https://github.com/matrix-org/synapse/issues/4591))
- Transfer Server ACLs on room upgrade. ([\#4608](https://github.com/matrix-org/synapse/issues/4608))
- Fix failure to start when not TLS certificate was given even if TLS was disabled. ([\#4618](https://github.com/matrix-org/synapse/issues/4618))
- Fix self-signed cert notice from generate-config. ([\#4625](https://github.com/matrix-org/synapse/issues/4625))
- Fix performance of `user_ips` table deduplication background update ([\#4626](https://github.com/matrix-org/synapse/issues/4626), [\#4627](https://github.com/matrix-org/synapse/issues/4627))

Internal Changes
----------------

- Change the user directory state query to use a filtered call to the db instead of a generic one. ([\#4462](https://github.com/matrix-org/synapse/issues/4462))
- Reject federation transactions if they include more than 50 PDUs or 100 EDUs. ([\#4513](https://github.com/matrix-org/synapse/issues/4513))
- Reduce duplication of ``synapse.app`` code. ([\#4567](https://github.com/matrix-org/synapse/issues/4567))
- Fix docker upload job to push -py2 images. ([\#4576](https://github.com/matrix-org/synapse/issues/4576))
- Add port configuration information to ACME instructions. ([\#4578](https://github.com/matrix-org/synapse/issues/4578))
- Update MSC1711 FAQ to calrify .well-known usage ([\#4584](https://github.com/matrix-org/synapse/issues/4584))
- Clean up default listener configuration ([\#4586](https://github.com/matrix-org/synapse/issues/4586))
- Clarifications for reverse proxy docs ([\#4607](https://github.com/matrix-org/synapse/issues/4607))
- Move ClientTLSOptionsFactory init out of `refresh_certificates` ([\#4611](https://github.com/matrix-org/synapse/issues/4611))
- Fail cleanly if listener config lacks a 'port' ([\#4616](https://github.com/matrix-org/synapse/issues/4616))
- Remove redundant entries from docker config ([\#4619](https://github.com/matrix-org/synapse/issues/4619))
- README updates ([\#4621](https://github.com/matrix-org/synapse/issues/4621))
2019-02-14 14:41:40 +00:00
Richard van der Hoff c475275926
Clarifications for reverse proxy docs (#4607)
Factor out the reverse proxy info to a separate file, add some more info on
reverse-proxying the federation port.
2019-02-11 11:44:28 +00:00
Erik Johnston 4588b0d64a
Update MSC1711_certificates_FAQ.md
Fix incorrect heading level
2019-02-08 09:37:16 +00:00
Erik Johnston acb2ac5863 Update MSC1711 FAQ to be explicit about well-known (#4584)
A surprising number of people are using the well-known method, and are
simply copying the example configuration. This is problematic as the
example includes an explicit port, which causes inbound federation
requests to have the HTTP Host header include the port, upsetting some
reverse proxies.

Given that, we update the well-known example to be more explicit about
the various ways you can set it up, and the consequence of using an
explict port.
2019-02-07 19:30:32 +00:00
Richard van der Hoff 7cadc4c918 cleanups 2019-02-07 19:29:20 +00:00
Richard van der Hoff 188ad47e73 Merge branch 'master' into erikj/msc1711_faq 2019-02-07 19:27:42 +00:00
Erik Johnston 9285d5c2ce Update MSC1711 FAQ to be explicit about well-known
A surprising number of people are using the well-known method, and are
simply copying the example configuration. This is problematic as the
example includes an explicit port, which causes inbound federation
requests to have the HTTP Host header include the port, upsetting some
reverse proxies.

Given that, we update the well-known example to be more explicit about
the various ways you can set it up, and the consequence of using an
explict port.
2019-02-07 19:24:11 +00:00
Richard van der Hoff 624b172e08 Merge remote-tracking branch 'origin/release-v0.99.0' 2019-02-07 19:18:26 +00:00
Andrew Morgan c17b128b83 Update ACME docs to include port instructions (#4578) 2019-02-07 19:18:08 +00:00
Richard van der Hoff 9b7aa543d9
clarify option 1 2019-02-07 18:46:02 +00:00
Richard van der Hoff b05dd4ac06 faq cleanups 2019-02-05 18:59:57 +00:00
Richard van der Hoff 39bf0ea2e8
Add notes on SRV and .well-known (#4573) 2019-02-05 18:11:26 +00:00