ced4784592
Fix inserting bytes as text
2019-10-08 16:18:43 +01:00
782dd72037
Synapse 1.4.0 (2019-10-03)
...
==========================
Bugfixes
--------
- Redact `client_secret` in server logs. ([\#6158](https://github.com/matrix-org/synapse/issues/6158 ))
Synapse 1.4.0rc2 (2019-10-02)
=============================
Bugfixes
--------
- Fix bug in background update that adds last seen information to the `devices` table, and improve its performance on Postgres. ([\#6135](https://github.com/matrix-org/synapse/issues/6135 ))
- Fix bad performance of censoring redactions background task. ([\#6141](https://github.com/matrix-org/synapse/issues/6141 ))
- Fix fetching censored redactions from DB, which caused APIs like initial sync to fail if it tried to include the censored redaction. ([\#6145](https://github.com/matrix-org/synapse/issues/6145 ))
- Fix exceptions when storing large retry intervals for down remote servers. ([\#6146](https://github.com/matrix-org/synapse/issues/6146 ))
Internal Changes
----------------
- Fix up sample config entry for `redaction_retention_period` option. ([\#6117](https://github.com/matrix-org/synapse/issues/6117 ))
Synapse 1.4.0rc1 (2019-09-26)
=============================
Note that this release includes significant changes around 3pid
verification. Administrators are reminded to review the [upgrade notes](UPGRADE.rst#upgrading-to-v140).
Features
--------
- Changes to 3pid verification:
- Add the ability to send registration emails from the homeserver rather than delegating to an identity server. ([\#5835](https://github.com/matrix-org/synapse/issues/5835 ), [\#5940](https://github.com/matrix-org/synapse/issues/5940 ), [\#5993](https://github.com/matrix-org/synapse/issues/5993 ), [\#5994](https://github.com/matrix-org/synapse/issues/5994 ), [\#5868](https://github.com/matrix-org/synapse/issues/5868 ))
- Replace `trust_identity_server_for_password_resets` config option with `account_threepid_delegates`, and make the `id_server` parameteter optional on `*/requestToken` endpoints, as per [MSC2263](https://github.com/matrix-org/matrix-doc/pull/2263 ). ([\#5876](https://github.com/matrix-org/synapse/issues/5876 ), [\#5969](https://github.com/matrix-org/synapse/issues/5969 ), [\#6028](https://github.com/matrix-org/synapse/issues/6028 ))
- Switch to using the v2 Identity Service `/lookup` API where available, with fallback to v1. (Implements [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134 ) plus `id_access_token authentication` for v2 Identity Service APIs from [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140 )). ([\#5897](https://github.com/matrix-org/synapse/issues/5897 ))
- Remove `bind_email` and `bind_msisdn` parameters from `/register` ala [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140 ). ([\#5964](https://github.com/matrix-org/synapse/issues/5964 ))
- Add `m.id_access_token` to `unstable_features` in `/versions` as per [MSC2264](https://github.com/matrix-org/matrix-doc/pull/2264 ). ([\#5974](https://github.com/matrix-org/synapse/issues/5974 ))
- Use the v2 Identity Service API for 3PID invites. ([\#5979](https://github.com/matrix-org/synapse/issues/5979 ))
- Add `POST /_matrix/client/unstable/account/3pid/unbind` endpoint from [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140 ) for unbinding a 3PID from an identity server without removing it from the homeserver user account. ([\#5980](https://github.com/matrix-org/synapse/issues/5980 ), [\#6062](https://github.com/matrix-org/synapse/issues/6062 ))
- Use `account_threepid_delegate.email` and `account_threepid_delegate.msisdn` for validating threepid sessions. ([\#6011](https://github.com/matrix-org/synapse/issues/6011 ))
- Allow homeserver to handle or delegate email validation when adding an email to a user's account. ([\#6042](https://github.com/matrix-org/synapse/issues/6042 ))
- Implement new Client Server API endpoints `/account/3pid/add` and `/account/3pid/bind` as per [MSC2290](https://github.com/matrix-org/matrix-doc/pull/2290 ). ([\#6043](https://github.com/matrix-org/synapse/issues/6043 ))
- Add an unstable feature flag for separate add/bind 3pid APIs. ([\#6044](https://github.com/matrix-org/synapse/issues/6044 ))
- Remove `bind` parameter from Client Server POST `/account` endpoint as per [MSC2290](https://github.com/matrix-org/matrix-doc/pull/2290/ ). ([\#6067](https://github.com/matrix-org/synapse/issues/6067 ))
- Add `POST /add_threepid/msisdn/submit_token` endpoint for proxying submitToken on an `account_threepid_handler`. ([\#6078](https://github.com/matrix-org/synapse/issues/6078 ))
- Add `submit_url` response parameter to `*/msisdn/requestToken` endpoints. ([\#6079](https://github.com/matrix-org/synapse/issues/6079 ))
- Add `m.require_identity_server` flag to /version's unstable_features. ([\#5972](https://github.com/matrix-org/synapse/issues/5972 ))
- Enhancements to OpenTracing support:
- Make OpenTracing work in worker mode. ([\#5771](https://github.com/matrix-org/synapse/issues/5771 ))
- Pass OpenTracing contexts between servers when transmitting EDUs. ([\#5852](https://github.com/matrix-org/synapse/issues/5852 ))
- OpenTracing for device list updates. ([\#5853](https://github.com/matrix-org/synapse/issues/5853 ))
- Add a tag recording a request's authenticated entity and corresponding servlet in OpenTracing. ([\#5856](https://github.com/matrix-org/synapse/issues/5856 ))
- Add minimum OpenTracing for client servlets. ([\#5983](https://github.com/matrix-org/synapse/issues/5983 ))
- Check at setup that OpenTracing is installed if it's enabled in the config. ([\#5985](https://github.com/matrix-org/synapse/issues/5985 ))
- Trace replication send times. ([\#5986](https://github.com/matrix-org/synapse/issues/5986 ))
- Include missing OpenTracing contexts in outbout replication requests. ([\#5982](https://github.com/matrix-org/synapse/issues/5982 ))
- Fix sending of EDUs when OpenTracing is enabled with an empty whitelist. ([\#5984](https://github.com/matrix-org/synapse/issues/5984 ))
- Fix invalid references to None while OpenTracing if the log context slips. ([\#5988](https://github.com/matrix-org/synapse/issues/5988 ), [\#5991](https://github.com/matrix-org/synapse/issues/5991 ))
- OpenTracing for room and e2e keys. ([\#5855](https://github.com/matrix-org/synapse/issues/5855 ))
- Add OpenTracing span over HTTP push processing. ([\#6003](https://github.com/matrix-org/synapse/issues/6003 ))
- Add an admin API to purge old rooms from the database. ([\#5845](https://github.com/matrix-org/synapse/issues/5845 ))
- Retry well-known lookups if we have recently seen a valid well-known record for the server. ([\#5850](https://github.com/matrix-org/synapse/issues/5850 ))
- Add support for filtered room-directory search requests over federation ([MSC2197](https://github.com/matrix-org/matrix-doc/pull/2197 ), in order to allow upcoming room directory query performance improvements. ([\#5859](https://github.com/matrix-org/synapse/issues/5859 ))
- Correctly retry all hosts returned from SRV when we fail to connect. ([\#5864](https://github.com/matrix-org/synapse/issues/5864 ))
- Add admin API endpoint for setting whether or not a user is a server administrator. ([\#5878](https://github.com/matrix-org/synapse/issues/5878 ))
- Enable cleaning up extremities with dummy events by default to prevent undue build up of forward extremities. ([\#5884](https://github.com/matrix-org/synapse/issues/5884 ))
- Add config option to sign remote key query responses with a separate key. ([\#5895](https://github.com/matrix-org/synapse/issues/5895 ))
- Add support for config templating. ([\#5900](https://github.com/matrix-org/synapse/issues/5900 ))
- Users with the type of "support" or "bot" are no longer required to consent. ([\#5902](https://github.com/matrix-org/synapse/issues/5902 ))
- Let synctl accept a directory of config files. ([\#5904](https://github.com/matrix-org/synapse/issues/5904 ))
- Increase max display name size to 256. ([\#5906](https://github.com/matrix-org/synapse/issues/5906 ))
- Add admin API endpoint for getting whether or not a user is a server administrator. ([\#5914](https://github.com/matrix-org/synapse/issues/5914 ))
- Redact events in the database that have been redacted for a week. ([\#5934](https://github.com/matrix-org/synapse/issues/5934 ))
- New prometheus metrics:
- `synapse_federation_known_servers`: represents the total number of servers your server knows about (i.e. is in rooms with), including itself. Enable by setting `metrics_flags.known_servers` to True in the configuration.([\#5981](https://github.com/matrix-org/synapse/issues/5981 ))
- `synapse_build_info`: exposes the Python version, OS version, and Synapse version of the running server. ([\#6005](https://github.com/matrix-org/synapse/issues/6005 ))
- Give appropriate exit codes when synctl fails. ([\#5992](https://github.com/matrix-org/synapse/issues/5992 ))
- Apply the federation blacklist to requests to identity servers. ([\#6000](https://github.com/matrix-org/synapse/issues/6000 ))
- Add `report_stats_endpoint` option to configure where stats are reported to, if enabled. Contributed by @Sorunome. ([\#6012](https://github.com/matrix-org/synapse/issues/6012 ))
- Add config option to increase ratelimits for room admins redacting messages. ([\#6015](https://github.com/matrix-org/synapse/issues/6015 ))
- Stop sending federation transactions to servers which have been down for a long time. ([\#6026](https://github.com/matrix-org/synapse/issues/6026 ))
- Make the process for mapping SAML2 users to matrix IDs more flexible. ([\#6037](https://github.com/matrix-org/synapse/issues/6037 ))
- Return a clearer error message when a timeout occurs when attempting to contact an identity server. ([\#6073](https://github.com/matrix-org/synapse/issues/6073 ))
- Prevent password reset's submit_token endpoint from accepting trailing slashes. ([\#6074](https://github.com/matrix-org/synapse/issues/6074 ))
- Return 403 on `/register/available` if registration has been disabled. ([\#6082](https://github.com/matrix-org/synapse/issues/6082 ))
- Explicitly log when a homeserver does not have the `trusted_key_servers` config field configured. ([\#6090](https://github.com/matrix-org/synapse/issues/6090 ))
- Add support for pruning old rows in `user_ips` table. ([\#6098](https://github.com/matrix-org/synapse/issues/6098 ))
Bugfixes
--------
- Don't create broken room when `power_level_content_override.users` does not contain `creator_id`. ([\#5633](https://github.com/matrix-org/synapse/issues/5633 ))
- Fix database index so that different backup versions can have the same sessions. ([\#5857](https://github.com/matrix-org/synapse/issues/5857 ))
- Fix Synapse looking for config options `password_reset_failure_template` and `password_reset_success_template`, when they are actually `password_reset_template_failure_html`, `password_reset_template_success_html`. ([\#5863](https://github.com/matrix-org/synapse/issues/5863 ))
- Fix stack overflow when recovering an appservice which had an outage. ([\#5885](https://github.com/matrix-org/synapse/issues/5885 ))
- Fix error message which referred to `public_base_url` instead of `public_baseurl`. Thanks to @aaronraimist for the fix! ([\#5909](https://github.com/matrix-org/synapse/issues/5909 ))
- Fix 404 for thumbnail download when `dynamic_thumbnails` is `false` and the thumbnail was dynamically generated. Fix reported by rkfg. ([\#5915](https://github.com/matrix-org/synapse/issues/5915 ))
- Fix a cache-invalidation bug for worker-based deployments. ([\#5920](https://github.com/matrix-org/synapse/issues/5920 ))
- Fix admin API for listing media in a room not being available with an external media repo. ([\#5966](https://github.com/matrix-org/synapse/issues/5966 ))
- Fix list media admin API always returning an error. ([\#5967](https://github.com/matrix-org/synapse/issues/5967 ))
- Fix room and user stats tracking. ([\#5971](https://github.com/matrix-org/synapse/issues/5971 ), [\#5998](https://github.com/matrix-org/synapse/issues/5998 ), [\#6029](https://github.com/matrix-org/synapse/issues/6029 ))
- Return a `M_MISSING_PARAM` if `sid` is not provided to `/account/3pid`. ([\#5995](https://github.com/matrix-org/synapse/issues/5995 ))
- `federation_certificate_verification_whitelist` now will not cause `TypeErrors` to be raised (a regression in 1.3). Additionally, it now supports internationalised domain names in their non-canonical representation. ([\#5996](https://github.com/matrix-org/synapse/issues/5996 ))
- Only count real users when checking for auto-creation of auto-join room. ([\#6004](https://github.com/matrix-org/synapse/issues/6004 ))
- Ensure support users can be registered even if MAU limit is reached. ([\#6020](https://github.com/matrix-org/synapse/issues/6020 ))
- Fix bug where login error was shown incorrectly on SSO fallback login. ([\#6024](https://github.com/matrix-org/synapse/issues/6024 ))
- Fix bug in calculating the federation retry backoff period. ([\#6025](https://github.com/matrix-org/synapse/issues/6025 ))
- Prevent exceptions being logged when extremity-cleanup events fail due to lack of user consent to the terms of service. ([\#6053](https://github.com/matrix-org/synapse/issues/6053 ))
- Remove POST method from password-reset `submit_token` endpoint until we implement `submit_url` functionality. ([\#6056](https://github.com/matrix-org/synapse/issues/6056 ))
- Fix logcontext spam on non-Linux platforms. ([\#6059](https://github.com/matrix-org/synapse/issues/6059 ))
- Ensure query parameters in email validation links are URL-encoded. ([\#6063](https://github.com/matrix-org/synapse/issues/6063 ))
- Fix a bug which caused SAML attribute maps to be overridden by defaults. ([\#6069](https://github.com/matrix-org/synapse/issues/6069 ))
- Fix the logged number of updated items for the `users_set_deactivated_flag` background update. ([\#6092](https://github.com/matrix-org/synapse/issues/6092 ))
- Add `sid` to `next_link` for email validation. ([\#6097](https://github.com/matrix-org/synapse/issues/6097 ))
- Threepid validity checks on msisdns should not be dependent on `threepid_behaviour_email`. ([\#6104](https://github.com/matrix-org/synapse/issues/6104 ))
- Ensure that servers which are not configured to support email address verification do not offer it in the registration flows. ([\#6107](https://github.com/matrix-org/synapse/issues/6107 ))
Updates to the Docker image
---------------------------
- Avoid changing `UID/GID` if they are already correct. ([\#5970](https://github.com/matrix-org/synapse/issues/5970 ))
- Provide `SYNAPSE_WORKER` envvar to specify python module. ([\#6058](https://github.com/matrix-org/synapse/issues/6058 ))
Improved Documentation
----------------------
- Convert documentation to markdown (from rst) ([\#5849](https://github.com/matrix-org/synapse/issues/5849 ))
- Update `INSTALL.md` to say that Python 2 is no longer supported. ([\#5953](https://github.com/matrix-org/synapse/issues/5953 ))
- Add developer documentation for using SAML2. ([\#6032](https://github.com/matrix-org/synapse/issues/6032 ))
- Add some notes on rolling back to v1.3.1. ([\#6049](https://github.com/matrix-org/synapse/issues/6049 ))
- Update the upgrade notes. ([\#6050](https://github.com/matrix-org/synapse/issues/6050 ))
Deprecations and Removals
-------------------------
- Remove shared-secret registration from `/_matrix/client/r0/register` endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5877](https://github.com/matrix-org/synapse/issues/5877 ))
- Deprecate the `trusted_third_party_id_servers` option. ([\#5875](https://github.com/matrix-org/synapse/issues/5875 ))
Internal Changes
----------------
- Lay the groundwork for structured logging output. ([\#5680](https://github.com/matrix-org/synapse/issues/5680 ))
- Retry well-known lookup before the cache expires, giving a grace period where the remote well-known can be down but we still use the old result. ([\#5844](https://github.com/matrix-org/synapse/issues/5844 ))
- Remove log line for debugging issue #5407 . ([\#5860](https://github.com/matrix-org/synapse/issues/5860 ))
- Refactor the Appservice scheduler code. ([\#5886](https://github.com/matrix-org/synapse/issues/5886 ))
- Compatibility with v2 Identity Service APIs other than /lookup. ([\#5892](https://github.com/matrix-org/synapse/issues/5892 ), [\#6013](https://github.com/matrix-org/synapse/issues/6013 ))
- Stop populating some unused tables. ([\#5893](https://github.com/matrix-org/synapse/issues/5893 ), [\#6047](https://github.com/matrix-org/synapse/issues/6047 ))
- Add missing index on `users_in_public_rooms` to improve the performance of directory queries. ([\#5894](https://github.com/matrix-org/synapse/issues/5894 ))
- Improve the logging when we have an error when fetching signing keys. ([\#5896](https://github.com/matrix-org/synapse/issues/5896 ))
- Add support for database engine-specific schema deltas, based on file extension. ([\#5911](https://github.com/matrix-org/synapse/issues/5911 ))
- Update Buildkite pipeline to use plugins instead of buildkite-agent commands. ([\#5922](https://github.com/matrix-org/synapse/issues/5922 ))
- Add link in sample config to the logging config schema. ([\#5926](https://github.com/matrix-org/synapse/issues/5926 ))
- Remove unnecessary parentheses in return statements. ([\#5931](https://github.com/matrix-org/synapse/issues/5931 ))
- Remove unused `jenkins/prepare_sytest.sh` file. ([\#5938](https://github.com/matrix-org/synapse/issues/5938 ))
- Move Buildkite pipeline config to the pipelines repo. ([\#5943](https://github.com/matrix-org/synapse/issues/5943 ))
- Remove unnecessary return statements in the codebase which were the result of a regex run. ([\#5962](https://github.com/matrix-org/synapse/issues/5962 ))
- Remove left-over methods from v1 registration API. ([\#5963](https://github.com/matrix-org/synapse/issues/5963 ))
- Cleanup event auth type initialisation. ([\#5975](https://github.com/matrix-org/synapse/issues/5975 ))
- Clean up dependency checking at setup. ([\#5989](https://github.com/matrix-org/synapse/issues/5989 ))
- Update OpenTracing docs to use the unified `trace` method. ([\#5776](https://github.com/matrix-org/synapse/issues/5776 ))
- Small refactor of function arguments and docstrings in` RoomMemberHandler`. ([\#6009](https://github.com/matrix-org/synapse/issues/6009 ))
- Remove unused `origin` argument on `FederationHandler.add_display_name_to_third_party_invite`. ([\#6010](https://github.com/matrix-org/synapse/issues/6010 ))
- Add a `failure_ts` column to the `destinations` database table. ([\#6016](https://github.com/matrix-org/synapse/issues/6016 ), [\#6072](https://github.com/matrix-org/synapse/issues/6072 ))
- Clean up some code in the retry logic. ([\#6017](https://github.com/matrix-org/synapse/issues/6017 ))
- Fix the structured logging tests stomping on the global log configuration for subsequent tests. ([\#6023](https://github.com/matrix-org/synapse/issues/6023 ))
- Clean up the sample config for SAML authentication. ([\#6064](https://github.com/matrix-org/synapse/issues/6064 ))
- Change mailer logging to reflect Synapse doesn't just do chat notifications by email now. ([\#6075](https://github.com/matrix-org/synapse/issues/6075 ))
- Move last-seen info into devices table. ([\#6089](https://github.com/matrix-org/synapse/issues/6089 ))
- Remove unused parameter to `get_user_id_by_threepid`. ([\#6099](https://github.com/matrix-org/synapse/issues/6099 ))
- Refactor the user-interactive auth handling. ([\#6105](https://github.com/matrix-org/synapse/issues/6105 ))
- Refactor code for calculating registration flows. ([\#6106](https://github.com/matrix-org/synapse/issues/6106 ))
-----BEGIN PGP SIGNATURE-----
iQJHBAABCgAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAl2V6NsTHGFuZHJld0Bh
bW9yZ2FuLnh5egAKCRCIhIgNLv5f9IUTEACE/gPcItzKAOsnEjCyBVzjwXcvqdZU
5FpEbnDqhVS20GBOctl+00QqHOeje8tDLmYy4yS/LH89iX+PQNI55M/4bmS5ZTYm
x13AsxgqJ11xDOPmkzLkXDF0HnzZngGeGSShsPFUUWfMF9pcCz+kMY88dJe69sfU
TK/VGIvp4MAcbeIbtgbInS7ZH1igntBLyfp48VQ0WCz6HeorJSxniRDKpRQ08Mc3
TbZsFZ/2ajvL+HSr6iSDUD32GTEfUERVpV7OrQnC9laa0Aidq0yRCMcRpBWJ8omP
g/240jDxsbSZkNe+6GPuiRjqFQuHw0KVb/A10BJRvEDCgVzIdsE5oEQLr3HKl2h3
q1HQewuc2tG/ns4vMHK+dhEVlc95yUI1UxrNafP0/hhpWRiRX1VfiWv08tKNP5GJ
7an7Oqah4yZWjyzVBhiMugmQnnaf2z5TymUaRcZRNfueTpUo4DfJ3as71JcOmIGc
Hg7gM1DOkLUMV5xoieW+7dKUqO8Nqv8y+q7qbK3LRT83HeE3AHXf3JmfB7/6+zi0
4coM5ZrGrp5A4PeGDVKPOT3VdmXK8r6buPqcJxUMaWAMziWGAEWROrGF+khvpUlW
NkLAF/XitjqIlDQEWpUzHP7vGXNt9m83/9J8wD4/ZAjcUDfW1KSSCbBmCR4s5ZAk
HVzNubTzKy2xlA==
=2TFh
-----END PGP SIGNATURE-----
Merge tag 'v1.4.0'
Synapse 1.4.0 (2019-10-03)
==========================
Bugfixes
--------
- Redact `client_secret` in server logs. ([\#6158](https://github.com/matrix-org/synapse/issues/6158 ))
Synapse 1.4.0rc2 (2019-10-02)
=============================
Bugfixes
--------
- Fix bug in background update that adds last seen information to the `devices` table, and improve its performance on Postgres. ([\#6135](https://github.com/matrix-org/synapse/issues/6135 ))
- Fix bad performance of censoring redactions background task. ([\#6141](https://github.com/matrix-org/synapse/issues/6141 ))
- Fix fetching censored redactions from DB, which caused APIs like initial sync to fail if it tried to include the censored redaction. ([\#6145](https://github.com/matrix-org/synapse/issues/6145 ))
- Fix exceptions when storing large retry intervals for down remote servers. ([\#6146](https://github.com/matrix-org/synapse/issues/6146 ))
Internal Changes
----------------
- Fix up sample config entry for `redaction_retention_period` option. ([\#6117](https://github.com/matrix-org/synapse/issues/6117 ))
Synapse 1.4.0rc1 (2019-09-26)
=============================
Note that this release includes significant changes around 3pid
verification. Administrators are reminded to review the [upgrade notes](UPGRADE.rst#upgrading-to-v140).
Features
--------
- Changes to 3pid verification:
- Add the ability to send registration emails from the homeserver rather than delegating to an identity server. ([\#5835](https://github.com/matrix-org/synapse/issues/5835 ), [\#5940](https://github.com/matrix-org/synapse/issues/5940 ), [\#5993](https://github.com/matrix-org/synapse/issues/5993 ), [\#5994](https://github.com/matrix-org/synapse/issues/5994 ), [\#5868](https://github.com/matrix-org/synapse/issues/5868 ))
- Replace `trust_identity_server_for_password_resets` config option with `account_threepid_delegates`, and make the `id_server` parameteter optional on `*/requestToken` endpoints, as per [MSC2263](https://github.com/matrix-org/matrix-doc/pull/2263 ). ([\#5876](https://github.com/matrix-org/synapse/issues/5876 ), [\#5969](https://github.com/matrix-org/synapse/issues/5969 ), [\#6028](https://github.com/matrix-org/synapse/issues/6028 ))
- Switch to using the v2 Identity Service `/lookup` API where available, with fallback to v1. (Implements [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134 ) plus `id_access_token authentication` for v2 Identity Service APIs from [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140 )). ([\#5897](https://github.com/matrix-org/synapse/issues/5897 ))
- Remove `bind_email` and `bind_msisdn` parameters from `/register` ala [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140 ). ([\#5964](https://github.com/matrix-org/synapse/issues/5964 ))
- Add `m.id_access_token` to `unstable_features` in `/versions` as per [MSC2264](https://github.com/matrix-org/matrix-doc/pull/2264 ). ([\#5974](https://github.com/matrix-org/synapse/issues/5974 ))
- Use the v2 Identity Service API for 3PID invites. ([\#5979](https://github.com/matrix-org/synapse/issues/5979 ))
- Add `POST /_matrix/client/unstable/account/3pid/unbind` endpoint from [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140 ) for unbinding a 3PID from an identity server without removing it from the homeserver user account. ([\#5980](https://github.com/matrix-org/synapse/issues/5980 ), [\#6062](https://github.com/matrix-org/synapse/issues/6062 ))
- Use `account_threepid_delegate.email` and `account_threepid_delegate.msisdn` for validating threepid sessions. ([\#6011](https://github.com/matrix-org/synapse/issues/6011 ))
- Allow homeserver to handle or delegate email validation when adding an email to a user's account. ([\#6042](https://github.com/matrix-org/synapse/issues/6042 ))
- Implement new Client Server API endpoints `/account/3pid/add` and `/account/3pid/bind` as per [MSC2290](https://github.com/matrix-org/matrix-doc/pull/2290 ). ([\#6043](https://github.com/matrix-org/synapse/issues/6043 ))
- Add an unstable feature flag for separate add/bind 3pid APIs. ([\#6044](https://github.com/matrix-org/synapse/issues/6044 ))
- Remove `bind` parameter from Client Server POST `/account` endpoint as per [MSC2290](https://github.com/matrix-org/matrix-doc/pull/2290/ ). ([\#6067](https://github.com/matrix-org/synapse/issues/6067 ))
- Add `POST /add_threepid/msisdn/submit_token` endpoint for proxying submitToken on an `account_threepid_handler`. ([\#6078](https://github.com/matrix-org/synapse/issues/6078 ))
- Add `submit_url` response parameter to `*/msisdn/requestToken` endpoints. ([\#6079](https://github.com/matrix-org/synapse/issues/6079 ))
- Add `m.require_identity_server` flag to /version's unstable_features. ([\#5972](https://github.com/matrix-org/synapse/issues/5972 ))
- Enhancements to OpenTracing support:
- Make OpenTracing work in worker mode. ([\#5771](https://github.com/matrix-org/synapse/issues/5771 ))
- Pass OpenTracing contexts between servers when transmitting EDUs. ([\#5852](https://github.com/matrix-org/synapse/issues/5852 ))
- OpenTracing for device list updates. ([\#5853](https://github.com/matrix-org/synapse/issues/5853 ))
- Add a tag recording a request's authenticated entity and corresponding servlet in OpenTracing. ([\#5856](https://github.com/matrix-org/synapse/issues/5856 ))
- Add minimum OpenTracing for client servlets. ([\#5983](https://github.com/matrix-org/synapse/issues/5983 ))
- Check at setup that OpenTracing is installed if it's enabled in the config. ([\#5985](https://github.com/matrix-org/synapse/issues/5985 ))
- Trace replication send times. ([\#5986](https://github.com/matrix-org/synapse/issues/5986 ))
- Include missing OpenTracing contexts in outbout replication requests. ([\#5982](https://github.com/matrix-org/synapse/issues/5982 ))
- Fix sending of EDUs when OpenTracing is enabled with an empty whitelist. ([\#5984](https://github.com/matrix-org/synapse/issues/5984 ))
- Fix invalid references to None while OpenTracing if the log context slips. ([\#5988](https://github.com/matrix-org/synapse/issues/5988 ), [\#5991](https://github.com/matrix-org/synapse/issues/5991 ))
- OpenTracing for room and e2e keys. ([\#5855](https://github.com/matrix-org/synapse/issues/5855 ))
- Add OpenTracing span over HTTP push processing. ([\#6003](https://github.com/matrix-org/synapse/issues/6003 ))
- Add an admin API to purge old rooms from the database. ([\#5845](https://github.com/matrix-org/synapse/issues/5845 ))
- Retry well-known lookups if we have recently seen a valid well-known record for the server. ([\#5850](https://github.com/matrix-org/synapse/issues/5850 ))
- Add support for filtered room-directory search requests over federation ([MSC2197](https://github.com/matrix-org/matrix-doc/pull/2197 ), in order to allow upcoming room directory query performance improvements. ([\#5859](https://github.com/matrix-org/synapse/issues/5859 ))
- Correctly retry all hosts returned from SRV when we fail to connect. ([\#5864](https://github.com/matrix-org/synapse/issues/5864 ))
- Add admin API endpoint for setting whether or not a user is a server administrator. ([\#5878](https://github.com/matrix-org/synapse/issues/5878 ))
- Enable cleaning up extremities with dummy events by default to prevent undue build up of forward extremities. ([\#5884](https://github.com/matrix-org/synapse/issues/5884 ))
- Add config option to sign remote key query responses with a separate key. ([\#5895](https://github.com/matrix-org/synapse/issues/5895 ))
- Add support for config templating. ([\#5900](https://github.com/matrix-org/synapse/issues/5900 ))
- Users with the type of "support" or "bot" are no longer required to consent. ([\#5902](https://github.com/matrix-org/synapse/issues/5902 ))
- Let synctl accept a directory of config files. ([\#5904](https://github.com/matrix-org/synapse/issues/5904 ))
- Increase max display name size to 256. ([\#5906](https://github.com/matrix-org/synapse/issues/5906 ))
- Add admin API endpoint for getting whether or not a user is a server administrator. ([\#5914](https://github.com/matrix-org/synapse/issues/5914 ))
- Redact events in the database that have been redacted for a week. ([\#5934](https://github.com/matrix-org/synapse/issues/5934 ))
- New prometheus metrics:
- `synapse_federation_known_servers`: represents the total number of servers your server knows about (i.e. is in rooms with), including itself. Enable by setting `metrics_flags.known_servers` to True in the configuration.([\#5981](https://github.com/matrix-org/synapse/issues/5981 ))
- `synapse_build_info`: exposes the Python version, OS version, and Synapse version of the running server. ([\#6005](https://github.com/matrix-org/synapse/issues/6005 ))
- Give appropriate exit codes when synctl fails. ([\#5992](https://github.com/matrix-org/synapse/issues/5992 ))
- Apply the federation blacklist to requests to identity servers. ([\#6000](https://github.com/matrix-org/synapse/issues/6000 ))
- Add `report_stats_endpoint` option to configure where stats are reported to, if enabled. Contributed by @Sorunome. ([\#6012](https://github.com/matrix-org/synapse/issues/6012 ))
- Add config option to increase ratelimits for room admins redacting messages. ([\#6015](https://github.com/matrix-org/synapse/issues/6015 ))
- Stop sending federation transactions to servers which have been down for a long time. ([\#6026](https://github.com/matrix-org/synapse/issues/6026 ))
- Make the process for mapping SAML2 users to matrix IDs more flexible. ([\#6037](https://github.com/matrix-org/synapse/issues/6037 ))
- Return a clearer error message when a timeout occurs when attempting to contact an identity server. ([\#6073](https://github.com/matrix-org/synapse/issues/6073 ))
- Prevent password reset's submit_token endpoint from accepting trailing slashes. ([\#6074](https://github.com/matrix-org/synapse/issues/6074 ))
- Return 403 on `/register/available` if registration has been disabled. ([\#6082](https://github.com/matrix-org/synapse/issues/6082 ))
- Explicitly log when a homeserver does not have the `trusted_key_servers` config field configured. ([\#6090](https://github.com/matrix-org/synapse/issues/6090 ))
- Add support for pruning old rows in `user_ips` table. ([\#6098](https://github.com/matrix-org/synapse/issues/6098 ))
Bugfixes
--------
- Don't create broken room when `power_level_content_override.users` does not contain `creator_id`. ([\#5633](https://github.com/matrix-org/synapse/issues/5633 ))
- Fix database index so that different backup versions can have the same sessions. ([\#5857](https://github.com/matrix-org/synapse/issues/5857 ))
- Fix Synapse looking for config options `password_reset_failure_template` and `password_reset_success_template`, when they are actually `password_reset_template_failure_html`, `password_reset_template_success_html`. ([\#5863](https://github.com/matrix-org/synapse/issues/5863 ))
- Fix stack overflow when recovering an appservice which had an outage. ([\#5885](https://github.com/matrix-org/synapse/issues/5885 ))
- Fix error message which referred to `public_base_url` instead of `public_baseurl`. Thanks to @aaronraimist for the fix! ([\#5909](https://github.com/matrix-org/synapse/issues/5909 ))
- Fix 404 for thumbnail download when `dynamic_thumbnails` is `false` and the thumbnail was dynamically generated. Fix reported by rkfg. ([\#5915](https://github.com/matrix-org/synapse/issues/5915 ))
- Fix a cache-invalidation bug for worker-based deployments. ([\#5920](https://github.com/matrix-org/synapse/issues/5920 ))
- Fix admin API for listing media in a room not being available with an external media repo. ([\#5966](https://github.com/matrix-org/synapse/issues/5966 ))
- Fix list media admin API always returning an error. ([\#5967](https://github.com/matrix-org/synapse/issues/5967 ))
- Fix room and user stats tracking. ([\#5971](https://github.com/matrix-org/synapse/issues/5971 ), [\#5998](https://github.com/matrix-org/synapse/issues/5998 ), [\#6029](https://github.com/matrix-org/synapse/issues/6029 ))
- Return a `M_MISSING_PARAM` if `sid` is not provided to `/account/3pid`. ([\#5995](https://github.com/matrix-org/synapse/issues/5995 ))
- `federation_certificate_verification_whitelist` now will not cause `TypeErrors` to be raised (a regression in 1.3). Additionally, it now supports internationalised domain names in their non-canonical representation. ([\#5996](https://github.com/matrix-org/synapse/issues/5996 ))
- Only count real users when checking for auto-creation of auto-join room. ([\#6004](https://github.com/matrix-org/synapse/issues/6004 ))
- Ensure support users can be registered even if MAU limit is reached. ([\#6020](https://github.com/matrix-org/synapse/issues/6020 ))
- Fix bug where login error was shown incorrectly on SSO fallback login. ([\#6024](https://github.com/matrix-org/synapse/issues/6024 ))
- Fix bug in calculating the federation retry backoff period. ([\#6025](https://github.com/matrix-org/synapse/issues/6025 ))
- Prevent exceptions being logged when extremity-cleanup events fail due to lack of user consent to the terms of service. ([\#6053](https://github.com/matrix-org/synapse/issues/6053 ))
- Remove POST method from password-reset `submit_token` endpoint until we implement `submit_url` functionality. ([\#6056](https://github.com/matrix-org/synapse/issues/6056 ))
- Fix logcontext spam on non-Linux platforms. ([\#6059](https://github.com/matrix-org/synapse/issues/6059 ))
- Ensure query parameters in email validation links are URL-encoded. ([\#6063](https://github.com/matrix-org/synapse/issues/6063 ))
- Fix a bug which caused SAML attribute maps to be overridden by defaults. ([\#6069](https://github.com/matrix-org/synapse/issues/6069 ))
- Fix the logged number of updated items for the `users_set_deactivated_flag` background update. ([\#6092](https://github.com/matrix-org/synapse/issues/6092 ))
- Add `sid` to `next_link` for email validation. ([\#6097](https://github.com/matrix-org/synapse/issues/6097 ))
- Threepid validity checks on msisdns should not be dependent on `threepid_behaviour_email`. ([\#6104](https://github.com/matrix-org/synapse/issues/6104 ))
- Ensure that servers which are not configured to support email address verification do not offer it in the registration flows. ([\#6107](https://github.com/matrix-org/synapse/issues/6107 ))
Updates to the Docker image
---------------------------
- Avoid changing `UID/GID` if they are already correct. ([\#5970](https://github.com/matrix-org/synapse/issues/5970 ))
- Provide `SYNAPSE_WORKER` envvar to specify python module. ([\#6058](https://github.com/matrix-org/synapse/issues/6058 ))
Improved Documentation
----------------------
- Convert documentation to markdown (from rst) ([\#5849](https://github.com/matrix-org/synapse/issues/5849 ))
- Update `INSTALL.md` to say that Python 2 is no longer supported. ([\#5953](https://github.com/matrix-org/synapse/issues/5953 ))
- Add developer documentation for using SAML2. ([\#6032](https://github.com/matrix-org/synapse/issues/6032 ))
- Add some notes on rolling back to v1.3.1. ([\#6049](https://github.com/matrix-org/synapse/issues/6049 ))
- Update the upgrade notes. ([\#6050](https://github.com/matrix-org/synapse/issues/6050 ))
Deprecations and Removals
-------------------------
- Remove shared-secret registration from `/_matrix/client/r0/register` endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5877](https://github.com/matrix-org/synapse/issues/5877 ))
- Deprecate the `trusted_third_party_id_servers` option. ([\#5875](https://github.com/matrix-org/synapse/issues/5875 ))
Internal Changes
----------------
- Lay the groundwork for structured logging output. ([\#5680](https://github.com/matrix-org/synapse/issues/5680 ))
- Retry well-known lookup before the cache expires, giving a grace period where the remote well-known can be down but we still use the old result. ([\#5844](https://github.com/matrix-org/synapse/issues/5844 ))
- Remove log line for debugging issue #5407 . ([\#5860](https://github.com/matrix-org/synapse/issues/5860 ))
- Refactor the Appservice scheduler code. ([\#5886](https://github.com/matrix-org/synapse/issues/5886 ))
- Compatibility with v2 Identity Service APIs other than /lookup. ([\#5892](https://github.com/matrix-org/synapse/issues/5892 ), [\#6013](https://github.com/matrix-org/synapse/issues/6013 ))
- Stop populating some unused tables. ([\#5893](https://github.com/matrix-org/synapse/issues/5893 ), [\#6047](https://github.com/matrix-org/synapse/issues/6047 ))
- Add missing index on `users_in_public_rooms` to improve the performance of directory queries. ([\#5894](https://github.com/matrix-org/synapse/issues/5894 ))
- Improve the logging when we have an error when fetching signing keys. ([\#5896](https://github.com/matrix-org/synapse/issues/5896 ))
- Add support for database engine-specific schema deltas, based on file extension. ([\#5911](https://github.com/matrix-org/synapse/issues/5911 ))
- Update Buildkite pipeline to use plugins instead of buildkite-agent commands. ([\#5922](https://github.com/matrix-org/synapse/issues/5922 ))
- Add link in sample config to the logging config schema. ([\#5926](https://github.com/matrix-org/synapse/issues/5926 ))
- Remove unnecessary parentheses in return statements. ([\#5931](https://github.com/matrix-org/synapse/issues/5931 ))
- Remove unused `jenkins/prepare_sytest.sh` file. ([\#5938](https://github.com/matrix-org/synapse/issues/5938 ))
- Move Buildkite pipeline config to the pipelines repo. ([\#5943](https://github.com/matrix-org/synapse/issues/5943 ))
- Remove unnecessary return statements in the codebase which were the result of a regex run. ([\#5962](https://github.com/matrix-org/synapse/issues/5962 ))
- Remove left-over methods from v1 registration API. ([\#5963](https://github.com/matrix-org/synapse/issues/5963 ))
- Cleanup event auth type initialisation. ([\#5975](https://github.com/matrix-org/synapse/issues/5975 ))
- Clean up dependency checking at setup. ([\#5989](https://github.com/matrix-org/synapse/issues/5989 ))
- Update OpenTracing docs to use the unified `trace` method. ([\#5776](https://github.com/matrix-org/synapse/issues/5776 ))
- Small refactor of function arguments and docstrings in` RoomMemberHandler`. ([\#6009](https://github.com/matrix-org/synapse/issues/6009 ))
- Remove unused `origin` argument on `FederationHandler.add_display_name_to_third_party_invite`. ([\#6010](https://github.com/matrix-org/synapse/issues/6010 ))
- Add a `failure_ts` column to the `destinations` database table. ([\#6016](https://github.com/matrix-org/synapse/issues/6016 ), [\#6072](https://github.com/matrix-org/synapse/issues/6072 ))
- Clean up some code in the retry logic. ([\#6017](https://github.com/matrix-org/synapse/issues/6017 ))
- Fix the structured logging tests stomping on the global log configuration for subsequent tests. ([\#6023](https://github.com/matrix-org/synapse/issues/6023 ))
- Clean up the sample config for SAML authentication. ([\#6064](https://github.com/matrix-org/synapse/issues/6064 ))
- Change mailer logging to reflect Synapse doesn't just do chat notifications by email now. ([\#6075](https://github.com/matrix-org/synapse/issues/6075 ))
- Move last-seen info into devices table. ([\#6089](https://github.com/matrix-org/synapse/issues/6089 ))
- Remove unused parameter to `get_user_id_by_threepid`. ([\#6099](https://github.com/matrix-org/synapse/issues/6099 ))
- Refactor the user-interactive auth handling. ([\#6105](https://github.com/matrix-org/synapse/issues/6105 ))
- Refactor code for calculating registration flows. ([\#6106](https://github.com/matrix-org/synapse/issues/6106 ))
2019-10-03 13:48:21 +01:00
ecb69d824a
1.4.0
2019-10-03 13:22:44 +01:00
0f46bf5737
Replace client_secret with <redacted> in server logs ( #6158 )
...
Replace `client_secret` query parameter values with `<redacted>` in the logs. Prevents a scenario where a MITM of server traffic can horde 3pids on their account.
2019-10-03 12:57:26 +01:00
aec1377d0b
1.4.0rc2
2019-10-02 13:55:00 +01:00
b4fe7e19c0
Merge pull request #6146 from matrix-org/erikj/fix_destination_retry_timings
...
Fix errors storing large retry intervals.
2019-10-02 11:05:22 +01:00
35f392bb29
Merge pull request #6145 from matrix-org/erikj/fix_censored_redactions
...
Fix fetching censored redactions from DB
2019-10-02 10:57:30 +01:00
2bc027ab71
Newsfile
2019-10-02 10:41:29 +01:00
f44f1d2e83
Fix errors storing large retry intervals.
...
We have set the max retry interval to a value larger than a postgres or
sqlite int can hold, which caused exceptions when updating the
destinations table.
To fix postgres we need to change the column to a bigint, and for sqlite
we lower the max interval to 2**62 (which is still incredibly long).
2019-10-02 10:36:27 +01:00
dfe7009639
Merge pull request #6141 from matrix-org/erikj/censor_redactions_fix
...
Fix censoring redactions performance
2019-10-02 10:20:35 +01:00
33d4ebdf78
Newsfile
2019-10-02 10:18:17 +01:00
ce7a3e7e27
Fix fetching censored redactions from DB
...
Fetching a censored redactions caused an exception due to the code
expecting redactions to have a `redact` key, which redacted redactions
don't have.
2019-10-02 10:14:01 +01:00
2b8352e638
Newsfile
2019-10-01 13:47:06 +01:00
5e8387af9e
Use `received_ts` to find uncensored redacted events
...
Joining against `events` and ordering by `stream_ordering` is
inefficient as it forced scanning the entirety of the redactions table.
This isn't the case if we use `redactions.received_ts` column as we can
then use an index.
2019-10-01 13:43:48 +01:00
898dde981b
Add received_ts column to redactions.
...
This will allow us to efficiently search for uncensored redactions in
the DB before a given time.
2019-10-01 13:43:48 +01:00
a27fb7d5ca
Don't repeatedly attempt to censor events we don't have.
...
Currently we don't set `have_censored` column if we don't have the
target event of a redaction, which means we repeatedly attempt to censor
the same non-existant event.
When we persist non-redacted events we unset the `have_censored` column
for any redactions that target said event.
2019-10-01 11:05:48 +01:00
479fbac96f
Merge pull request #6135 from matrix-org/erikj/fixup_devices_last_seen_query
...
Fix `devices_last_seen` background update.
2019-10-01 10:16:57 +01:00
129343cd8a
Newsfile
2019-09-30 12:00:34 +01:00
9267741a5f
Fix `devices_last_seen` background update.
...
Fixes #6134 .
2019-09-30 11:58:36 +01:00
6d0f559fb3
Merge pull request #6117 from matrix-org/erikj/fix_sample_config
...
Fix 'redaction_retention_period' sampel config to match guidelines
2019-09-26 17:14:23 +01:00
9d99eade7c
Newsfile
2019-09-26 16:46:21 +01:00
3423633d50
Fix 'redaction_retention_period' sampel config to match guidelines
2019-09-26 16:43:52 +01:00
f25c5ee1fe
s/month/week/ to match config
2019-09-26 15:57:02 +01:00
e755128006
Update CHANGES.md
...
formatting tweak
2019-09-26 15:29:31 +01:00
e04c235907
more changelog updates
2019-09-26 14:59:01 +01:00
5384c43626
Changelog formatting
2019-09-26 13:39:09 +01:00
3fbca80a8d
changelog
2019-09-26 13:23:48 +01:00
8b8f8c7b3c
Explicitly log when a homeserver does not have a trusted key server configured ( #6090 )
2019-09-26 12:57:01 +01:00
1b051f1245
Merge commit '1b23f991a' into release-v1.4.0
2019-09-26 12:35:26 +01:00
1b23f991ab
Clarify upgrade notes ahead of 1.4.0 release
2019-09-26 12:30:10 +01:00
2927c6bc4c
bump version
2019-09-26 12:29:59 +01:00
034db2ba21
Fix dummy event insertion consent bug ( #6053 )
...
Fixes #5905
2019-09-26 11:47:53 +01:00
d64b70ada2
Merge pull request #6098 from matrix-org/erikj/cleanup_user_ips_2
...
Prune old rows in user_ips tables.
2019-09-26 10:23:54 +01:00
a96318127d
Update comments and docstring
2019-09-25 18:17:39 +01:00
4fb3c129aa
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips_2
2019-09-25 17:53:13 +01:00
9614d3c9d1
Merge pull request #6089 from matrix-org/erikj/cleanup_user_ips
...
Move last seen info into devices table
2019-09-25 17:42:39 +01:00
a4f3ca48b5
Enable cleaning up extremities with dummy events by default to prevent undue build up of forward extremities. ( #5884 )
2019-09-25 17:27:35 +01:00
39b50ad42a
Review comments
2019-09-25 17:22:33 +01:00
d2bd0bc6b1
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips
2019-09-25 17:16:28 +01:00
50572db837
Use if `is not None`
...
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-09-25 17:00:23 +01:00
77dc7093a7
Threepid validity checks on msisdns should not be dependent on 'threepid_behaviour_email'. ( #6104 )
...
Fixes #6103
2019-09-25 12:29:35 +01:00
990928abde
Stop advertising unsupported flows for registration ( #6107 )
...
If email or msisdn verification aren't supported, let's stop advertising them
for registration.
Fixes #6100 .
2019-09-25 12:10:26 +01:00
2cd98812ba
Refactor the user-interactive auth handling ( #6105 )
...
Pull the checkers out to their own classes, rather than having them lost in a
massive 1000-line class which does everything.
This is also preparation for some more intelligent advertising of flows, as per #6100
2019-09-25 11:33:03 +01:00
8004d6ca2f
Refactor code for calculating registration flows ( #6106 )
...
because, frankly, it looked like it was written by an axe-murderer.
This should be a non-functional change, except that where `m.login.dummy` was
previously advertised *before* `m.login.terms`, it will now be advertised
afterwards. AFAICT that should have no effect, and will be more consistent with
the flows that involve passing a 3pid.
2019-09-25 11:32:05 +01:00
f99a9c9cb0
Merge pull request #6092 from matrix-org/babolivier/background_update_deactivated_return
...
Fix the return value in the users_set_deactivated_flag background job
2019-09-24 17:11:12 +01:00
4f6bbe9d0d
Merge pull request #6037 from matrix-org/rav/saml_mapping_work
...
Update the process for mapping SAML2 users to matrix IDs
2019-09-24 17:04:54 +01:00
9b7c4f4078
Merge remote-tracking branch 'origin/develop' into rav/saml_mapping_work
2019-09-24 17:03:50 +01:00
566ac40939
remove unused parameter to get_user_id_by_threepid ( #6099 )
...
Added in #5377 , apparently in error
2019-09-24 17:01:09 +01:00
f8b02c5420
Newsfile
2019-09-24 15:59:43 +01:00
faac453f08
Test that pruning of old user IPs works
2019-09-24 15:53:17 +01:00
Erik Johnston