276ae5c63e
add some logging to the rooms stats updates, to try to track down a flaky test ( #6167 )
2019-10-07 14:41:39 +01:00
97e2722723
Merge pull request #6175 from matrix-org/babolivier/fix_unique_user_filter_index
...
Fix unique_user_filter_index schema update
2019-10-07 13:48:43 +01:00
aa7a003074
Changelog
2019-10-07 13:16:54 +01:00
c8e6c308c6
Fix unique_user_filter_index schema update
2019-10-07 13:15:35 +01:00
86f4705866
Merge pull request #6159 from matrix-org/erikj/cache_memberships
...
Cache room membership lookups in _get_joined_users_from_context
2019-10-07 13:15:00 +01:00
5119a4cac7
Fix bug where we didn't pull out event ID
2019-10-07 12:21:17 +01:00
ae0b78cb1f
Merge pull request #6147 from matrix-org/babolivier/3pid-invite-revoked
...
Don't 500 when trying to exchange a revoked 3PID invite
2019-10-04 12:09:05 +01:00
21d51ab598
Typo
2019-10-04 11:21:24 +01:00
4676732ca0
Lint
2019-10-04 11:18:28 +01:00
81d51ce48b
Incorporate review
2019-10-04 11:16:19 +01:00
13c4345c84
Update `user_filters` table to have a unique index, and non-null columns ( #1172 )
2019-10-04 10:34:16 +01:00
39b40d6d99
media/thumbnailer: Better quality for 1-bit / 8-bit color palette images ( #2142 )
...
Pillow will use nearest neighbour as the resampling algorithm if the
source image is either 1-bit or a color palette using 8 bits. If we
convert to RGB before scaling, we'll probably get a better result.
2019-10-04 09:34:52 +01:00
6511071837
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cache_memberships
2019-10-03 17:51:06 +01:00
66537e10ce
add some metrics on the federation sender ( #6160 )
2019-10-03 17:47:20 +01:00
0186ec9df7
Fixup newsfile
2019-10-03 17:46:43 +01:00
ab8a64772b
Remove unused variable
2019-10-03 17:42:32 +01:00
4fc60f12de
Newsfile
2019-10-03 17:35:50 +01:00
693156aaf4
Don't regenerate numeric user ID if registration fails.
...
This causes huge amounts of DB IO if registrations start to fail e.g.
because the DB is struggling with IO.
2019-10-03 17:33:54 +01:00
91f61fc6d7
Use the right error....
2019-10-03 17:28:31 +01:00
84691da6c3
pep8
2019-10-03 17:27:18 +01:00
a9610cdf02
Fixup names and comments
2019-10-03 17:26:56 +01:00
d89ebf7c25
cachedList descriptor doesn't like typing
2019-10-03 17:23:11 +01:00
0ccf0ffc85
Newsfile
2019-10-03 17:12:24 +01:00
c8145af8a9
Cache room membership lookups in _get_joined_users_from_context
2019-10-03 17:11:04 +01:00
6018bec919
Merge branch 'master' into develop
2019-10-03 13:48:45 +01:00
782dd72037
Synapse 1.4.0 (2019-10-03)
...
==========================
Bugfixes
--------
- Redact `client_secret` in server logs. ([\#6158](https://github.com/matrix-org/synapse/issues/6158 ))
Synapse 1.4.0rc2 (2019-10-02)
=============================
Bugfixes
--------
- Fix bug in background update that adds last seen information to the `devices` table, and improve its performance on Postgres. ([\#6135](https://github.com/matrix-org/synapse/issues/6135 ))
- Fix bad performance of censoring redactions background task. ([\#6141](https://github.com/matrix-org/synapse/issues/6141 ))
- Fix fetching censored redactions from DB, which caused APIs like initial sync to fail if it tried to include the censored redaction. ([\#6145](https://github.com/matrix-org/synapse/issues/6145 ))
- Fix exceptions when storing large retry intervals for down remote servers. ([\#6146](https://github.com/matrix-org/synapse/issues/6146 ))
Internal Changes
----------------
- Fix up sample config entry for `redaction_retention_period` option. ([\#6117](https://github.com/matrix-org/synapse/issues/6117 ))
Synapse 1.4.0rc1 (2019-09-26)
=============================
Note that this release includes significant changes around 3pid
verification. Administrators are reminded to review the [upgrade notes](UPGRADE.rst#upgrading-to-v140).
Features
--------
- Changes to 3pid verification:
- Add the ability to send registration emails from the homeserver rather than delegating to an identity server. ([\#5835](https://github.com/matrix-org/synapse/issues/5835 ), [\#5940](https://github.com/matrix-org/synapse/issues/5940 ), [\#5993](https://github.com/matrix-org/synapse/issues/5993 ), [\#5994](https://github.com/matrix-org/synapse/issues/5994 ), [\#5868](https://github.com/matrix-org/synapse/issues/5868 ))
- Replace `trust_identity_server_for_password_resets` config option with `account_threepid_delegates`, and make the `id_server` parameteter optional on `*/requestToken` endpoints, as per [MSC2263](https://github.com/matrix-org/matrix-doc/pull/2263 ). ([\#5876](https://github.com/matrix-org/synapse/issues/5876 ), [\#5969](https://github.com/matrix-org/synapse/issues/5969 ), [\#6028](https://github.com/matrix-org/synapse/issues/6028 ))
- Switch to using the v2 Identity Service `/lookup` API where available, with fallback to v1. (Implements [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134 ) plus `id_access_token authentication` for v2 Identity Service APIs from [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140 )). ([\#5897](https://github.com/matrix-org/synapse/issues/5897 ))
- Remove `bind_email` and `bind_msisdn` parameters from `/register` ala [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140 ). ([\#5964](https://github.com/matrix-org/synapse/issues/5964 ))
- Add `m.id_access_token` to `unstable_features` in `/versions` as per [MSC2264](https://github.com/matrix-org/matrix-doc/pull/2264 ). ([\#5974](https://github.com/matrix-org/synapse/issues/5974 ))
- Use the v2 Identity Service API for 3PID invites. ([\#5979](https://github.com/matrix-org/synapse/issues/5979 ))
- Add `POST /_matrix/client/unstable/account/3pid/unbind` endpoint from [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140 ) for unbinding a 3PID from an identity server without removing it from the homeserver user account. ([\#5980](https://github.com/matrix-org/synapse/issues/5980 ), [\#6062](https://github.com/matrix-org/synapse/issues/6062 ))
- Use `account_threepid_delegate.email` and `account_threepid_delegate.msisdn` for validating threepid sessions. ([\#6011](https://github.com/matrix-org/synapse/issues/6011 ))
- Allow homeserver to handle or delegate email validation when adding an email to a user's account. ([\#6042](https://github.com/matrix-org/synapse/issues/6042 ))
- Implement new Client Server API endpoints `/account/3pid/add` and `/account/3pid/bind` as per [MSC2290](https://github.com/matrix-org/matrix-doc/pull/2290 ). ([\#6043](https://github.com/matrix-org/synapse/issues/6043 ))
- Add an unstable feature flag for separate add/bind 3pid APIs. ([\#6044](https://github.com/matrix-org/synapse/issues/6044 ))
- Remove `bind` parameter from Client Server POST `/account` endpoint as per [MSC2290](https://github.com/matrix-org/matrix-doc/pull/2290/ ). ([\#6067](https://github.com/matrix-org/synapse/issues/6067 ))
- Add `POST /add_threepid/msisdn/submit_token` endpoint for proxying submitToken on an `account_threepid_handler`. ([\#6078](https://github.com/matrix-org/synapse/issues/6078 ))
- Add `submit_url` response parameter to `*/msisdn/requestToken` endpoints. ([\#6079](https://github.com/matrix-org/synapse/issues/6079 ))
- Add `m.require_identity_server` flag to /version's unstable_features. ([\#5972](https://github.com/matrix-org/synapse/issues/5972 ))
- Enhancements to OpenTracing support:
- Make OpenTracing work in worker mode. ([\#5771](https://github.com/matrix-org/synapse/issues/5771 ))
- Pass OpenTracing contexts between servers when transmitting EDUs. ([\#5852](https://github.com/matrix-org/synapse/issues/5852 ))
- OpenTracing for device list updates. ([\#5853](https://github.com/matrix-org/synapse/issues/5853 ))
- Add a tag recording a request's authenticated entity and corresponding servlet in OpenTracing. ([\#5856](https://github.com/matrix-org/synapse/issues/5856 ))
- Add minimum OpenTracing for client servlets. ([\#5983](https://github.com/matrix-org/synapse/issues/5983 ))
- Check at setup that OpenTracing is installed if it's enabled in the config. ([\#5985](https://github.com/matrix-org/synapse/issues/5985 ))
- Trace replication send times. ([\#5986](https://github.com/matrix-org/synapse/issues/5986 ))
- Include missing OpenTracing contexts in outbout replication requests. ([\#5982](https://github.com/matrix-org/synapse/issues/5982 ))
- Fix sending of EDUs when OpenTracing is enabled with an empty whitelist. ([\#5984](https://github.com/matrix-org/synapse/issues/5984 ))
- Fix invalid references to None while OpenTracing if the log context slips. ([\#5988](https://github.com/matrix-org/synapse/issues/5988 ), [\#5991](https://github.com/matrix-org/synapse/issues/5991 ))
- OpenTracing for room and e2e keys. ([\#5855](https://github.com/matrix-org/synapse/issues/5855 ))
- Add OpenTracing span over HTTP push processing. ([\#6003](https://github.com/matrix-org/synapse/issues/6003 ))
- Add an admin API to purge old rooms from the database. ([\#5845](https://github.com/matrix-org/synapse/issues/5845 ))
- Retry well-known lookups if we have recently seen a valid well-known record for the server. ([\#5850](https://github.com/matrix-org/synapse/issues/5850 ))
- Add support for filtered room-directory search requests over federation ([MSC2197](https://github.com/matrix-org/matrix-doc/pull/2197 ), in order to allow upcoming room directory query performance improvements. ([\#5859](https://github.com/matrix-org/synapse/issues/5859 ))
- Correctly retry all hosts returned from SRV when we fail to connect. ([\#5864](https://github.com/matrix-org/synapse/issues/5864 ))
- Add admin API endpoint for setting whether or not a user is a server administrator. ([\#5878](https://github.com/matrix-org/synapse/issues/5878 ))
- Enable cleaning up extremities with dummy events by default to prevent undue build up of forward extremities. ([\#5884](https://github.com/matrix-org/synapse/issues/5884 ))
- Add config option to sign remote key query responses with a separate key. ([\#5895](https://github.com/matrix-org/synapse/issues/5895 ))
- Add support for config templating. ([\#5900](https://github.com/matrix-org/synapse/issues/5900 ))
- Users with the type of "support" or "bot" are no longer required to consent. ([\#5902](https://github.com/matrix-org/synapse/issues/5902 ))
- Let synctl accept a directory of config files. ([\#5904](https://github.com/matrix-org/synapse/issues/5904 ))
- Increase max display name size to 256. ([\#5906](https://github.com/matrix-org/synapse/issues/5906 ))
- Add admin API endpoint for getting whether or not a user is a server administrator. ([\#5914](https://github.com/matrix-org/synapse/issues/5914 ))
- Redact events in the database that have been redacted for a week. ([\#5934](https://github.com/matrix-org/synapse/issues/5934 ))
- New prometheus metrics:
- `synapse_federation_known_servers`: represents the total number of servers your server knows about (i.e. is in rooms with), including itself. Enable by setting `metrics_flags.known_servers` to True in the configuration.([\#5981](https://github.com/matrix-org/synapse/issues/5981 ))
- `synapse_build_info`: exposes the Python version, OS version, and Synapse version of the running server. ([\#6005](https://github.com/matrix-org/synapse/issues/6005 ))
- Give appropriate exit codes when synctl fails. ([\#5992](https://github.com/matrix-org/synapse/issues/5992 ))
- Apply the federation blacklist to requests to identity servers. ([\#6000](https://github.com/matrix-org/synapse/issues/6000 ))
- Add `report_stats_endpoint` option to configure where stats are reported to, if enabled. Contributed by @Sorunome. ([\#6012](https://github.com/matrix-org/synapse/issues/6012 ))
- Add config option to increase ratelimits for room admins redacting messages. ([\#6015](https://github.com/matrix-org/synapse/issues/6015 ))
- Stop sending federation transactions to servers which have been down for a long time. ([\#6026](https://github.com/matrix-org/synapse/issues/6026 ))
- Make the process for mapping SAML2 users to matrix IDs more flexible. ([\#6037](https://github.com/matrix-org/synapse/issues/6037 ))
- Return a clearer error message when a timeout occurs when attempting to contact an identity server. ([\#6073](https://github.com/matrix-org/synapse/issues/6073 ))
- Prevent password reset's submit_token endpoint from accepting trailing slashes. ([\#6074](https://github.com/matrix-org/synapse/issues/6074 ))
- Return 403 on `/register/available` if registration has been disabled. ([\#6082](https://github.com/matrix-org/synapse/issues/6082 ))
- Explicitly log when a homeserver does not have the `trusted_key_servers` config field configured. ([\#6090](https://github.com/matrix-org/synapse/issues/6090 ))
- Add support for pruning old rows in `user_ips` table. ([\#6098](https://github.com/matrix-org/synapse/issues/6098 ))
Bugfixes
--------
- Don't create broken room when `power_level_content_override.users` does not contain `creator_id`. ([\#5633](https://github.com/matrix-org/synapse/issues/5633 ))
- Fix database index so that different backup versions can have the same sessions. ([\#5857](https://github.com/matrix-org/synapse/issues/5857 ))
- Fix Synapse looking for config options `password_reset_failure_template` and `password_reset_success_template`, when they are actually `password_reset_template_failure_html`, `password_reset_template_success_html`. ([\#5863](https://github.com/matrix-org/synapse/issues/5863 ))
- Fix stack overflow when recovering an appservice which had an outage. ([\#5885](https://github.com/matrix-org/synapse/issues/5885 ))
- Fix error message which referred to `public_base_url` instead of `public_baseurl`. Thanks to @aaronraimist for the fix! ([\#5909](https://github.com/matrix-org/synapse/issues/5909 ))
- Fix 404 for thumbnail download when `dynamic_thumbnails` is `false` and the thumbnail was dynamically generated. Fix reported by rkfg. ([\#5915](https://github.com/matrix-org/synapse/issues/5915 ))
- Fix a cache-invalidation bug for worker-based deployments. ([\#5920](https://github.com/matrix-org/synapse/issues/5920 ))
- Fix admin API for listing media in a room not being available with an external media repo. ([\#5966](https://github.com/matrix-org/synapse/issues/5966 ))
- Fix list media admin API always returning an error. ([\#5967](https://github.com/matrix-org/synapse/issues/5967 ))
- Fix room and user stats tracking. ([\#5971](https://github.com/matrix-org/synapse/issues/5971 ), [\#5998](https://github.com/matrix-org/synapse/issues/5998 ), [\#6029](https://github.com/matrix-org/synapse/issues/6029 ))
- Return a `M_MISSING_PARAM` if `sid` is not provided to `/account/3pid`. ([\#5995](https://github.com/matrix-org/synapse/issues/5995 ))
- `federation_certificate_verification_whitelist` now will not cause `TypeErrors` to be raised (a regression in 1.3). Additionally, it now supports internationalised domain names in their non-canonical representation. ([\#5996](https://github.com/matrix-org/synapse/issues/5996 ))
- Only count real users when checking for auto-creation of auto-join room. ([\#6004](https://github.com/matrix-org/synapse/issues/6004 ))
- Ensure support users can be registered even if MAU limit is reached. ([\#6020](https://github.com/matrix-org/synapse/issues/6020 ))
- Fix bug where login error was shown incorrectly on SSO fallback login. ([\#6024](https://github.com/matrix-org/synapse/issues/6024 ))
- Fix bug in calculating the federation retry backoff period. ([\#6025](https://github.com/matrix-org/synapse/issues/6025 ))
- Prevent exceptions being logged when extremity-cleanup events fail due to lack of user consent to the terms of service. ([\#6053](https://github.com/matrix-org/synapse/issues/6053 ))
- Remove POST method from password-reset `submit_token` endpoint until we implement `submit_url` functionality. ([\#6056](https://github.com/matrix-org/synapse/issues/6056 ))
- Fix logcontext spam on non-Linux platforms. ([\#6059](https://github.com/matrix-org/synapse/issues/6059 ))
- Ensure query parameters in email validation links are URL-encoded. ([\#6063](https://github.com/matrix-org/synapse/issues/6063 ))
- Fix a bug which caused SAML attribute maps to be overridden by defaults. ([\#6069](https://github.com/matrix-org/synapse/issues/6069 ))
- Fix the logged number of updated items for the `users_set_deactivated_flag` background update. ([\#6092](https://github.com/matrix-org/synapse/issues/6092 ))
- Add `sid` to `next_link` for email validation. ([\#6097](https://github.com/matrix-org/synapse/issues/6097 ))
- Threepid validity checks on msisdns should not be dependent on `threepid_behaviour_email`. ([\#6104](https://github.com/matrix-org/synapse/issues/6104 ))
- Ensure that servers which are not configured to support email address verification do not offer it in the registration flows. ([\#6107](https://github.com/matrix-org/synapse/issues/6107 ))
Updates to the Docker image
---------------------------
- Avoid changing `UID/GID` if they are already correct. ([\#5970](https://github.com/matrix-org/synapse/issues/5970 ))
- Provide `SYNAPSE_WORKER` envvar to specify python module. ([\#6058](https://github.com/matrix-org/synapse/issues/6058 ))
Improved Documentation
----------------------
- Convert documentation to markdown (from rst) ([\#5849](https://github.com/matrix-org/synapse/issues/5849 ))
- Update `INSTALL.md` to say that Python 2 is no longer supported. ([\#5953](https://github.com/matrix-org/synapse/issues/5953 ))
- Add developer documentation for using SAML2. ([\#6032](https://github.com/matrix-org/synapse/issues/6032 ))
- Add some notes on rolling back to v1.3.1. ([\#6049](https://github.com/matrix-org/synapse/issues/6049 ))
- Update the upgrade notes. ([\#6050](https://github.com/matrix-org/synapse/issues/6050 ))
Deprecations and Removals
-------------------------
- Remove shared-secret registration from `/_matrix/client/r0/register` endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5877](https://github.com/matrix-org/synapse/issues/5877 ))
- Deprecate the `trusted_third_party_id_servers` option. ([\#5875](https://github.com/matrix-org/synapse/issues/5875 ))
Internal Changes
----------------
- Lay the groundwork for structured logging output. ([\#5680](https://github.com/matrix-org/synapse/issues/5680 ))
- Retry well-known lookup before the cache expires, giving a grace period where the remote well-known can be down but we still use the old result. ([\#5844](https://github.com/matrix-org/synapse/issues/5844 ))
- Remove log line for debugging issue #5407 . ([\#5860](https://github.com/matrix-org/synapse/issues/5860 ))
- Refactor the Appservice scheduler code. ([\#5886](https://github.com/matrix-org/synapse/issues/5886 ))
- Compatibility with v2 Identity Service APIs other than /lookup. ([\#5892](https://github.com/matrix-org/synapse/issues/5892 ), [\#6013](https://github.com/matrix-org/synapse/issues/6013 ))
- Stop populating some unused tables. ([\#5893](https://github.com/matrix-org/synapse/issues/5893 ), [\#6047](https://github.com/matrix-org/synapse/issues/6047 ))
- Add missing index on `users_in_public_rooms` to improve the performance of directory queries. ([\#5894](https://github.com/matrix-org/synapse/issues/5894 ))
- Improve the logging when we have an error when fetching signing keys. ([\#5896](https://github.com/matrix-org/synapse/issues/5896 ))
- Add support for database engine-specific schema deltas, based on file extension. ([\#5911](https://github.com/matrix-org/synapse/issues/5911 ))
- Update Buildkite pipeline to use plugins instead of buildkite-agent commands. ([\#5922](https://github.com/matrix-org/synapse/issues/5922 ))
- Add link in sample config to the logging config schema. ([\#5926](https://github.com/matrix-org/synapse/issues/5926 ))
- Remove unnecessary parentheses in return statements. ([\#5931](https://github.com/matrix-org/synapse/issues/5931 ))
- Remove unused `jenkins/prepare_sytest.sh` file. ([\#5938](https://github.com/matrix-org/synapse/issues/5938 ))
- Move Buildkite pipeline config to the pipelines repo. ([\#5943](https://github.com/matrix-org/synapse/issues/5943 ))
- Remove unnecessary return statements in the codebase which were the result of a regex run. ([\#5962](https://github.com/matrix-org/synapse/issues/5962 ))
- Remove left-over methods from v1 registration API. ([\#5963](https://github.com/matrix-org/synapse/issues/5963 ))
- Cleanup event auth type initialisation. ([\#5975](https://github.com/matrix-org/synapse/issues/5975 ))
- Clean up dependency checking at setup. ([\#5989](https://github.com/matrix-org/synapse/issues/5989 ))
- Update OpenTracing docs to use the unified `trace` method. ([\#5776](https://github.com/matrix-org/synapse/issues/5776 ))
- Small refactor of function arguments and docstrings in` RoomMemberHandler`. ([\#6009](https://github.com/matrix-org/synapse/issues/6009 ))
- Remove unused `origin` argument on `FederationHandler.add_display_name_to_third_party_invite`. ([\#6010](https://github.com/matrix-org/synapse/issues/6010 ))
- Add a `failure_ts` column to the `destinations` database table. ([\#6016](https://github.com/matrix-org/synapse/issues/6016 ), [\#6072](https://github.com/matrix-org/synapse/issues/6072 ))
- Clean up some code in the retry logic. ([\#6017](https://github.com/matrix-org/synapse/issues/6017 ))
- Fix the structured logging tests stomping on the global log configuration for subsequent tests. ([\#6023](https://github.com/matrix-org/synapse/issues/6023 ))
- Clean up the sample config for SAML authentication. ([\#6064](https://github.com/matrix-org/synapse/issues/6064 ))
- Change mailer logging to reflect Synapse doesn't just do chat notifications by email now. ([\#6075](https://github.com/matrix-org/synapse/issues/6075 ))
- Move last-seen info into devices table. ([\#6089](https://github.com/matrix-org/synapse/issues/6089 ))
- Remove unused parameter to `get_user_id_by_threepid`. ([\#6099](https://github.com/matrix-org/synapse/issues/6099 ))
- Refactor the user-interactive auth handling. ([\#6105](https://github.com/matrix-org/synapse/issues/6105 ))
- Refactor code for calculating registration flows. ([\#6106](https://github.com/matrix-org/synapse/issues/6106 ))
-----BEGIN PGP SIGNATURE-----
iQJHBAABCgAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAl2V6NsTHGFuZHJld0Bh
bW9yZ2FuLnh5egAKCRCIhIgNLv5f9IUTEACE/gPcItzKAOsnEjCyBVzjwXcvqdZU
5FpEbnDqhVS20GBOctl+00QqHOeje8tDLmYy4yS/LH89iX+PQNI55M/4bmS5ZTYm
x13AsxgqJ11xDOPmkzLkXDF0HnzZngGeGSShsPFUUWfMF9pcCz+kMY88dJe69sfU
TK/VGIvp4MAcbeIbtgbInS7ZH1igntBLyfp48VQ0WCz6HeorJSxniRDKpRQ08Mc3
TbZsFZ/2ajvL+HSr6iSDUD32GTEfUERVpV7OrQnC9laa0Aidq0yRCMcRpBWJ8omP
g/240jDxsbSZkNe+6GPuiRjqFQuHw0KVb/A10BJRvEDCgVzIdsE5oEQLr3HKl2h3
q1HQewuc2tG/ns4vMHK+dhEVlc95yUI1UxrNafP0/hhpWRiRX1VfiWv08tKNP5GJ
7an7Oqah4yZWjyzVBhiMugmQnnaf2z5TymUaRcZRNfueTpUo4DfJ3as71JcOmIGc
Hg7gM1DOkLUMV5xoieW+7dKUqO8Nqv8y+q7qbK3LRT83HeE3AHXf3JmfB7/6+zi0
4coM5ZrGrp5A4PeGDVKPOT3VdmXK8r6buPqcJxUMaWAMziWGAEWROrGF+khvpUlW
NkLAF/XitjqIlDQEWpUzHP7vGXNt9m83/9J8wD4/ZAjcUDfW1KSSCbBmCR4s5ZAk
HVzNubTzKy2xlA==
=2TFh
-----END PGP SIGNATURE-----
Merge tag 'v1.4.0'
Synapse 1.4.0 (2019-10-03)
==========================
Bugfixes
--------
- Redact `client_secret` in server logs. ([\#6158](https://github.com/matrix-org/synapse/issues/6158 ))
Synapse 1.4.0rc2 (2019-10-02)
=============================
Bugfixes
--------
- Fix bug in background update that adds last seen information to the `devices` table, and improve its performance on Postgres. ([\#6135](https://github.com/matrix-org/synapse/issues/6135 ))
- Fix bad performance of censoring redactions background task. ([\#6141](https://github.com/matrix-org/synapse/issues/6141 ))
- Fix fetching censored redactions from DB, which caused APIs like initial sync to fail if it tried to include the censored redaction. ([\#6145](https://github.com/matrix-org/synapse/issues/6145 ))
- Fix exceptions when storing large retry intervals for down remote servers. ([\#6146](https://github.com/matrix-org/synapse/issues/6146 ))
Internal Changes
----------------
- Fix up sample config entry for `redaction_retention_period` option. ([\#6117](https://github.com/matrix-org/synapse/issues/6117 ))
Synapse 1.4.0rc1 (2019-09-26)
=============================
Note that this release includes significant changes around 3pid
verification. Administrators are reminded to review the [upgrade notes](UPGRADE.rst#upgrading-to-v140).
Features
--------
- Changes to 3pid verification:
- Add the ability to send registration emails from the homeserver rather than delegating to an identity server. ([\#5835](https://github.com/matrix-org/synapse/issues/5835 ), [\#5940](https://github.com/matrix-org/synapse/issues/5940 ), [\#5993](https://github.com/matrix-org/synapse/issues/5993 ), [\#5994](https://github.com/matrix-org/synapse/issues/5994 ), [\#5868](https://github.com/matrix-org/synapse/issues/5868 ))
- Replace `trust_identity_server_for_password_resets` config option with `account_threepid_delegates`, and make the `id_server` parameteter optional on `*/requestToken` endpoints, as per [MSC2263](https://github.com/matrix-org/matrix-doc/pull/2263 ). ([\#5876](https://github.com/matrix-org/synapse/issues/5876 ), [\#5969](https://github.com/matrix-org/synapse/issues/5969 ), [\#6028](https://github.com/matrix-org/synapse/issues/6028 ))
- Switch to using the v2 Identity Service `/lookup` API where available, with fallback to v1. (Implements [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134 ) plus `id_access_token authentication` for v2 Identity Service APIs from [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140 )). ([\#5897](https://github.com/matrix-org/synapse/issues/5897 ))
- Remove `bind_email` and `bind_msisdn` parameters from `/register` ala [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140 ). ([\#5964](https://github.com/matrix-org/synapse/issues/5964 ))
- Add `m.id_access_token` to `unstable_features` in `/versions` as per [MSC2264](https://github.com/matrix-org/matrix-doc/pull/2264 ). ([\#5974](https://github.com/matrix-org/synapse/issues/5974 ))
- Use the v2 Identity Service API for 3PID invites. ([\#5979](https://github.com/matrix-org/synapse/issues/5979 ))
- Add `POST /_matrix/client/unstable/account/3pid/unbind` endpoint from [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140 ) for unbinding a 3PID from an identity server without removing it from the homeserver user account. ([\#5980](https://github.com/matrix-org/synapse/issues/5980 ), [\#6062](https://github.com/matrix-org/synapse/issues/6062 ))
- Use `account_threepid_delegate.email` and `account_threepid_delegate.msisdn` for validating threepid sessions. ([\#6011](https://github.com/matrix-org/synapse/issues/6011 ))
- Allow homeserver to handle or delegate email validation when adding an email to a user's account. ([\#6042](https://github.com/matrix-org/synapse/issues/6042 ))
- Implement new Client Server API endpoints `/account/3pid/add` and `/account/3pid/bind` as per [MSC2290](https://github.com/matrix-org/matrix-doc/pull/2290 ). ([\#6043](https://github.com/matrix-org/synapse/issues/6043 ))
- Add an unstable feature flag for separate add/bind 3pid APIs. ([\#6044](https://github.com/matrix-org/synapse/issues/6044 ))
- Remove `bind` parameter from Client Server POST `/account` endpoint as per [MSC2290](https://github.com/matrix-org/matrix-doc/pull/2290/ ). ([\#6067](https://github.com/matrix-org/synapse/issues/6067 ))
- Add `POST /add_threepid/msisdn/submit_token` endpoint for proxying submitToken on an `account_threepid_handler`. ([\#6078](https://github.com/matrix-org/synapse/issues/6078 ))
- Add `submit_url` response parameter to `*/msisdn/requestToken` endpoints. ([\#6079](https://github.com/matrix-org/synapse/issues/6079 ))
- Add `m.require_identity_server` flag to /version's unstable_features. ([\#5972](https://github.com/matrix-org/synapse/issues/5972 ))
- Enhancements to OpenTracing support:
- Make OpenTracing work in worker mode. ([\#5771](https://github.com/matrix-org/synapse/issues/5771 ))
- Pass OpenTracing contexts between servers when transmitting EDUs. ([\#5852](https://github.com/matrix-org/synapse/issues/5852 ))
- OpenTracing for device list updates. ([\#5853](https://github.com/matrix-org/synapse/issues/5853 ))
- Add a tag recording a request's authenticated entity and corresponding servlet in OpenTracing. ([\#5856](https://github.com/matrix-org/synapse/issues/5856 ))
- Add minimum OpenTracing for client servlets. ([\#5983](https://github.com/matrix-org/synapse/issues/5983 ))
- Check at setup that OpenTracing is installed if it's enabled in the config. ([\#5985](https://github.com/matrix-org/synapse/issues/5985 ))
- Trace replication send times. ([\#5986](https://github.com/matrix-org/synapse/issues/5986 ))
- Include missing OpenTracing contexts in outbout replication requests. ([\#5982](https://github.com/matrix-org/synapse/issues/5982 ))
- Fix sending of EDUs when OpenTracing is enabled with an empty whitelist. ([\#5984](https://github.com/matrix-org/synapse/issues/5984 ))
- Fix invalid references to None while OpenTracing if the log context slips. ([\#5988](https://github.com/matrix-org/synapse/issues/5988 ), [\#5991](https://github.com/matrix-org/synapse/issues/5991 ))
- OpenTracing for room and e2e keys. ([\#5855](https://github.com/matrix-org/synapse/issues/5855 ))
- Add OpenTracing span over HTTP push processing. ([\#6003](https://github.com/matrix-org/synapse/issues/6003 ))
- Add an admin API to purge old rooms from the database. ([\#5845](https://github.com/matrix-org/synapse/issues/5845 ))
- Retry well-known lookups if we have recently seen a valid well-known record for the server. ([\#5850](https://github.com/matrix-org/synapse/issues/5850 ))
- Add support for filtered room-directory search requests over federation ([MSC2197](https://github.com/matrix-org/matrix-doc/pull/2197 ), in order to allow upcoming room directory query performance improvements. ([\#5859](https://github.com/matrix-org/synapse/issues/5859 ))
- Correctly retry all hosts returned from SRV when we fail to connect. ([\#5864](https://github.com/matrix-org/synapse/issues/5864 ))
- Add admin API endpoint for setting whether or not a user is a server administrator. ([\#5878](https://github.com/matrix-org/synapse/issues/5878 ))
- Enable cleaning up extremities with dummy events by default to prevent undue build up of forward extremities. ([\#5884](https://github.com/matrix-org/synapse/issues/5884 ))
- Add config option to sign remote key query responses with a separate key. ([\#5895](https://github.com/matrix-org/synapse/issues/5895 ))
- Add support for config templating. ([\#5900](https://github.com/matrix-org/synapse/issues/5900 ))
- Users with the type of "support" or "bot" are no longer required to consent. ([\#5902](https://github.com/matrix-org/synapse/issues/5902 ))
- Let synctl accept a directory of config files. ([\#5904](https://github.com/matrix-org/synapse/issues/5904 ))
- Increase max display name size to 256. ([\#5906](https://github.com/matrix-org/synapse/issues/5906 ))
- Add admin API endpoint for getting whether or not a user is a server administrator. ([\#5914](https://github.com/matrix-org/synapse/issues/5914 ))
- Redact events in the database that have been redacted for a week. ([\#5934](https://github.com/matrix-org/synapse/issues/5934 ))
- New prometheus metrics:
- `synapse_federation_known_servers`: represents the total number of servers your server knows about (i.e. is in rooms with), including itself. Enable by setting `metrics_flags.known_servers` to True in the configuration.([\#5981](https://github.com/matrix-org/synapse/issues/5981 ))
- `synapse_build_info`: exposes the Python version, OS version, and Synapse version of the running server. ([\#6005](https://github.com/matrix-org/synapse/issues/6005 ))
- Give appropriate exit codes when synctl fails. ([\#5992](https://github.com/matrix-org/synapse/issues/5992 ))
- Apply the federation blacklist to requests to identity servers. ([\#6000](https://github.com/matrix-org/synapse/issues/6000 ))
- Add `report_stats_endpoint` option to configure where stats are reported to, if enabled. Contributed by @Sorunome. ([\#6012](https://github.com/matrix-org/synapse/issues/6012 ))
- Add config option to increase ratelimits for room admins redacting messages. ([\#6015](https://github.com/matrix-org/synapse/issues/6015 ))
- Stop sending federation transactions to servers which have been down for a long time. ([\#6026](https://github.com/matrix-org/synapse/issues/6026 ))
- Make the process for mapping SAML2 users to matrix IDs more flexible. ([\#6037](https://github.com/matrix-org/synapse/issues/6037 ))
- Return a clearer error message when a timeout occurs when attempting to contact an identity server. ([\#6073](https://github.com/matrix-org/synapse/issues/6073 ))
- Prevent password reset's submit_token endpoint from accepting trailing slashes. ([\#6074](https://github.com/matrix-org/synapse/issues/6074 ))
- Return 403 on `/register/available` if registration has been disabled. ([\#6082](https://github.com/matrix-org/synapse/issues/6082 ))
- Explicitly log when a homeserver does not have the `trusted_key_servers` config field configured. ([\#6090](https://github.com/matrix-org/synapse/issues/6090 ))
- Add support for pruning old rows in `user_ips` table. ([\#6098](https://github.com/matrix-org/synapse/issues/6098 ))
Bugfixes
--------
- Don't create broken room when `power_level_content_override.users` does not contain `creator_id`. ([\#5633](https://github.com/matrix-org/synapse/issues/5633 ))
- Fix database index so that different backup versions can have the same sessions. ([\#5857](https://github.com/matrix-org/synapse/issues/5857 ))
- Fix Synapse looking for config options `password_reset_failure_template` and `password_reset_success_template`, when they are actually `password_reset_template_failure_html`, `password_reset_template_success_html`. ([\#5863](https://github.com/matrix-org/synapse/issues/5863 ))
- Fix stack overflow when recovering an appservice which had an outage. ([\#5885](https://github.com/matrix-org/synapse/issues/5885 ))
- Fix error message which referred to `public_base_url` instead of `public_baseurl`. Thanks to @aaronraimist for the fix! ([\#5909](https://github.com/matrix-org/synapse/issues/5909 ))
- Fix 404 for thumbnail download when `dynamic_thumbnails` is `false` and the thumbnail was dynamically generated. Fix reported by rkfg. ([\#5915](https://github.com/matrix-org/synapse/issues/5915 ))
- Fix a cache-invalidation bug for worker-based deployments. ([\#5920](https://github.com/matrix-org/synapse/issues/5920 ))
- Fix admin API for listing media in a room not being available with an external media repo. ([\#5966](https://github.com/matrix-org/synapse/issues/5966 ))
- Fix list media admin API always returning an error. ([\#5967](https://github.com/matrix-org/synapse/issues/5967 ))
- Fix room and user stats tracking. ([\#5971](https://github.com/matrix-org/synapse/issues/5971 ), [\#5998](https://github.com/matrix-org/synapse/issues/5998 ), [\#6029](https://github.com/matrix-org/synapse/issues/6029 ))
- Return a `M_MISSING_PARAM` if `sid` is not provided to `/account/3pid`. ([\#5995](https://github.com/matrix-org/synapse/issues/5995 ))
- `federation_certificate_verification_whitelist` now will not cause `TypeErrors` to be raised (a regression in 1.3). Additionally, it now supports internationalised domain names in their non-canonical representation. ([\#5996](https://github.com/matrix-org/synapse/issues/5996 ))
- Only count real users when checking for auto-creation of auto-join room. ([\#6004](https://github.com/matrix-org/synapse/issues/6004 ))
- Ensure support users can be registered even if MAU limit is reached. ([\#6020](https://github.com/matrix-org/synapse/issues/6020 ))
- Fix bug where login error was shown incorrectly on SSO fallback login. ([\#6024](https://github.com/matrix-org/synapse/issues/6024 ))
- Fix bug in calculating the federation retry backoff period. ([\#6025](https://github.com/matrix-org/synapse/issues/6025 ))
- Prevent exceptions being logged when extremity-cleanup events fail due to lack of user consent to the terms of service. ([\#6053](https://github.com/matrix-org/synapse/issues/6053 ))
- Remove POST method from password-reset `submit_token` endpoint until we implement `submit_url` functionality. ([\#6056](https://github.com/matrix-org/synapse/issues/6056 ))
- Fix logcontext spam on non-Linux platforms. ([\#6059](https://github.com/matrix-org/synapse/issues/6059 ))
- Ensure query parameters in email validation links are URL-encoded. ([\#6063](https://github.com/matrix-org/synapse/issues/6063 ))
- Fix a bug which caused SAML attribute maps to be overridden by defaults. ([\#6069](https://github.com/matrix-org/synapse/issues/6069 ))
- Fix the logged number of updated items for the `users_set_deactivated_flag` background update. ([\#6092](https://github.com/matrix-org/synapse/issues/6092 ))
- Add `sid` to `next_link` for email validation. ([\#6097](https://github.com/matrix-org/synapse/issues/6097 ))
- Threepid validity checks on msisdns should not be dependent on `threepid_behaviour_email`. ([\#6104](https://github.com/matrix-org/synapse/issues/6104 ))
- Ensure that servers which are not configured to support email address verification do not offer it in the registration flows. ([\#6107](https://github.com/matrix-org/synapse/issues/6107 ))
Updates to the Docker image
---------------------------
- Avoid changing `UID/GID` if they are already correct. ([\#5970](https://github.com/matrix-org/synapse/issues/5970 ))
- Provide `SYNAPSE_WORKER` envvar to specify python module. ([\#6058](https://github.com/matrix-org/synapse/issues/6058 ))
Improved Documentation
----------------------
- Convert documentation to markdown (from rst) ([\#5849](https://github.com/matrix-org/synapse/issues/5849 ))
- Update `INSTALL.md` to say that Python 2 is no longer supported. ([\#5953](https://github.com/matrix-org/synapse/issues/5953 ))
- Add developer documentation for using SAML2. ([\#6032](https://github.com/matrix-org/synapse/issues/6032 ))
- Add some notes on rolling back to v1.3.1. ([\#6049](https://github.com/matrix-org/synapse/issues/6049 ))
- Update the upgrade notes. ([\#6050](https://github.com/matrix-org/synapse/issues/6050 ))
Deprecations and Removals
-------------------------
- Remove shared-secret registration from `/_matrix/client/r0/register` endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5877](https://github.com/matrix-org/synapse/issues/5877 ))
- Deprecate the `trusted_third_party_id_servers` option. ([\#5875](https://github.com/matrix-org/synapse/issues/5875 ))
Internal Changes
----------------
- Lay the groundwork for structured logging output. ([\#5680](https://github.com/matrix-org/synapse/issues/5680 ))
- Retry well-known lookup before the cache expires, giving a grace period where the remote well-known can be down but we still use the old result. ([\#5844](https://github.com/matrix-org/synapse/issues/5844 ))
- Remove log line for debugging issue #5407 . ([\#5860](https://github.com/matrix-org/synapse/issues/5860 ))
- Refactor the Appservice scheduler code. ([\#5886](https://github.com/matrix-org/synapse/issues/5886 ))
- Compatibility with v2 Identity Service APIs other than /lookup. ([\#5892](https://github.com/matrix-org/synapse/issues/5892 ), [\#6013](https://github.com/matrix-org/synapse/issues/6013 ))
- Stop populating some unused tables. ([\#5893](https://github.com/matrix-org/synapse/issues/5893 ), [\#6047](https://github.com/matrix-org/synapse/issues/6047 ))
- Add missing index on `users_in_public_rooms` to improve the performance of directory queries. ([\#5894](https://github.com/matrix-org/synapse/issues/5894 ))
- Improve the logging when we have an error when fetching signing keys. ([\#5896](https://github.com/matrix-org/synapse/issues/5896 ))
- Add support for database engine-specific schema deltas, based on file extension. ([\#5911](https://github.com/matrix-org/synapse/issues/5911 ))
- Update Buildkite pipeline to use plugins instead of buildkite-agent commands. ([\#5922](https://github.com/matrix-org/synapse/issues/5922 ))
- Add link in sample config to the logging config schema. ([\#5926](https://github.com/matrix-org/synapse/issues/5926 ))
- Remove unnecessary parentheses in return statements. ([\#5931](https://github.com/matrix-org/synapse/issues/5931 ))
- Remove unused `jenkins/prepare_sytest.sh` file. ([\#5938](https://github.com/matrix-org/synapse/issues/5938 ))
- Move Buildkite pipeline config to the pipelines repo. ([\#5943](https://github.com/matrix-org/synapse/issues/5943 ))
- Remove unnecessary return statements in the codebase which were the result of a regex run. ([\#5962](https://github.com/matrix-org/synapse/issues/5962 ))
- Remove left-over methods from v1 registration API. ([\#5963](https://github.com/matrix-org/synapse/issues/5963 ))
- Cleanup event auth type initialisation. ([\#5975](https://github.com/matrix-org/synapse/issues/5975 ))
- Clean up dependency checking at setup. ([\#5989](https://github.com/matrix-org/synapse/issues/5989 ))
- Update OpenTracing docs to use the unified `trace` method. ([\#5776](https://github.com/matrix-org/synapse/issues/5776 ))
- Small refactor of function arguments and docstrings in` RoomMemberHandler`. ([\#6009](https://github.com/matrix-org/synapse/issues/6009 ))
- Remove unused `origin` argument on `FederationHandler.add_display_name_to_third_party_invite`. ([\#6010](https://github.com/matrix-org/synapse/issues/6010 ))
- Add a `failure_ts` column to the `destinations` database table. ([\#6016](https://github.com/matrix-org/synapse/issues/6016 ), [\#6072](https://github.com/matrix-org/synapse/issues/6072 ))
- Clean up some code in the retry logic. ([\#6017](https://github.com/matrix-org/synapse/issues/6017 ))
- Fix the structured logging tests stomping on the global log configuration for subsequent tests. ([\#6023](https://github.com/matrix-org/synapse/issues/6023 ))
- Clean up the sample config for SAML authentication. ([\#6064](https://github.com/matrix-org/synapse/issues/6064 ))
- Change mailer logging to reflect Synapse doesn't just do chat notifications by email now. ([\#6075](https://github.com/matrix-org/synapse/issues/6075 ))
- Move last-seen info into devices table. ([\#6089](https://github.com/matrix-org/synapse/issues/6089 ))
- Remove unused parameter to `get_user_id_by_threepid`. ([\#6099](https://github.com/matrix-org/synapse/issues/6099 ))
- Refactor the user-interactive auth handling. ([\#6105](https://github.com/matrix-org/synapse/issues/6105 ))
- Refactor code for calculating registration flows. ([\#6106](https://github.com/matrix-org/synapse/issues/6106 ))
2019-10-03 13:48:21 +01:00
ecb69d824a
1.4.0
2019-10-03 13:22:44 +01:00
0f46bf5737
Replace client_secret with <redacted> in server logs ( #6158 )
...
Replace `client_secret` query parameter values with `<redacted>` in the logs. Prevents a scenario where a MITM of server traffic can horde 3pids on their account.
2019-10-03 12:57:26 +01:00
8a5e8e829b
Lint (again)
2019-10-03 11:30:43 +01:00
ebcb6a30d7
Lint
2019-10-03 11:29:07 +01:00
6527fa18c1
Add test case
2019-10-03 11:24:36 +01:00
dca7e32d3d
Merge pull request #6154 from matrix-org/erikj/fix_appservice_pagination
...
Fix appservice room list pagination
2019-10-02 16:50:32 +01:00
22a6ffdf91
Merge pull request #6153 from matrix-org/erikj/fix_room_list_non_federatable
...
Fix not showing non-federatable rooms to remote room list queries
2019-10-02 16:09:15 +01:00
5d8ffdfe61
Merge pull request #6148 from matrix-org/erikj/find_next_generated
...
Bound find_next_generated_user_id DB query.
2019-10-02 16:09:02 +01:00
5be4083306
Newsfile
2019-10-02 15:48:36 +01:00
7a5f080f91
Fix appservice room list pagination
2019-10-02 15:47:22 +01:00
5b9e5c27d8
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fix_room_list_non_federatable
2019-10-02 15:38:54 +01:00
824df3ead4
Merge pull request #6152 from matrix-org/erikj/fix_room_list
...
Fix public room list pagination.
2019-10-02 15:38:05 +01:00
ed73f04bef
Newsfile
2019-10-02 15:24:33 +01:00
4c4f44930d
Fix not showing non-federatable rooms to remote room list queries
2019-10-02 15:20:36 +01:00
8e32240e6b
Newsfile
2019-10-02 15:12:17 +01:00
03cf4385e0
Fix public room list pagination.
...
We incorrectly used `room_id` as to bound the result set, even though we
order by `joined_members, room_id`, leading to incorrect results after
pagination.
2019-10-02 15:11:17 +01:00
baf12bc02a
Merge branch 'master' into develop
2019-10-02 14:41:02 +01:00
a5166e4d5f
Land improved room list based on room stats ( #6019 )
...
Use room_stats and room_state for room directory search
2019-10-02 14:08:35 +01:00
aec1377d0b
1.4.0rc2
2019-10-02 13:55:00 +01:00
864f144543
Fix up some typechecking ( #6150 )
...
* type checking fixes
* changelog
2019-10-02 05:29:01 -07:00
24efea338d
Changelog
2019-10-02 12:20:03 +01:00
972c9f65d7
Lint
2019-10-02 12:17:46 +01:00
2a1470cd05
Fix yields and copy instead of move push rules on room upgrade ( #6144 )
...
Copy push rules during a room upgrade from the old room to the new room, instead of deleting them from the old room.
For instance, we've defined upgrading of a room multiple times to be possible, and push rules won't be transferred on the second upgrade if they're deleted during the first.
Also fix some missing yields that probably broke things quite a bit.
2019-10-02 12:04:22 +01:00
de1823b521
Newsfile
2019-10-02 11:45:31 +01:00
Richard van der Hoff