Commit Graph

21394 Commits

Author SHA1 Message Date
David Robertson 1a1abdda42
Last batch of Pydantic for synapse/rest/client/account.py (#13832)
* Validation for `/add_threepid/msisdn/submit_token`

* Don't validate deprecated endpoint

* Changelog
2022-09-21 22:23:44 +01:00
Brendan Abolivier efabf44c76
Add version flag for MSC3881 (#13860) 2022-09-21 18:18:44 +02:00
Brendan Abolivier ccca14140a
Track device IDs for pushers (#13831)
Second half of the MSC3881 implementation
2022-09-21 15:31:53 +00:00
Hugh Nimmo-Smith 0fd2f2d460
Implementation of MSC3882 login token request (#13722) 2022-09-21 15:12:29 +00:00
villepeh 269eddad6f
Add `worker_main_http_uri` to the contrib bash script (#13772)
* Add worker_main_http_uri, replace >> with >

Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com>
Co-authored-by: Erik Johnston <erik@matrix.org>
2022-09-21 15:58:46 +01:00
Brendan Abolivier 8ae42ab8fa
Support enabling/disabling pushers (from MSC3881) (#13799)
Partial implementation of MSC3881
2022-09-21 14:39:01 +00:00
Mathieu Velten 6bd8763804
Add cache invalidation across workers to module API (#13667)
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
2022-09-21 15:32:01 +02:00
Peter Scheu 16e1a9d9a7
Correct documentation for map_user_attributes of OpenID Mapping Providers (#13836)
Co-authored-by: David Robertson <davidr@element.io>
2022-09-21 13:08:16 +00:00
Quentin Gliech e0804ef898
Improve the `synapse.api.auth.Auth` mock used in unit tests. (#13809)
To return the proper type (`Requester`) instead of a `dict`.
2022-09-21 12:40:34 +00:00
reivilibre a35842caec
Fix the release script not publishing binary wheels. (#13850) 2022-09-21 09:43:08 +00:00
Olivier Wilkinson (reivilibre) 2b522cceb6 Merge branch 'release-v1.68' into develop 2022-09-20 14:54:05 +01:00
Quentin Gliech 85fc7ea1a1
Remove the `complete_sso_login` method from the Module API which was deprecated in Synapse 1.13.0. (#13843)
Signed-off-by: Quentin Gliech <quenting@element.io>
2022-09-20 15:18:07 +02:00
David Robertson fff9b955fa
Generate separate snapshots for logical databases (#13792)
* Generate separate snapshots for sqlite, postgres and common
* Cleanup postgres dbs in the TRAP
* Say which logical DB we're applying updates to
* Run background updates on the state DB
* Add new option for accepting a SCHEMA_NUMBER
2022-09-20 14:14:12 +01:00
Olivier Wilkinson (reivilibre) cfe486b041 Tweak changelog 2022-09-20 12:21:59 +01:00
Olivier Wilkinson (reivilibre) ec4dca6064 Edit description of cache invalidation bug 2022-09-20 12:11:45 +01:00
Erik Johnston 42d261c32f
Port the push rule classes to Rust. (#13768) 2022-09-20 12:10:31 +01:00
Olivier Wilkinson (reivilibre) bffb71b04a Edit changelog in response to feedback 2022-09-20 12:05:15 +01:00
Olivier Wilkinson (reivilibre) e124b24ad9 1.68.0rc1 2022-09-20 11:30:48 +01:00
Denis c802ef1411
Don't include redundant prev_state in new events (#13791) 2022-09-20 09:44:38 +01:00
Eric Eastwood 44be42338e
Add support to purge rows from MSC2716 and other tables when purging a room (#13825)
`event_failed_pull_attempts` added in https://github.com/matrix-org/synapse/pull/13589

MSC2716 related tables added in:

 - https://github.com/matrix-org/synapse/pull/10245/files#diff-3d42dfb44d02f7de3aada105e0bdc1cc9dd7f953cbf0f36c5d0f50827bf0320aR1
    - Renamed in https://github.com/matrix-org/synapse/pull/10838/files#diff-2730bfbe9e688b55e46f9371aefe67dac2bd2b2b7d9d6b92774eea1fcfae156dR1
 - https://github.com/matrix-org/synapse/pull/10498/files#diff-c52bbfbb5921a3f6f023b24343668479d966fac164f13b7c39d2197ce3afa7a5R1
2022-09-16 10:56:56 -05:00
Mathieu Velten d5292b8017
Fix Docker build when Rust .so has been build locally first (#13811)
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
2022-09-16 15:38:54 +00:00
David Robertson 642c4b253d
Compare ported to unported PG schemas in portdb test job (#13808) 2022-09-16 16:25:54 +01:00
David Robertson 5e84461653
Minor speedups to CI linting (#13827) 2022-09-16 16:18:32 +01:00
Sean Quah d64e85197a
Remove error spam when users query the keys of departed remote users (#13826)
The error message introduced in #13749 has turned out to be very spammy.
Remove it for now.
2022-09-16 16:16:05 +01:00
Mathieu Velten 384dca53d6
complement: init postgres DB directly inside the target image (#13819)
Doing so in the base postgres image doesn't work with buildah because
changes in a declared VOLUME in the Dockerfile is supposed to be
discarded, cf https://docs.docker.com/engine/reference/builder/#volume

Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
2022-09-16 17:12:45 +02:00
Quentin Gliech 74f60cec92
Add an admin API endpoint to find a user based on its external ID in an auth provider. (#13810) 2022-09-16 12:29:03 +00:00
reivilibre f7a77ad717
Update request log format documentation to mention the format used when the authenticated user is controlling another user. (#13794) 2022-09-16 11:48:41 +00:00
Sean Quah b73cbb8215
Avoid putting rejected events in room state (#13723)
Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-16 12:45:04 +01:00
Eric Eastwood 6986bcbf39
Document common fix of Poetry problems by removing egg-info (#13785)
`matrix_synapse.egg-info/`

Mentioned at https://matrix.to/#/!vcyiEtMVHIhWXcJAfl:sw1v.org/$aKy_IjrKwb70aTVZWeW_6zt0k7OIZ1YkyZpkP9uiRaM?via=matrix.org&via=element.io&via=beeper.com and many other places.
2022-09-15 16:28:03 -05:00
Eric Eastwood 5093cbf88d
Be able to correlate timeouts in reverse-proxy layer in front of Synapse (pull request ID from header) (#13801)
Fix https://github.com/matrix-org/synapse/issues/13685

New config:

```diff
  listeners:
    - port: 8008
      tls: false
      type: http
      x_forwarded: true
+     request_id_header: "cf-ray"
      bind_addresses: ['::1', '127.0.0.1', '0.0.0.0']
```
2022-09-15 15:32:25 -05:00
Eric Eastwood 140af0cdb6
Record any exception when processing a pulled event (#13814)
Part of https://github.com/matrix-org/synapse/issues/13700 and https://github.com/matrix-org/synapse/issues/13356

Follow-up to https://github.com/matrix-org/synapse/pull/13589
2022-09-15 14:40:49 -05:00
Patrick Cloke b2b0c85279
Support providing an index predicate for upserts. (#13822)
This is useful to upsert against a table which has a unique
partial index while avoiding conflicts.
2022-09-15 18:28:48 +00:00
David Robertson 742f9f9d78
A third batch of Pydantic validation for rest/client/account.py (#13736) 2022-09-15 18:36:02 +01:00
Andrew Morgan 918c74bfb5
Add a `MXCUri` class to make working with mxc uri's easier. (#13162) 2022-09-15 12:57:16 +00:00
Eric Eastwood 957e3d74fc
Keep track when we try and fail to process a pulled event (#13589)
We can follow-up this PR with:

 1. Only try to backfill from an event if we haven't tried recently -> https://github.com/matrix-org/synapse/issues/13622
 1. When we decide to backfill that event again, process it in the background so it doesn't block and make `/messages` slow when we know it will probably fail again -> https://github.com/matrix-org/synapse/issues/13623
 1. Generally track failures everywhere we try and fail to pull an event over federation -> https://github.com/matrix-org/synapse/issues/13700

Fix https://github.com/matrix-org/synapse/issues/13621

Part of https://github.com/matrix-org/synapse/issues/13356

Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.qv7cj51sv9i5)
2022-09-14 13:57:50 -05:00
Patrick Cloke 666ae87729
Update event push action and receipt tables to support threads. (#13753)
Adds a `thread_id` column to the `event_push_actions`, `event_push_actions_staging`,
and `event_push_summary` tables. This will notifications to be segmented by the thread
in a future pull request. The `thread_id` column stores the root event ID or the special
value `"main"`.

The `thread_id` column for `event_push_actions` and `event_push_summary` is
backfilled with `"main"` for all existing rows. New entries into `event_push_actions`
and `event_push_actions_staging` will get the proper thread ID.

`receipts_linearized` and `receipts_graph` also gain a `thread_id` column, which is similar,
except `NULL` is a special value meaning the receipt is "unthreaded".

See MSC3771 and MSC3773 for where this data will be useful.
2022-09-14 17:11:16 +00:00
Patrick Cloke f2d12ccabe
Use partial indices on SQLIte. (#13802)
Partial indices have been supported since SQLite 3.8, but Synapse
now requires >= 3.27, so we can enable support for them.

This requires rebuilding previous indices which were partial on
PostgreSQL, but not on SQLite.
2022-09-14 12:01:42 -04:00
reivilibre 6302753012
Deduplicate `is_server_notices_room`. (#13780) 2022-09-14 15:53:18 +00:00
reivilibre cf65433de2
Fix a memory leak when running the unit tests. (#13798) 2022-09-14 15:29:05 +00:00
Quentin Gliech eaed4e6113
Remove unused method in `synapse.api.auth.Auth`. (#13795)
Clean-up from b19060a29b (#13094)
and 73af10f419 (#13093) which removed
all callers.
2022-09-14 10:33:54 -04:00
David Robertson 51a77e990b
Remove incorrect migration file from `state` logical DB (#13788)
* Remove incorrect migration file from `state` logical DB

The table `ex_outlier_stream` is part of the `main` logical DB; it
should not have been created in the `state` logical DB. We remove this
migration now as a tidy-up.

Note: we cannot `DROP TABLE IF EXISTS ex_outlier_stream` in a new
migration, because some (most) instances of Synapse host both of these
logical DBs on the same DB cluster.

* Changelog
2022-09-14 14:16:12 +01:00
Sean Quah c73774467e
Fix bug in device list caching when remote users leave rooms (#13749)
When a remote user leaves the last room shared with the homeserver, we
have to mark their device list as unsubscribed, otherwise we would hold
on to a stale device list in our cache. Crucially, the device list would
remain cached even after the remote user rejoined the room, which could
lead to E2EE failures until the next change to the remote user's device
list.

Fixes #13651.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-14 10:42:57 +01:00
reivilibre 21687ec189
Fix a long-standing spec compliance bug where Synapse would accept a trailing slash on the end of `/get_missing_events` federation requests. (#13789)
* Don't accept a trailing slash on the end of /get_missing_events

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-09-14 09:28:12 +01:00
Mathieu Velten 12dacecabd
Make sequence `cache_invalidation_stream_seq` begin at `2` (#13766)
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-09-13 16:14:28 +02:00
Erik Johnston 9772e362aa Merge branch 'master' into develop 2022-09-13 12:11:53 +01:00
David Robertson b60d47ab2c
Updates to the schema dump script (#13770) 2022-09-13 10:53:11 +01:00
David Robertson 540afb0bfc
Simplify CI tests DAG (#13784)
* Simplify CI tests DAG

* Changelog
2022-09-13 10:17:23 +01:00
Richard van der Hoff 41df25bbbd
installation.md: require libpq on M1 macs (#13480) 2022-09-13 09:01:21 +00:00
Erik Johnston 80bb098d87 Fixup changelog 2022-09-13 09:55:10 +01:00
Erik Johnston 4b678b20a2 1.67.0 2022-09-13 09:20:28 +01:00