Mirrors
/
synapse
mirror of https://github.com/element-hq/synapse.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
18,965 Commits 937 Branches 734 Tags 447 MiB
Commit Graph

4483 Commits

Author SHA1 Message Date
Sorunome d936371b69
Implement knock feature (#6739) 
This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403

Signed-off-by: Sorunome mail@sorunome.de
Signed-off-by: Andrew Morgan andrewm@element.io
 2021-06-09 19:39:51 +01:00
Patrick Cloke 11846dff8c
Limit the number of in-flight /keys/query requests from a single device. (#10144) 2021-06-09 07:05:32 -04:00
Richard van der Hoff 1bf83a191b
Clean up the interface for injecting opentracing over HTTP (#10143) 
* Remove unused helper functions

* Clean up the interface for injecting opentracing over HTTP

* changelog
 2021-06-09 11:33:00 +01:00
Patrick Cloke c7f3fb2745
Add type hints to the federation server transport. (#10080) 2021-06-08 11:19:25 -04:00
Andrew Morgan 8df9941cc2 1.36.0rc1 2021-06-08 14:09:00 +01:00
Erik Johnston 1092718cac
Fix logging context when opening new DB connection (#10141) 
Fixes #10140
 2021-06-08 13:49:29 +01:00
Patrick Cloke 9e4610cc27
Correct type hints for parse_string(s)_from_args. (#10137) 2021-06-08 08:30:48 -04:00
Dan Callahan 7dc14730d9
Name release branches just after major.minor (#10013) 
With the prior format, 1.33.0 / 1.33.1 / 1.33.2 got separate branches:

    release-v1.33.0
    release-v1.33.1
    release-v1.33.2

Under the new model, all three would share a common branch:

    release-v1.33

As before, RCs and actual releases exist as tags on these branches.

This better reflects our support model, e.g., that the "1.33" series had
a formal release followed by two patches / updates.

Signed-off-by: Dan Callahan <danc@element.io>
 2021-06-08 11:44:50 +01:00
Erik Johnston c842c581ed
When joining a remote room limit the number of events we concurrently check signatures/hashes for (#10117) 
If we do hundreds of thousands at once the memory overhead can easily reach 500+ MB.
 2021-06-08 11:07:46 +01:00
Erik Johnston a0101fc021
Handle /backfill returning no events (#10133) 
Fixes #10123
 2021-06-08 10:37:01 +01:00
Richard van der Hoff 0acb5010ec
More database opentracing (#10136) 
Add a couple of extra logs/spans, to give a bit of a better idea.
 2021-06-07 18:01:32 +01:00
Richard van der Hoff b2557cbf42
opentracing: use a consistent name for background processes (#10135) 
... otherwise we tend to get a namespace clash between the bg process and the
functions that it calls.
 2021-06-07 17:57:49 +01:00
Rohan Sharma beb251e3ee
Make link in docs use HTTPS (#10130) 
Fixes #10121

Signed-off-by: Rohan Sharma <rhnsharma5113@gmail.com>
 2021-06-07 16:35:02 +01:00
Chris Castle 543e423fce
Fix broken link to README at root of repo (#10132) 
Signed-off-by: Chris Castle chris@crc.io
 2021-06-07 16:31:39 +01:00
14mRh4X0r 8942e23a69
Always update AS last_pos, even on no events (#10107) 
Fixes #1834.

`get_new_events_for_appservice` internally calls `get_events_as_list`, which will filter out any rejected events. If all returned events are filtered out, `_notify_interested_services` will return without updating the last handled stream position. If there are 100 consecutive such events, processing will halt altogether.

Breaking the loop is now done by checking whether we're up-to-date with `current_max` in the loop condition, instead of relying on an empty `events` list.


Signed-off-by: Willem Mulder <14mRh4X0r@gmail.com>
 2021-06-07 15:42:05 +01:00
Dirk Klimpel d558292548
Add missing type hints to the admin API servlets (#10105) 2021-06-07 15:12:34 +01:00
Richard van der Hoff fa1db8f156
Delete completes to-device messages earlier in /sync (#10124) 
I hope this will improve
https://github.com/matrix-org/synapse/issues/9564.
 2021-06-07 09:19:06 +01:00
Erik Johnston a0cd8ae8cb
Don't try and backfill the same room in parallel. (#10116) 
If backfilling is slow then the client may time out and retry, causing
Synapse to start a new `/backfill` before the existing backfill has
finished, duplicating work.
 2021-06-04 10:47:58 +01:00
Erik Johnston c96ab31dff
Limit number of events in a replication request (#10118) 
Fixes #9956.
 2021-06-04 10:35:47 +01:00
Richard van der Hoff d8be7d493d
Enable Prometheus metrics for the jaeger client library (#10112) 2021-06-04 09:25:33 +01:00
Andrew Morgan fd9856e4a9
Compile and render Synapse's docs into a browsable, mobile-friendly and searchable website (#10086) 2021-06-03 17:20:40 +01:00
Richard van der Hoff 9eea4646be
Add OpenTracing for database activity. (#10113) 
This adds quite a lot of OpenTracing decoration for database activity. Specifically it adds tracing at four different levels:

 * emit a span for each "interaction" - ie, the top level database function that we tend to call "transaction", but isn't really, because it can end up as multiple transactions.
 * emit a span while we hold a database connection open
 * emit a span for each database transaction - actual actual transaction.
 * emit a span for each database query.

I'm aware this might be quite a lot of overhead, but even just running it on a local Synapse it looks really interesting, and I hope the overhead can be offset just by turning down the sampling frequency and finding other ways of tracing requests of interest (eg, the `force_tracing_for_users` setting).
 2021-06-03 16:31:56 +01:00
Richard van der Hoff 1d143074c5
Improve opentracing annotations for Notifier (#10111) 
The existing tracing reports an error each time there is a timeout, which isn't
really representative.

Additionally, we log things about the way `wait_for_events` works
(eg, the result of the callback) to the *parent* span, which is confusing.
 2021-06-03 16:01:30 +01:00
Andrew Morgan 73636cab69
Convert admin api docs to markdown (#10089) 
So that they render nicely in mdbook (see #10086), and so that we no longer have a mix of structured text languages in our documentation (excluding files outside of `docs/`).
 2021-06-03 14:06:03 +01:00
Travis Ralston 5325f0308c
r0.6.1 support: /rooms/:roomId/aliases endpoint (#9224) 
[MSC2432](https://github.com/matrix-org/matrix-doc/pull/2432) added this endpoint originally but it has since been included in the spec for nearly a year. 

This is progress towards https://github.com/matrix-org/synapse/issues/8334
 2021-06-03 13:50:49 +01:00
Patrick Cloke d7a646abca Merge branch 'master' into develop 2021-06-03 08:43:38 -04:00
Patrick Cloke 57c01dca29 1.35.1 2021-06-03 08:18:22 -04:00
Patrick Cloke 36a7ff0c86 Do not show invite-only rooms in spaces summary (unless joined/invited). (#10109) 2021-06-03 08:04:01 -04:00
Dirk Klimpel 0284d2a297
Add new admin APIs to remove media by media ID from quarantine. (#10044) 
Related to: #6681, #5956, #10040

Signed-off-by: Dirk Klimpel dirk@klimpel.org
 2021-06-02 18:50:35 +01:00
Richard van der Hoff bf6fd9f4fd
github actions: summarize Sytest results in an easy-to-read format (#10094) 
... using the script from matrix-org/sytest#1052
 2021-06-02 17:10:37 +01:00
Erik Johnston fc3d2dc269
Rewrite the KeyRing (#10035) 2021-06-02 16:37:59 +01:00
Patrick Cloke 3cf6b34b4e
Do not show invite-only rooms in spaces summary (unless joined/invited). (#10109) 2021-06-02 11:31:41 -04:00
Erik Johnston 4deaebfe00
Make /sync do less state res (#10102) 2021-06-02 15:48:17 +01:00
Andrew Morgan a8372ad591 1.35.0 2021-06-01 13:23:55 +01:00
Erik Johnston 408ecf8ece
Announce deprecation of experimental `msc2858_enabled` option. (#10101) 
c.f. https://github.com/matrix-org/synapse/pull/9617 and https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md

Fixes #9627.
 2021-06-01 13:19:50 +01:00
Richard van der Hoff b4b2fd2ece
add a cache to have_seen_event (#9953) 
Empirically, this helped my server considerably when handling gaps in Matrix HQ. The problem was that we would repeatedly call have_seen_events for the same set of (50K or so) auth_events, each of which would take many minutes to complete, even though it's only an index scan.
 2021-06-01 12:04:47 +01:00
Brad Murray 10e6d2abce
Fix opentracing inject to use the SpanContext, not the Span (#10074) 
Signed-off-by: Brad Murray brad@beeper.com
 2021-06-01 08:40:26 +01:00
Erik Johnston 258a9a9e8b 1.35.0rc3 2021-05-28 17:06:05 +01:00
Erik Johnston 6b6c6a02db Merge remote-tracking branch 'origin/release-v1.35.0' into release-v1.35 2021-05-28 17:03:14 +01:00
Brendan Abolivier 9408b86f5c Limit the number of events sent over replication when persisting events. (#10082) 2021-05-28 16:29:09 +01:00
Erik Johnston 1641c5c707 Log method and path when dropping request due to size limit (#10091) 2021-05-28 16:29:09 +01:00
Erik Johnston 84cf3e47a0
Allow response of `/send_join` to be larger. (#10093) 
Fixes #10087.
 2021-05-28 16:28:01 +01:00
Richard van der Hoff ed53bf314f
Set opentracing priority before setting other tags (#10092) 
... because tags on spans which aren't being sampled get thrown away.
 2021-05-28 16:14:08 +01:00
Erik Johnston 3f96dbbda7
Log method and path when dropping request due to size limit (#10091) 2021-05-28 15:57:53 +01:00
Eric Eastwood ac3e02d089
Add `parse_strings_from_args` to get `prev_events` array (#10048) 
Split out from https://github.com/matrix-org/synapse/pull/9247

Strings:

 - `parse_string`
 - `parse_string_from_args`
 - `parse_strings_from_args`

For comparison with ints:

 - `parse_integer`
 - `parse_integer_from_args`

Previous discussions:

 - https://github.com/matrix-org/synapse/pull/9247#discussion_r573195687
 - https://github.com/matrix-org/synapse/pull/9247#discussion_r574214156
 - https://github.com/matrix-org/synapse/pull/9247#discussion_r573264791

Signed-off-by: Eric Eastwood <erice@element.io>
 2021-05-28 14:19:06 +01:00
Erik Johnston 5eed6348ce
Move some more endpoints off master (#10084) 2021-05-27 22:45:43 +01:00
Callum Brown 8fb9af570f
Make reason and score optional for report_event (#10077) 
Implements MSC2414: https://github.com/matrix-org/matrix-doc/pull/2414
See #8551 

Signed-off-by: Callum Brown <callum@calcuode.com>
 2021-05-27 18:42:23 +01:00
Brendan Abolivier f828a70be3
Limit the number of events sent over replication when persisting events. (#10082) 2021-05-27 17:10:58 +01:00
Erik Johnston b1bc26a909 1.35.0rc2 2021-05-27 14:46:24 +01:00
Erik Johnston 78b5102ae7
Fix up `BatchingQueue` (#10078) 
Fixes #10068
 2021-05-27 14:32:31 +01:00
Patrick Cloke 8e15c92c2f
Pass the origin when calculating the spaces summary over GET. (#10079) 
Fixes a bug due to conflicting PRs which were merged. (One added a new caller to
a method, the other added a new parameter to the same method.)
 2021-05-27 08:52:28 -04:00
Denis Kasak d9f44fd0b9
Clarify security note regarding the domain Synapse is hosted on. (#9221) 2021-05-27 11:41:16 +00:00
Richard van der Hoff dcbfec919b
Improve the error message printed by synctl when synapse fails to start. (#10059) 2021-05-27 10:35:06 +01:00
Richard van der Hoff 5447a76332
Remove redundant, unmaintained `convert_server_keys` script. (#10055) 2021-05-27 10:34:55 +01:00
Richard van der Hoff fe5dad46b0
Remove redundant code to reload tls cert (#10054) 
we don't need to reload the tls cert if we don't have any tls listeners.

Follow-up to #9280.
 2021-05-27 10:34:24 +01:00
Richard van der Hoff 224f2f949b
Combine `LruCache.invalidate` and `invalidate_many` (#9973) 
* Make `invalidate` and `invalidate_many` do the same thing

... so that we can do either over the invalidation replication stream, and also
because they always confused me a bit.

* Kill off `invalidate_many`

* changelog
 2021-05-27 10:33:56 +01:00
Patrick Cloke f42e4c4eb9
Remove the experimental spaces enabled flag. (#10063) 
In lieu of just always enabling the unstable spaces endpoint and
unstable room version.
 2021-05-26 14:35:16 -04:00
Dan Callahan 49df2c28e3
Fix GitHub Actions lint for newsfragments (#10069) 
* Fix GitHub Actions lint for newsfragments

Signed-off-by: Dan Callahan <danc@element.io>
 2021-05-26 14:14:43 +01:00
Aaron Raimist f95e7a03fa
Tweak wording of database recommendation in INSTALL.md (#10057) 
* Tweak wording of database recommendation in INSTALL.md

Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2021-05-26 13:29:02 +01:00
Dan Callahan 913a761a53
Tell CircleCI to build Docker images from `main` (#9906) 
The `only` field takes a string or list of strings per the Circle docs:
https://circleci.com/docs/2.0/configuration-reference/#branches

Signed-off-by: Dan Callahan <danc@element.io>
 2021-05-26 13:16:06 +01:00
Dirk Klimpel 65e6c64d83
Add an admin API for unprotecting local media from quarantine (#10040) 
Signed-off-by: Dirk Klimpel dirk@klimpel.org
 2021-05-26 11:19:47 +01:00
Aaron Raimist 3e1beb75e6
Update CAPTCHA documentation to mention turning off verify origin feature (#10046) 
* Update CAPTCHA documentation to mention turning off verify origin

Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2021-05-26 10:55:30 +01:00
Erik Johnston 557635f69a 1.35.0rc1 2021-05-25 11:00:13 +01:00
Patrick Cloke 7d90d6ce9b
Run complement with Synapse workers manually. (#10039) 
Adds an option to complement.sh to run Synapse in worker
mode (instead of the default monolith mode).
 2021-05-24 15:32:45 -04:00
Patrick Cloke 7adcb20fc0
Add missing type hints to synapse.util (#9982) 2021-05-24 15:32:01 -04:00
Sergio Miguéns 22a8838f62
Fix docker image to not log at `/homeserver.log` (#10045) 
Fixes #9970

Signed-off-by: Sergio Miguéns Iglesias lonyelon@lony.xyz
 2021-05-24 20:23:54 +01:00
Jerin J Titus 057ce7b754
Remove tls_fingerprints option (#9280) 
Signed-off-by: Jerin J Titus <72017981+jerinjtitus@users.noreply.github.com>
 2021-05-24 17:43:30 +01:00
Richard van der Hoff 82eacb0e07
Fix --no-daemonize for synctl with workers (#9995) 2021-05-24 14:03:30 +01:00
Richard van der Hoff daca7b2794
Fix off-by-one-error in synapse_port_db (#9991) 
fixes #9979
 2021-05-24 14:03:00 +01:00
Richard van der Hoff c0df6bae06
Remove `keylen` from `LruCache`. (#9993) 
`keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it.

The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`).

Commits should be independently reviewable.
 2021-05-24 14:02:01 +01:00
Patrick Cloke 316f89e87f
Enable experimental spaces by default. (#10011) 
The previous spaces_enabled flag now defaults to true and
is exposed in the sample config.
 2021-05-24 08:57:14 -04:00
Dirk Klimpel 387c297489
Add missing entry to the table of contents of room admin API (#10043) 2021-05-24 07:37:30 -04:00
Eric Eastwood 5f1198a67e
Fix `get_state_ids_for_event` return type typo to match what the function actually does (#10050) 
It looks like a typo copy/paste from `get_state_for_event` above.
 2021-05-24 10:43:33 +01:00
Erik Johnston 3e831f24ff
Don't hammer the database for destination retry timings every ~5mins (#10036) 2021-05-21 17:57:08 +01:00
Michael Telatynski e8ac9ac8ca
Fix /upload 500'ing when presented a very large image (#10029) 
* Fix /upload 500'ing when presented a very large image

Catch DecompressionBombError and re-raise as ThumbnailErrors

* Set PIL's MAX_IMAGE_PIXELS to match homeserver.yaml

to get it to bomb out quicker, to load less into memory
in the case of super large images

* Add changelog entry for 10029
 2021-05-21 18:31:59 +02:00
Andrew Morgan 21bd230831
Add a test for update_presence (#10033) 
https://github.com/matrix-org/synapse/issues/9962 uncovered that we accidentally removed all but one of the presence updates that we store in the database when persisting multiple updates. This could cause users' presence state to be stale.

The bug was fixed in #10014, and this PR just adds a test that failed on the old code, and was used to initially verify the bug.

The test attempts to insert some presence into the database in a batch using `PresenceStore.update_presence`, and then simply pulls it out again.
 2021-05-21 17:29:14 +01:00
Patrick Cloke c5413d0e9e
Remove unused properties from the SpaceSummaryHandler. (#10038) 2021-05-21 12:02:01 -04:00
Marek Matys 6a8643ff3d
Fixed removal of new presence stream states (#10014) 
Fixes: https://github.com/matrix-org/synapse/issues/9962

This is a fix for above problem.

I fixed it by swaping the order of insertion of new records and deletion of old ones. This ensures that we don't delete fresh database records as we do deletes before inserts.

Signed-off-by: Marek Matys <themarcq@gmail.com>
 2021-05-21 12:02:06 +01:00
Erik Johnston 7958eadcd1
Add a batching queue implementation. (#10017) 2021-05-21 11:20:51 +01:00
Erik Johnston 1c6a19002c
Add `Keyring.verify_events_for_server` and reduce memory usage (#10018) 
Also add support for giving a callback to generate the JSON object to
verify. This should reduce memory usage, as we no longer have the event
in memory in dict form (which has a large memory footprint) for extend
periods of time.
 2021-05-20 16:25:11 +01:00
Erik Johnston 64887f06fc
Use ijson to parse the response to `/send_join`, reducing memory usage. (#9958) 
Instead of parsing the full response to `/send_join` into Python objects (which can be huge for large rooms) and *then* parsing that into events, we instead use ijson to stream parse the response directly into `EventBase` objects.
 2021-05-20 16:11:48 +01:00
Patrick Cloke 551d2c3f4b
Allow a user who could join a restricted room to see it in spaces summary. (#9922) 
This finishes up the experimental implementation of MSC3083 by showing
the restricted rooms in the spaces summary (from MSC2946).
 2021-05-20 11:10:36 -04:00
Andrew Morgan d983ced596 Merge branch 'master' of github.com:matrix-org/synapse into develop 2021-05-20 14:39:46 +01:00
Javier Junquera Sánchez 141b073c7b
Update user_directory.md (#10016) 
Signed-off-by: Javier Junquera Sánchez <javier@junquera.io>
 2021-05-20 14:24:19 +01:00
Erik Johnston 9c76d0561b
Update the contrib grafana dashboard (#10001) 2021-05-19 11:47:16 +01:00
Savyasachee Jha 5bba1b4905
Hardened systemd unit files (#9803) 
Signed-off-by: Savyasachee Jha savya.jha@hawkradius.com
 2021-05-19 11:44:16 +01:00
Patrick Cloke ac6bfcd52f
Refactor checking restricted join rules (#10007) 
To be more consistent with similar code. The check now automatically
raises an AuthError instead of passing back a boolean. It also absorbs
some shared logic between callers.
 2021-05-18 12:17:04 -04:00
Andrew Morgan 4d6e5a5e99
Use a database table to hold the users that should have full presence sent to them, instead of something in-memory (#9823) 2021-05-18 14:13:45 +01:00
Patrick Cloke 206a7b5f12
Fix the allowed range of valid ordering characters for spaces. (#10002) 
\x7F was meant to be \0x7E (~) this was originally incorrect
in MSC1772.
 2021-05-17 09:59:17 -04:00
Patrick Cloke 9752849e2b
Clarify comments in the space summary handler. (#9974) 2021-05-17 09:01:19 -04:00
Brendan Abolivier 41ac128fd3
Split multiplart email sending into a dedicated handler (#9977) 
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
 2021-05-17 12:33:38 +02:00
Richard van der Hoff 6660912226
Update postgres docs (#9989) 2021-05-14 13:14:48 +01:00
Richard van der Hoff 6482075c95
Run `black` on the scripts (#9981) 
Turns out these scripts weren't getting linted.
 2021-05-14 11:46:35 +01:00
Richard van der Hoff 5090f26b63
Minor `@cachedList` enhancements (#9975) 
- use a tuple rather than a list for the iterable that is passed into the
  wrapped function, for performance

- test that we can pass an iterable and that keys are correctly deduped.
 2021-05-14 11:12:36 +01:00
Dan Callahan 52ed9655ed
Remove unnecessary SystemRandom from SQLBaseStore (#9987) 
It's not obvious that instances of SQLBaseStore each need their own
instances of random.SystemRandom(); let's just use random directly.

Introduced by 52839886d6

Signed-off-by: Dan Callahan <danc@element.io>
 2021-05-14 10:59:10 +01:00
Dan Callahan ebdef256b3
Remove superfluous call to bool() (#9986) 
Our strtobool already returns a bool, so no need to re-cast here

Signed-off-by: Dan Callahan <danc@element.io>
 2021-05-14 10:58:57 +01:00
Dan Callahan bd918d874f
Simplify exception handling in is_ascii. (#9985) 
We can get away with just catching UnicodeError here.

    ⋮
    +-- ValueError
    |    +-- UnicodeError
    |         +-- UnicodeDecodeError
    |         +-- UnicodeEncodeError
    |         +-- UnicodeTranslateError
    ⋮

https://docs.python.org/3/library/exceptions.html#exception-hierarchy

Signed-off-by: Dan Callahan <danc@element.io>
 2021-05-14 10:58:52 +01:00
Dan Callahan 498084228b
Use Python's secrets module instead of random (#9984) 
Functionally identical, but more obviously cryptographically secure.
...Explicit is better than implicit?

Avoids needing to know that SystemRandom() implies a CSPRNG, and
complies with the big scary red box on the documentation for random:

> Warning:
>   The pseudo-random generators of this module should not be used for
>   security purposes. For security or cryptographic uses, see the
>   secrets module.

https://docs.python.org/3/library/random.html

Signed-off-by: Dan Callahan <danc@element.io>
 2021-05-14 10:58:46 +01:00
Richard van der Hoff c14f99be46
Support enabling opentracing by user (#9978) 
Add a config option which allows enabling opentracing by user id, eg for
debugging requests made by a test user.
 2021-05-14 10:51:08 +01:00
Andrew Morgan 976216959b
Update minimum supported version in postgres.md (#9988) 2021-05-14 09:21:00 +01:00
Patrick Cloke d19bccdbec
Update SSO mapping providers documentation about unique IDs. (#9980) 2021-05-13 14:37:20 -04:00
Brendan Abolivier 47806b0869 1.34.0rc1 2021-05-12 16:59:46 +01:00
Brendan Abolivier a683028d81
Correctly ratelimit invites when creating a room (#9968) 
* Correctly ratelimit invites when creating a room

Also allow ratelimiting for more than one action at a time.
 2021-05-12 16:05:28 +02:00
Richard van der Hoff 7562d887e1
Change the format of access tokens away from macaroons (#5588) 2021-05-12 15:04:51 +01:00
Erik Johnston affaffb0ab
Run cache_joined_hosts_for_event in background (#9951) 2021-05-12 13:17:11 +01:00
Richard van der Hoff 63fb220e5f
Tests for to-device messages (#9965) 2021-05-11 18:01:11 +01:00
Patrick Cloke 27c375f812
Sort child events according to MSC1772 for the spaces summary API. (#9954) 
This should help ensure that equivalent results are achieved between
homeservers querying for the summary of a space.

This implements modified MSC1772 rules, according to MSC2946.
The different is that the origin_server_ts of the m.room.create event
is not used as a tie-breaker since this might not be known if the
homeserver is not part of the room.
 2021-05-11 12:57:39 -04:00
Patrick Cloke f4833e0c06
Support fetching the spaces summary via GET over federation. (#9947) 
Per changes in MSC2946, the C-S and S-S APIs for spaces summary
should use GET requests.

Until this is stable, the POST endpoints still exist.

This does not switch federation requests to use the GET version yet
since it is newly added and already deployed servers might not support
it. When switching to the stable endpoint we should switch to GET
requests.
 2021-05-11 12:21:43 -04:00
Patrick Cloke 28c6841102
Send the `m.room.create` stripped event with invites (support MSC1772). (#9966) 
MSC1772 specifies the m.room.create event should be sent as part
of the invite_state. This was done optionally behind an experimental
flag, but is now done by default due to MSC1772 being approved.
 2021-05-11 10:58:58 -04:00
Brendan Abolivier 652a6b094d Merge branch 'master' into develop 2021-05-11 14:15:30 +01:00
Aaron Raimist dc6366a9bd
Add config option to hide device names over federation (#9945) 
Now that cross signing exists there is much less of a need for other people to look at devices and verify them individually. This PR adds a config option to allow you to prevent device display names from being shared with other servers.

Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2021-05-11 14:03:23 +01:00
Brendan Abolivier 86fb71431c
1.33.2 2021-05-11 14:01:32 +01:00
Richard van der Hoff b378d98c8f
Add debug logging for issue #9533 (#9959) 
Hopefully this will help us track down where to-device messages are getting
lost/delayed.
 2021-05-11 11:04:03 +01:00
Richard van der Hoff 7967b36efe
Fix `m.room_key_request` to-device messages (#9961) 
fixes #9960
 2021-05-11 11:02:56 +01:00
Erik Johnston 2b2985b5cf
Improve performance of backfilling in large rooms. (#9935) 
We were pulling the full auth chain for the room out of the DB each time
we backfilled, which can be *huge* for large rooms and is totally
unnecessary.
 2021-05-10 13:29:02 +01:00
Richard van der Hoff 51065c44bb
Fix port_db on empty db (#9930) 
... and test it.
 2021-05-10 13:02:55 +01:00
Erik Johnston 6c84778549
Always cache 'event_to_prev_state_group' (#9950) 
Fixes regression in send PDU times introduced in #9905.
 2021-05-07 14:54:09 +01:00
Richard van der Hoff 765473567c
Fix make_full_schema to create the db with the right options and user (#9931) 2021-05-07 14:01:57 +01:00
Erik Johnston b65ecaff9b Merge remote-tracking branch 'origin/release-v1.33.2' into develop 2021-05-07 13:27:19 +01:00
Erik Johnston 4df26abf28
Unpin attrs dep after new version has been released (#9946) 
c.f. #9936
 2021-05-07 12:57:21 +01:00
Richard van der Hoff 25f43faa70
Reorganise the database schema directories (#9932) 
The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate.

It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.
 2021-05-07 10:22:05 +01:00
Erik Johnston 8771b1337d
Export jemalloc stats to prometheus when used (#9882) 2021-05-06 15:54:07 +01:00
Erik Johnston eba431c539
Revert "Leave out optional keys from /sync (#9919)" (#9940) 
This reverts commit e9eb3549d3.
 2021-05-06 15:06:35 +01:00
Erik Johnston ac88aca7f7 1.33.1 2021-05-06 14:06:38 +01:00
Erik Johnston 24f07a83e6
Pin attrs to <21.1.0 (#9937) 
Fixes #9936
 2021-05-06 14:06:06 +01:00
Patrick Cloke d783880083
Include the time of the create event in Spaces Summary. (#9928) 
This is an update based on changes to MSC2946. The origin_server_ts
of the m.room.create event is copied into the creation_ts field for each
room returned from the spaces summary.
 2021-05-05 13:33:05 -04:00
Erik Johnston 37623e3382
Increase perf of handling presence when joining large rooms. (#9916) 2021-05-05 17:27:05 +01:00
Patrick Cloke e2a443550e
Support stable MSC1772 spaces identifiers. (#9915) 
Support both the unstable and stable identifiers. A future release
will disable the unstable identifiers.
 2021-05-05 11:56:51 -04:00
Erik Johnston ef889c98a6
Optionally track memory usage of each LruCache (#9881) 
This will double count slightly in the presence of interned strings. It's off by default as it can consume a lot of resources.
 2021-05-05 16:54:36 +01:00
Erik Johnston 1fb9a2d0bf
Limit how often GC happens by time. (#9902) 
Synapse can be quite memory intensive, and unless care is taken to tune
the GC thresholds it can end up thrashing, causing noticable performance
problems for large servers. We fix this by limiting how often we GC a
given generation, regardless of current counts/thresholds.

This does not help with the reverse problem where the thresholds are set
too high, but that should only happen in situations where they've been
manually configured.

Adds a `gc_min_seconds_between` config option to override the defaults.

Fixes #9890.
 2021-05-05 16:53:45 +01:00
Erik Johnston de8f0a03a3
Don't set the external cache if its been done recently (#9905) 2021-05-05 16:53:22 +01:00
Erik Johnston d0aee697ac
Use get_current_users_in_room from store and not StateHandler (#9910) 2021-05-05 16:49:34 +01:00
Christopher May-Townsend d5305000f1
Docker healthcheck timings - add startup delay and changed interval (#9913) 
* Add healthcheck startup delay by 5secs and reduced interval check to 15s
to reduce waiting time for docker aware edge routers bringing an
instance online
 2021-05-05 17:33:04 +02:00
DeepBlueV7.X e9eb3549d3
Leave out optional keys from /sync (#9919) 
This leaves out all optional keys from /sync. This should be fine for all clients tested against conduit already, but it may break some clients, as such we should check, that at least most of them don't break horribly and maybe back out some of the individual changes. (We can probably always leave out groups for example, while the others may cause more issues.)

Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
 2021-05-05 14:37:56 +01:00
Brendan Abolivier 0644ac0989 1.33.0 2021-05-05 14:15:54 +01:00
Erik Johnston e3bc4617fc
Time external cache response time (#9904) 2021-05-04 15:14:22 +01:00
Andrew Morgan b85821aca2
Add port parameter to the sample config for psycopg2 args (#9911) 
Adds the `port` option with the default value to the sample config file.
 2021-05-04 13:28:59 +01:00
Dan Callahan 56c4b47df3
Build Debian packages for Ubuntu 21.04 Hirsute (#9909) 
Signed-off-by: Dan Callahan <danc@element.io>
 2021-04-30 15:36:05 +01:00
Andrew Morgan 4d624f467a Synapse 1.33.0rc2 (2021-04-29) 
==============================
 
 Bugfixes
 --------
 
 - Fix tight loop when handling presence replication when using workers. Introduced in v1.33.0rc1. ([\#9900](https://github.com/matrix-org/synapse/issues/9900))
 -----BEGIN PGP SIGNATURE-----
 
 iQJHBAABCgAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmCKtVATHGFuZHJld0Bh
 bW9yZ2FuLnh5egAKCRCIhIgNLv5f9Ok1D/40l93bxGBFtSNEHlqem7lZRLY2On9j
 AUnzIMqtJTLUgXf67vKIYqtTWkRjTdLIhIuINwhRmB0K23vcRrONnnyRLdhUC4Hl
 v6qLlPBHzupAJQPRcy3FGIVwsloAZ3JqVhV8AG0gXBRA0Dc34qYXh1DlA8EQ8mW/
 l2umQIJI562xLNss7c5tpbCQxi7mq6OmTZ+Nwh45VRDavKEQQzSb9sPfs5w8IGLA
 ViYosUoXoXEWus/j60E38LSA/v1YcEE+Uyf5+LieV4b2c305xKj8gIbZqjLHR2K5
 U9IrUp4QlYpPiebfGn7EmiPnOnlWdR9soIVZMzJ/m29tlyRSL5MBagtWA2fFlXo6
 PywW1R/VsnLTdOuWY/t+et11iI4lVa7VSpywEcbMKGgmGKexDnfB10K7jHw0YICq
 0pyLgy8JWyGUk/xpH2W1pvak8vyyN/zI/H09bDgGu721Qr/EPiHWF6RADNASpUVW
 w10SnO1udVc8ajugNwpX2oOQA7i5uiEuTdyFUHYfZ/aSpaCvdxM+rL3v0K955bNc
 /AGjSbs9ITe+JJQDiAJsWCYc+XLkR4h5KHVaZm2XlusbYb4iobK1DflkWXnQFjsi
 rlTcUH3sIFot3i9mjBBvXdvmzuI9qyx81/Sh8gRunPYT5TPhwaXKfyMBKE93/kif
 4bKoHIXVcJvAqA==
 =sboq
 -----END PGP SIGNATURE-----

Merge tag 'v1.33.0rc2' into develop

Synapse 1.33.0rc2 (2021-04-29)
==============================

Bugfixes
--------

- Fix tight loop when handling presence replication when using workers. Introduced in v1.33.0rc1. ([\#9900](https://github.com/matrix-org/synapse/issues/9900))
 2021-04-29 14:35:14 +01:00
Patrick Cloke bb4b11846f
Add missing type hints to handlers and fix a Spam Checker type hint. (#9896) 
The user_may_create_room_alias method on spam checkers
declared the room_alias parameter as a str when in reality it is
passed a RoomAlias object.
 2021-04-29 07:17:28 -04:00
Andrew Morgan e9444cc74d 1.33.0rc2 2021-04-29 11:45:37 +01:00
ThibF 0085dc5abc
Delete room endpoint (#9889) 
Support the delete of a room through DELETE request and mark
previous request as deprecated through documentation.

Signed-off-by: Thibault Ferrante <thibault.ferrante@pm.me>
 2021-04-29 10:31:45 +01:00
Erik Johnston 802560211a Merge remote-tracking branch 'origin/release-v1.33.0' into develop 2021-04-28 14:43:10 +01:00
Erik Johnston e4ab8676b4
Fix tight loop handling presence replication. (#9900) 
Only affects workers. Introduced in #9819.

Fixes #9899.
 2021-04-28 14:42:50 +01:00
Patrick Cloke 10a08ab88a
Use the parent's logging context name for runWithConnection. (#9895) 
This fixes a regression where the logging context for runWithConnection
was reported as runWithConnection instead of the connection name,
e.g. "POST-XYZ".
 2021-04-28 07:44:52 -04:00
Andrew Morgan fa6679e794 Synapse 1.33.0rc1 (2021-04-28) 
==============================
 
 Features
 --------
 
 - Update experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083): restricting room access via group membership. ([\#9800](https://github.com/matrix-org/synapse/issues/9800), [\#9814](https://github.com/matrix-org/synapse/issues/9814))
 - Add experimental support for handling presence on a worker. ([\#9819](https://github.com/matrix-org/synapse/issues/9819), [\#9820](https://github.com/matrix-org/synapse/issues/9820), [\#9828](https://github.com/matrix-org/synapse/issues/9828), [\#9850](https://github.com/matrix-org/synapse/issues/9850))
 - Return a new template when an user attempts to renew their account multiple times with the same token, stating that their account is set to expire. This replaces the invalid token template that would previously be shown in this case. This change concerns the optional account validity feature. ([\#9832](https://github.com/matrix-org/synapse/issues/9832))
 
 Bugfixes
 --------
 
 - Fixes the OIDC SSO flow when using a `public_baseurl` value including a non-root URL path. ([\#9726](https://github.com/matrix-org/synapse/issues/9726))
 - Fix thumbnail generation for some sites with non-standard content types. Contributed by @rkfg. ([\#9788](https://github.com/matrix-org/synapse/issues/9788))
 - Add some sanity checks to identity server passed to 3PID bind/unbind endpoints. ([\#9802](https://github.com/matrix-org/synapse/issues/9802))
 - Limit the size of HTTP responses read over federation. ([\#9833](https://github.com/matrix-org/synapse/issues/9833))
 - Fix a bug which could cause Synapse to get stuck in a loop of resyncing device lists. ([\#9867](https://github.com/matrix-org/synapse/issues/9867))
 - Fix a long-standing bug where errors from federation did not propagate to the client. ([\#9868](https://github.com/matrix-org/synapse/issues/9868))
 
 Improved Documentation
 ----------------------
 
 - Add a note to the docker docs mentioning that we mirror upstream's supported Docker platforms. ([\#9801](https://github.com/matrix-org/synapse/issues/9801))
 
 Internal Changes
 ----------------
 
 - Add a dockerfile for running Synapse in worker-mode under Complement. ([\#9162](https://github.com/matrix-org/synapse/issues/9162))
 - Apply `pyupgrade` across the codebase. ([\#9786](https://github.com/matrix-org/synapse/issues/9786))
 - Move some replication processing out of `generic_worker`. ([\#9796](https://github.com/matrix-org/synapse/issues/9796))
 - Replace `HomeServer.get_config()` with inline references. ([\#9815](https://github.com/matrix-org/synapse/issues/9815))
 - Rename some handlers and config modules to not duplicate the top-level module. ([\#9816](https://github.com/matrix-org/synapse/issues/9816))
 - Fix a long-standing bug which caused `max_upload_size` to not be correctly enforced. ([\#9817](https://github.com/matrix-org/synapse/issues/9817))
 - Reduce CPU usage of the user directory by reusing existing calculated room membership. ([\#9821](https://github.com/matrix-org/synapse/issues/9821))
 - Small speed up for joining large remote rooms. ([\#9825](https://github.com/matrix-org/synapse/issues/9825))
 - Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9838](https://github.com/matrix-org/synapse/issues/9838))
 - Only store the raw data in the in-memory caches, rather than objects that include references to e.g. the data stores. ([\#9845](https://github.com/matrix-org/synapse/issues/9845))
 - Limit length of accepted email addresses. ([\#9855](https://github.com/matrix-org/synapse/issues/9855))
 - Remove redundant `synapse.types.Collection` type definition. ([\#9856](https://github.com/matrix-org/synapse/issues/9856))
 - Handle recently added rate limits correctly when using `--no-rate-limit` with the demo scripts. ([\#9858](https://github.com/matrix-org/synapse/issues/9858))
 - Disable invite rate-limiting by default when running the unit tests. ([\#9871](https://github.com/matrix-org/synapse/issues/9871))
 - Pass a reactor into `SynapseSite` to make testing easier. ([\#9874](https://github.com/matrix-org/synapse/issues/9874))
 - Make `DomainSpecificString` an `attrs` class. ([\#9875](https://github.com/matrix-org/synapse/issues/9875))
 - Add type hints to `synapse.api.auth` and `synapse.api.auth_blocking` modules. ([\#9876](https://github.com/matrix-org/synapse/issues/9876))
 - Remove redundant `_PushHTTPChannel` test class. ([\#9878](https://github.com/matrix-org/synapse/issues/9878))
 - Remove backwards-compatibility code for Python versions < 3.6. ([\#9879](https://github.com/matrix-org/synapse/issues/9879))
 - Small performance improvement around handling new local presence updates. ([\#9887](https://github.com/matrix-org/synapse/issues/9887))
 -----BEGIN PGP SIGNATURE-----
 
 iQJHBAABCgAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmCJQmgTHGFuZHJld0Bh
 bW9yZ2FuLnh5egAKCRCIhIgNLv5f9EKWD/9MZG4ngjECrk3QMBlYqaipGl/l7wcl
 8vBOTxiEqIgVJxHLCRFBbtFOItdxB4YkrorIo2fqshkk+lv4CRtTD9jEKHRvS4T6
 7p1icRdRTv2K94OkJ8R9jMFlmywZFU87oHPfI2xUYg6hvOKrR+RwPvIjA7c24UZt
 6MJqDhgGDlZD7/hQdQof9O4oOJzzIgLJPk6o8E42y6c0bLlPPKgH3sh0vlenDLfE
 15thRCOeiP237YYvXSdbr7G3PI66Efhq4BwQowSrgFg+B0BR68l4747iSIeWmLQJ
 Ow6QLLFPCOhgAPC4amp9PdaaV/9NYiBcvNxlOvyQAVl/+ioEiATNHnzdNtDouCQo
 nDdSHw0Mt9D3i+rxpu5Pf0gZN9dXRGiczqnq5QKL8+EvT/4FLYeluEeduuy3rE+G
 o5OTCd3EajxynzjftuopeysNAw6zeDpbulZoTCeCumxBL2+wAod2PsyY5Ei9Gapn
 iJvExNOJX4OlkFc67jO2CK8o3sUTNEDeIDWCQ0fVKAwIt7T45ebTA/UzDNWg7YzN
 EyUp+3NZcUZBskgMB5hpcijPJoXNYzWZPews73vMPV7AfQRxDzU5xrM9AdbadBDd
 Idv6wT9ssDUA+M0aKnafvoSJZ+qE85mi2x0rRsueZNd9uO9/QHIXKIb+4PvVpj3C
 BJnZf34m568AGQ==
 =smAk
 -----END PGP SIGNATURE-----

Merge tag 'v1.33.0rc1' into develop

Synapse 1.33.0rc1 (2021-04-28)
==============================

Features
--------

- Update experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083): restricting room access via group membership. ([\#9800](https://github.com/matrix-org/synapse/issues/9800), [\#9814](https://github.com/matrix-org/synapse/issues/9814))
- Add experimental support for handling presence on a worker. ([\#9819](https://github.com/matrix-org/synapse/issues/9819), [\#9820](https://github.com/matrix-org/synapse/issues/9820), [\#9828](https://github.com/matrix-org/synapse/issues/9828), [\#9850](https://github.com/matrix-org/synapse/issues/9850))
- Return a new template when an user attempts to renew their account multiple times with the same token, stating that their account is set to expire. This replaces the invalid token template that would previously be shown in this case. This change concerns the optional account validity feature. ([\#9832](https://github.com/matrix-org/synapse/issues/9832))

Bugfixes
--------

- Fixes the OIDC SSO flow when using a `public_baseurl` value including a non-root URL path. ([\#9726](https://github.com/matrix-org/synapse/issues/9726))
- Fix thumbnail generation for some sites with non-standard content types. Contributed by @rkfg. ([\#9788](https://github.com/matrix-org/synapse/issues/9788))
- Add some sanity checks to identity server passed to 3PID bind/unbind endpoints. ([\#9802](https://github.com/matrix-org/synapse/issues/9802))
- Limit the size of HTTP responses read over federation. ([\#9833](https://github.com/matrix-org/synapse/issues/9833))
- Fix a bug which could cause Synapse to get stuck in a loop of resyncing device lists. ([\#9867](https://github.com/matrix-org/synapse/issues/9867))
- Fix a long-standing bug where errors from federation did not propagate to the client. ([\#9868](https://github.com/matrix-org/synapse/issues/9868))

Improved Documentation
----------------------

- Add a note to the docker docs mentioning that we mirror upstream's supported Docker platforms. ([\#9801](https://github.com/matrix-org/synapse/issues/9801))

Internal Changes
----------------

- Add a dockerfile for running Synapse in worker-mode under Complement. ([\#9162](https://github.com/matrix-org/synapse/issues/9162))
- Apply `pyupgrade` across the codebase. ([\#9786](https://github.com/matrix-org/synapse/issues/9786))
- Move some replication processing out of `generic_worker`. ([\#9796](https://github.com/matrix-org/synapse/issues/9796))
- Replace `HomeServer.get_config()` with inline references. ([\#9815](https://github.com/matrix-org/synapse/issues/9815))
- Rename some handlers and config modules to not duplicate the top-level module. ([\#9816](https://github.com/matrix-org/synapse/issues/9816))
- Fix a long-standing bug which caused `max_upload_size` to not be correctly enforced. ([\#9817](https://github.com/matrix-org/synapse/issues/9817))
- Reduce CPU usage of the user directory by reusing existing calculated room membership. ([\#9821](https://github.com/matrix-org/synapse/issues/9821))
- Small speed up for joining large remote rooms. ([\#9825](https://github.com/matrix-org/synapse/issues/9825))
- Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9838](https://github.com/matrix-org/synapse/issues/9838))
- Only store the raw data in the in-memory caches, rather than objects that include references to e.g. the data stores. ([\#9845](https://github.com/matrix-org/synapse/issues/9845))
- Limit length of accepted email addresses. ([\#9855](https://github.com/matrix-org/synapse/issues/9855))
- Remove redundant `synapse.types.Collection` type definition. ([\#9856](https://github.com/matrix-org/synapse/issues/9856))
- Handle recently added rate limits correctly when using `--no-rate-limit` with the demo scripts. ([\#9858](https://github.com/matrix-org/synapse/issues/9858))
- Disable invite rate-limiting by default when running the unit tests. ([\#9871](https://github.com/matrix-org/synapse/issues/9871))
- Pass a reactor into `SynapseSite` to make testing easier. ([\#9874](https://github.com/matrix-org/synapse/issues/9874))
- Make `DomainSpecificString` an `attrs` class. ([\#9875](https://github.com/matrix-org/synapse/issues/9875))
- Add type hints to `synapse.api.auth` and `synapse.api.auth_blocking` modules. ([\#9876](https://github.com/matrix-org/synapse/issues/9876))
- Remove redundant `_PushHTTPChannel` test class. ([\#9878](https://github.com/matrix-org/synapse/issues/9878))
- Remove backwards-compatibility code for Python versions < 3.6. ([\#9879](https://github.com/matrix-org/synapse/issues/9879))
- Small performance improvement around handling new local presence updates. ([\#9887](https://github.com/matrix-org/synapse/issues/9887))
 2021-04-28 12:12:29 +01:00
Erik Johnston 391bfe9a7b
Reduce memory footprint of caches (#9886) 2021-04-28 11:59:28 +01:00
Andrew Morgan 787de3190f 1.33.0rc1 2021-04-28 11:43:33 +01:00
Andrew Morgan 4e0fd35bc9 Revert "Experimental Federation Speedup (#9702)" 
This reverts commit 05e8c70c05.
 2021-04-28 11:38:33 +01:00
Erik Johnston dd2d32dcdb
Add type hints to presence handler (#9885) 2021-04-28 11:07:47 +01:00