Commit Graph

800 Commits

Author SHA1 Message Date
Erik Johnston 8771b1337d
Export jemalloc stats to prometheus when used (#9882) 2021-05-06 15:54:07 +01:00
Erik Johnston ef889c98a6
Optionally track memory usage of each LruCache (#9881)
This will double count slightly in the presence of interned strings. It's off by default as it can consume a lot of resources.
2021-05-05 16:54:36 +01:00
Erik Johnston 1fb9a2d0bf
Limit how often GC happens by time. (#9902)
Synapse can be quite memory intensive, and unless care is taken to tune
the GC thresholds it can end up thrashing, causing noticable performance
problems for large servers. We fix this by limiting how often we GC a
given generation, regardless of current counts/thresholds.

This does not help with the reverse problem where the thresholds are set
too high, but that should only happen in situations where they've been
manually configured.

Adds a `gc_min_seconds_between` config option to override the defaults.

Fixes #9890.
2021-05-05 16:53:45 +01:00
Richard van der Hoff 3ff2251754
Improved validation for received requests (#9817)
* Simplify `start_listening` callpath

* Correctly check the size of uploaded files
2021-04-23 19:20:44 +01:00
Richard van der Hoff 59d24c5bef
pass a reactor into SynapseSite (#9874) 2021-04-23 17:06:47 +01:00
Erik Johnston 9d25a0ae65
Split presence out of master (#9820) 2021-04-23 12:21:55 +01:00
Richard van der Hoff 5a153772c1
remove `HomeServer.get_config` (#9815)
Every single time I want to access the config object, I have to remember
whether or not we use `get_config`. Let's just get rid of it.
2021-04-14 19:09:08 +01:00
Erik Johnston 00a6db9676
Move some replication processing out of generic_worker (#9796)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-04-14 17:06:06 +01:00
Jonathan de Jong 4b965c862d
Remove redundant "coding: utf-8" lines (#9786)
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Andrew Morgan 04819239ba
Add a Synapse Module for configuring presence update routing (#9491)
At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though.

This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around.

A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync. 

The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being:

* Sending state for a specific set or all known users to a defined set of local and remote users.
* The ability to trigger an initial sync for specific users, so they receive all current state.
2021-04-06 14:38:30 +01:00
Patrick Cloke da75d2ea1f
Add type hints for the federation sender. (#9681)
Includes an abstract base class which both the FederationSender
and the FederationRemoteSendQueue must implement.
2021-03-29 11:43:20 -04:00
Richard van der Hoff 7c8402ddb8
Suppress CryptographyDeprecationWarning (#9698)
This warning is somewhat confusing to users, so let's suppress it
2021-03-26 17:33:55 +00:00
Brendan Abolivier 0b56481caa
Fix lint 2021-03-19 16:11:08 +01:00
Brendan Abolivier 066c703729
Move support for MSC3026 behind an experimental flag 2021-03-18 18:37:19 +01:00
Brendan Abolivier 405aeb0b2c
Implement MSC3026: busy presence state 2021-03-18 16:34:47 +01:00
Jonathan de Jong 27d2820c33
Enable flake8-bugbear, but disable most checks. (#9499)
* Adds B00 to ignored checks.
* Fixes remaining issues.
2021-03-16 14:19:27 -04:00
Richard van der Hoff 4db07f9aef
Set X-Forwarded-Proto header when frontend-proxy proxies a request (#9539)
Should fix some remaining warnings
2021-03-03 18:49:08 +00:00
Jonathan de Jong e12077a78a
Allow bytecode again (#9502)
In #75, bytecode was disabled (from a bit of FUD back in `python<2.4` days, according to dev chat), I think it's safe enough to enable it again.

Added in `__pycache__/` and `.pyc`/`.pyd` to `.gitignore`, to extra-insure compiled files don't get committed.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-02-26 18:30:54 +00:00
Erik Johnston 2927921942
Clean up `ShardedWorkerHandlingConfig` (#9466)
* Split ShardedWorkerHandlingConfig

This is so that we have a type level understanding of when it is safe to
call `get_instance(..)` (as opposed to `should_handle(..)`).

* Remove special cases in ShardedWorkerHandlingConfig.

`ShardedWorkerHandlingConfig` tried to handle the various different ways
it was possible to configure federation senders and pushers. This led to
special cases that weren't hit during testing.

To fix this the handling of the different cases is moved from there and
`generic_worker` into the worker config class. This allows us to have
the logic in one place and allows the rest of the code to ignore the
different cases.
2021-02-24 13:23:18 +00:00
Erik Johnston 66f4949e7f
Fix deleting pushers when using sharded pushers. (#9465) 2021-02-22 21:14:42 +00:00
Eric Eastwood 0a00b7ff14
Update black, and run auto formatting over the codebase (#9381)
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Richard van der Hoff 18ab35284a Merge branch 'social_login' into develop 2021-02-01 17:28:37 +00:00
Jan Christian Grünhage 43dd93bb26
Add phone home stats for encrypted messages. (#9283)
Signed-off-by: Jan Christian Grünhage <jan.christian@gruenhage.xyz>
2021-02-01 17:06:22 +00:00
Richard van der Hoff 9c715a5f19
Fix SSO on workers (#9271)
Fixes #8966.

* Factor out build_synapse_client_resource_tree

Start a function which will mount resources common to all workers.

* Move sso init into build_synapse_client_resource_tree

... so that we don't have to do it for each worker

* Fix SSO-login-via-a-worker

Expose the SSO login endpoints on workers, like the documentation says.

* Update workers config for new endpoints

Add documentation for endpoints recently added (#8942, #9017, #9262)

* remove submit_token from workers endpoints list

this *doesn't* work on workers (yet).

* changelog

* Add a comment about the odd path for SAML2Resource
2021-02-01 15:47:59 +00:00
Richard van der Hoff f78d07bf00
Split out a separate endpoint to complete SSO registration (#9262)
There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.
2021-02-01 13:15:51 +00:00
Ivan Shapovalov 13c7ab8181
Fixes for PyPy compatibility (#9270)
* synapse.app.base: only call gc.freeze() on CPython

gc.freeze() is an implementation detail of CPython garbage collector,
and notably does not exist on PyPy.

Rather than playing whack-a-mole and skipping the call when under PyPy,
simply restrict it to CPython because the whole gc module is
implementation-defined.

Signed-off-by: Ivan Shapovalov <intelfx@intelfx.name>
2021-01-30 17:22:05 +00:00
Erik Johnston 6633a4015a
Allow moving account data and receipts streams off master (#9104) 2021-01-18 15:47:59 +00:00
Richard van der Hoff 21a296cd5a
Split OidcProvider out of OidcHandler (#9107)
The idea here is that we will have an instance of OidcProvider for each
configured IdP, with OidcHandler just doing the marshalling of them.

For now it's still hardcoded with a single provider.
2021-01-14 13:29:17 +00:00
Patrick Cloke d1eb1b96e8
Register the /devices endpoint on workers. (#9092) 2021-01-13 12:35:40 -05:00
Erik Johnston c9195744a4
Move more encryption endpoints off master (#9068) 2021-01-11 18:01:27 +00:00
Richard van der Hoff 671138f658
Clean up exception handling in the startup code (#9059)
Factor out the exception handling in the startup code to a utility function,
and fix the some logging and exit code stuff.
2021-01-11 15:55:05 +00:00
Richard van der Hoff 7db2622d30
Remove unused SynapseService (#9058) 2021-01-11 10:24:22 +00:00
Erik Johnston b530eaa262
Allow running sendToDevice on workers (#9044) 2021-01-07 20:19:26 +00:00
Richard van der Hoff 111b673fc1
Add initial support for a "pick your IdP" page (#9017)
During login, if there are multiple IdPs enabled, offer the user a choice of
IdPs.
2021-01-05 11:25:28 +00:00
Patrick Cloke 68bb26da69
Allow redacting events on workers (#8994)
Adds the redacts endpoint to workers that have the client listener.
2020-12-29 07:40:12 -05:00
Richard van der Hoff 28877fade9
Implement a username picker for synapse (#8942)
The final part (for now) of my work to implement a username picker in synapse itself. The idea is that we allow
`UsernameMappingProvider`s to return `localpart=None`, in which case, rather than redirecting the browser
back to the client, we redirect to a username-picker resource, which allows the user to enter a username.
We *then* complete the SSO flow (including doing the client permission checks).

The static resources for the username picker itself (in 
https://github.com/matrix-org/synapse/tree/rav/username_picker/synapse/res/username_picker)
are essentially lifted wholesale from
https://github.com/matrix-org/matrix-synapse-saml-mozilla/tree/master/matrix_synapse_saml_mozilla/res. 
As the comment says, we might want to think about making them customisable, but that can be a follow-up. 

Fixes #8876.
2020-12-18 14:19:46 +00:00
Erik Johnston 80a992d7b9
Fix deadlock on SIGHUP (#8918)
Fixes #8892
2020-12-10 16:56:05 +00:00
Richard van der Hoff ab7a24cc6b
Better formatting for config errors from modules (#8874)
The idea is that the parse_config method of extension modules can raise either a ConfigError or a JsonValidationError,
and it will be magically turned into a legible error message. There's a few components to it:

* Separating the "path" and the "message" parts of a ConfigError, so that we can fiddle with the path bit to turn it
   into an absolute path.
* Generally improving the way ConfigErrors get printed.
* Passing in the config path to load_module so that it can wrap any exceptions that get caught appropriately.
2020-12-08 14:04:35 +00:00
Patrick Cloke 30fba62108
Apply an IP range blacklist to push and key revocation requests. (#8821)
Replaces the `federation_ip_range_blacklist` configuration setting with an
`ip_range_blacklist` setting with wider scope. It now applies to:

* Federation
* Identity servers
* Push notifications
* Checking key validitity for third-party invite events

The old `federation_ip_range_blacklist` setting is still honored if present, but
with reduced scope (it only applies to federation and identity servers).
2020-12-02 11:09:24 -05:00
Erik Johnston 382b4e83f1
Defer SIGHUP handlers to reactor. (#8817)
We can get a SIGHUP at any point, including times where we are not in a
sane state. By deferring calling the handlers until the next reactor
tick we ensure that we don't get unexpected conflicts, e.g. trying to
flush logs from the signal handler while the code was in the process of
writing a log entry.

Fixes #8769.
2020-11-26 11:18:10 +00:00
Richard van der Hoff fb56dfdccd
Fix SIGHUP handler (#8697)
Fixes:

```
builtins.TypeError: _reload_logging_config() takes 1 positional argument but 2 were given
```
2020-11-06 11:42:07 +00:00
Erik Johnston 921a3f8a59
Fix not sending events over federation when using sharded event persisters (#8536)
* Fix outbound federaion with multiple event persisters.

We incorrectly notified federation senders that the minimum persisted
stream position had advanced when we got an `RDATA` from an event
persister.

Notifying of federation senders already correctly happens in the
notifier, so we just delete the offending line.

* Change some interfaces to use RoomStreamToken.

By enforcing use of `RoomStreamTokens` we make it less likely that
people pass in random ints that they got from somewhere random.
2020-10-14 13:27:51 +01:00
Patrick Cloke fe0f4a3591
Move additional tasks to the background worker, part 3 (#8489) 2020-10-09 07:37:51 -04:00
Patrick Cloke c9c0ad5e20
Remove the deprecated Handlers object (#8494)
All handlers now available via get_*_handler() methods on the HomeServer.
2020-10-09 07:24:34 -04:00
Patrick Cloke e4f72ddc44
Move additional tasks to the background worker (#8458) 2020-10-07 11:27:56 -04:00
Patrick Cloke 8dbf62fada
Include the configured log level in phone home stats. (#8477)
By reporting the log level of the synapse logger as a string.
2020-10-07 11:13:38 -04:00
Richard van der Hoff 4f0637346a
Combine `SpamCheckerApi` with the more generic `ModuleApi`. (#8464)
Lots of different module apis is not easy to maintain.

Rather than adding yet another ModuleApi(hs, hs.get_auth_handler()) incantation, first add an hs.get_module_api() method and use it where possible.
2020-10-07 12:03:26 +01:00
Erik Johnston e3debf9682
Add logging on startup/shutdown (#8448)
This is so we can tell what is going on when things are taking a while to start up.

The main change here is to ensure that transactions that are created during startup get correctly logged like normal transactions.
2020-10-02 15:20:45 +01:00
Patrick Cloke 62894673e6
Allow background tasks to be run on a separate worker. (#8369) 2020-10-02 08:23:15 -04:00
Patrick Cloke 8a4a4186de
Simplify super() calls to Python 3 syntax. (#8344)
This converts calls like super(Foo, self) -> super().

Generated with:

    sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
2020-09-18 09:56:44 -04:00