Commit Graph

72 Commits

Author SHA1 Message Date
Erik Johnston 7a5873277e
Add support for evicting cache entries based on last access time. (#10205) 2021-07-05 16:32:12 +01:00
Richard van der Hoff 107c06081f
Ensure that errors during startup are written to the logs and the console. (#10191)
* Defer stdio redirection until we are about to start the reactor

* Catch and handle exceptions during startup
2021-06-21 11:41:25 +01:00
Brendan Abolivier 1b3e398bea
Standardise the module interface (#10062)
This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.
2021-06-18 12:15:52 +01:00
Brendan Abolivier 08c8469322
Remove support for ACME v1 (#10194)
Fixes #9778

ACME v1 has been fully decommissioned for existing installs on June 1st 2021(see https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/27), so we can now safely remove it from Synapse.
2021-06-17 18:56:48 +01:00
Richard van der Hoff 9cf6e0eae7
Rip out the DNS lookup limiter (#10190)
As I've written in various places in the past (#7113, #9865) I'm pretty sure this is doing nothing useful at all.
2021-06-17 16:22:41 +01:00
Richard van der Hoff fe5dad46b0
Remove redundant code to reload tls cert (#10054)
we don't need to reload the tls cert if we don't have any tls listeners.

Follow-up to #9280.
2021-05-27 10:34:24 +01:00
Erik Johnston 8771b1337d
Export jemalloc stats to prometheus when used (#9882) 2021-05-06 15:54:07 +01:00
Richard van der Hoff 3ff2251754
Improved validation for received requests (#9817)
* Simplify `start_listening` callpath

* Correctly check the size of uploaded files
2021-04-23 19:20:44 +01:00
Jonathan de Jong 4b965c862d
Remove redundant "coding: utf-8" lines (#9786)
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Richard van der Hoff 7c8402ddb8
Suppress CryptographyDeprecationWarning (#9698)
This warning is somewhat confusing to users, so let's suppress it
2021-03-26 17:33:55 +00:00
Eric Eastwood 0a00b7ff14
Update black, and run auto formatting over the codebase (#9381)
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Ivan Shapovalov 13c7ab8181
Fixes for PyPy compatibility (#9270)
* synapse.app.base: only call gc.freeze() on CPython

gc.freeze() is an implementation detail of CPython garbage collector,
and notably does not exist on PyPy.

Rather than playing whack-a-mole and skipping the call when under PyPy,
simply restrict it to CPython because the whole gc module is
implementation-defined.

Signed-off-by: Ivan Shapovalov <intelfx@intelfx.name>
2021-01-30 17:22:05 +00:00
Richard van der Hoff 671138f658
Clean up exception handling in the startup code (#9059)
Factor out the exception handling in the startup code to a utility function,
and fix the some logging and exit code stuff.
2021-01-11 15:55:05 +00:00
Erik Johnston 80a992d7b9
Fix deadlock on SIGHUP (#8918)
Fixes #8892
2020-12-10 16:56:05 +00:00
Erik Johnston 382b4e83f1
Defer SIGHUP handlers to reactor. (#8817)
We can get a SIGHUP at any point, including times where we are not in a
sane state. By deferring calling the handlers until the next reactor
tick we ensure that we don't get unexpected conflicts, e.g. trying to
flush logs from the signal handler while the code was in the process of
writing a log entry.

Fixes #8769.
2020-11-26 11:18:10 +00:00
Richard van der Hoff fb56dfdccd
Fix SIGHUP handler (#8697)
Fixes:

```
builtins.TypeError: _reload_logging_config() takes 1 positional argument but 2 were given
```
2020-11-06 11:42:07 +00:00
Erik Johnston e3debf9682
Add logging on startup/shutdown (#8448)
This is so we can tell what is going on when things are taking a while to start up.

The main change here is to ensure that transactions that are created during startup get correctly logged like normal transactions.
2020-10-02 15:20:45 +01:00
Patrick Cloke 62894673e6
Allow background tasks to be run on a separate worker. (#8369) 2020-10-02 08:23:15 -04:00
Patrick Cloke c619253db8
Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
Richard van der Hoff 8027166dd5 Add a comment about _LimitedHostnameResolver 2020-08-29 00:06:00 +01:00
Erik Johnston a7bdf98d01
Rename database classes to make some sense (#8033) 2020-08-05 21:38:57 +01:00
Richard van der Hoff 916cf2d439
re-implement daemonize (#8011)
This has long been something I've wanted to do. Basically the `Daemonize` code
is both too flexible and not flexible enough, in that it offers a bunch of
features that we don't use (changing UID, closing FDs in the child, logging to
syslog) and doesn't offer a bunch that we could do with (redirecting stdout/err
to a file instead of /dev/null; having the parent not exit until the child is
running).

As a first step, I've lifted the Daemonize code and removed the bits we don't
use. This should be a non-functional change. Fixing everything else will come
later.
2020-08-04 10:03:41 +01:00
Richard van der Hoff 03619324fc
Create a ListenerConfig object (#7681)
This ended up being a bit more invasive than I'd hoped for (not helped by
generic_worker duplicating some of the code from homeserver), but hopefully
it's an improvement.

The idea is that, rather than storing unstructured `dict`s in the config for
the listener configurations, we instead parse it into a structured
`ListenerConfig` object.
2020-06-16 12:44:07 +01:00
Erik Johnston 1a1da60ad2
Fix new flake8 errors (#7470) 2020-05-12 11:20:48 +01:00
Erik Johnston 37f6823f5b
Add instance name to RDATA/POSITION commands (#7364)
This is primarily for allowing us to send those commands from workers, but for now simply allows us to ignore echoed RDATA/POSITION commands that we sent (we get echoes of sent commands when using redis). Currently we log a WARNING on the master process every time we receive an echoed RDATA.
2020-04-29 16:23:08 +01:00
Richard van der Hoff 443162e577
Move pusherpool startup into _base.setup (#7104)
This should be safe to do on all workers/masters because it is guarded by
a config option which will ensure it is only actually done on the worker
assigned as a pusher.
2020-03-19 09:48:45 +00:00
Patrick Cloke 509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
Erik Johnston fc87d2ffb3
Freeze allocated objects on startup. (#6953)
This may make gc go a bit faster as the gc will know things like
caches/data stores etc. are frozen without having to check.
2020-02-19 15:09:00 +00:00
Richard van der Hoff 0b794cbd7b
Fix sdnotify with acme enabled (#6571)
If acme was enabled, the sdnotify startup hook would never be run because we
would try to add it to a hook which had already fired.

There's no need to delay it: we can sdnotify as soon as we've started the
listeners.
2019-12-19 14:52:52 +00:00
Erik Johnston 756d4942f5 Move DB pool and helper functions into dedicated Database class 2019-12-05 10:46:37 +00:00
Amber Brown 864f144543
Fix up some typechecking (#6150)
* type checking fixes

* changelog
2019-10-02 05:29:01 -07:00
Amber Brown 7dc398586c
Implement a structured logging output system. (#5680) 2019-08-28 21:18:53 +10:00
Richard van der Hoff 412c6e21a8
Drop dependency on sdnotify (#5871)
... to save OSes which don't use it from having to maintain a port.

Fixes #5865.
2019-08-17 09:09:52 +01:00
Erik Johnston c560b791e1 Add process hooks to tell systemd our state.
Fixes #5676.
2019-07-22 14:52:18 +01:00
Amber Brown 7ad1d76356
Support Prometheus_client 0.4.0+ (#5636) 2019-07-18 23:57:15 +10:00
Erik Johnston e8c53b07f2 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_api_cmd 2019-07-15 14:13:22 +01:00
Jorik Schellekens 38a6d3eea7
Add basic opentracing support (#5544)
* Configure and initialise tracer

Includes config options for the tracer and sets up JaegerClient.

* Scope manager using LogContexts

We piggy-back our tracer scopes by using log context.
The current log context gives us the current scope. If new scope is
created we create a stack of scopes in the context.

* jaeger is a dependency now

* Carrier inject and extraction for Twisted Headers

* Trace federation requests on the way in and out.

The span is created in _started_processing and closed in
_finished_processing because we need a meaningful log context.

* Create logcontext for new scope.

Instead of having a stack of scopes in a logcontext we create a new
context for a new scope if the current logcontext already has a scope.

* Remove scope from logcontext if logcontext is top level

* Disable tracer if not configured

* typo

* Remove dependence on jaeger internals

* bools

* Set service name

* :Explicitely state that the tracer is disabled

* Black is the new black

* Newsfile

* Code style

* Use the new config setup.

* Generate config.

* Copyright

* Rename config to opentracing

* Remove user whitelisting

* Empty whitelist by default

* User ConfigError instead of RuntimeError

* Use isinstance

* Use tag constants for opentracing.

* Remove debug comment and no need to explicitely record error

* Two errors a "s(c)entry"

* Docstrings!

* Remove debugging brainslip

* Homeserver Whitlisting

* Better opentracing config comment

* linting

* Inclue worker name in service_name

* Make opentracing an optional dependency

* Neater config retreival

* Clean up dummy tags

* Instantiate tracing as object instead of global class

* Inlcude opentracing as a homeserver member.

* Thread opentracing to the request level

* Reference opetnracing through hs

* Instantiate dummy opentracin g for tests.

* About to revert, just keeping the unfinished changes just in case

* Revert back to global state, commit number:

9ce4a3d906

* Use class level methods in tracerutils

* Start and stop requests spans in a place where we
have access to the authenticated entity

* Seen it, isort it

* Make sure to close the active span.

* I'm getting black and blue from this.

* Logger formatting

Co-Authored-By: Erik Johnston <erik@matrix.org>

* Outdated comment

* Import opentracing at the top

* Return a contextmanager

* Start tracing client requests from the servlet

* Return noop context manager if not tracing

* Explicitely say that these are federation requests

* Include servlet name in client requests

* Use context manager

* Move opentracing to logging/

* Seen it, isort it again!

* Ignore twisted return exceptions on context exit

* Escape the scope

* Scopes should be entered to make them useful.

* Nicer decorator names

* Just one init, init?

* Don't need to close something that isn't open

* Docs make you smarter
2019-07-11 10:36:03 +01:00
Amber Brown 463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ (#5606) 2019-07-04 00:07:04 +10:00
Richard van der Hoff cb8d568cf9 Fix 'utime went backwards' errors on daemonization. (#5609)
* Fix 'utime went backwards' errors on daemonization.

Fixes #5608

* remove spurious debug
2019-07-03 22:40:45 +10:00
Erik Johnston 9f3c0a8556 Add basic admin cmd app 2019-07-02 17:12:48 +01:00
Richard van der Hoff 6cda36777b Drop support for cpu_affinity (#5525)
This has no useful purpose on python3, and is generally a source of confusion.
2019-06-22 11:01:55 +10:00
Amber Brown 32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Richard van der Hoff f76d407ef3
Fix dropped logcontexts during high outbound traffic. (#5277)
Fixes #5271.
2019-05-29 09:17:33 +01:00
Erik Johnston a0fc256d65 Limit in flight DNS requests
This is to work around a bug in twisted where a large number of
concurrent DNS requests cause it to tight loop forever.

c.f. https://twistedmatrix.com/trac/ticket/9620#ticket
2019-04-09 17:23:42 +01:00
Erik Johnston 9ad448c1e5 Correctly handle all command line options 2019-03-14 13:32:14 +00:00
Erik Johnston d154f5a055
Merge pull request #4632 from matrix-org/erikj/basic_sentry
Add basic optional sentry.io integration
2019-02-18 17:22:45 +00:00
Erik Johnston 6cb415b63f Fixup comments and add warning 2019-02-13 16:15:11 +00:00
Erik Johnston 309f3bb322
Update synapse/app/_base.py
Co-Authored-By: richvdh <1389908+richvdh@users.noreply.github.com>
2019-02-13 13:24:27 +00:00
Richard van der Hoff 2a5a15aff8 Improve logging around listening services
I wanted to bring listen_tcp into line with listen_ssl in terms of returning a
list of ports, and wanted to check that was a safe thing to do - hence the
logging in `refresh_certificate`.

Also, pull the 'Synapse now listening' message up to homeserver.py, because it
was being duplicated everywhere else.
2019-02-13 11:58:54 +00:00
Richard van der Hoff e3a0300431 Special-case the default bind_addresses for metrics listener
turns out it doesn't really support ipv6, so let's hack around that by only
listening on ipv4 by default.
2019-02-13 11:48:56 +00:00