Commit Graph

156 Commits

Author SHA1 Message Date
Patrick Cloke 1619802228
Various clean-ups to the logging context code (#8935) 2020-12-14 14:19:47 -05:00
David Teller f14428b25c
Allow spam-checker modules to be provide async methods. (#8890)
Spam checker modules can now provide async methods. This is implemented
in a backwards-compatible manner.
2020-12-11 14:05:15 -05:00
Erik Johnston 427ede619f
Add metrics for tracking 3PID /requestToken requests. (#8712)
The main use case is to see how many requests are being made, and how
many are second/third/etc attempts. If there are large number of retries
then that likely indicates a delivery problem.
2020-11-13 12:03:51 +00:00
Erik Johnston 2b7c180879
Start fewer opentracing spans (#8640)
#8567 started a span for every background process. This is good as it means all Synapse code that gets run should be in a span (unless in the sentinel logging context), but it means we generate about 15x the number of spans as we did previously.

This PR attempts to reduce that number by a) not starting one for send commands to Redis, and b) deferring starting background processes until after we're sure they're necessary.

I don't really know how much this will help.
2020-10-26 09:30:19 +00:00
Patrick Cloke 34a5696f93
Fix typos and spelling errors. (#8639) 2020-10-23 12:38:40 -04:00
Erik Johnston 1fcdbeb3ab
Start an opentracing span for background processes. (#8567)
This should reduce the number of `There was no active span` errors we
see.

Fixes #8510.
2020-10-19 12:26:26 +01:00
Richard van der Hoff 6d2d42f8fb Rewrite BucketCollector
This was a bit unweildy for what I wanted: in particular, I wanted to assign
each measurement straight into a bucket, rather than storing an intermediate
Counter which didn't do any bucketing at all.

I've replaced it with something that is hopefully a bit easier to use.

(I'm not entirely sure what the difference between a HistogramMetricFamily and
a GaugeHistogramMetricFamily is, but given our counters can go down as well as
up the latter *sounds* more accurate?)
2020-09-30 16:49:15 +01:00
Richard van der Hoff 1c8ca2c543 Fix _exposition.py to stop stripping samples
Our hacked-up `_exposition.py` was stripping out some samples it shouldn't
have been. Put them back in, to more closely match the upstream
`exposition.py`.
2020-09-30 16:45:43 +01:00
Richard van der Hoff ceafb5a1c6
Drop support for ancient prometheus_client (#8426)
Drop compatibility hacks for prometheus-client pre 0.4.0. Debian stretch and
Fedora 31 both have newer versions, so hopefully this will be ok.
2020-09-30 16:42:05 +01:00
Patrick Cloke aec294ee0d
Use slots in attrs classes where possible (#8296)
slots use less memory (and attribute access is faster) while slightly
limiting the flexibility of the class attributes. This focuses on objects
which are instantiated "often" and for short periods of time.
2020-09-14 12:50:06 -04:00
Patrick Cloke c619253db8
Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
Patrick Cloke d89692ea84
Convert runWithConnection to async. (#8121) 2020-08-19 07:09:24 -04:00
Patrick Cloke c36228c403
Convert run_as_background_process inner function to async. (#8032) 2020-08-06 08:20:42 -04:00
Richard van der Hoff 8ca39bd2c3
Improve stacktraces from exceptions in background processes (#7808)
use `Failure()` to fish out the real exception.
2020-07-09 13:01:33 +01:00
Erik Johnston a99658074d
Add some metrics for inbound and outbound federation processing times (#7755) 2020-06-30 16:58:06 +01:00
Christian Svensson 8bbe87f42d
Set Content-Length for Metrics requests (#7730)
HTTP requires the response to contain a Content-Length header unless chunked encoding is being used.
Prometheus metrics endpoint did not set this, causing software such as prometheus-proxy to not be able to scrape synapse for metrics.

Signed-off-by: Christian Svensson <blue@cmd.nu>
2020-06-23 18:06:01 +01:00
Patrick Cloke bd6dc17221
Replace iteritems/itervalues/iterkeys with native versions. (#7692) 2020-06-15 07:03:36 -04:00
Erik Johnston f5353eff21
Make inflight background metrics more efficient. (#7597)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2020-05-29 13:25:32 +01:00
Ivan Shapovalov ac481a738e
synapse.metrics: implement detailed memory usage reporting on PyPy (#7536)
PyPy's gc.get_stats() returns an object containing detailed allocator statistics
which could be beneficial to collect as metrics.

Signed-off-by: Ivan Shapovalov <intelfx@intelfx.name>
2020-05-22 11:08:41 +01:00
Amber Brown 7cb8b4bc67
Allow configuration of Synapse's cache without using synctl or environment variables (#6391) 2020-05-11 18:45:23 +01:00
Richard van der Hoff 8c75667ad7
Add prometheus metrics for the number of active pushers (#7103) 2020-03-19 10:00:24 +00:00
Patrick Cloke 509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
Amber Brown 864f144543
Fix up some typechecking (#6150)
* type checking fixes

* changelog
2019-10-02 05:29:01 -07:00
Richard van der Hoff a96318127d Update comments and docstring 2019-09-25 18:17:39 +01:00
Erik Johnston 367158a609 Add wrap_as_background_process decorator.
This does the same thing as `run_as_background_process` but means we
don't need to create superfluous functions.
2019-09-24 15:53:17 +01:00
Amber Brown b617864cd9
Fix for structured logging tests stomping on logs (#6023) 2019-09-13 02:29:55 +10:00
Amber Brown aeb9b2179e
Add a build info metric to Prometheus (#6005) 2019-09-10 00:14:58 +10:00
Amber Brown 7ad1d76356
Support Prometheus_client 0.4.0+ (#5636) 2019-07-18 23:57:15 +10:00
Amber Brown 463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ (#5606) 2019-07-04 00:07:04 +10:00
Amber Brown 071150ce19
Don't log GC 0s at INFO (#5557) 2019-06-28 21:45:33 +10:00
Amber Brown 32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Erik Johnston 3ed595e327 Prometheus histograms are cumalative 2019-06-14 14:07:32 +01:00
Amber H. Brown a10c8dae85 fix prometheus rendering error 2019-06-14 21:09:33 +10:00
Amber Brown 6312d6cc7c
Expose statistics on extrems to prometheus (#5384) 2019-06-13 22:40:52 +10:00
Richard van der Hoff 82ca6d1f9f
Add metrics for number of outgoing EDUs, by type (#4695) 2019-02-20 14:13:14 +00:00
Erik Johnston 7c570bff74 Fix exception in background metrics collection
We attempted to iterate through a list on a separate thread without
doing the necessary copying.
2018-10-03 11:28:01 +01:00
Erik Johnston 94ae1dea3c Add missing logger 2018-09-20 17:05:34 +01:00
Erik Johnston 9ea408441f Handle exceptions thrown by background tasks
Fixes #3921
2018-09-20 16:15:21 +01:00
Erik Johnston d0f6c1ce21 Remove spurious comment 2018-09-14 15:12:36 +01:00
Erik Johnston 0a81038ea0 Add in flight real time metrics for Measure blocks 2018-09-14 15:08:37 +01:00
Erik Johnston 3f6762f0bb isort 2018-08-21 09:38:38 +01:00
Erik Johnston 1058d14127 Make the in flight background process metrics thread safe 2018-08-20 17:27:24 +01:00
Richard van der Hoff bab94da79c fix metric name 2018-08-07 22:11:45 +01:00
Richard van der Hoff 53bca4690b more metrics for the federation and appservice senders 2018-08-07 19:09:48 +01:00
Richard van der Hoff 03751a6420 Fix some looping_call calls which were broken in #3604
It turns out that looping_call does check the deferred returned by its
callback, and (at least in the case of client_ips), we were relying on this,
and I broke it in #3604.

Update run_as_background_process to return the deferred, and make sure we
return it to clock.looping_call.
2018-07-26 11:48:08 +01:00
Richard van der Hoff 6e3fc657b4 Resource tracking for background processes
This introduces a mechanism for tracking resource usage by background
processes, along with an example of how it will be used.

This will help address #3518, but more importantly will give us better insights
into things which are happening but not being shown up by the request metrics.

We *could* do this with Measure blocks, but:
 - I think having them pulled out as a completely separate metric class will
   make it easier to distinguish top-level processes from those which are
   nested.

 - I want to be able to report on in-flight background processes, and I don't
   think we want to do this for *all* Measure blocks.
2018-07-18 10:50:33 +01:00
Amber Brown 49af402019 run isort 2018-07-09 16:09:20 +10:00
Amber Brown 6350bf925e
Attempt to be more performant on PyPy (#3462) 2018-06-28 14:49:57 +01:00
Richard van der Hoff cbbfaa4be8 Fix description of "python_gc_time" metric 2018-06-21 10:02:42 +01:00
Matthew Hodgson ccfdaf68be spell gauge correctly 2018-06-16 07:10:34 +01:00