Commit Graph

767 Commits

Author SHA1 Message Date
Sean Quah 91f2bd0907 Prevent the media store from writing outside of the configured directory
Also tighten validation of server names by forbidding invalid characters
in IPv6 addresses and empty domain labels.
2021-11-19 13:39:15 +00:00
Sean Quah 84fac0f814
Add type annotations to `synapse.metrics` (#10847) 2021-11-17 19:07:02 +00:00
Patrick Cloke 7468723697
Add most missing type hints to synapse.util (#11328) 2021-11-16 08:47:36 -05:00
Patrick Cloke b64b6d12d4
Add more type hints to synapse.util. (#11321) 2021-11-12 13:43:06 +00:00
Patrick Cloke 5cace20bf1
Add missing type hints to `synapse.app`. (#11287) 2021-11-10 15:06:54 -05:00
Richard van der Hoff 46d0937447
ObservableDeferred: run observers in order (#11229) 2021-11-02 00:17:35 +00:00
Erik Johnston 7004f43da1
Move DNS lookups into separate thread pool (#11177)
This is to stop large bursts of lookups starving out other users of the
thread pools.

Fixes #11049.
2021-10-26 13:45:38 +01:00
David Robertson 797ee7812d
Relax `ignore-missing-imports` for modules that have stubs now and update mypy (#11006)
Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details.
We therefore pull in stub packages in setup.py

Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them.

The rest of this change consists of fixups to make the newer mypy + stubs pass CI.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-10-08 14:49:41 +01:00
Sean Quah 49a683d871
Fix long-standing bug where `ReadWriteLock` could drop logging contexts (#10993)
Use `PreserveLoggingContext()` to ensure that logging contexts are not
lost when exiting a read/write lock.

When exiting a read/write lock, callbacks on a `Deferred` are triggered
as a signal to any waiting coroutines. Any waiting coroutine that
becomes runnable is likely to follow the Synapse logging context rules
and will restore its own logging context, then either run to completion
or await another `Deferred`, resetting the logging context in the
process.
2021-10-08 12:27:16 +01:00
David Robertson f8d0f72b27
More types for synapse.util, part 1 (#10888)
The following modules now pass `disallow_untyped_defs`:

* synapse.util.caches.cached_call 
* synapse.util.caches.lrucache
* synapse.util.caches.response_cache 
* synapse.util.caches.stream_change_cache
* synapse.util.caches.ttlcache pass
* synapse.util.daemonize
* synapse.util.patch_inline_callbacks pass `no-untyped-defs`
* synapse.util.versionstring

Additional typing in synapse.util.metrics. Didn't get this to pass `no-untyped-defs`, think I'll need to watch #10847
2021-10-06 11:20:49 +01:00
Patrick Cloke a0f48ee89d
Use direct references for configuration variables (part 7). (#10959) 2021-10-04 07:18:54 -04:00
Hillery Shay 0f007fe009
Update utility code to handle C implementations of frozendict (#10902)
* update _handle_frozendict to work with c implementations of frozen dict

* add changelog

* add clarifying comment to _handle_frozendict
2021-09-28 09:13:23 -07:00
Erik Johnston 707d5e4e48
Encode JSON responses on a thread in C, mk2 (#10905)
Currently we use `JsonEncoder.iterencode` to write JSON responses, which ensures that we don't block the main reactor thread when encoding huge objects. The downside to this is that `iterencode` falls back to using a pure Python encoder that is *much* less efficient and can easily burn a lot of CPU for huge responses. To fix this, while still ensuring we don't block the reactor loop, we encode the JSON on a threadpool using the standard `JsonEncoder.encode` functions, which is backed by a C library.

Doing so, however, requires `respond_with_json` to have access to the reactor, which it previously didn't. There are two ways of doing this:

1. threading through the reactor object, which is a bit fiddly as e.g. `DirectServeJsonResource` doesn't currently take a reactor, but is exposed to modules and so is a PITA to change; or
2. expose the reactor in `SynapseRequest`, which requires updating a bunch of servlet types.

I went with the latter as that is just a mechanical change, and I think makes sense as a request already has a reactor associated with it (via its http channel).
2021-09-28 09:37:58 +00:00
David Robertson 724aef9a87
Opt out of cache expiry for `get_users_who_share_room_with_user` (#10826)
* Allow LruCaches to opt out of time-based expiry
* Don't expire `get_users_who_share_room` & friends
2021-09-22 14:21:58 +01:00
David Robertson a2d7195e01
Track why we're evicting from caches (#10829)
So we can see distinguish between "evicting because the cache is too big" and "evicting because the cache entries haven't been recently used".
2021-09-22 10:59:52 +01:00
Patrick Cloke 3eba047d38
Add type hints to state database module. (#10823) 2021-09-15 09:54:13 -04:00
Patrick Cloke 01c88a09cd
Use direct references for some configuration variables (#10798)
Instead of proxying through the magic getter of the RootConfig
object. This should be more performant (and is more explicit).
2021-09-13 13:07:12 -04:00
reivilibre 524b8ead77
Add types to synapse.util. (#10601) 2021-09-10 17:03:18 +01:00
Azrenbeth 6e895366ea
Add config option to use non-default manhole password and keys (#10643) 2021-09-06 16:08:03 +01:00
reivilibre 19e51b14d2
Manhole: wrap coroutines in `defer.ensureDeferred` automatically (#10602) 2021-08-16 18:11:48 +01:00
Patrick Cloke 1de26b3467
Convert Transaction and Edu object to attrs (#10542)
Instead of wrapping the JSON into an object, this creates concrete
instances for Transaction and Edu. This allows for improved type
hints and simplified code.
2021-08-06 09:39:59 -04:00
V02460 b7f7ca24b1
Remove shebang line from module files (#10415)
Signed-off-by: Kai A. Hiller <V02460@gmail.com>
2021-07-29 21:34:14 +01:00
Richard van der Hoff 858363d0b7
Generics for `ObservableDeferred` (#10491)
Now that `Deferred` is a generic class, let's update `ObeservableDeferred` to
follow suit.
2021-07-28 19:55:50 +00:00
Richard van der Hoff d9cb658c78
Fix up type hints for Twisted 21.7 (#10490)
Mostly this involves decorating a few Deferred declarations with extra type hints. We wrap the types in quotes to avoid runtime errors when running against older versions of Twisted that don't have generics on Deferred.
2021-07-28 12:04:11 +00:00
Richard van der Hoff 9643dfde6a
improve typing annotations in CachedCall (#10450)
tighten up some of the typing in CachedCall, which is going to be needed when
Twisted 21.7 brings better typing on Deferred.
2021-07-28 12:25:12 +01:00
Jonathan de Jong 95e47b2e78
[pyupgrade] `synapse/` (#10348)
This PR is tantamount to running 
```
pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"`
```

Part of #9744
2021-07-19 15:28:05 +01:00
Jonathan de Jong bdfde6dca1
Use inline type hints in `http/federation/`, `storage/` and `util/` (#10381) 2021-07-15 12:46:54 -04:00
Erik Johnston 7a5873277e
Add support for evicting cache entries based on last access time. (#10205) 2021-07-05 16:32:12 +01:00
Brendan Abolivier 1b3e398bea
Standardise the module interface (#10062)
This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.
2021-06-18 12:15:52 +01:00
Erik Johnston 36c426e294
Add debug logging when we enter/exit Measure block (#10183)
It can be helpful to know when trying to track down slow requests.
2021-06-16 13:29:54 +01:00
Richard van der Hoff d7808a2dde
Extend `ResponseCache` to pass a context object into the callback (#10157)
This is the first of two PRs which seek to address #8518. This first PR lays the groundwork by extending ResponseCache; a second PR (#10158) will update the SyncHandler to actually use it, and fix the bug.

The idea here is that we allow the callback given to ResponseCache.wrap to decide whether its result should be cached or not. We do that by (optionally) passing a ResponseCacheContext into it, which it can modify.
2021-06-14 10:26:09 +01:00
Erik Johnston c842c581ed
When joining a remote room limit the number of events we concurrently check signatures/hashes for (#10117)
If we do hundreds of thousands at once the memory overhead can easily reach 500+ MB.
2021-06-08 11:07:46 +01:00
Erik Johnston 78b5102ae7
Fix up `BatchingQueue` (#10078)
Fixes #10068
2021-05-27 14:32:31 +01:00
Richard van der Hoff 224f2f949b
Combine `LruCache.invalidate` and `invalidate_many` (#9973)
* Make `invalidate` and `invalidate_many` do the same thing

... so that we can do either over the invalidation replication stream, and also
because they always confused me a bit.

* Kill off `invalidate_many`

* changelog
2021-05-27 10:33:56 +01:00
Patrick Cloke 7adcb20fc0
Add missing type hints to synapse.util (#9982) 2021-05-24 15:32:01 -04:00
Richard van der Hoff c0df6bae06
Remove `keylen` from `LruCache`. (#9993)
`keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it.

The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`).

Commits should be independently reviewable.
2021-05-24 14:02:01 +01:00
Erik Johnston 3e831f24ff
Don't hammer the database for destination retry timings every ~5mins (#10036) 2021-05-21 17:57:08 +01:00
Erik Johnston 7958eadcd1
Add a batching queue implementation. (#10017) 2021-05-21 11:20:51 +01:00
Richard van der Hoff 5090f26b63
Minor `@cachedList` enhancements (#9975)
- use a tuple rather than a list for the iterable that is passed into the
  wrapped function, for performance

- test that we can pass an iterable and that keys are correctly deduped.
2021-05-14 11:12:36 +01:00
Dan Callahan bd918d874f
Simplify exception handling in is_ascii. (#9985)
We can get away with just catching UnicodeError here.

    ⋮
    +-- ValueError
    |    +-- UnicodeError
    |         +-- UnicodeDecodeError
    |         +-- UnicodeEncodeError
    |         +-- UnicodeTranslateError
    ⋮

https://docs.python.org/3/library/exceptions.html#exception-hierarchy

Signed-off-by: Dan Callahan <danc@element.io>
2021-05-14 10:58:52 +01:00
Dan Callahan 498084228b
Use Python's secrets module instead of random (#9984)
Functionally identical, but more obviously cryptographically secure.
...Explicit is better than implicit?

Avoids needing to know that SystemRandom() implies a CSPRNG, and
complies with the big scary red box on the documentation for random:

> Warning:
>   The pseudo-random generators of this module should not be used for
>   security purposes. For security or cryptographic uses, see the
>   secrets module.

https://docs.python.org/3/library/random.html

Signed-off-by: Dan Callahan <danc@element.io>
2021-05-14 10:58:46 +01:00
Richard van der Hoff 7562d887e1
Change the format of access tokens away from macaroons (#5588) 2021-05-12 15:04:51 +01:00
Brendan Abolivier 652a6b094d Merge branch 'master' into develop 2021-05-11 14:15:30 +01:00
Richard van der Hoff 03318a766c
Merge pull request from GHSA-x345-32rc-8h85
* tests for push rule pattern matching

* tests for acl pattern matching

* factor out common `re.escape`

* Factor out common re.compile

* Factor out common anchoring code

* add word_boundary support to `glob_to_regex`

* Use `glob_to_regex` in push rule evaluator

NB that this drops support for character classes. I don't think anyone ever
used them.

* Improve efficiency of globs with multiple wildcards

The idea here is that we compress multiple `*` globs into a single `.*`. We
also need to consider `?`, since `*?*` is as hard to implement efficiently as
`**`.

* add assertion on regex pattern

* Fix mypy

* Simplify glob_to_regex

* Inline the glob_to_regex helper function

Signed-off-by: Dan Callahan <danc@element.io>

* Moar comments

Signed-off-by: Dan Callahan <danc@element.io>

Co-authored-by: Dan Callahan <danc@element.io>
2021-05-11 11:47:23 +02:00
Erik Johnston ef889c98a6
Optionally track memory usage of each LruCache (#9881)
This will double count slightly in the presence of interned strings. It's off by default as it can consume a lot of resources.
2021-05-05 16:54:36 +01:00
Erik Johnston 391bfe9a7b
Reduce memory footprint of caches (#9886) 2021-04-28 11:59:28 +01:00
Andrew Morgan fe604a022a
Remove various bits of compatibility code for Python <3.6 (#9879)
I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.
2021-04-27 13:13:07 +01:00
Erik Johnston 177dae2704
Limit length of accepted email addresses (#9855) 2021-04-22 17:49:11 +01:00
Richard van der Hoff 294c675033
Remove `synapse.types.Collection` (#9856)
This is no longer required, since we have dropped support for Python 3.5.
2021-04-22 16:43:50 +01:00
Andrew Morgan 6982db9651 Merge branch 'master' into develop 2021-04-20 14:55:16 +01:00
Patrick Cloke b076bc276e
Always use the name as the log ID. (#9829)
As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used.

This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
2021-04-20 14:19:00 +01:00
Denis Kasak e694a598f8
Sanity check identity server passed to bind/unbind. (#9802)
Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
2021-04-19 17:21:46 +01:00
Jonathan de Jong 4b965c862d
Remove redundant "coding: utf-8" lines (#9786)
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Jonathan de Jong 2ca4e349e9
Bugbear: Add Mutable Parameter fixes (#9682)
Part of #9366

Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-04-08 22:38:54 +01:00
Patrick Cloke 44bb881096
Add type hints to expiring cache. (#9730) 2021-04-06 08:58:18 -04:00
Jonathan de Jong e2b8a90897
Update mypy configuration: `no_implicit_optional = True` (#9742) 2021-04-05 09:10:18 -04:00
Patrick Cloke 01dd90b0f0
Add type hints to DictionaryCache and TTLCache. (#9442) 2021-03-29 12:15:33 -04:00
Erik Johnston b5efcb577e
Make it possible to use dmypy (#9692)
Running `dmypy run` will do a `mypy` check while spinning up a daemon
that makes rerunning `dmypy run` a lot faster.

`dmypy` doesn't support `follow_imports = silent` and has
`local_partial_types` enabled, so this PR enables those options and
fixes the issues that were newly raised. Note that `local_partial_types`
will be enabled by default in upcoming mypy releases.
2021-03-26 16:49:46 +00:00
Jonathan de Jong 4c3827f2c1
Enable addtional flake8-bugbear linting checks. (#9659) 2021-03-24 09:34:30 -04:00
Ankit Dobhal d66f9070cd
Fixed code misc. quality issues (#9649)
- Merge 'isinstance' calls.
- Remove unnecessary dict call outside of comprehension.
- Use 'sys.exit()' calls.
2021-03-22 11:18:13 -04:00
Jonathan de Jong 9898470e7d
Add logging to ObservableDeferred callbacks (#9523) 2021-03-09 11:09:31 +00:00
Jonathan de Jong d6196efafc
Add ResponseCache tests. (#9458) 2021-03-08 14:00:07 -05:00
Richard van der Hoff 7eb6e39a8f
Record the SSO Auth Provider in the login token (#9510)
This great big stack of commits is a a whole load of hoop-jumping to make it easier to store additional values in login tokens, and then to actually store the SSO Identity Provider in the login token. (Making use of that data will follow in a subsequent PR.)
2021-03-04 14:44:22 +00:00
Patrick Cloke aee10768d8 Revert "Fix #8518 (sync requests being cached wrongly on timeout) (#9358)"
This reverts commit f5c93fc993.

This is being backed out due to a regression (#9507) and additional
review feedback being provided.
2021-03-02 09:43:34 -05:00
Jonathan de Jong f5c93fc993
Fix #8518 (sync requests being cached wrongly on timeout) (#9358)
This fixes #8518 by adding a conditional check on `SyncResult` in a function when `prev_stream_token == current_stream_token`, as a sanity check. In `CachedResponse.set.<remove>()`, the result is immediately popped from the cache if the conditional function returns "false".

This prevents the caching of a timed-out `SyncResult` (that has `next_key` as the stream key that produced that `SyncResult`). The cache is prevented from returning a `SyncResult` that makes the client request the same stream key over and over again, effectively making it stuck in a loop of requesting and getting a response immediately for as long as the cache keeps those values.

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-02-24 13:57:00 +00:00
Eric Eastwood 0a00b7ff14
Update black, and run auto formatting over the codebase (#9381)
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Richard van der Hoff 3b754aea27
Clean up caching/locking of OIDC metadata load (#9362)
Ensure that we lock correctly to prevent multiple concurrent metadata load
requests, and generally clean up the way we construct the metadata cache.
2021-02-16 16:27:38 +00:00
Patrick Cloke 7950aa8a27 Fix some typos. 2021-02-12 11:14:12 -05:00
Patrick Cloke 2c9b4a5f16 Synapse 1.27.0rc2 (2021-02-11)
==============================
 
 Features
 --------
 
 - Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297))
 
 Bugfixes
 --------
 
 - Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302))
 - Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313))
 
 Improved Documentation
 ----------------------
 
 - Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAlX4AACgkQM/xY9qcR
 MEjxhg/+JzOGDmgxy4Vm9oU84htkkJsflU755ykxhwbIjVy2j+07pMgtWMmebXIS
 /tWjinwRB9OgKIK+j5RWdYzz/GRTkLCbafTLhhNOWsnaCO7nnAfZFozLOMd9g2qN
 309cS8efZkxGCISGSRm1QQKjQDx6HHx7nZVAwpqb778Q9TMrBYRAQeCk9yws7FfL
 GsP5YrBI42n84YhvflWA8J8QRYkeJJaggVWzE0XvwdI6raY1BmxMKcDvyl9iKpGb
 /QXsdketD3eJxdOlU92O5ZxRXmL00bdyAFdw2+J3Y+wjoGBnC5njKSFG3j8Z2UcP
 xIdB6w/zyGoPLKd4s7tkDI3axE7qrwFetA4NytannWGEHZ9q5tgOd5PA1kTeKYvn
 ao2os4pKMjWQOHiWLskqZVXgmoW5Kb2zgyZU/vhFAz8ImuidFULOjpEytAWqtt1B
 iuL/fRlM3z0BpwQNd3NBsblYTpZri8gdTp7ULJYtaKpT2MeG0sg3swJAptUjw9L4
 awbQuBFZ8o/E/0xwiGieSxbR6b6Nz5WiCyLaeMz6b3Y4YdomttOAZ6pL6E9y8ygU
 2wKYleSmg+tbaBKKZR5CSCwXOduSYPhWw7oqsfEfvT0NCQLuItfGNFZqxRzsf73H
 lCV0wwxNZBv8kRyMs6KivMvSI6SKinAjXS8AjzHE402ozdwOCv0=
 =NSyp
 -----END PGP SIGNATURE-----

Merge tag 'v1.27.0rc2' into develop

Synapse 1.27.0rc2 (2021-02-11)
==============================

Features
--------

- Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297))

Bugfixes
--------

- Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302))
- Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313))

Improved Documentation
----------------------

- Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
2021-02-11 11:56:03 -05:00
Patrick Cloke e40d88cff3
Backout changes for automatically calculating the public baseurl. (#9313)
This breaks some people's configurations (if their Client-Server API
is not accessed via port 443).
2021-02-11 11:16:54 -05:00
Patrick Cloke 3f58fc848d
Type hints and validation improvements. (#9321)
* Adds type hints to the groups servlet and stringutils code.
* Assert the maximum length of some input values for spec compliance.
2021-02-08 13:59:54 -05:00
Richard van der Hoff 18ab35284a Merge branch 'social_login' into develop 2021-02-01 17:28:37 +00:00
Richard van der Hoff 4167494c90
Replace username picker with a template (#9275)
There's some prelimiary work here to pull out the construction of a jinja environment to a separate function.

I wanted to load the template at display time rather than load time, so that it's easy to update on the fly. Honestly, I think we should do this with all our templates: the risk of ending up with malformed templates is far outweighed by the improved turnaround time for an admin trying to update them.
2021-02-01 15:52:50 +00:00
Andrew Morgan a64c29926e
Pass a dict, instead of None, to modules if a None config is specified in the homeserver config (#9229)
If a Synapse module's config block were empty in YAML, thus being translated to a `Nonetype` in Python, then some modules could fail as that None ends up getting passed to their `parse_config` method. Modules are expected to accept a `dict` instead.

This PR ensures that if the user does end up specifying an empty config block (such as what [the default oidc config in the sample config](5310808d3b/docs/sample_config.yaml (L1816-L1845)) states) then `None` is not passed to the module. An empty dict is passed instead.

This code assumes that no existing modules are relying on receiving a `None` config block, but I'd really hope that they aren't.
2021-01-27 11:49:31 +00:00
Erik Johnston 056327457f
Fix chain cover update to handle events with duplicate auth events (#9210) 2021-01-22 19:44:08 +00:00
Richard van der Hoff 0cd2938bc8
Support icons for Identity Providers (#9154) 2021-01-20 08:15:14 -05:00
Richard van der Hoff 9ffac2bef1
Remote dependency on distutils (#9125)
`distutils` is pretty much deprecated these days, and replaced with
`setuptools`. It's also annoying because it's you can't `pip install` it, and
it's hard to figure out which debian package we should depend on to make sure
it's there.

Since we only use it for a tiny function anyway, let's just vendor said
function into our codebase.
2021-01-15 15:59:20 +00:00
Erik Johnston 1a08e0cdab
Fix event chain bg update. (#9118)
We passed in a graph to `sorted_topologically` which didn't have an
entry for each node (as we dropped nodes with no edges).
2021-01-14 18:57:32 +00:00
Erik Johnston 1315a2e8be
Use a chain cover index to efficiently calculate auth chain difference (#8868) 2021-01-11 16:09:22 +00:00
Erik Johnston a03d71dc9d
Fix "Starting metrics collection from sentinel context" errors (#9053) 2021-01-08 14:33:53 +00:00
Patrick Cloke 1b4d5d6acf
Empty iterables should count towards cache usage. (#9028) 2021-01-06 12:33:20 -05:00
Patrick Cloke 06fefe0bb1
Add type hints to the logging context code. (#8939) 2021-01-05 08:06:55 -05:00
David Teller f14428b25c
Allow spam-checker modules to be provide async methods. (#8890)
Spam checker modules can now provide async methods. This is implemented
in a backwards-compatible manner.
2020-12-11 14:05:15 -05:00
Richard van der Hoff ab7a24cc6b
Better formatting for config errors from modules (#8874)
The idea is that the parse_config method of extension modules can raise either a ConfigError or a JsonValidationError,
and it will be magically turned into a legible error message. There's a few components to it:

* Separating the "path" and the "message" parts of a ConfigError, so that we can fiddle with the path bit to turn it
   into an absolute path.
* Generally improving the way ConfigErrors get printed.
* Passing in the config path to load_module so that it can wrap any exceptions that get caught appropriately.
2020-12-08 14:04:35 +00:00
Richard van der Hoff cbc82aa09f
Implement and use an @lru_cache decorator (#8595)
We don't always need the full power of a DeferredCache.
2020-10-30 11:43:17 +00:00
Richard van der Hoff c97da1e45d
Merge pull request #8678 from matrix-org/rav/fix_frozen_events
Fix serialisation errors when using third-party event rules.
2020-10-28 20:41:42 +00:00
Richard van der Hoff b6ca69e4f1 Remove frozendict_json_encoder and support frozendicts everywhere
Not being able to serialise `frozendicts` is fragile, and it's annoying to have
to think about which serialiser you want. There's no real downside to
supporting frozendicts, so let's just have one json encoder.
2020-10-28 15:56:57 +00:00
Dan Callahan aff1eb7c67
Tell Black to format code for Python 3.5 (#8664)
This allows trailing commas in multi-line arg lists.

Minor, but we might as well keep our formatting current with regard to
our minimum supported Python version.

Signed-off-by: Dan Callahan <danc@element.io>
2020-10-27 23:26:36 +00:00
Richard van der Hoff b28aaeb3a5
Optimise CacheDescriptor (#8594)
don't bother constricting a CacheContext unless we need one.
2020-10-21 22:57:45 +01:00
Richard van der Hoff c13820bcee fix failure case 2020-10-21 18:54:53 +01:00
Richard van der Hoff 2b3af01791 optimise DeferredCache.set 2020-10-21 17:55:53 +01:00
Richard van der Hoff 1f4269700c Push some deferred wrangling down into DeferredCache 2020-10-21 15:39:25 +01:00
Richard van der Hoff 96e7d3c4a0
Fix 'LruCache' object has no attribute '_on_resize' (#8591)
We need to make sure we are readu for the `set_cache_factor` callback.
2020-10-19 21:13:50 +01:00
Richard van der Hoff 903d11c43a
Add `DeferredCache.get_immediate` method (#8568)
* Add `DeferredCache.get_immediate` method

A bunch of things that are currently calling `DeferredCache.get` are only
really interested in the result if it's completed. We can optimise and simplify
this case.

* Remove unused 'default' parameter to DeferredCache.get()

* another get_immediate instance
2020-10-19 15:00:12 +01:00
Richard van der Hoff 97647b33c2
Replace DeferredCache with LruCache where possible (#8563)
Most of these uses don't need a full-blown DeferredCache; LruCache is lighter and more appropriate.
2020-10-19 12:20:29 +01:00
Richard van der Hoff 6d7b22041d review comments 2020-10-16 16:25:15 +01:00
Richard van der Hoff 995cc615a0
Apply suggestions from code review
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2020-10-16 16:14:42 +01:00
Richard van der Hoff 0ec0bc3886 type annotations for LruCache 2020-10-16 15:56:39 +01:00
Richard van der Hoff 3ee17585cd
Make LruCache register its own metrics (#8561)
rather than have everything that instantiates an LruCache manage metrics
separately, have LruCache do it itself.
2020-10-16 15:51:57 +01:00
Richard van der Hoff 8075504a60
Enable mypy for synapse.util.caches (#8547)
This seemed to entail dragging in a type stub for SortedList.
2020-10-15 11:44:39 +01:00