Commit Graph

757 Commits

Author SHA1 Message Date
Sean Quah 79e7c2c426
Fix edge case where a `Linearizer` could get stuck (#12358)
Just after a task acquires a contended `Linearizer` lock, it sleeps.
If the task is cancelled during this sleep, we need to release the lock.

Signed-off-by: Sean Quah <seanq@element.io>
2022-04-05 17:19:16 +01:00
Sean Quah 800ba87cc8
Refactor and convert `Linearizer` to async (#12357)
Refactor and convert `Linearizer` to async. This makes a `Linearizer`
cancellation bug easier to fix.

Also refactor to use an async context manager, which eliminates an
unlikely footgun where code that doesn't immediately use the context
manager could forget to release the lock.

Signed-off-by: Sean Quah <seanq@element.io>
2022-04-05 15:43:52 +01:00
Patrick Cloke 11df4ec6c2
Add more type hints to the main state store. (#12267) 2022-03-31 13:38:09 -04:00
David Robertson a2b00a4486
Bump `black` and `click` versions (#12320) 2022-03-29 10:41:19 +00:00
David Robertson bf9d549e3a
Try to detect borked package installations. (#12244)
* Try to detect borked package installations.

Fixes #12223.

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-03-18 19:03:46 +00:00
Sean Quah 2fcf4b3f6c
Add cancellation support to `@cached` and `@cachedList` decorators (#12183)
These decorators mostly support cancellation already. Add cancellation
tests and fix use of finished logging contexts by delaying cancellation,
as suggested by @erikjohnston.

Signed-off-by: Sean Quah <seanq@element.io>
2022-03-14 19:04:29 +00:00
Sean Quah 605d161d7d
Add cancellation support to `ReadWriteLock` (#12120)
Also convert `ReadWriteLock` to use async context managers.

Signed-off-by: Sean Quah <seanq@element.io>
2022-03-14 18:49:07 +00:00
Sean Quah 90b2327066
Add `delay_cancellation` utility function (#12180)
`delay_cancellation` behaves like `stop_cancellation`, except it
delays `CancelledError`s until the original `Deferred` resolves.
This is handy for unifying cleanup paths and ensuring that uncancelled
coroutines don't use finished logcontexts.

Signed-off-by: Sean Quah <seanq@element.io>
2022-03-14 17:52:15 +00:00
Brendan Abolivier e6a106fd5e
Implement a Jinja2 filter to extract localparts from email addresses (#12212) 2022-03-11 15:15:11 +00:00
Patrick Cloke bc9dff1d95
Remove unnecessary pass statements. (#12206) 2022-03-11 07:06:21 -05:00
reivilibre 72e7f1c420
Remove workaround introduced in Synapse v1.50.0rc1 for Mjolnir compatibility. Breaks compatibility with Mjolnir v1.3.1 and earlier. (#11700) 2022-03-10 15:53:23 +00:00
Patrick Cloke 690cb4f3b3
Allow for ignoring some arguments when caching. (#12189)
* `@cached` can now take an `uncached_args` which is an iterable of names to not use in the cache key.
* Requires `@cached`, @cachedList` and `@lru_cache` to use keyword arguments for clarity.
* Asserts that keyword-only arguments in cached functions are not accepted. (I tested this briefly and I don't believe this works properly.)
2022-03-09 18:07:41 +00:00
Olivier Wilkinson (reivilibre) 0dc9c5653c Merge branch 'master' into develop 2022-03-08 15:37:35 +00:00
reivilibre 2eef234ae3
Fix a bug introduced in 1.54.0rc1 which meant that Synapse would refuse to start if pre-release versions of dependencies were installed. (#12177)
* Add failing test to characterise the regression #12176

* Permit pre-release versions of specified packages

* Newsfile (bugfix)

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-03-08 10:47:28 +00:00
Richard van der Hoff 00a67f831a Merge remote-tracking branch 'origin/release-v1.54' into develop 2022-03-04 22:40:51 +00:00
Sean Quah 75574726a7
Add type hints for `ObservableDeferred` attributes (#12159)
Signed-off-by: Sean Quah <seanq@element.io>
2022-03-04 15:37:02 +00:00
David Robertson cea1b58c4a
Don't impose version checks on dev extras at runtime (#12129)
* Fix incorrect argument in test case

* Add copyright header

* Docstring and __all__

* Exclude dev depenencies

* Use changelog from #12088

* Include version in error messages

This will hopefully distinguish between the version of the source code
and the version of the distribution package that is installed.

* Linter script is your friend
2022-03-03 12:47:55 +00:00
David Robertson 313581e4e9
Use importlib.metadata to read requirements (#12088)
* Pull runtime dep checks into their own module
* Reimplement `check_requirements` using `importlib`

I've tried to make this clearer. We start by working out which of
Synapse's requirements we need to be installed here and now. I was
surprised that there wasn't an easier way to see which packages were
installed by a given extra.

I've pulled out the error messages into functions that deal with "is
this for an extra or not". And I've rearranged the loop over two
different sets of requirements into one loop with a "must be instaled"
flag.

I hope you agree that this is clearer.

* Test cases
2022-03-01 17:44:41 +00:00
Sean Quah 4d6b6c17c8
Fix rare error in `ReadWriteLock` when writers complete immediately (#12105)
Signed-off-by: Sean Quah <seanq@element.io>
2022-03-01 15:27:15 +00:00
Sean Quah 91bc15c772
Add `stop_cancellation` utility function (#12106) 2022-03-01 13:51:03 +00:00
Richard van der Hoff 5458eb8551
Fix 'Unhandled error in Deferred' (#12089)
* Fix 'Unhandled error in Deferred'

Fixes a CRITICAL "Unhandled error in Deferred" log message which happened when
a function wrapped with `@cachedList` failed

* Minor optimisation to cachedListDescriptor

we can avoid re-using `missing`, which saves looking up entries in
`deferreds_map`, and means we don't need to copy it.

* Improve type annotation on CachedListDescriptor
2022-03-01 09:51:38 +00:00
Richard van der Hoff 9d11fee8f2
Improve exception handling for concurrent execution (#12109)
* fix incorrect unwrapFirstError import

this was being imported from the wrong place

* Refactor `concurrently_execute` to use `yieldable_gather_results`

* Improve exception handling in `yieldable_gather_results`

Try to avoid swallowing so many stack traces.

* mark unwrapFirstError deprecated

* changelog
2022-03-01 09:34:30 +00:00
Sean Quah e6acd3cf4f
Upgrade mypy to version 0.931 (#12030)
Upgrade mypy to 0.931, mypy-zope to 0.3.5 and fix new complaints.
2022-02-18 15:57:26 +00:00
Erik Johnston 0dbbe33a65
Track cache invalidations (#12000)
Currently we only track evictions due to size or time constraints.
2022-02-15 14:31:04 +00:00
David Robertson 4ae956c8bb
Use version string helper from matrix-common (#11979)
* Require latest matrix-common
* Use the common function
2022-02-14 13:12:22 +00:00
Patrick Cloke d0e78af35e
Add missing type hints to synapse.replication. (#11938) 2022-02-08 11:03:08 -05:00
Brendan Abolivier 0640f8ebaa
Add a callback to allow modules to deny 3PID (#11854)
Part of the Tchap Synapse mainlining.

This allows modules to implement extra logic to figure out whether a given 3PID can be added to the local homeserver. In the Tchap use case, this will allow a Synapse module to interface with the custom endpoint /internal_info.
2022-02-08 11:20:32 +01:00
David Robertson f510fba4ba
Describe `prune_unread_entries` in docstrings (#11876)
Should have been caught in #10826.
2022-02-02 15:11:23 +00:00
Patrick Cloke 10a88ba91c
Use auto_attribs/native type hints for attrs classes. (#11692) 2022-01-13 13:49:28 +00:00
reivilibre eec34b1f2a
Work around Mjolnir compatibility issue by adding an import for `glob_to_regex` in `synapse.util`, where it moved from. (#11696) 2022-01-06 16:36:26 +00:00
reivilibre 84bfe47b01
Re-apply: Move glob_to_regex and re_word_boundary to matrix-python-common #11505 (#11687)
Co-authored-by: Sean Quah <seanq@element.io>
2022-01-05 11:41:49 +00:00
Shay fcfe67578f
Update to the current version of Black and run it on Synapse codebase (#11596)
* update black version

* run updated version of black on code

* newsfragment

* enumerate python versions
2021-12-23 20:22:15 -08:00
Richard van der Hoff c3e38b88f2
Improve opentracing support for `ResponseCache` (#11607)
This adds some opentracing annotations to ResponseCache, to make it easier to see what's going on; in particular, it adds a link back to the initial trace which is actually doing the work of generating the response.
2021-12-20 18:12:08 +00:00
Sean Quah 0147b3de20
Add missing type hints to `synapse.logging.context` (#11556) 2021-12-14 17:35:28 +00:00
Sean Quah 088d748f2c
Revert "Move `glob_to_regex` and `re_word_boundary` to `matrix-python-common` (#11505) (#11527)
This reverts commit a77c369897.
2021-12-07 13:51:11 +00:00
Sean Quah a77c369897
Move `glob_to_regex` and `re_word_boundary` to `matrix-python-common` (#11505) 2021-12-06 11:36:08 +00:00
Patrick Cloke 7b62791e00
Clean-up get_version_string (#11468) 2021-12-01 12:43:32 -05:00
Sean Quah 7ff22d6da4
Fix `LruCache` corruption bug with a `size_callback` that can return 0 (#11454)
When all entries in an `LruCache` have a size of 0 according to the
provided `size_callback`, and `drop_from_cache` is called on a cache
node, the node would be unlinked from the LRU linked list but remain in
the cache dictionary. An assertion would be later be tripped due to the
inconsistency.

Avoid unintentionally calling `__len__` and use a strict `is None`
check instead when unwrapping the weak reference.
2021-11-30 16:28:02 +00:00
Sean Quah 5a0b652d36
Eliminate a few `Any`s in `LruCache` type hints (#11453) 2021-11-30 15:39:07 +00:00
Sean Quah 454c3d7694 Merge branch 'master' into develop 2021-11-23 13:06:56 +00:00
Sean Quah 91f2bd0907 Prevent the media store from writing outside of the configured directory
Also tighten validation of server names by forbidding invalid characters
in IPv6 addresses and empty domain labels.
2021-11-19 13:39:15 +00:00
Sean Quah 84fac0f814
Add type annotations to `synapse.metrics` (#10847) 2021-11-17 19:07:02 +00:00
Patrick Cloke 7468723697
Add most missing type hints to synapse.util (#11328) 2021-11-16 08:47:36 -05:00
Patrick Cloke b64b6d12d4
Add more type hints to synapse.util. (#11321) 2021-11-12 13:43:06 +00:00
Patrick Cloke 5cace20bf1
Add missing type hints to `synapse.app`. (#11287) 2021-11-10 15:06:54 -05:00
Richard van der Hoff 46d0937447
ObservableDeferred: run observers in order (#11229) 2021-11-02 00:17:35 +00:00
Erik Johnston 7004f43da1
Move DNS lookups into separate thread pool (#11177)
This is to stop large bursts of lookups starving out other users of the
thread pools.

Fixes #11049.
2021-10-26 13:45:38 +01:00
David Robertson 797ee7812d
Relax `ignore-missing-imports` for modules that have stubs now and update mypy (#11006)
Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details.
We therefore pull in stub packages in setup.py

Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them.

The rest of this change consists of fixups to make the newer mypy + stubs pass CI.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-10-08 14:49:41 +01:00
Sean Quah 49a683d871
Fix long-standing bug where `ReadWriteLock` could drop logging contexts (#10993)
Use `PreserveLoggingContext()` to ensure that logging contexts are not
lost when exiting a read/write lock.

When exiting a read/write lock, callbacks on a `Deferred` are triggered
as a signal to any waiting coroutines. Any waiting coroutine that
becomes runnable is likely to follow the Synapse logging context rules
and will restore its own logging context, then either run to completion
or await another `Deferred`, resetting the logging context in the
process.
2021-10-08 12:27:16 +01:00
David Robertson f8d0f72b27
More types for synapse.util, part 1 (#10888)
The following modules now pass `disallow_untyped_defs`:

* synapse.util.caches.cached_call 
* synapse.util.caches.lrucache
* synapse.util.caches.response_cache 
* synapse.util.caches.stream_change_cache
* synapse.util.caches.ttlcache pass
* synapse.util.daemonize
* synapse.util.patch_inline_callbacks pass `no-untyped-defs`
* synapse.util.versionstring

Additional typing in synapse.util.metrics. Didn't get this to pass `no-untyped-defs`, think I'll need to watch #10847
2021-10-06 11:20:49 +01:00