Commit Graph

314 Commits

Author SHA1 Message Date
Erik Johnston ffad4fe35b Don't update event cache hit ratio from get_joined_users
Otherwise the hit ration of plain get_events gets completely skewed by
calls to get_joined_users* functions.
2017-05-08 16:06:17 +01:00
Erik Johnston d2d8ed4884 Optimise caches with single key 2017-05-04 14:18:46 +01:00
Richard van der Hoff 2e996271fe Instantiate DeferredTimedOutError correctly
Call `super` correctly, so that we correctly initialise the `errcode` field.

Fixes https://github.com/matrix-org/synapse/issues/2179.
2017-05-02 13:26:17 +01:00
Erik Johnston d9aa645f86 Reduce size of joined_user cache
The _get_joined_users_from_context cache stores a mapping from user_id
to avatar_url and display_name. Instead of storing those in a dict,
store them in a namedtuple as that uses much less memory.

We also try converting the string to ascii to further reduce the size.
2017-04-25 14:38:51 +01:00
Erik Johnston efab1dadde Remove DEBUG_CACHES 2017-04-25 10:54:09 +01:00
Erik Johnston 119cb9bbcf Reduce cache size by not storing deferreds
Currently the cache descriptors store deferreds rather than raw values,
this is a simple way of triggering only one database hit and sharing the
result if two callers attempt to get the same value.

However, there are a few caches that simply store a mapping from string
to string (or int). These caches can have a large number of entries,
under the assumption that each entry is small. However, the size of a
deferred (specifically the size of ObservableDeferred) is signigicantly
larger than that of the raw value, 2kb vs 32b.

This PR therefore changes the cache descriptors to store the raw values
rather than the deferreds.

As a side effect cached storage function now either return a deferred or
the actual value, as the cached list decriptor already does. This is
fine as we always end up just yield'ing on the returned value
eventually, which handles that case correctly.
2017-04-25 10:23:11 +01:00
Erik Johnston d134d0935e Only intern ascii strings 2017-04-24 14:07:48 +01:00
Richard van der Hoff e2eebf1696 Fix fixme in preserve_fn
`preserve_fn` is no longer used as a decorator anywhere, so we can safely fix a
fixme therein.
2017-04-03 15:38:02 +01:00
Erik Johnston 4d17add8de Remove unused instance variable 2017-03-31 09:38:27 +01:00
Erik Johnston 5b5b171f3e Docs 2017-03-30 17:05:53 +01:00
Erik Johnston b282fe7170 Revert log context change 2017-03-30 17:03:59 +01:00
Erik Johnston 6194a64ae9 Doc new instance variables 2017-03-30 14:19:10 +01:00
Erik Johnston 014fee93b3 Manually calculate cache key as getcallargs is expensive
This is because getcallargs recomputes the getargspec, amongst other
things, which we don't need to do as its already been done
2017-03-30 14:14:46 +01:00
Erik Johnston 86780a8bc3 Don't convert to deferreds when not necessary 2017-03-30 14:14:36 +01:00
Richard van der Hoff f9b4bb05e0 Fix the logcontext handling in the cache wrappers (#2077)
The cache wrappers had a habit of leaking the logcontext into the reactor while
the lookup function was running, and then not restoring it correctly when the
lookup function had completed. It's all the fault of
`preserve_context_over_{fn,deferred}` which are basically a bit broken.
2017-03-30 13:22:24 +01:00
Richard van der Hoff 9397edb28b Merge pull request #2050 from matrix-org/rav/federation_backoff
push federation retry limiter down to matrixfederationclient
2017-03-23 22:27:01 +00:00
Richard van der Hoff 06ce7335e9 Merge pull request #2052 from matrix-org/rav/time_bound_deferred
Fix time_bound_deferred to throw the right exception
2017-03-23 22:22:54 +00:00
Richard van der Hoff 5a16cb4bf0 Ignore backoff history for invites, aliases, and roomdirs
Add a param to the federation client which lets us ignore historical backoff
data for federation queries, and set it for a handful of operations.
2017-03-23 12:23:22 +00:00
Richard van der Hoff b88a323ffb Fix time_bound_deferred to throw the right exception
Due to a failure to instantiate DeferredTimedOutError, time_bound_deferred
would throw a CancelledError when the deferred timed out, which was rather
confusing.
2017-03-23 12:07:11 +00:00
Richard van der Hoff 4bd597d9fc push federation retry limiter down to matrixfederationclient
rather than having to instrument everywhere we make a federation call,
make the MatrixFederationHttpClient manage the retry limiter.
2017-03-23 09:28:46 +00:00
Richard van der Hoff 19b9366d73 Fix a couple of logcontext leaks
Use preserve_fn to correctly manage the logcontexts around things we don't want
to yield on.
2017-03-23 00:17:46 +00:00
Richard van der Hoff 95f21c7a66 Fix caching of remote servers' signature keys
The `@cached` decorator on `KeyStore._get_server_verify_key` was missing
its `num_args` parameter, which meant that it was returning the wrong key for
any server which had more than one recorded key.

By way of a fix, change the default for `num_args` to be *all* arguments. To
implement that, factor out a common base class for `CacheDescriptor` and `CacheListDescriptor`.
2017-03-22 15:11:30 +00:00
Richard van der Hoff bd08ee7a46 Merge pull request #2026 from matrix-org/rav/logcontext_docs
Logcontext docs
2017-03-20 12:05:21 +00:00
Richard van der Hoff f40c2db05a Stop preserve_fn leaking context into the reactor
Fix a bug in ``logcontext.preserve_fn`` which made it leak context into the
reactor, and add a test for it.

Also, get rid of ``logcontext.reset_context_after_deferred``, which tried to do
the same thing but had its own, different, set of bugs.
2017-03-18 00:07:43 +00:00
Richard van der Hoff d2d146a314 Logcontext docs 2017-03-17 23:59:28 +00:00
Richard van der Hoff 2abe85d50e Merge pull request #2016 from matrix-org/rav/queue_pdus_during_join
Queue up federation PDUs while a room join is in progress
2017-03-17 11:32:44 +00:00
Richard van der Hoff 29ed09e80a Fix assertion to stop transaction queue getting wedged
... and update some docstrings to correctly reflect the types being used.

get_new_device_msgs_for_remote can return a long under some circumstances,
which was being stored in last_device_list_stream_id_by_dest, and was then
upsetting things on the next loop.
2017-03-15 12:16:55 +00:00
Richard van der Hoff b5d1c68beb Implement reset_context_after_deferred
to correctly reset the context when we fire off a deferred we aren't going to
wait for.
2017-03-15 02:21:07 +00:00
David Baker 73a5f06652 Support registration / login with phone number
Changes from https://github.com/matrix-org/synapse/pull/1971
2017-03-13 17:27:51 +00:00
Erik Johnston 7eae6eaa2f Revert "Support registration & login with phone number" 2017-03-13 09:59:33 +00:00
Erik Johnston 3545e17f43 Add setdefault key to ExpiringCache 2017-03-10 10:30:49 +00:00
David Baker 9d0d40fc15 Docs 2017-03-08 19:05:29 +00:00
David Baker 3edc57296d Incorrectly copied copyright
This file post-dates OM
2017-03-08 19:00:51 +00:00
David Baker 1c99934b28 pep8 2017-03-08 11:58:20 +00:00
David Baker a9e2b9ec16 Add msisdn util file 2017-03-08 11:53:36 +00:00
Erik Johnston 6b61060b51 Comment 2017-02-02 14:47:15 +00:00
Erik Johnston 9efcc3f3be Comment 2017-02-02 13:50:22 +00:00
Erik Johnston df4ecff5a9 Correctly raise exceptions for ratelimitng. Ratelimit on 401 2017-02-01 15:42:19 +00:00
Erik Johnston fe08db2713 Remove explicit < 400 check as apparently this is confusing 2017-01-31 15:21:32 +00:00
Erik Johnston 4c0ec15bdc Comment 2017-01-31 13:53:46 +00:00
Erik Johnston 85c590105f Comment 2017-01-31 13:46:38 +00:00
Erik Johnston ae7a132f38 Better handle 404 response for federation /send/ 2017-01-31 13:40:09 +00:00
Erik Johnston c430111d0e Update LruCache size estimate on clear 2017-01-18 14:55:23 +00:00
Erik Johnston 380dba1020 Measure metrics of string_cache 2017-01-17 17:04:46 +00:00
Erik Johnston 37b4c7d8a9 Fix typo in return type 2017-01-17 14:43:32 +00:00
Erik Johnston d6c75cb7c2 Rename and comment tree_to_leaves_iterator 2017-01-17 11:47:03 +00:00
Erik Johnston 1ccd5676e3 Remove needless call to evict() 2017-01-17 11:42:26 +00:00
Erik Johnston f85b6ca494 Speed up cache size calculation
Instead of calculating the size of the cache repeatedly, which can take
a long time now that it can use a callback, instead cache the size and
update that on insertion and deletion.

This requires changing the cache descriptors to have two caches, one for
pending deferreds and the other for the actual values. There's no reason
to evict from the pending deferreds as they won't take up any more
memory.
2017-01-17 11:18:13 +00:00
Erik Johnston 6d00213e80 Use OrderedDict in ExpiringCache 2017-01-16 15:33:22 +00:00
Erik Johnston 46aebbbcbf Add support for 'iterable' to ExpiringCache 2017-01-16 14:57:23 +00:00