Commit Graph

555 Commits

Author SHA1 Message Date
Matthew Hodgson ab9f844aaf
Add federation_domain_whitelist option (#2820)
Add federation_domain_whitelist

gives a way to restrict which domains your HS is allowed to federate with.
useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network
2018-01-22 19:11:18 +01:00
Matthew Hodgson d84f65255e
Merge pull request #2813 from matrix-org/matthew/registrations_require_3pid
add registrations_require_3pid and allow_local_3pids
2018-01-22 13:57:22 +00:00
Matthew Hodgson 8fe253f19b fix PR nitpicking 2018-01-19 18:23:45 +00:00
Matthew Hodgson 447f4f0d5f rewrite based on PR feedback:
* [ ] split config options into allowed_local_3pids and registrations_require_3pid
 * [ ] simplify and comment logic for picking registration flows
 * [ ] fix docstring and move check_3pid_allowed into a new util module
 * [ ] use check_3pid_allowed everywhere

@erikjohnston PTAL
2018-01-19 15:33:55 +00:00
Erik Johnston b6dc7044a9
Merge pull request #2804 from matrix-org/erikj/file_consumer
Add decent impl of a FileConsumer
2018-01-18 16:31:33 +00:00
Richard van der Hoff d57765fc8a Fix bugs in block metrics
... which I introduced in #2785
2018-01-18 12:24:42 +00:00
Erik Johnston be0dfcd4a2 Do logcontexts correctly 2018-01-18 11:57:57 +00:00
Erik Johnston 1432f7ccd5 Move test stuff to tests 2018-01-18 11:57:57 +00:00
Erik Johnston 2f18a2647b Make all fields private 2018-01-18 11:57:54 +00:00
Erik Johnston dc519602ac Ensure we registerProducer isn't called twice 2018-01-18 11:07:17 +00:00
Erik Johnston 17b54389fe Fix _notify_empty typo 2018-01-18 11:05:34 +00:00
Erik Johnston 28b338ed9b Move definition of paused_producer to __init__ 2018-01-18 11:04:41 +00:00
Erik Johnston a177325b49 Fix comments 2018-01-18 11:02:43 +00:00
Erik Johnston bc67e7d260 Add decent impl of a FileConsumer
Twisted core doesn't have a general purpose one, so we need to write one
ourselves.

Features:
- All writing happens in background thread
- Supports both push and pull producers
- Push producers get paused if the consumer falls behind
2018-01-17 16:43:03 +00:00
Richard van der Hoff 3d12d97415 Track DB scheduling delay per-request
For each request, track the amount of time spent waiting for a db
connection. This entails adding it to the LoggingContext and we may as well add
metrics for it while we are passing.
2018-01-16 17:23:32 +00:00
Richard van der Hoff 6324b65f08 Track db txn time in millisecs
... to reduce the amount of floating-point foo we do.
2018-01-16 15:53:18 +00:00
Richard van der Hoff 44a498418c Optimise LoggingContext creation and copying
It turns out that the only thing we use the __dict__ of LoggingContext for is
`request`, and given we create lots of LoggingContexts and then copy them every
time we do a db transaction or log line, using the __dict__ seems a bit
redundant. Let's try to optimise things by making the request attribute
explicit.
2018-01-16 15:49:42 +00:00
Richard van der Hoff 39f4e29d01 Reorganise request and block metrics
In order to circumvent the number of duplicate foo:count metrics increasing
without bounds, it's time for a rearrangement.

The following are all deprecated, and replaced with synapse_util_metrics_block_count:
  synapse_util_metrics_block_timer:count
  synapse_util_metrics_block_ru_utime:count
  synapse_util_metrics_block_ru_stime:count
  synapse_util_metrics_block_db_txn_count:count
  synapse_util_metrics_block_db_txn_duration:count

The following are all deprecated, and replaced with synapse_http_server_response_count:
   synapse_http_server_requests
   synapse_http_server_response_time:count
   synapse_http_server_response_ru_utime:count
   synapse_http_server_response_ru_stime:count
   synapse_http_server_response_db_txn_count:count
   synapse_http_server_response_db_txn_duration:count

The following are renamed (the old metrics are kept for now, but deprecated):

  synapse_util_metrics_block_timer:total ->
     synapse_util_metrics_block_time_seconds

  synapse_util_metrics_block_ru_utime:total ->
     synapse_util_metrics_block_ru_utime_seconds

  synapse_util_metrics_block_ru_stime:total ->
     synapse_util_metrics_block_ru_stime_seconds

  synapse_util_metrics_block_db_txn_count:total ->
     synapse_util_metrics_block_db_txn_count

  synapse_util_metrics_block_db_txn_duration:total ->
     synapse_util_metrics_block_db_txn_duration_seconds

  synapse_http_server_response_time:total ->
     synapse_http_server_response_time_seconds

  synapse_http_server_response_ru_utime:total ->
     synapse_http_server_response_ru_utime_seconds

  synapse_http_server_response_ru_stime:total ->
     synapse_http_server_response_ru_stime_seconds

   synapse_http_server_response_db_txn_count:total ->
      synapse_http_server_response_db_txn_count

   synapse_http_server_response_db_txn_duration:total
      synapse_http_server_response_db_txn_duration_seconds
2018-01-15 17:09:44 +00:00
Richard van der Hoff b2cd6accf5 Remove __PreservingContextDeferred too 2017-11-14 23:00:10 +00:00
Richard van der Hoff 7e6fa29cb5 Remove preserve_context_over_{fn, deferred}
Both of these functions ae known to leak logcontexts. Replace the remaining
calls to them and kill them off.
2017-11-14 11:22:42 +00:00
Richard van der Hoff bf993db11c Logging and logcontext fixes for Limiter
Add some logging to the Limiter in a similar spirit to the Linearizer, to help
debug issues.

Also fix a logcontext leak.

Also refactor slightly to avoid throwing exceptions.
2017-11-07 00:48:57 +00:00
Richard van der Hoff 0be99858f3 fix vars named `l`
E741 says "do not use variables named ‘l’, ‘O’, or ‘I’".
2017-10-23 15:56:38 +01:00
Richard van der Hoff eaaabc6c4f replace 'except:' with 'except Exception:'
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Richard van der Hoff 2e9f5ea31a Fix logcontext handling for persist_events
* don't use preserve_context_over_deferred, which is known broken.

* remove a redundant preserve_fn.

* add/improve some comments
2017-10-17 10:59:30 +01:00
Richard van der Hoff cc794d60e7 Merge pull request #2532 from matrix-org/rav/fix_linearizer
Fix stackoverflow and logcontexts from linearizer
2017-10-11 17:29:32 +01:00
Richard van der Hoff f30c4ed2bc logformatter: fix AttributeError
make sure we have the relevant fields before we try to log them.
2017-10-11 17:26:17 +01:00
Richard van der Hoff 4fad8efbfb Fix stackoverflow and logcontexts from linearizer
1. make it not blow out the stack when there are more than 50 things waiting
   for a lock. Fixes https://github.com/matrix-org/synapse/issues/2505.

2. Make it not mess up the log contexts.
2017-10-11 15:05:05 +01:00
Richard van der Hoff 3cc852d339 Fancy logformatter to format exceptions better
This is a bit of an experimental change at this point; the idea is to see if it
helps us track down where our stack overflows are coming from by logging the
stack when the exception was caught and turned into a Failure. (We'll also need
edf2704420).

If we deploy this, we'll be able to enable it via the log config yaml.
2017-10-09 17:44:42 +01:00
Richard van der Hoff 148428ce76 Fix logcontext handling for concurrently_execute
Avoid preserve_context_over_deferred, which is broken.
2017-10-06 22:24:28 +01:00
David Baker 8ad5f34908 pep8 2017-09-26 19:21:41 +01:00
David Baker 9fd086e506 unnecessary parens 2017-09-26 17:59:46 +01:00
David Baker 0b03a97708 Add module_loader.py 2017-09-26 17:56:41 +01:00
Erik Johnston 495f075b41 Increase default cache factor size. 2017-07-04 09:58:32 +01:00
Erik Johnston b5e8d529e6 Define CACHE_SIZE_FACTOR once 2017-07-04 09:56:44 +01:00
Erik Johnston c72058bcc6 Use an ExpiringCache for storing registration sessions
This is because pruning them was a significant performance drain on
matrix.org
2017-06-29 14:08:37 +01:00
Erik Johnston efc2b7db95 Rewrite conditional 2017-06-09 13:35:15 +01:00
Erik Johnston eed59dcc1e Fix has_any_entity_changed
Occaisonally has_any_entity_changed would throw the error: "Set changed
size during iteration" when taking the max of the `sorteddict`. While
its uncertain how that happens, its quite inefficient to iterate over
the entire dict anyway so we change to using the more traditional
`bisect_*` functions.
2017-06-09 11:44:01 +01:00
Erik Johnston 304880d185 Add stream change cache 2017-05-31 15:46:36 +01:00
Erik Johnston bd7bb5df71 Pull out if statement from for loop 2017-05-22 15:12:19 +01:00
Erik Johnston e3417a06e2 Update list cache to handle one arg case
We update the normal cache descriptors to handle caches with a single
argument specially so that the key wasn't a 1-tuple. We need to update
the cache list to be aware of this.
2017-05-22 15:04:42 +01:00
Erik Johnston bbfe4e996c Make get_state_groups_from_groups faster.
Most of the time was spent copying a dict to filter out sentinel values
that indicated that keys did not exist in the dict. The sentinel values
were added to ensure that we cached the non-existence of keys.

By updating DictionaryCache to keep track of which keys were known to
not exist itself we can remove a dictionary copy.
2017-05-17 15:12:15 +01:00
Erik Johnston ffad4fe35b Don't update event cache hit ratio from get_joined_users
Otherwise the hit ration of plain get_events gets completely skewed by
calls to get_joined_users* functions.
2017-05-08 16:06:17 +01:00
Erik Johnston d2d8ed4884 Optimise caches with single key 2017-05-04 14:18:46 +01:00
Richard van der Hoff 2e996271fe Instantiate DeferredTimedOutError correctly
Call `super` correctly, so that we correctly initialise the `errcode` field.

Fixes https://github.com/matrix-org/synapse/issues/2179.
2017-05-02 13:26:17 +01:00
Erik Johnston d9aa645f86 Reduce size of joined_user cache
The _get_joined_users_from_context cache stores a mapping from user_id
to avatar_url and display_name. Instead of storing those in a dict,
store them in a namedtuple as that uses much less memory.

We also try converting the string to ascii to further reduce the size.
2017-04-25 14:38:51 +01:00
Erik Johnston efab1dadde Remove DEBUG_CACHES 2017-04-25 10:54:09 +01:00
Erik Johnston 119cb9bbcf Reduce cache size by not storing deferreds
Currently the cache descriptors store deferreds rather than raw values,
this is a simple way of triggering only one database hit and sharing the
result if two callers attempt to get the same value.

However, there are a few caches that simply store a mapping from string
to string (or int). These caches can have a large number of entries,
under the assumption that each entry is small. However, the size of a
deferred (specifically the size of ObservableDeferred) is signigicantly
larger than that of the raw value, 2kb vs 32b.

This PR therefore changes the cache descriptors to store the raw values
rather than the deferreds.

As a side effect cached storage function now either return a deferred or
the actual value, as the cached list decriptor already does. This is
fine as we always end up just yield'ing on the returned value
eventually, which handles that case correctly.
2017-04-25 10:23:11 +01:00
Erik Johnston d134d0935e Only intern ascii strings 2017-04-24 14:07:48 +01:00
Richard van der Hoff e2eebf1696 Fix fixme in preserve_fn
`preserve_fn` is no longer used as a decorator anywhere, so we can safely fix a
fixme therein.
2017-04-03 15:38:02 +01:00
Erik Johnston 4d17add8de Remove unused instance variable 2017-03-31 09:38:27 +01:00
Erik Johnston 5b5b171f3e Docs 2017-03-30 17:05:53 +01:00
Erik Johnston b282fe7170 Revert log context change 2017-03-30 17:03:59 +01:00
Erik Johnston 6194a64ae9 Doc new instance variables 2017-03-30 14:19:10 +01:00
Erik Johnston 014fee93b3 Manually calculate cache key as getcallargs is expensive
This is because getcallargs recomputes the getargspec, amongst other
things, which we don't need to do as its already been done
2017-03-30 14:14:46 +01:00
Erik Johnston 86780a8bc3 Don't convert to deferreds when not necessary 2017-03-30 14:14:36 +01:00
Richard van der Hoff f9b4bb05e0 Fix the logcontext handling in the cache wrappers (#2077)
The cache wrappers had a habit of leaking the logcontext into the reactor while
the lookup function was running, and then not restoring it correctly when the
lookup function had completed. It's all the fault of
`preserve_context_over_{fn,deferred}` which are basically a bit broken.
2017-03-30 13:22:24 +01:00
Richard van der Hoff 9397edb28b Merge pull request #2050 from matrix-org/rav/federation_backoff
push federation retry limiter down to matrixfederationclient
2017-03-23 22:27:01 +00:00
Richard van der Hoff 06ce7335e9 Merge pull request #2052 from matrix-org/rav/time_bound_deferred
Fix time_bound_deferred to throw the right exception
2017-03-23 22:22:54 +00:00
Richard van der Hoff 5a16cb4bf0 Ignore backoff history for invites, aliases, and roomdirs
Add a param to the federation client which lets us ignore historical backoff
data for federation queries, and set it for a handful of operations.
2017-03-23 12:23:22 +00:00
Richard van der Hoff b88a323ffb Fix time_bound_deferred to throw the right exception
Due to a failure to instantiate DeferredTimedOutError, time_bound_deferred
would throw a CancelledError when the deferred timed out, which was rather
confusing.
2017-03-23 12:07:11 +00:00
Richard van der Hoff 4bd597d9fc push federation retry limiter down to matrixfederationclient
rather than having to instrument everywhere we make a federation call,
make the MatrixFederationHttpClient manage the retry limiter.
2017-03-23 09:28:46 +00:00
Richard van der Hoff 19b9366d73 Fix a couple of logcontext leaks
Use preserve_fn to correctly manage the logcontexts around things we don't want
to yield on.
2017-03-23 00:17:46 +00:00
Richard van der Hoff 95f21c7a66 Fix caching of remote servers' signature keys
The `@cached` decorator on `KeyStore._get_server_verify_key` was missing
its `num_args` parameter, which meant that it was returning the wrong key for
any server which had more than one recorded key.

By way of a fix, change the default for `num_args` to be *all* arguments. To
implement that, factor out a common base class for `CacheDescriptor` and `CacheListDescriptor`.
2017-03-22 15:11:30 +00:00
Richard van der Hoff bd08ee7a46 Merge pull request #2026 from matrix-org/rav/logcontext_docs
Logcontext docs
2017-03-20 12:05:21 +00:00
Richard van der Hoff f40c2db05a Stop preserve_fn leaking context into the reactor
Fix a bug in ``logcontext.preserve_fn`` which made it leak context into the
reactor, and add a test for it.

Also, get rid of ``logcontext.reset_context_after_deferred``, which tried to do
the same thing but had its own, different, set of bugs.
2017-03-18 00:07:43 +00:00
Richard van der Hoff d2d146a314 Logcontext docs 2017-03-17 23:59:28 +00:00
Richard van der Hoff 2abe85d50e Merge pull request #2016 from matrix-org/rav/queue_pdus_during_join
Queue up federation PDUs while a room join is in progress
2017-03-17 11:32:44 +00:00
Richard van der Hoff 29ed09e80a Fix assertion to stop transaction queue getting wedged
... and update some docstrings to correctly reflect the types being used.

get_new_device_msgs_for_remote can return a long under some circumstances,
which was being stored in last_device_list_stream_id_by_dest, and was then
upsetting things on the next loop.
2017-03-15 12:16:55 +00:00
Richard van der Hoff b5d1c68beb Implement reset_context_after_deferred
to correctly reset the context when we fire off a deferred we aren't going to
wait for.
2017-03-15 02:21:07 +00:00
David Baker 73a5f06652 Support registration / login with phone number
Changes from https://github.com/matrix-org/synapse/pull/1971
2017-03-13 17:27:51 +00:00
Erik Johnston 7eae6eaa2f Revert "Support registration & login with phone number" 2017-03-13 09:59:33 +00:00
Erik Johnston 3545e17f43 Add setdefault key to ExpiringCache 2017-03-10 10:30:49 +00:00
David Baker 9d0d40fc15 Docs 2017-03-08 19:05:29 +00:00
David Baker 3edc57296d Incorrectly copied copyright
This file post-dates OM
2017-03-08 19:00:51 +00:00
David Baker 1c99934b28 pep8 2017-03-08 11:58:20 +00:00
David Baker a9e2b9ec16 Add msisdn util file 2017-03-08 11:53:36 +00:00
Erik Johnston 6b61060b51 Comment 2017-02-02 14:47:15 +00:00
Erik Johnston 9efcc3f3be Comment 2017-02-02 13:50:22 +00:00
Erik Johnston df4ecff5a9 Correctly raise exceptions for ratelimitng. Ratelimit on 401 2017-02-01 15:42:19 +00:00
Erik Johnston fe08db2713 Remove explicit < 400 check as apparently this is confusing 2017-01-31 15:21:32 +00:00
Erik Johnston 4c0ec15bdc Comment 2017-01-31 13:53:46 +00:00
Erik Johnston 85c590105f Comment 2017-01-31 13:46:38 +00:00
Erik Johnston ae7a132f38 Better handle 404 response for federation /send/ 2017-01-31 13:40:09 +00:00
Erik Johnston c430111d0e Update LruCache size estimate on clear 2017-01-18 14:55:23 +00:00
Erik Johnston 380dba1020 Measure metrics of string_cache 2017-01-17 17:04:46 +00:00
Erik Johnston 37b4c7d8a9 Fix typo in return type 2017-01-17 14:43:32 +00:00
Erik Johnston d6c75cb7c2 Rename and comment tree_to_leaves_iterator 2017-01-17 11:47:03 +00:00
Erik Johnston 1ccd5676e3 Remove needless call to evict() 2017-01-17 11:42:26 +00:00
Erik Johnston f85b6ca494 Speed up cache size calculation
Instead of calculating the size of the cache repeatedly, which can take
a long time now that it can use a callback, instead cache the size and
update that on insertion and deletion.

This requires changing the cache descriptors to have two caches, one for
pending deferreds and the other for the actual values. There's no reason
to evict from the pending deferreds as they won't take up any more
memory.
2017-01-17 11:18:13 +00:00
Erik Johnston 6d00213e80 Use OrderedDict in ExpiringCache 2017-01-16 15:33:22 +00:00
Erik Johnston 46aebbbcbf Add support for 'iterable' to ExpiringCache 2017-01-16 14:57:23 +00:00
Erik Johnston 2fae34bd2c Optionally measure size of cache by sum of length of values 2017-01-13 17:46:17 +00:00
Erik Johnston bf5c9706d9 Remove full_twisted_stacktraces option
The debug 'full_twisted_stacktraces' flag caused synapse to rewrite
twisted deferreds to always fire the callback on the next reactor tick.
This was to force the deferred to always store the stacktraces on
exceptions, and thus be more likely to have a full stacktrace when it
reaches the final error handlers and gets printed to the logs.

Dynamically rewriting things is generally bad, and in particular this
change violates assumptions of various bits of Twisted. This wouldn't
necessarily be so bad, but it turns out this option has been turned on
on some production servers.

Turning the option can cause e.g. #1778.

For now, lets just entirely nuke this option.
2017-01-12 10:32:52 +00:00
Erik Johnston f477370c0c Add paranoia exception catch in Linearizer 2017-01-10 14:04:13 +00:00
Mark Haines dd3df11c55 More logging for the linearizer and for get_events 2017-01-05 12:32:47 +00:00
Mark Haines 62ce3034f3 s/aquire/acquire/g 2016-12-30 20:04:44 +00:00
Mark Haines 0aff09f6c9 Add more useful logging when we block fetching events 2016-12-30 20:00:44 +00:00
Erik Johnston d53a80af25 Merge pull request #1620 from matrix-org/erikj/concurrent_room_access
Limit the number of events that can be created on a given room concurrently
2016-12-12 10:30:23 +00:00
Erik Johnston fbaf868f62 Correctly handle timeout errors 2016-12-09 16:30:29 +00:00
Erik Johnston 11bfe438a2 Use correct var 2016-11-24 15:26:53 +00:00
Erik Johnston aaecffba3a Correctly handle 500's and 429 on federation 2016-11-24 15:04:49 +00:00
Erik Johnston 90565d015e Invalidate retry cache in both directions 2016-11-22 17:45:44 +00:00
Erik Johnston f8ee66250a Handle sending events and device messages over federation 2016-11-17 15:48:04 +00:00
Erik Johnston d56c39cf24 Use external ldap auth pacakge 2016-11-15 13:03:19 +00:00
Kegan Dougal 3991b4cbdb Clean transactions based on time. Add HttpTransactionCache tests. 2016-11-14 11:19:24 +00:00
Erik Johnston 64038b806c Comments 2016-11-11 10:42:08 +00:00
Erik Johnston d073cb7ead Add Limiter: limit concurrent access to resource 2016-11-10 16:29:51 +00:00
Erik Johnston 27d3f2e7ab Explicitly set authentication mode in ldap3
This only makes a difference for versions of ldap3 before 1.0, but a)
its best to be explicit and b) there are distributions that package
ancient versions for ldap3 (e.g. debian).
2016-11-08 14:35:25 +00:00
Erik Johnston 850b103b36 Implement pluggable password auth
Allows delegating the password auth to an external module. This also
moves the LDAP auth to using this system, allowing it to be removed from
the synapse tree entirely in the future.
2016-10-03 10:36:40 +01:00
Erik Johnston 955f34d23e Change get_pos_of_last_change to return upper bound 2016-09-15 15:12:07 +01:00
Erik Johnston cb3edec6af Use stream_change cache to make get_forward_extremeties_for_room cache more effective 2016-09-15 14:28:13 +01:00
Erik Johnston 7356d52e73 Fix up push to use get_current_state_ids 2016-08-25 18:35:49 +01:00
Erik Johnston 9219139351 Preserve some logcontexts 2016-08-24 11:58:40 +01:00
Erik Johnston e65bc7d315 Merge pull request #1031 from matrix-org/erikj/measure_notifier
Add more Measure blocks
2016-08-22 12:13:07 +01:00
Erik Johnston 8731197e54 Only abort Measure on Exceptions 2016-08-19 18:23:45 +01:00
Erik Johnston 45fd2c8942 Ensure invalidation list does not grow unboundedly 2016-08-19 16:09:16 +01:00
Erik Johnston c0d7d9d642 Rename to on_invalidate 2016-08-19 15:13:58 +01:00
Erik Johnston dc76a3e909 Make cache_context an explicit option 2016-08-19 15:02:38 +01:00
Erik Johnston ba214a5e32 Remove lru option 2016-08-19 14:17:11 +01:00
Erik Johnston 4161ff2fc4 Add concept of cache contexts 2016-08-19 14:17:07 +01:00
Erik Johnston ca8abfbf30 Clean up TransactionQueue 2016-08-10 16:24:16 +01:00
Erik Johnston 3bc9629be5 Measure federation send transaction resources 2016-08-10 10:56:38 +01:00
Erik Johnston 24f36469bc Add federation /version API 2016-08-05 16:36:07 +01:00
Erik Johnston 7b0f6293f2 Merge pull request #940 from matrix-org/erikj/fed_state_cache
Cache federation state responses
2016-08-02 15:21:37 +01:00
Mark Haines bf81e38d36 Fix retry utils to check if the exception is a subclass of CME 2016-07-28 10:47:02 +01:00
David Baker 389c890f14 Don't include name of room for invites in push
Avoids insane pushes like, "Bob invited you to invite from Bob"
2016-07-28 10:20:47 +01:00
Matthew Hodgson 242c52d607 typo 2016-07-26 10:09:25 +02:00
Erik Johnston 248e6770ca Cache federation state responses 2016-07-21 10:30:12 +01:00
Erik Johnston 7335f0adda Add ReadWriteLock 2016-07-05 15:23:17 +01:00
David Baker 870c45913e Use similar naming we use in email notifs for push
Fixes https://github.com/vector-im/vector-web/issues/1654
2016-06-24 11:41:11 +01:00
Mark Haines 9f1800fba8 Remove registered_users from the distributor.
The only place that was observed was to set the profile. I've made it
so that the profile is set within store.register in the same transaction
that creates the user.

This required some slight changes to the registration code for upgrading
guest users, since it previously relied on the distributor swallowing errors
if the profile already existed.
2016-06-17 19:14:16 +01:00
Erik Johnston 3b096c5f5c Merge branch 'erikj/cache_perf' of github.com:matrix-org/synapse into develop 2016-06-03 12:00:33 +01:00
Erik Johnston 58a224a651 Pull out update_results_dict 2016-06-03 11:47:07 +01:00
Erik Johnston 73c7112433 Change CacheMetrics to be quicker
We change it so that each cache has an individual CacheMetric, instead
of having one global CacheMetric. This means that when a cache tries to
increment a counter it does not need to go through so many indirections.
2016-06-03 11:26:52 +01:00
Erik Johnston e043ede4a2 Small optimisation to CacheListDescriptor 2016-06-03 11:19:22 +01:00
Erik Johnston 597013caa5 Make cachedList go a bit faster 2016-06-03 11:13:29 +01:00
Matthew Hodgson 680f1d9387 catch thinko in presentable names 2016-05-23 22:55:11 +01:00
David Baker c7c75e87dc Docstring 2016-04-29 19:47:35 +01:00
David Baker ec9cbe847d pep8 newline 2016-04-29 10:07:30 +01:00
David Baker acded821c4 Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs 2016-04-29 10:05:20 +01:00
David Baker ebbabc4986 Handle room invites in email notifs 2016-04-28 11:49:36 +01:00
David Baker 7b4715bad7 More variable calculation for email notifs
Include name of the person we're sending to and add summary text at the top giving an overview of what's happened.
2016-04-25 18:27:04 +01:00
Mark Haines 72e2fafa20 Add a metrics listener and a ssh listener to the pusher 2016-04-25 17:34:25 +01:00
Mark Haines f22f46f4f9 Move the listenTCP call outside the manhole function 2016-04-25 14:59:21 +01:00
David Baker 290f125a13 Typo 2016-04-25 14:42:59 +01:00
David Baker 05e49ffbdf No we don't: it's just the display name 2016-04-22 18:44:17 +01:00
David Baker c5b3c6e101 Sort member events
So names of people in a room are given in order
2016-04-22 18:33:36 +01:00
David Baker e8701e64b9 Implement group-of-people names 2016-04-22 17:28:42 +01:00
Mark Haines 5905f36f05 Split out setting up the manhole to a separate file 2016-04-22 17:09:15 +01:00
Mark Haines 9e7aa98c22 Split out create_resource_tree to a separate file 2016-04-22 15:40:51 +01:00
David Baker c10ed26c30 Flesh out email templating
Mostly WIP porting the room name calculation logic from the web client so our room names in the email mirror the clients.
2016-04-21 19:19:07 +01:00
Erik Johnston eb8619e256 Create log context in Measure if one doesn't exist 2016-04-18 16:08:32 +01:00
Erik Johnston cb9c465707 Use SynapseError 504 for Timeout errors 2016-04-15 10:21:32 +01:00
Mark Haines 82d7eea7e3 Move the versionstring code out of app.homeserver into util 2016-04-11 14:57:09 +01:00
Mark Haines 7e2f971c08 Remove some unused functions (#711)
* Remove some unused functions

* get_room_events_stream is only used in tests

* is_exclusive_room might actually be something we want
2016-04-08 14:01:56 +01:00
Mark Haines b9ee5650b0 Move all the wrapper functions for distributor.fire
Move the functions inside the distributor and import them
where needed. This reduces duplication and makes it possible
for flake8 to detect when the functions aren't used in a
given file.
2016-04-08 11:01:38 +01:00
Erik Johnston 95ac3078da Rename things 2016-04-07 16:07:16 +01:00
Erik Johnston ee5aef6c72 Log contexts and squash things together 2016-04-07 15:34:21 +01:00
Erik Johnston 639cd07d6d Add comment 2016-04-07 14:24:12 +01:00
Erik Johnston af03ecf352 Deduplicate joins 2016-04-07 14:19:02 +01:00
Mark Haines 87f2dec8d4 Make the cache objects be per instance rather than being global 2016-04-06 13:08:05 +01:00
Erik Johnston 3f4eb4c924 Comment 2016-04-01 14:15:27 +01:00
Erik Johnston 8d73cd502b Add concurrently_execute function 2016-04-01 14:06:00 +01:00
Mark Haines 77cba688ed Fix typo 2016-03-24 18:02:37 +00:00
Mark Haines 191c7bef6b Deduplicate identical /sync requests 2016-03-24 17:47:31 +00:00
Erik Johnston 8122ad7bab Simplify intern_dict 2016-03-23 16:41:54 +00:00
Erik Johnston acdfef7b14 Intern all the things 2016-03-23 16:25:54 +00:00
Erik Johnston 75daede92f String intern 2016-03-23 14:53:53 +00:00
Erik Johnston c4a8cbd15a Make LruCache use a dedicated _Node class 2016-03-22 16:06:21 +00:00
Erik Johnston a547e2df85 Return list, not generator. 2016-03-14 15:30:19 +00:00
Mark Haines 239badea9b Use syntax that works on both py2.7 and py3 2016-03-07 20:13:10 +00:00
Erik Johnston 374f9b2f07 Limit stream change cache size too 2016-03-01 13:30:15 +00:00
Erik Johnston ce2cdced61 Move cache size fiddling to descriptors only. Fix tests 2016-03-01 13:21:46 +00:00
Erik Johnston 910fc0f28f Add enviroment variable SYNAPSE_CACHE_FACTOR, default it to 0.1 2016-03-01 12:56:39 +00:00
Erik Johnston 72165e5b77 Reraise exception 2016-03-01 11:00:10 +00:00
Erik Johnston ff2d7551c7 Correct cache miss detection 2016-03-01 10:59:17 +00:00
Erik Johnston 278d6c0527 Report size of ExpiringCache 2016-02-23 16:46:21 +00:00
Erik Johnston c77dae7a1a Change the way we figure out presence updates for small deltas 2016-02-23 14:54:40 +00:00
Erik Johnston 5614b4dafb Add presence metrics 2016-02-19 09:50:54 +00:00
Erik Johnston b8cdec92c7 WheelTimer: Don't scan list, use index. 2016-02-18 16:33:07 +00:00
Erik Johnston e5999bfb1a Initial cut 2016-02-17 15:40:50 +00:00
Erik Johnston 9e7900da1e Add wheeltimer impl 2016-02-17 14:29:28 +00:00
Erik Johnston e557dc80b8 Merge pull request #566 from matrix-org/erikj/logcontext
Don't bother copying records on parent context
2016-02-10 11:41:45 +00:00
Erik Johnston 4eb8f9ca8a Remove comment 2016-02-10 11:29:21 +00:00
Erik Johnston 00c9ad49df s/parent_context/previous_context/ 2016-02-10 11:25:19 +00:00
Erik Johnston 9777c5f49a Set parent context on instansiation 2016-02-10 11:23:32 +00:00
Erik Johnston 82631c5f94 Fix unit tests 2016-02-09 13:50:37 +00:00
Erik Johnston 9daa4e2a85 Don't create new logging context 2016-02-09 11:06:19 +00:00
Erik Johnston 3e2fcd67b2 Don't bother copying records on parent context 2016-02-09 10:50:31 +00:00
Erik Johnston 241b71852e Fix bug in util.metrics.Measure 2016-02-09 10:28:13 +00:00
Erik Johnston 97294ef2fd Create new context when measuring 2016-02-09 10:12:00 +00:00
Erik Johnston c486b7b41c Change logcontext warns to debug 2016-02-09 09:20:38 +00:00
Erik Johnston 2c1fbea531 Fix up logcontexts 2016-02-08 14:26:45 +00:00
Erik Johnston 13e6262659 Add metrics to pushers 2016-02-08 14:26:45 +00:00
Erik Johnston d4f72a5bfb Allowing tagging log contexts 2016-02-03 13:52:27 +00:00
Daniel Wagner-Hall d83d004ccd Fix flake8 warnings for new flake8 2016-02-02 17:18:50 +00:00
Erik Johnston e70165039c If stream pos is greater then earliest known key and entity hasn't changed, then entity hasn't changed 2016-01-29 16:41:32 +00:00
Erik Johnston 18579534ea Prefill stream change caches 2016-01-29 14:37:59 +00:00
Erik Johnston b18114e19e Merge pull request #536 from matrix-org/erikj/sync
Make /sync "better".
2016-01-29 13:04:51 +00:00
Erik Johnston fb7299800f Directly set self.value 2016-01-29 11:29:14 +00:00
Erik Johnston c046630c33 Remove spurious self.size 2016-01-29 11:17:54 +00:00
Erik Johnston a30364c1f9 Correctly bookkeep the size of TreeCache 2016-01-29 10:44:46 +00:00
Erik Johnston 766526e114 Make TreeCache keep track of its own size. 2016-01-29 10:11:21 +00:00
Erik Johnston 50e18938a9 Reset size on clear 2016-01-29 10:00:45 +00:00
Erik Johnston 3f5dd18bd4 If the same as the earliest key, assume nothing has changed. 2016-01-28 18:11:41 +00:00
Erik Johnston 40431251cb Correctly update _entity_to_key 2016-01-28 18:05:43 +00:00
Erik Johnston 82cf3a8043 Fix inequalities 2016-01-28 17:44:04 +00:00
Erik Johnston 0663c5bd52 Include cache hits with has_entity_changed 2016-01-28 17:27:28 +00:00
Erik Johnston 45cf827c8f Change name and doc has_entity_changed 2016-01-28 16:39:18 +00:00
Erik Johnston 00cb3eb24b Cache tags and account data 2016-01-28 16:37:41 +00:00
Erik Johnston c23a8c7833 Ensure keys to RoomStreamChangeCache are ints 2016-01-28 15:55:26 +00:00
Erik Johnston e1941442d4 Invalidate caches properly. Remove unused arg 2016-01-28 15:02:41 +00:00
Erik Johnston b97f6626b6 Add cache to room stream 2016-01-27 17:33:26 +00:00
David Baker 7cd418d38e Don't add the member functiopn if we're not using treecache 2016-01-22 13:40:37 +00:00
David Baker cd80019eec docs 2016-01-22 12:21:13 +00:00
David Baker d552861346 Revert all the bits changing keys of eeverything that used LRUCaches to tuples 2016-01-22 12:18:14 +00:00
David Baker 10f76dc5da Make LRU cache not default to treecache & add options to use it 2016-01-22 12:10:33 +00:00
David Baker 5b142788d2 Add __contains__ 2016-01-22 11:49:59 +00:00
David Baker eaa836e8ca Docs for treecache 2016-01-22 11:47:22 +00:00
David Baker 8acc5cb60f Add invalidate_many here too 2016-01-22 11:22:32 +00:00
David Baker 330be18ec5 peppate 2016-01-21 19:17:32 +00:00
David Baker f1f8122120 Change LRUCache to be tree-based so we can delete subtrees. 2016-01-21 19:16:25 +00:00
Matthew Hodgson 6c28ac260c copyrights 2016-01-07 04:26:29 +00:00
Mark Haines d12c00bdc3 Add some docstring explaining the snapshot cache does 2015-12-23 15:18:11 +00:00
Mark Haines 7fa71e3267 Add a unit test for the snapshot cache 2015-12-23 11:48:03 +00:00
Mark Haines 9ac417fa88 Add a cache for initialSync responses that expires after 5 minutes 2015-12-22 18:27:56 +00:00
Mark Haines 6a5ff5f223 Track the time spent in the database per request.
and track the number of transactions that request started.
2015-12-07 17:56:11 +00:00
Mark Haines 3dd1630848 Add a setter for the current log context.
Move the resource tracking inside that setter so that it is easier
to make sure that the resource tracking isn't double counting the
resource usage.
2015-12-07 10:51:18 +00:00
Mark Haines 5231737369 Add comments to explain why we are hardcoding RUSAGE_THREAD 2015-12-04 11:53:38 +00:00
Mark Haines d6059bdd2a Fix warnings 2015-12-04 11:34:23 +00:00
Mark Haines 48a2526d62 Track the cpu used in the main thread by each logging context 2015-12-03 21:03:01 +00:00
Mark Haines a412b9a465 Run the background updates when starting synapse. 2015-11-10 15:50:58 +00:00
Daniel Wagner-Hall 2cebe53545 Exchange 3pid invites for m.room.member invites 2015-11-05 16:43:19 +00:00
Erik Johnston 5bc690408d Merge pull request #340 from matrix-org/erikj/server_retries
Retry dead servers a lot less often
2015-11-05 16:15:50 +00:00
Mark Haines c452dabc3d Remove the LockManager class because it wasn't being used 2015-11-04 14:08:15 +00:00
Erik Johnston eacb068ac2 Retry dead servers a lot less often 2015-11-02 16:56:30 +00:00
Daniel Wagner-Hall 9261ef3a15 Merge pull request #312 from matrix-org/daniel/3pidinvites
Stuff signed data in a standalone object
2015-10-19 15:52:34 +01:00
Mark Haines 9371a35e89 Merge pull request #306 from matrix-org/markjh/unused_methods
Remove some login classes from synapse.
2015-10-16 18:18:41 +01:00
Daniel Wagner-Hall 0e5239ffc3 Stuff signed data in a standalone object
Makes both generating it in sydent, and verifying it here, simpler at
the cost of some repetition
2015-10-16 17:45:48 +01:00
Daniel Wagner-Hall c225d63e9e Add signing host and keyname to signatures 2015-10-16 15:07:56 +01:00
Daniel Wagner-Hall b8dd5b1a2d Verify third party ID server certificates 2015-10-16 14:54:54 +01:00
Daniel Wagner-Hall 643b5fcdc8 Look for keys on the right objects 2015-10-15 13:10:30 +01:00
Daniel Wagner-Hall f38df51e8d Merge branch 'develop' into daniel/3pidinvites 2015-10-15 11:51:55 +01:00
Mark Haines 5338220d3a synapse.util.emailutils was unused 2015-10-15 10:39:33 +01:00
Daniel Wagner-Hall d82c5f7b5c Use more descriptive error code 2015-10-13 18:02:00 +01:00
Daniel Wagner-Hall 0c38e8637f Remove unnecessary class-wrapping 2015-10-13 18:00:38 +01:00
Mark Haines 32d66738b0 Fix pep8 warnings. 2015-10-13 17:18:29 +01:00
Mark Haines 7639c3d9e5 Bounce all deferreds through the reactor to make debugging easier.
If all deferreds wait a reactor tick before resolving then there is
always a chance to add an errback to the deferred so that stacktraces
get reported, rather than being discarded.
2015-10-13 17:13:04 +01:00
Daniel Wagner-Hall 17dffef5ec Move event contents into third_party_layout field 2015-10-13 15:48:12 +01:00
Daniel Wagner-Hall 5b3e9713dd Implement third party identifier invites 2015-10-01 17:49:52 +01:00