911db96658
Merge pull request #4073 from matrix-org/rav/require_psutil
...
Make psutil an explicit dependency
2018-10-22 12:33:21 +01:00
a6f421e812
Run MAU queries as background processes
...
Fixes #3820
2018-10-20 02:14:35 +01:00
e1728dfcbe
Make scripts/ and scripts-dev/ pass pyflakes (and the rest of the codebase on py3) ( #4068 )
2018-10-20 11:16:55 +11:00
e5b52d0f94
Make psutil an explicit dependency
...
As of #4027 , we require psutil to be installed, so it should be in our
dependency list. We can also remove some of the conditional import code
introduced by #992 .
Fixes #4062 .
2018-10-19 21:51:15 +01:00
06bc8d2fe5
synapse/app: frontend_proxy.py: actually make workers work on py3
2018-10-14 20:08:39 +03:00
381d2cfdf0
Make workers work on Py3 ( #4027 )
2018-10-13 00:14:08 +11:00
19475cf337
Remove redundant call to start_get_pdu_cache
...
I think this got forgotten in #3932 . We were getting away with it because it
was the last call in this function.
2018-09-28 12:01:23 +01:00
dc5db01ff2
Replaced all occurences of e.message with str(e)
...
Signed-off-by: Schnuffle <schnuffle@github.com>
2018-09-27 13:38:50 +02:00
66a1d57adb
Merge pull request #3948 from matrix-org/rav/no_symlink_synctl
...
Move synctl into top dir to avoid a symlink
2018-09-26 21:41:58 +10:00
c53336986d
Move synctl into top dir to avoid a symlink
...
symlinks apparently break setuptools on python3 and alpine
(https://bugs.python.org/issue31940 ), so let's stop using a symlink and just
use the file directly.
2018-09-25 11:19:27 +01:00
8601c24287
Fix some instances of ExpiringCache not expiring cache items
...
ExpiringCache required that `start()` be called before it would actually
start expiring entries. A number of places didn't do that.
This PR removes `start` from ExpiringCache, and automatically starts
backround reaping process on creation instead.
2018-09-21 14:19:46 +01:00
5baa087312
typo
2018-09-17 17:37:56 +01:00
b58714789f
make pip happy?
2018-09-17 17:35:54 +01:00
9a1cceeca9
Use a string for versions
2018-09-17 17:09:06 +01:00
2b39494cd5
Add python_version phone home stat
2018-09-17 16:35:18 +01:00
8decd6233d
improve naming
2018-09-12 16:22:15 +01:00
0ddf486724
expose number of real reserved users
2018-09-12 11:58:52 +01:00
0b07f02e19
Make sure that we close db connections opened during init
...
We should explicitly close any db connections we open, because failing to do so
can block other transactions as per
https://github.com/matrix-org/synapse/issues/3682 .
Let's also try to factor out some of the boilerplate by having server classes
define their datastore class rather than duplicating the whole of `setup`.
2018-08-28 13:39:49 +01:00
764030cf63
Merge pull request #3659 from matrix-org/erikj/split_profiles
...
Allow profile updates to happen on workers
2018-08-22 11:35:55 +01:00
3cef867cc1
Merge pull request #3709 from matrix-org/rav/logcontext_for_replication_commands
...
Logcontexts for replication command handlers
2018-08-17 16:22:07 +01:00
c144252a8c
Merge pull request #3710 from matrix-org/rav/logcontext_for_pusher_updates
...
Fix logcontexts for running pushers
2018-08-17 16:21:49 +01:00
c334ca67bb
Integrate presence from hotfixes ( #3694 )
2018-08-18 01:08:45 +10:00
ab822a2d1f
Add some fixmes
2018-08-17 15:31:50 +01:00
91cdb6de08
Call UserDirectoryHandler methods directly
...
Turns out that the user directory handling is fairly racey as a bunch
of stuff assumes that the processing happens on master, which it doesn't
when there is a synapse.app.user_dir worker. So lets just call the
function directly until we actually get round to fixing it, since it
doesn't make the situation any worse.
2018-08-17 15:26:13 +01:00
782689bd40
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles
2018-08-17 14:15:48 +01:00
521d369e7a
remove errant yield
2018-08-17 10:12:11 +01:00
0e8d78f6aa
Logcontexts for replication command handlers
...
Run the handlers for replication commands as background processes. This should
improve the visibility in our metrics, and reduce the number of "running db
transaction from sentinel context" warnings.
Ideally it means converting the things that fire off deferreds into the night
into things that actually return a Deferred when they are done. I've made a bit
of a stab at this, but it will probably be leaky.
2018-08-17 00:43:43 +01:00
66f7dc8c87
Fix logcontexts for running pushers
...
First of all, avoid resetting the logcontext before running the pushers, to fix
the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning.
Instead, give them their own "background process" logcontexts.
2018-08-17 00:32:39 +01:00
bcfeb44afe
call reap on start up and fix under reaping bug
2018-08-16 22:55:32 +01:00
7d5b1a60a3
Fix inbound federation on reader worker
...
Inbound federation requires calculating push, which in turn relies on
having access to account data.
2018-08-16 15:32:20 +01:00
ef184caf30
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation
2018-08-15 14:25:46 +01:00
773db62a22
Rename slave TransactionStore to SlaveTransactionStore
2018-08-15 14:17:06 +01:00
e5962f845c
pep8
2018-08-14 16:36:14 +01:00
e7d091fb86
combine mau metrics into one group
2018-08-14 16:26:55 +01:00
b37c472419
Rename async to async_helpers because `async` is a keyword on Python 3.7 ( #3678 )
2018-08-10 23:50:21 +10:00
3c0213a217
Merge pull request #3439 from vojeroen/send_sni_for_federation_requests
...
send SNI for federation requests
2018-08-10 12:23:54 +01:00
484a0ebdfc
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles
2018-08-09 10:16:29 +01:00
5785b93711
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation
2018-08-09 10:16:16 +01:00
e8eba2b4e3
implement reserved users for mau limits
2018-08-07 17:49:43 +01:00
495cb100d1
Allow profile changes to happen on workers
2018-08-07 10:50:26 +01:00
96a9a29645
Pull in necessary stores in federation_reader
2018-08-06 15:23:57 +01:00
e26dbd82ef
Add replication APIs for persisting federation events
2018-08-06 15:02:28 +01:00
950807d93a
fix caching and tests
2018-08-03 13:49:53 +01:00
c0affa7b4f
update generate_monthly_active_users, and reap_monthly_active_users
2018-08-02 23:03:01 +01:00
d766f26de9
Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker
2018-08-01 17:49:41 +01:00
b7f203a566
count_monthly_users is now async
2018-08-01 16:17:42 +01:00
2c54f1c225
remove need to plot limit_usage_by_mau
2018-08-01 11:46:59 +01:00
c507fa15ce
only need to loop if mau limiting is enabled
2018-08-01 10:20:42 +01:00
6ef983ce5c
api into monthly_active_users table
2018-07-31 16:36:24 +01:00
df2235e7fa
coding style
2018-07-31 13:16:20 +01:00
21276ff846
remove errant logging
2018-07-30 22:42:12 +01:00
9b13817e06
factor out metrics from __init__ to app/homeserver
2018-07-30 22:07:07 +01:00
7041cd872b
Merge branch 'develop' into send_sni_for_federation_requests
2018-07-27 09:17:11 +01:00
03751a6420
Fix some looping_call calls which were broken in #3604
...
It turns out that looping_call does check the deferred returned by its
callback, and (at least in the case of client_ips), we were relying on this,
and I broke it in #3604 .
Update run_as_background_process to return the deferred, and make sure we
return it to clock.looping_call.
2018-07-26 11:48:08 +01:00
371da42ae4
Wrap a number of things that run in the background
...
This will reduce the number of "Starting db connection from sentinel context"
warnings, and will help with our metrics.
2018-07-25 09:41:12 +01:00
8b8c4f34a3
Replace usage of get_current_toke with StreamToken.START
...
This allows us to handle /context/ requests on the client_reader worker
without having to pull in all the various stream handlers (e.g.
precence, typing, pushers etc). The only thing the token gets used for
is pagination, and that ignores everything but the room portion of the
token.
2018-07-24 16:49:17 +01:00
3188973857
Pull out did_forget to worker store
2018-07-24 16:49:14 +01:00
0b0b24cb82
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/client_apis_move
2018-07-23 13:21:15 +01:00
3132b89f12
Make the rest of the .iterwhatever go away ( #3562 )
2018-07-21 15:47:18 +10:00
9b596177ae
Add some room read only APIs to client_reader
2018-07-18 15:33:03 +01:00
505530f36a
Merge remote-tracking branch 'upstream/develop' into send_sni_for_federation_requests
...
# Conflicts:
# synapse/crypto/context_factory.py
2018-07-14 20:24:46 +02:00
49af402019
run isort
2018-07-09 16:09:20 +10:00
3d605853c8
send SNI for federation requests
2018-06-24 22:38:43 +02:00
245d53d32a
Write a clear restart indicator in logs
...
I'm fed up with never being able to find the point a server restarted in the
logs.
2018-06-20 15:33:14 +01:00
0fde1896cd
Merge pull request #3389 from turt2live/travis/name_metrics
...
Use the correct flag (enable_metrics) when warning about an incorrect metrics setup
2018-06-13 23:50:10 +10:00
45768d1640
Use the RegistryProxy for appservices too
...
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-06-12 12:55:48 -06:00
12285a1a76
The flag is named enable_metrics, not collect_metrics
...
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-06-12 12:51:31 -06:00
042eedfa2b
Add hacky cache factor override system
2018-06-04 15:39:28 +01:00
71477f3317
Merge pull request #3264 from matrix-org/neil/sign-up-stats
...
daily user type phone home stats
2018-06-01 13:42:01 +00:00
febe0ec8fd
Run Prometheus on a different port, optionally. ( #3274 )
2018-05-31 19:04:50 +10:00
85ba83eb51
fixes
2018-05-22 16:28:23 -05:00
d8cb7225d2
daily user type phone home stats
2018-05-22 18:09:09 +01:00
a8990fa2ec
Merge remote-tracking branch 'origin/develop' into 3218-official-prom
2018-05-22 10:50:26 -05:00
b5b2d5d64b
Fix dependency on jinja2
...
Delay the import of ConsentResource, so that we can get away without jinja2 if
people don't have the consent resource enabled.
Fixes #3259
2018-05-22 14:03:45 +01:00
02c1d29133
look at the Prometheus metrics instead
2018-05-21 17:02:20 -05:00
dc8930ea9e
Merge pull request #3163 from matrix-org/cohort_analytics
...
user visit data
2018-05-16 10:09:24 +00:00
a2204cc9cc
remove unused method recurring_user_daily_visit_stats
2018-05-16 09:47:20 +01:00
8030a825c8
Merge pull request #3213 from matrix-org/rav/consent_handler
...
ConsentResource to gather policy consent from users
2018-05-16 07:19:18 +01:00
c92a8aa578
pep8
2018-05-15 17:31:11 +01:00
05ac15ae82
Limit query load of generate_user_daily_visits
...
The aim is to keep track of when it was last called and only query from that point in time
2018-05-15 17:01:33 +01:00
47815edcfa
ConsentResource to gather policy consent from users
...
Hopefully there are enough comments and docs in this that it makes sense on its
own.
2018-05-15 15:11:59 +01:00
f077e97914
instead of inserting user daily visit data at the end of the day, instead insert incrementally through the day
2018-05-14 13:50:58 +01:00
977765bde2
Merge branch 'develop' of https://github.com/matrix-org/synapse into cohort_analytics
2018-05-14 09:31:42 +01:00
318711e139
Set Server header in SynapseRequest
...
(instead of everywhere that writes a response. Or rather, the subset of places
which write responses where we haven't forgotten it).
This also means that we don't have to have the mysterious version_string
attribute in anything with a request handler.
Unfortunately it does mean that we have to pass the version string wherever we
instantiate a SynapseSite, which has been c&ped 150 times, but that is code
that ought to be cleaned up anyway really.
2018-05-10 18:50:27 +01:00
5917562b60
10 mins seems more reasonable that every minute
2018-05-01 12:12:22 +01:00
fc149b4eeb
Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background
2018-04-27 14:31:23 +01:00
2a13af23bc
Use run_in_background in preference to preserve_fn
...
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
2018-04-27 12:55:51 +01:00
9255a6cb17
Improve exception handling for background processes
...
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.
This is unsatisfactory for a number of reasons:
- logging on garbage collection is best-effort and may happen some time after
the error, if at all
- it can be hard to figure out where the error actually happened.
- it is logged as a scary CRITICAL error which (a) I always forget to grep for
and (b) it's not really CRITICAL if a background process we don't care about
fails.
So this is an attempt to add exception handling to everything we fire off into
the background.
2018-04-27 11:07:40 +01:00
617bf40924
Generate user daily stats
2018-04-25 17:37:29 +01:00
36c59ce669
Use six.itervalues in some places
...
There's more where that came from
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 20:39:43 +02:00
87478c5a60
Merge pull request #3061 from NotAFile/add-some-byte-strings
...
Add b prefixes to some strings that are bytes in py3
2018-04-09 23:54:05 +01:00
15e8ed874f
more verbosity in synctl
2018-04-06 09:28:36 +01:00
68b0ee4e8d
Merge pull request #3041 from matrix-org/r30_stats
...
R30 stats
2018-04-05 15:37:37 +01:00
0e5f479fc0
Review comments
...
Use iteritems over item to loop over dict
formatting
2018-04-05 12:16:46 +01:00
e4570c53dd
phone home cache size configurations
2018-04-04 16:46:58 +01:00
6168351877
Add b prefixes to some strings that are bytes in py3
...
This has no effect on python2
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-04 13:48:51 +02:00
725a72ec5a
Merge pull request #3000 from NotAFile/change-except-style
...
Replace old style error catching with 'as' keyword
2018-04-04 10:45:22 +01:00
86932be2cb
Support multi client R30 for psql
2018-03-28 14:36:53 +01:00
33f6195d9a
Handle review comments
2018-03-28 14:25:25 +01:00
792d340572
rename stat to future proof
2018-03-28 12:25:02 +01:00
Richard van der Hoff