Commit Graph

432 Commits

Author SHA1 Message Date
Richard van der Hoff 3c0213a217
Merge pull request #3439 from vojeroen/send_sni_for_federation_requests
send SNI for federation requests
2018-08-10 12:23:54 +01:00
Neil Johnson e8eba2b4e3 implement reserved users for mau limits 2018-08-07 17:49:43 +01:00
Neil Johnson 950807d93a fix caching and tests 2018-08-03 13:49:53 +01:00
Neil Johnson c0affa7b4f update generate_monthly_active_users, and reap_monthly_active_users 2018-08-02 23:03:01 +01:00
Neil Johnson d766f26de9 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker 2018-08-01 17:49:41 +01:00
Neil Johnson b7f203a566 count_monthly_users is now async 2018-08-01 16:17:42 +01:00
Neil Johnson 2c54f1c225 remove need to plot limit_usage_by_mau 2018-08-01 11:46:59 +01:00
Neil Johnson c507fa15ce only need to loop if mau limiting is enabled 2018-08-01 10:20:42 +01:00
Neil Johnson 6ef983ce5c api into monthly_active_users table 2018-07-31 16:36:24 +01:00
Neil Johnson df2235e7fa coding style 2018-07-31 13:16:20 +01:00
Neil Johnson 21276ff846 remove errant logging 2018-07-30 22:42:12 +01:00
Neil Johnson 9b13817e06 factor out metrics from __init__ to app/homeserver 2018-07-30 22:07:07 +01:00
Richard van der Hoff 7041cd872b
Merge branch 'develop' into send_sni_for_federation_requests 2018-07-27 09:17:11 +01:00
Richard van der Hoff 03751a6420 Fix some looping_call calls which were broken in #3604
It turns out that looping_call does check the deferred returned by its
callback, and (at least in the case of client_ips), we were relying on this,
and I broke it in #3604.

Update run_as_background_process to return the deferred, and make sure we
return it to clock.looping_call.
2018-07-26 11:48:08 +01:00
Richard van der Hoff 371da42ae4 Wrap a number of things that run in the background
This will reduce the number of "Starting db connection from sentinel context"
warnings, and will help with our metrics.
2018-07-25 09:41:12 +01:00
Amber Brown 3132b89f12
Make the rest of the .iterwhatever go away (#3562) 2018-07-21 15:47:18 +10:00
Jeroen 505530f36a Merge remote-tracking branch 'upstream/develop' into send_sni_for_federation_requests
# Conflicts:
#	synapse/crypto/context_factory.py
2018-07-14 20:24:46 +02:00
Amber Brown 49af402019 run isort 2018-07-09 16:09:20 +10:00
Jeroen 3d605853c8 send SNI for federation requests 2018-06-24 22:38:43 +02:00
Richard van der Hoff 245d53d32a Write a clear restart indicator in logs
I'm fed up with never being able to find the point a server restarted in the
logs.
2018-06-20 15:33:14 +01:00
Travis Ralston 12285a1a76 The flag is named enable_metrics, not collect_metrics
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-06-12 12:51:31 -06:00
Neil Johnson 71477f3317
Merge pull request #3264 from matrix-org/neil/sign-up-stats
daily user type phone home stats
2018-06-01 13:42:01 +00:00
Amber Brown febe0ec8fd
Run Prometheus on a different port, optionally. (#3274) 2018-05-31 19:04:50 +10:00
Amber Brown 85ba83eb51 fixes 2018-05-22 16:28:23 -05:00
Neil Johnson d8cb7225d2 daily user type phone home stats 2018-05-22 18:09:09 +01:00
Amber Brown a8990fa2ec Merge remote-tracking branch 'origin/develop' into 3218-official-prom 2018-05-22 10:50:26 -05:00
Richard van der Hoff b5b2d5d64b Fix dependency on jinja2
Delay the import of ConsentResource, so that we can get away without jinja2 if
people don't have the consent resource enabled.

Fixes #3259
2018-05-22 14:03:45 +01:00
Amber Brown 02c1d29133 look at the Prometheus metrics instead 2018-05-21 17:02:20 -05:00
Neil Johnson dc8930ea9e
Merge pull request #3163 from matrix-org/cohort_analytics
user visit data
2018-05-16 10:09:24 +00:00
Neil Johnson a2204cc9cc remove unused method recurring_user_daily_visit_stats 2018-05-16 09:47:20 +01:00
Richard van der Hoff 8030a825c8
Merge pull request #3213 from matrix-org/rav/consent_handler
ConsentResource to gather policy consent from users
2018-05-16 07:19:18 +01:00
Neil Johnson c92a8aa578 pep8 2018-05-15 17:31:11 +01:00
Neil Johnson 05ac15ae82 Limit query load of generate_user_daily_visits
The aim is to keep track of when it was last called and only query from that point in time
2018-05-15 17:01:33 +01:00
Richard van der Hoff 47815edcfa ConsentResource to gather policy consent from users
Hopefully there are enough comments and docs in this that it makes sense on its
own.
2018-05-15 15:11:59 +01:00
Neil Johnson f077e97914 instead of inserting user daily visit data at the end of the day, instead insert incrementally through the day 2018-05-14 13:50:58 +01:00
Neil Johnson 977765bde2 Merge branch 'develop' of https://github.com/matrix-org/synapse into cohort_analytics 2018-05-14 09:31:42 +01:00
Richard van der Hoff 318711e139 Set Server header in SynapseRequest
(instead of everywhere that writes a response. Or rather, the subset of places
which write responses where we haven't forgotten it).

This also means that we don't have to have the mysterious version_string
attribute in anything with a request handler.

Unfortunately it does mean that we have to pass the version string wherever we
instantiate a SynapseSite, which has been c&ped 150 times, but that is code
that ought to be cleaned up anyway really.
2018-05-10 18:50:27 +01:00
Neil Johnson 5917562b60 10 mins seems more reasonable that every minute 2018-05-01 12:12:22 +01:00
Neil Johnson 617bf40924 Generate user daily stats 2018-04-25 17:37:29 +01:00
Neil Johnson 68b0ee4e8d
Merge pull request #3041 from matrix-org/r30_stats
R30 stats
2018-04-05 15:37:37 +01:00
Neil Johnson 0e5f479fc0 Review comments
Use iteritems over item to loop over dict
formatting
2018-04-05 12:16:46 +01:00
Jan Christian Grünhage e4570c53dd phone home cache size configurations 2018-04-04 16:46:58 +01:00
Neil Johnson 86932be2cb Support multi client R30 for psql 2018-03-28 14:36:53 +01:00
Michael Kaye 33f6195d9a Handle review comments 2018-03-28 14:25:25 +01:00
Neil Johnson 792d340572 rename stat to future proof 2018-03-28 12:25:02 +01:00
Michael Kaye 4ceaa7433a As daemonizing will make a new process, defer call to init. 2018-03-28 12:19:01 +01:00
Neil Johnson a32d2548d9 query and call for r30 stats 2018-03-28 10:39:13 +01:00
Michael Kaye ef520d8d0e Include coarse CPU and Memory use in stats callbacks.
This requires the psutil module, and is still opt-in based on the report_stats
config option.
2018-03-27 17:56:03 +01:00
Matthew Hodgson 8cbbfaefc1 404 correctly on missing paths via NoResource
fixes https://github.com/matrix-org/synapse/issues/2043 and https://github.com/matrix-org/synapse/issues/2029
2018-03-23 10:32:50 +00:00
Erik Johnston cb9f8e527c s/replication_client/federation_client/ 2018-03-13 13:26:52 +00:00
Erik Johnston 265b993b8a Split replication layer into two 2018-03-13 10:55:47 +00:00
Erik Johnston 24dd73028a Add replication http endpoint for event sending 2018-02-07 10:32:32 +00:00
Richard van der Hoff 25adde9a04 Factor out get_db_conn to HomeServer base class
This function is identical to all subclasses, so we may as well push it up to
the base class to reduce duplication (and make use of it in the tests)
2018-01-26 00:56:49 +00:00
Silke df0f602796 Implement listen_tcp method in remaining workers
Signed-off-by: Silke <silke@slxh.eu>
2017-12-18 20:00:42 +01:00
Silke 26cd3f5690 Remove logger argument and do not catch replication listener
Signed-off-by: Silke <silke@slxh.eu>
2017-12-18 20:00:42 +01:00
Silke Hofstra ed48ecc58c Add methods for listening on multiple addresses
Add listen_tcp and listen_ssl which implement Twisted's reactor.listenTCP
and reactor.listenSSL for multiple addresses.

Signed-off-by: Silke Hofstra <silke@slxh.eu>
2017-12-17 13:15:48 +01:00
Silke Hofstra 37d1a90025 Allow binds to both :: and 0.0.0.0
Binding on 0.0.0.0 when :: is specified in the bind_addresses is now allowed.
This causes a warning explaining the behaviour.
Configuration changed to match.

See #2232

Signed-off-by: Silke Hofstra <silke@slxh.eu>
2017-12-17 13:10:31 +01:00
Richard van der Hoff 68ca864141 Add config option to disable media_repo on main synapse
... to stop us doing the cache cleanup jobs on the master.
2017-11-22 16:20:27 +00:00
Richard van der Hoff e1fd4751de Build MediaRepositoryResource as a homeserver dependency
This avoids the scenario where we have four different PreviewUrlResources
configured on a single app, each of which have their own caches and cache
clearing jobs.
2017-11-22 16:19:49 +00:00
David Baker d4553818a0
Merge pull request #2627 from matrix-org/rav/custom_rest_endpoints
Add a hook for custom rest endpoints
2017-11-02 15:18:37 +00:00
Richard van der Hoff fcdfc911ee Add a hook for custom rest endpoints
Let the user specify custom modules which can be used for implementing extra
endpoints.
2017-11-02 14:36:55 +00:00
Richard van der Hoff 6650a07ede Factor out _configure_named_resource
This was a bit of a code vomit, so let's factor it out to preserve some sanity
2017-11-02 14:33:37 +00:00
Erik Johnston 3fed5bb25f Move quit_with_error 2017-10-02 17:59:34 +01:00
Richard van der Hoff 10d8b701a1 Allow configuration of CPU affinity
Make it possible to set the CPU affinity in the config file, so that we don't
need to remember to do it manually every time.
2017-08-15 17:08:28 +01:00
Richard van der Hoff 543c794a76 Factor out common application start
We have 10 copies of this code, and I don't really want to update each one
separately.
2017-08-15 17:04:40 +01:00
Erik Johnston 5ddd199870 Typo 2017-06-15 10:49:10 +01:00
Erik Johnston 4b461a6931 Add some more stats 2017-06-15 09:39:39 +01:00
Erik Johnston 617304b2cf Fix phone home stats 2017-06-14 19:47:15 +01:00
Matthew Wolff d4040e9e28
Queried CONDITIONAL_REQUIREMENTS 2017-04-18 16:19:48 -05:00
Matthew Wolff 54f9a4cb59
Fixed travis build failure
Signed-off-by: Matthew Wolff <matthewjwolff@gmail.com>
2017-04-17 01:38:27 -05:00
Matthew Wolff 8e780b113d
web_server_root documentation fix
Signed-off-by: Matthew Wolff <matthewjwolff@gmail.com>
2017-04-17 00:49:11 -05:00
Erik Johnston 28a4649785 Remove HTTP replication APIs 2017-04-11 09:52:11 +01:00
Erik Johnston 3ba2859e0c Add tcp replication listener type and hook it up 2017-03-30 13:31:10 +01:00
Richard van der Hoff 067b00d49d Run the reactor with the sentinel logcontext
This fixes a class of 'Unexpected logcontext' messages, which were happening
because the logcontext was somewhat arbitrarily swapping between the sentinel
and the `run` logcontext.
2017-03-18 00:07:43 +00:00
Richard van der Hoff d84bd51e95 Refactor logger config for workers
- to make it easier to add more config options.
2017-03-10 15:34:01 +00:00
Erik Johnston edd6cdfc9a Restore default bind address 2017-01-10 17:21:41 +00:00
Johannes Löthberg 702c020e58 Fix check for bind_address
The empty string is a valid setting for the bind_address option, so
explicitly check for None here instead.

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-20 01:37:50 +01:00
Johannes Löthberg 7dfd70fc83 Add support for specifying multiple bind addresses
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-18 21:51:56 +01:00
Mark Haines 689f4cb914 Update comment 2016-08-22 16:17:31 +01:00
Mark Haines c8f9b45bc2 Add usage stats to prometheus monitoring 2016-08-22 15:34:38 +01:00
Erik Johnston 24f36469bc Add federation /version API 2016-08-05 16:36:07 +01:00
Erik Johnston 66868119dc Add metrics for psutil derived memory usage 2016-07-20 16:00:21 +01:00
Mark Haines 13e334506c Remove the legacy v0 content upload API.
The existing content can still be downloaded. The last upload to the
matrix.org server was in January 2015, so it is probably safe to remove
the upload API.
2016-06-21 11:47:39 +01:00
Mark Haines 7dbb473339 Add function to load config without generating it
Renames ``load_config`` to ``load_or_generate_config``
Adds a method called ``load_config`` that just loads the
config.

The main synapse.app.homeserver will continue to use
``load_or_generate_config`` to retain backwards compat.
However new worker processes can use ``load_config`` to
load the config avoiding some of the cruft needed to generate
the config.

As the new ``load_config`` method is expected to be used by new
configs it removes support for the legacy commandline overrides
that ``load_or_generate_config`` supports
2016-06-09 18:50:38 +01:00
Erik Johnston dded389ac1 Allow setting of gc.set_thresholds 2016-06-07 15:45:56 +01:00
Mark Haines f22f46f4f9 Move the listenTCP call outside the manhole function 2016-04-25 14:59:21 +01:00
Mark Haines 5905f36f05 Split out setting up the manhole to a separate file 2016-04-22 17:09:15 +01:00
Mark Haines 62607d5452 Merge branch 'develop' into markjh/split_out_site.py
Conflicts:
	synapse/app/homeserver.py
2016-04-22 16:26:57 +01:00
Mark Haines e856036f4c Move SynapseSite to its own file 2016-04-22 16:09:55 +01:00
Mark Haines 9e7aa98c22 Split out create_resource_tree to a separate file 2016-04-22 15:40:51 +01:00
Mark Haines 82d7eea7e3 Move the versionstring code out of app.homeserver into util 2016-04-11 14:57:09 +01:00
Erik Johnston 8aab9d87fa Don't require config to create database 2016-04-06 14:15:45 +01:00
Mark Haines 239badea9b Use syntax that works on both py2.7 and py3 2016-03-07 20:13:10 +00:00
Daniel Wagner-Hall 95481e7ba7 Merge pull request #571 from matrix-org/daniel/asids
Mark AS users with their AS's ID
2016-03-03 13:56:28 +00:00
Mark Haines 60a0f81c7a Add a /replication API for extracting the updates that happened on
synapse

This is necessary for replicating the data in synapse to be visible to a
separate service because presence and typing notifications aren't stored
in a database so won't be visible to another process.

This API can be used to either get the raw data by requesting the tables
themselves or to just receive notifications for updates by following the
streams meta-stream.

Returns updates for each table requested a JSON array of arrays with a
row for each row in the table.

Each table is prefixed by a header row with the: name of the table,
current stream_id position for the table, number of rows, number of
columns and the names of the columns.
This is followed by the rows that have been added to the server since
the requester last asked.

The API has a timeout and is hooked up to the notifier so that a slave
can long poll for updates.
2016-03-01 14:49:41 +00:00
Daniel Wagner-Hall 763360594d Mark AS users with their AS's ID 2016-02-11 17:26:42 +00:00
Erik Johnston 2c1fbea531 Fix up logcontexts 2016-02-08 14:26:45 +00:00
Daniel Wagner-Hall 737c4223ef Host /media/r0 as well as /media/v1 2016-02-05 10:47:46 +00:00
Daniel Wagner-Hall 6a9f1209df Error if macaroon key is missing from config
Currently we store all access tokens in the DB, and fall back to that
check if we can't validate the macaroon, so our fallback works here, but
for guests, their macaroons don't get persisted, so we don't get to
find them in the database. Each restart, we generate a new ephemeral
key, so guests lose access after each server restart.

I tried to fix up the config stuff to be less insane, but gave up, so
instead I bolt on yet another piece of custom one-off insanity.

Also, add some basic tests for config generation and loading.
2016-02-05 01:58:23 +00:00
Erik Johnston f8aae79a72 Simplify get_rooms 2016-02-03 13:24:35 +00:00
Daniel Wagner-Hall d83d004ccd Fix flake8 warnings for new flake8 2016-02-02 17:18:50 +00:00
Erik Johnston 167d1df699 Merge pull request #534 from matrix-org/erikj/setup
Add a Homeserver.setup method
2016-01-28 14:45:36 +00:00
Erik Johnston 7ed2bbeb11 Clean up a bit. Add comment 2016-01-28 14:32:05 +00:00
Erik Johnston a6477d5933 Remove chdir 2016-01-28 09:19:55 +00:00
Erik Johnston fa90c180ee Merge branch 'develop' of github.com:matrix-org/synapse into erikj/setup 2016-01-27 17:37:33 +00:00
Erik Johnston 9fda8b5193 Don't turn on profiling 2016-01-26 18:27:23 +00:00
Erik Johnston 87f9477b10 Add a Homeserver.setup method.
This is for setting up dependencies that require work on startup. This
is useful for the DataStore that wants to read a bunch from the database
before initiliazing.
2016-01-26 15:51:06 +00:00
Erik Johnston 9959d9ece8 Remove redundated BaseHomeServer 2016-01-26 13:52:29 +00:00
Erik Johnston 44b4fc5f50 Use compiled regex 2016-01-13 11:47:32 +00:00
Erik Johnston 8f8b884430 Don't log urlencoded access_tokens 2016-01-08 17:48:08 +00:00
Erik Johnston 33bef689c1 Turn on core dumps 2016-01-07 15:34:30 +00:00
Erik Johnston 5727922106 Merge pull request #473 from matrix-org/erikj/ssh_manhole
Change manhole to use ssh
2016-01-07 14:36:16 +00:00
Erik Johnston c9ae1d1ee5 Change manhole to use ssh 2016-01-07 13:59:02 +00:00
Matthew Hodgson 6c28ac260c copyrights 2016-01-07 04:26:29 +00:00
Daniel Wagner-Hall 11a974da21 Add /_matrix/versions to report supported versions 2016-01-06 18:08:52 +00:00
Daniel Wagner-Hall bf32922e5a Log when starting stats reporting 2016-01-06 14:13:34 +00:00
Daniel Wagner-Hall 797691f908 Log on stats scheduling 2016-01-06 14:04:27 +00:00
Daniel Wagner-Hall 7a8ba4c9a0 Actually host r0 and unstable prefixes 2015-12-08 15:26:52 +00:00
Mark Haines 6a5ff5f223 Track the time spent in the database per request.
and track the number of transactions that request started.
2015-12-07 17:56:11 +00:00
Mark Haines d6059bdd2a Fix warnings 2015-12-04 11:34:23 +00:00
Mark Haines a412b9a465 Run the background updates when starting synapse. 2015-11-10 15:50:58 +00:00
Erik Johnston fa1cf5ef34 Move static folder into synapse
This is because otherwise it won't get picked up by python packaging.

This also fixes the problem where the "static" folder was found if
synapse wasn't started from that directory.
2015-10-26 15:37:44 +00:00
Mark Haines 9020860479 Only turn on the twisted deferred debugging if full_twisted_stacktraces is set in the config 2015-10-13 17:50:44 +01:00
Mark Haines 7639c3d9e5 Bounce all deferreds through the reactor to make debugging easier.
If all deferreds wait a reactor tick before resolving then there is
always a chance to add an errback to the deferred so that stacktraces
get reported, rather than being discarded.
2015-10-13 17:13:04 +01:00
Erik Johnston 17c80c8a3d rename schema_prepare to prepare_database 2015-10-13 13:56:22 +01:00
Erik Johnston ec398af41c Expose error more nicely 2015-10-13 11:43:43 +01:00
Mark Haines 7a5818ed81 Note that GzipFile was removed in comment that referenced it 2015-09-22 18:27:22 +01:00
Mark Haines 184ba0968a synapse/app/homeserver.py:GzipFile was unused 2015-09-22 18:25:30 +01:00
Daniel Wagner-Hall b6e0303c83 Catch stats-reporting errors 2015-09-22 13:34:29 +01:00
Daniel Wagner-Hall 7213588083 Implement configurable stats reporting
SYN-287

This requires that HS owners either opt in or out of stats reporting.

When --generate-config is passed, --report-stats must be specified
If an already-generated config is used, and doesn't have the
report_stats key, it is requested to be set.
2015-09-22 12:57:40 +01:00
Mark Haines 0b5c9adeb5 Merge pull request #267 from matrix-org/markjh/missing_requirements
Print an example "pip install" line for a missing requirement
2015-09-18 18:52:08 +01:00
Daniel Wagner-Hall 2c746382e0 Merge branch 'daniel/insecureclient' into develop 2015-09-09 14:27:30 +01:00
Daniel Wagner-Hall 6485f03d91 Fix random formatting 2015-09-09 13:05:00 +01:00
Daniel Wagner-Hall 81a93ddcc8 Allow configuration to ignore invalid SSL certs
This will be useful for sytest, and sytest only, hence the aggressive
config key name.
2015-09-09 12:02:07 +01:00
Mark Haines 73e4ad4b8b Merge branch 'master' into develop
Conflicts:
	setup.py
2015-09-07 15:06:46 +01:00
Erik Johnston 1002bbd732 Change log level to info 2015-09-03 09:51:01 +01:00
Mark Haines 3cdfd37d95 Print an example "pip install" line for a missing requirement 2015-09-01 16:47:26 +01:00
Matthew Hodgson d71af2ee12 don't log the whole DB config (including postgres password...) 2015-08-29 22:23:21 +01:00
Erik Johnston d33f31d741 Print the correct pip install line when failing due to lack of matrix-angular-sdk 2015-08-25 15:33:23 +01:00
Erik Johnston 37403ab06c Update the log message 2015-08-25 14:19:09 +01:00
Erik Johnston d9088c923f Remove dependency on matrix-angular-sdk 2015-08-25 13:34:50 +01:00
Erik Johnston 86cef6a91b Allow specifying a directory to host a web client from 2015-08-25 12:01:23 +01:00
Paul "LeoNerd" Evans efe60d5e8c Only print the pidfile path on startup if requested by a commandline flag 2015-08-07 16:36:42 +01:00
Erik Johnston 0e58d19163 Merge pull request #187 from matrix-org/erikj/sanitize_logging
Sanitize logging
2015-06-19 11:35:59 +01:00
Erik Johnston eb928c9f52 Add site_tag to logger 2015-06-19 10:16:48 +01:00
Eric Myhre 9e5a353663 Make upload dir a configurable path.
Fixes SYN-425.

Signed-off-by: Eric Myhre <hash@exultant.us>
2015-06-18 23:38:20 -05:00
Erik Johnston aaa749d366 Disable twisted access logging. Move access logging to SynapseRequest object 2015-06-15 18:18:05 +01:00
Erik Johnston cee69441d3 Log more when we have processed the request 2015-06-15 17:11:44 +01:00
Erik Johnston b5209c5744 Create SynapseRequest that overrides __repr__ to not print access_token 2015-06-15 16:37:04 +01:00
Erik Johnston a005b7269a Add backwards compat support for metrics, manhole and webclient config options 2015-06-12 17:44:23 +01:00