Commit Graph

7230 Commits

Author SHA1 Message Date
Matthew Hodgson 2d9ab533f9 fix SQL when searching all users 2018-01-17 15:58:52 +00:00
Richard van der Hoff 390093d45e Split resolve_events into two functions
... so that the return type doesn't depend on the arg types
2018-01-17 15:44:31 +00:00
Erik Johnston 2fb3a28c98
Remove lost comment 2018-01-17 14:59:44 +00:00
Richard van der Hoff a7e4ff9cca
Merge pull request #2795 from matrix-org/rav/track_db_scheduling
Track DB scheduling delay per-request
2018-01-17 14:29:37 +00:00
Richard van der Hoff f884cfffb9
Merge pull request #2797 from matrix-org/rav/user_id_checking
Sanity checking for user ids
2018-01-17 14:29:12 +00:00
Richard van der Hoff a5213df1f7 Sanity checking for user ids
Check the user_id passed to a couple of APIs for validity, to avoid
"IndexError: list index out of range" exception which looks scary and results
in a 500 rather than a more useful error.

Fixes #1432, among other things
2018-01-17 14:28:54 +00:00
Richard van der Hoff e8f7541d3f Merge remote-tracking branch 'origin/develop' into rav/track_db_scheduling 2018-01-17 14:01:57 +00:00
Richard van der Hoff fb6563b4be
Merge pull request #2793 from matrix-org/rav/db_txn_time_in_millis
Track db txn time in millisecs
2018-01-17 13:52:42 +00:00
Richard van der Hoff 1954e867b4
Merge pull request #2796 from matrix-org/rav/fix_closed_connection_errors
Fix 'NoneType' object has no attribute 'writeHeaders'
2018-01-17 13:52:24 +00:00
Richard van der Hoff 11ab2f56f5 Merge branch 'develop' into rav/fix_closed_connection_errors 2018-01-17 11:25:11 +00:00
Richard van der Hoff 0486a7814a Merge branch 'develop' into rav/rework_run_interaction 2018-01-17 11:24:38 +00:00
Richard van der Hoff 90c14da992 Merge branch 'develop' into rav/db_txn_time_in_millis 2018-01-17 11:19:55 +00:00
Richard van der Hoff 38506773eb Merge remote-tracking branch 'origin/develop' into rav/optimise_logging_context 2018-01-17 10:57:13 +00:00
Erik Johnston 300edc2348 Update last access time when thumbnails are viewed 2018-01-17 10:24:43 +00:00
Erik Johnston 05f98a2224 Keep track of last access time for local media 2018-01-17 10:24:43 +00:00
Erik Johnston 3cb2dabaad
Merge pull request #2789 from matrix-org/erikj/fix_thumbnails
Fix thumbnailing remote files
2018-01-17 10:22:20 +00:00
Erik Johnston d728c47142 Add docstring 2018-01-17 10:06:14 +00:00
Richard van der Hoff 936482d507 Fix 'NoneType' object has no attribute 'writeHeaders'
Avoid throwing a (harmless) exception when we try to write an error response to
an http request where the client has disconnected.

This comes up as a CRITICAL error in the logs which tends to mislead people
into thinking there's an actual problem
2018-01-16 17:58:16 +00:00
Richard van der Hoff 3d12d97415 Track DB scheduling delay per-request
For each request, track the amount of time spent waiting for a db
connection. This entails adding it to the LoggingContext and we may as well add
metrics for it while we are passing.
2018-01-16 17:23:32 +00:00
Richard van der Hoff 0f5d2cc37c Merge branch 'rav/db_txn_time_in_millis' into rav/track_db_scheduling 2018-01-16 17:09:15 +00:00
Richard van der Hoff 8615f19d20 rework runInteraction in terms of runConnection
... so that we can share the code
2018-01-16 17:08:29 +00:00
Matthew Hodgson 5e97ca7ee6 fix typo 2018-01-16 16:52:35 +00:00
Erik Johnston d863f68cab Use local vars 2018-01-16 16:24:15 +00:00
Erik Johnston 6368e5c0ab Change _generate_thumbnails to take media_type 2018-01-16 16:17:38 +00:00
Erik Johnston 0a90d9ede4 Move setting of file_id up to caller 2018-01-16 16:03:05 +00:00
Richard van der Hoff 6324b65f08 Track db txn time in millisecs
... to reduce the amount of floating-point foo we do.
2018-01-16 15:53:18 +00:00
Richard van der Hoff 44a498418c Optimise LoggingContext creation and copying
It turns out that the only thing we use the __dict__ of LoggingContext for is
`request`, and given we create lots of LoggingContexts and then copy them every
time we do a db transaction or log line, using the __dict__ seems a bit
redundant. Let's try to optimise things by making the request attribute
explicit.
2018-01-16 15:49:42 +00:00
Erik Johnston 5dfc83704b Fix typo 2018-01-16 14:32:56 +00:00
Richard van der Hoff febdca4b37
Merge pull request #2785 from matrix-org/rav/reorganise_metrics_again
Reorganise request and block metrics
2018-01-16 14:10:21 +00:00
Richard van der Hoff f5f89fda21
Merge pull request #2790 from matrix-org/rav/preserve_event_logcontext_leak
Fix a logcontext leak in persist_events
2018-01-16 14:09:13 +00:00
Erik Johnston 307f88dfb6 Fix up log lines 2018-01-16 13:53:52 +00:00
Richard van der Hoff 807e848f0f
Merge pull request #2787 from matrix-org/rav/worker_event_counts
Metrics for events processed in appservice and fed sender
2018-01-16 13:14:09 +00:00
Richard van der Hoff 4a31a61ef9
Merge pull request #2786 from matrix-org/rav/rdata_metrics
Metrics for number of RDATA commands received
2018-01-16 13:13:53 +00:00
Erik Johnston 9795b9ebb1 Correctly use server_name/file_id when generating/fetching remote thumbnails 2018-01-16 12:02:06 +00:00
Erik Johnston c5b589f2e8 Log when we respond with 404 2018-01-16 12:01:40 +00:00
Richard van der Hoff 64ddec1bc0 Fix a logcontext leak in persist_events
ObserveableDeferred expects its callbacks to be called without any
logcontexts, whereas it turns out we were calling them with the logcontext of
the request which initiated the persistence loop.

It seems wrong that we are attributing work done in the persistence loop to the
request that happened to initiate it, so let's solve this by dropping the
logcontext for it.

(I'm not sure this actually causes any real problems other than messages in the
debug log, but let's clean it up anyway)
2018-01-16 11:47:36 +00:00
Erik Johnston a4c5e4a645 Fix thumbnailing remote files 2018-01-16 11:37:50 +00:00
Erik Johnston 1159abbdd2
Merge pull request #2767 from matrix-org/erikj/media_storage_refactor
Refactor MediaRepository to separate out storage
2018-01-16 10:23:50 +00:00
Richard van der Hoff a027c2af8d Metrics for events processed in appservice and fed sender
More metrics I wished I'd had
2018-01-15 18:23:24 +00:00
Richard van der Hoff 5c3c32f16f Metrics for number of RDATA commands received
I found myself wishing we had this.
2018-01-15 17:45:55 +00:00
Richard van der Hoff 39f4e29d01 Reorganise request and block metrics
In order to circumvent the number of duplicate foo:count metrics increasing
without bounds, it's time for a rearrangement.

The following are all deprecated, and replaced with synapse_util_metrics_block_count:
  synapse_util_metrics_block_timer:count
  synapse_util_metrics_block_ru_utime:count
  synapse_util_metrics_block_ru_stime:count
  synapse_util_metrics_block_db_txn_count:count
  synapse_util_metrics_block_db_txn_duration:count

The following are all deprecated, and replaced with synapse_http_server_response_count:
   synapse_http_server_requests
   synapse_http_server_response_time:count
   synapse_http_server_response_ru_utime:count
   synapse_http_server_response_ru_stime:count
   synapse_http_server_response_db_txn_count:count
   synapse_http_server_response_db_txn_duration:count

The following are renamed (the old metrics are kept for now, but deprecated):

  synapse_util_metrics_block_timer:total ->
     synapse_util_metrics_block_time_seconds

  synapse_util_metrics_block_ru_utime:total ->
     synapse_util_metrics_block_ru_utime_seconds

  synapse_util_metrics_block_ru_stime:total ->
     synapse_util_metrics_block_ru_stime_seconds

  synapse_util_metrics_block_db_txn_count:total ->
     synapse_util_metrics_block_db_txn_count

  synapse_util_metrics_block_db_txn_duration:total ->
     synapse_util_metrics_block_db_txn_duration_seconds

  synapse_http_server_response_time:total ->
     synapse_http_server_response_time_seconds

  synapse_http_server_response_ru_utime:total ->
     synapse_http_server_response_ru_utime_seconds

  synapse_http_server_response_ru_stime:total ->
     synapse_http_server_response_ru_stime_seconds

   synapse_http_server_response_db_txn_count:total ->
      synapse_http_server_response_db_txn_count

   synapse_http_server_response_db_txn_duration:total
      synapse_http_server_response_db_txn_duration_seconds
2018-01-15 17:09:44 +00:00
Richard van der Hoff 992018d1c0 mechanism to render metrics with alternative names 2018-01-15 17:04:39 +00:00
Richard van der Hoff 80fa610f9c Add some comments to metrics classes 2018-01-15 16:52:52 +00:00
Richard van der Hoff 19d274085f Make Counter render floats
Prometheus handles all metrics as floats, and sometimes we store non-integer
values in them (notably, durations in seconds), so let's render them as floats
too.

(Note that the standard client libraries also treat Counters as floats.)
2018-01-12 23:49:44 +00:00
Richard van der Hoff 21bf87a146 Reinstate media download on thumbnail request
We need to actually download the remote media when we get a request for a
thumbnail.
2018-01-12 15:38:06 +00:00
Erik Johnston 694f1c1b18 Fix up comments 2018-01-12 15:02:46 +00:00
Erik Johnston e21370ba54 Correctly reraise exception 2018-01-12 14:44:02 +00:00
Erik Johnston 85a4d78213 Make Responder a context manager 2018-01-12 13:32:03 +00:00
Erik Johnston dcc8eded41 Add missing class var 2018-01-12 13:16:27 +00:00
Erik Johnston fefeb0ab0e
Merge pull request #2774 from matrix-org/erikj/synctl
When using synctl with workers, don't start the main synapse automatically
2018-01-12 12:00:07 +00:00
Erik Johnston 81391fa162 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/media_storage_refactor 2018-01-12 11:28:49 +00:00
Erik Johnston 1e4edd1717 Remove unnecessary condition 2018-01-12 11:28:32 +00:00
Erik Johnston c6c009603c Remove unused variables 2018-01-12 11:24:05 +00:00
Erik Johnston 4d88958cf6 Make class var local 2018-01-12 11:23:54 +00:00
Erik Johnston 227c491510 Comments 2018-01-12 11:22:41 +00:00
Erik Johnston f4d93ae424 Actually make it work 2018-01-12 10:39:27 +00:00
Erik Johnston f68e4cf690 Refactor 2018-01-12 10:11:12 +00:00
Richard van der Hoff 5f23b6d5ea
Merge pull request #2766 from matrix-org/rav/room_event
Add /room/{id}/event/{id} to synapse
2018-01-11 14:47:18 +00:00
Erik Johnston 7cd34512d8 When using synctl with workers, don't start the main synapse automatically 2018-01-11 11:37:39 +00:00
Erik Johnston 07ab948c38 Merge branch 'master' of github.com:matrix-org/synapse into develop 2018-01-11 11:23:40 +00:00
Erik Johnston 825a07a974
Merge pull request #2773 from matrix-org/erikj/hash_bg
Do bcrypt hashing in a background thread
2018-01-10 18:11:41 +00:00
Erik Johnston f8e1ab5fee Do bcrypt hashing in a background thread 2018-01-10 18:01:28 +00:00
Michael Telatynski 5f07f5694c fix order of operations derp and also use `.get` to default to {}
Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>
2018-01-10 15:11:35 +00:00
Michael Telatynski 8c9d5b4873 Fix publicised groups API (singular) over federation
which was missing its fed client API, since there is no other API
it might as well reuse the bulk one and unwrap it

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>
2018-01-10 15:04:51 +00:00
Richard van der Hoff c175a5f0f2
Merge pull request #2770 from matrix-org/rav/fix_request_metrics
Update http request metrics before calling servlet
2018-01-10 14:53:40 +00:00
Richard van der Hoff d90e8ea444 Update http request metrics before calling servlet
Make sure that we set the servlet name in the metrics object *before* calling
the servlet, in case the servlet throws an exception.
2018-01-09 18:27:35 +00:00
Erik Johnston 8f03aa9f61 Add StorageProvider concept 2018-01-09 16:16:12 +00:00
Erik Johnston 2442e9876c Make PreviewUrlResource use MediaStorage 2018-01-09 16:15:07 +00:00
Erik Johnston 9d30a7691c Make ThumbnailResource use MediaStorage 2018-01-09 16:15:07 +00:00
Erik Johnston 9e20840e02 Use MediaStorage for remote media 2018-01-09 16:15:07 +00:00
Erik Johnston dd3092c3a3 Use MediaStorage for local files 2018-01-09 16:15:07 +00:00
Erik Johnston ada470bccb Add MediaStorage class 2018-01-09 16:15:07 +00:00
Erik Johnston 1ee787912b Add some helper classes 2018-01-09 16:15:07 +00:00
Erik Johnston 47ca5eb882 Split out add_file_headers 2018-01-09 16:15:07 +00:00
Erik Johnston b6c9deffda Remove dead TODO 2018-01-09 15:53:23 +00:00
Richard van der Hoff 51c9d9ed65 Add /room/{id}/event/{id} to synapse
Turns out that there is a valid usecase for retrieving event by id (notably
having received a push), but event ids should be scoped to room, so /event/{id}
is wrong.
2018-01-09 14:39:12 +00:00
Erik Johnston b30cd5b107 Remove dead code related to default thumbnails 2018-01-09 14:38:33 +00:00
Richard van der Hoff cb66a2d387
Merge pull request #2763 from matrix-org/rav/fix_config_uts
Fix broken config UTs
2018-01-09 12:08:08 +00:00
Richard van der Hoff 0211464ba2 Fix broken config UTs
https://github.com/matrix-org/synapse/pull/2755 broke log-config generation,
which in turn broke the unit tests.
2018-01-09 11:28:33 +00:00
Richard van der Hoff 3a556f1ea0 Make indentation of generated log config consistent
(we had a mix of 2- and 4-space indents)
2018-01-09 11:27:19 +00:00
Travis Ralston eccfc8e928
Fix templating error with unban permission message
Fixes https://github.com/matrix-org/synapse/issues/2759

Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-07 19:52:58 -07:00
Richard van der Hoff 840f72356e
Remove 'verbosity'/'log_file' from generated cfg
... because these only really exist to confuse people nowadays.

Also bring log config more into line with the generated log config, by making `level_for_storage`
apply to the `synapse.storage.SQL` logger rather than `synapse.storage`.
2018-01-05 12:30:28 +00:00
Erik Johnston 864a6d2977 Bump version and changelog 2018-01-05 10:54:01 +00:00
Richard van der Hoff 6e375f4597
Merge pull request #2744 from matrix-org/rav/login_logging
Better logging when login can't find a 3pid
2018-01-05 09:54:39 +00:00
Richard van der Hoff bd91857028 Check missing fields in event_from_pdu_json
Return a 400 rather than a 500 when somebody messes up their send_join
2017-12-30 18:40:19 +00:00
Richard van der Hoff 3079f80d4a Factor out `event_from_pdu_json`
turns out we have two copies of this, and neither needs to be an instance
method
2017-12-30 18:40:19 +00:00
Richard van der Hoff 65abc90fb6 federation_server: clean up imports 2017-12-30 18:40:19 +00:00
Richard van der Hoff a7b726ad18 federation_client: clean up imports 2017-12-30 18:40:19 +00:00
Richard van der Hoff 75c1b8df01 Better logging when login can't find a 3pid 2017-12-20 19:31:00 +00:00
Richard van der Hoff 3f9f1c50f3
Merge pull request #2683 from seckrv/fix_pwd_auth_prov_typo
synapse/config/password_auth_providers: Fixed bracket typo
2017-12-18 22:37:21 +00:00
Silke df0f602796 Implement listen_tcp method in remaining workers
Signed-off-by: Silke <silke@slxh.eu>
2017-12-18 20:00:42 +01:00
Silke 26cd3f5690 Remove logger argument and do not catch replication listener
Signed-off-by: Silke <silke@slxh.eu>
2017-12-18 20:00:42 +01:00
Silke Hofstra ed48ecc58c Add methods for listening on multiple addresses
Add listen_tcp and listen_ssl which implement Twisted's reactor.listenTCP
and reactor.listenSSL for multiple addresses.

Signed-off-by: Silke Hofstra <silke@slxh.eu>
2017-12-17 13:15:48 +01:00
Silke Hofstra 37d1a90025 Allow binds to both :: and 0.0.0.0
Binding on 0.0.0.0 when :: is specified in the bind_addresses is now allowed.
This causes a warning explaining the behaviour.
Configuration changed to match.

See #2232

Signed-off-by: Silke Hofstra <silke@slxh.eu>
2017-12-17 13:10:31 +01:00
Willem Mulder 3e59143ba8 Adapt the default config to bind on IPv6.
Most deployments are on Linux (or Mac OS), so this would actually bind
on both IPv4 and IPv6.

Resolves #1886.

Signed-off-by: Willem Mulder <willemmaster@hotmail.com>
2017-12-17 13:07:37 +01:00
Erik Johnston 069ae2a5d6 Bump changelog and version 2017-12-13 15:11:22 +00:00
Erik Johnston d8a6c734fa Merge branch 'develop' of github.com:matrix-org/synapse into erikj/createroom_content 2017-12-07 14:24:01 +00:00
Erik Johnston ef045dcd71 Copy dict in update_membership too 2017-12-07 14:17:15 +00:00
Matthew Hodgson 33cb7ef0b7
Merge pull request #2723 from matrix-org/matthew/search-all-local-users
Add all local users to the user_directory and optionally search them
2017-12-05 11:09:47 +00:00
Matthew Hodgson cdc2cb5d11 fix StoreError syntax 2017-12-05 11:09:31 +00:00