Commit Graph

726 Commits

Author SHA1 Message Date
Andrew Morgan cd36a1283b New test, fix issues 2019-03-20 14:00:39 +00:00
Andrew Morgan c69df5d5d3 Fix comments. v0.99.2 -> v0.99.3 2019-03-20 11:27:18 +00:00
Andrew Morgan 551ea11559 Just return if not doing any trailing slash shennanigans 2019-03-20 11:07:36 +00:00
Andrew Morgan a8ad39eec7 lint 2019-03-18 17:47:39 +00:00
Andrew Morgan 621e7f37f1 Better exception handling 2019-03-18 17:45:54 +00:00
Andrew Morgan ecea5af491 Correct var name 2019-03-13 21:21:03 +00:00
Andrew Morgan b2df0e8e2c receiving a 400 caused an exception. handle it 2019-03-13 21:08:10 +00:00
Andrew Morgan 9a2e22fd41 is this what purgatory feels like 2019-03-13 20:29:38 +00:00
Andrew Morgan 45524f2f5e i should have given up x2 2019-03-13 20:17:39 +00:00
Andrew Morgan 8d16ffaf7a i should have given up 2019-03-13 20:03:10 +00:00
Andrew Morgan 26f8e2d099 there comes a time when you should give up. but you dont 2019-03-13 19:49:08 +00:00
Andrew Morgan 5ca857ad84 as above 2019-03-13 19:35:23 +00:00
Andrew Morgan 7c0295f13c no kwargs today 2019-03-13 19:27:10 +00:00
Andrew Morgan 66cdb840a6 Or perhaps I was the one who was drunk 2019-03-13 19:18:25 +00:00
Andrew Morgan bec313818c go home python, you're drunk 2019-03-13 19:10:56 +00:00
Andrew Morgan c991e7aec7 Syntax checker is bork 2019-03-13 19:08:08 +00:00
Andrew Morgan c2d848b80d Destructure again 2019-03-13 19:04:43 +00:00
Andrew Morgan ee8ba397e8 Are you happy now 2019-03-13 18:48:31 +00:00
Andrew Morgan 9dd0e34679 Syntax test 2019-03-13 18:45:17 +00:00
Andrew Morgan 220607a618 Remove testing code 2019-03-13 18:43:40 +00:00
Andrew Morgan 660b77f362 Add missing docstring detail 2019-03-13 18:38:16 +00:00
Andrew Morgan 5526b054aa Fix syntax issues 2019-03-13 18:35:21 +00:00
Andrew Morgan 09626bfd39 Switch to wrapper function around _send_request 2019-03-13 18:26:06 +00:00
Andrew Morgan 7d053cfe10 Retry on 400:M_UNRECOGNIZED 2019-03-13 12:10:33 +00:00
Andrew Morgan 7e75d9644b Fix paranthesis indent 2019-03-13 11:15:23 +00:00
Andrew Morgan cf301e37d8 Add workaround note 2019-03-13 11:14:43 +00:00
Andrew Morgan 0ea8582f8b Cleaner way of implementing trailing slashes 2019-03-12 14:11:11 +00:00
Andrew Morgan 64ff11019e Retry certain federation requests on 404 2019-03-08 18:22:47 +00:00
Andrew Morgan ac61b45a75 Minor docstring fixes for MatrixFederationAgent (#4765) 2019-02-28 16:24:01 +00:00
Erik Johnston c7b333c545 Log tracebacks correctly 2019-02-25 16:56:41 +00:00
Richard van der Hoff c88bc53903
Fix TaskStopped exceptions when outbound requests time out (#4690) 2019-02-20 11:35:52 +00:00
Erik Johnston fe725f7e45 Cleanup top level request exception logging
Firstly, we always logged that the request was being handled via
`JsonResource._async_render`, so we change that to use the servlet name
we add to the request.

Secondly, we pass the exception information to the logger rather than
formatting it manually. This makes it consistent with other exception
logging, allwoing logging hooks and formatters to access the exception
information.
2019-02-18 15:11:04 +00:00
Richard van der Hoff 5d27730a73
Move ClientTLSOptionsFactory init out of refresh_certificates (#4611)
It's nothing to do with refreshing the certificates. No idea why it was here.
2019-02-11 18:03:30 +00:00
Richard van der Hoff ef43a03fc5
Merge pull request #4546 from matrix-org/rav/silence_critical_error_from_federation
Fix noisy "twisted.internet.task.TaskStopped" errors in logs
2019-02-01 14:37:22 +00:00
Richard van der Hoff fa794980ec
Merge pull request #4544 from matrix-org/rav/skip_invalid_well_known
Treat an invalid .well-known the same as an absent one
2019-02-01 13:18:36 +00:00
Richard van der Hoff e9779a6f8f Fix b'ab' noise in logs 2019-02-01 12:34:31 +00:00
Richard van der Hoff f0ba34f581 Fix noisy "twisted.internet.task.TaskStopped" errors in logs
Fixes #4003
2019-02-01 12:22:57 +00:00
Richard van der Hoff 8a21b03fba Treat an invalid .well-known the same as an absent one
... basically, carry on and fall back to SRV etc.
2019-02-01 11:37:31 +00:00
Richard van der Hoff 3c8a41140e Cache failures to parse .well-known
Also add a Measure block around the .well-known fetch
2019-02-01 00:37:52 +00:00
Richard van der Hoff 24d59c7568 better logging for federation connections 2019-01-31 23:18:20 +00:00
Richard van der Hoff d428b46346 Update federation routing logic to check .well-known before SRV 2019-01-31 23:14:18 +00:00
Richard van der Hoff a79034aedf
Merge pull request #4521 from matrix-org/rav/fed_routing/cleanups
Tiny .well-known fixes
2019-01-30 11:47:24 +00:00
Richard van der Hoff c7b24ac3d0
Follow redirects on .well-known (#4520) 2019-01-30 11:43:33 +00:00
Richard van der Hoff 09a1a6b55e fix exception text 2019-01-30 10:58:52 +00:00
Richard van der Hoff 928c50b59a Also jitter the invalid cache period 2019-01-30 10:58:52 +00:00
Richard van der Hoff bc5f6e1797
Add a caching layer to .well-known responses (#4516) 2019-01-30 10:55:25 +00:00
Richard van der Hoff cc2d650ef7
Relax requirement for a content-type on .well-known (#4511) 2019-01-29 16:49:17 +00:00
Richard van der Hoff 99e36d5e24
Implement MSC1708 (.well-known lookups for server routing) (#4489) 2019-01-29 13:53:02 +00:00
Richard van der Hoff 3bd0f1a4a3 docstrings for _RoutingResult 2019-01-28 12:43:09 +00:00
Richard van der Hoff 0fd5b3b53e Handle IP literals explicitly
We don't want to be doing .well-known lookups on these guys.
2019-01-28 10:34:30 +00:00
Richard van der Hoff 51958df766 MatrixFederationAgent: factor out routing logic
This is going to get too big and unmanageable.
2019-01-28 10:34:30 +00:00
Richard van der Hoff d840019192
Fix idna and ipv6 literal handling in MatrixFederationAgent (#4487)
Turns out that the library does a better job of parsing URIs than our
reinvented wheel. Who knew.

There are two things going on here. The first is that, unlike
parse_server_name, URI.fromBytes will strip off square brackets from IPv6
literals, which means that it is valid input to ClientTLSOptionsFactory and
HostnameEndpoint.

The second is that we stay in `bytes` throughout (except for the argument to
ClientTLSOptionsFactory), which avoids the weirdness of (sometimes) ending up
with idna-encoded values being held in `unicode` variables. TBH it probably
would have been ok but it made the tests fragile.
2019-01-28 09:56:59 +00:00
Richard van der Hoff 8520bc3109
Fix Host header sent by MatrixFederationAgent (#4468)
Move the Host header logic down here so that (a) it is used if we reuse the
agent elsewhere, and (b) we can mess about with it with .well-known.
2019-01-25 12:38:16 +00:00
Richard van der Hoff 4a6e863843
Merge pull request #4464 from matrix-org/rav/fix_srv_lookup
MatrixFederationAgent: Look up the right SRV record
2019-01-24 13:57:51 +00:00
Richard van der Hoff f4697b5ec1
Fix UnboundLocalError in post_urlencoded_get_json (#4460)
This could cause exceptions if the id server returned 4xx responses.
2019-01-24 13:38:29 +00:00
Richard van der Hoff afd69a0920 Look up the right SRV record 2019-01-24 13:31:43 +00:00
Richard van der Hoff 7021784d46 put resolve_service in an object
this makes it easier to stub things out for tests.
2019-01-22 20:35:12 +00:00
Richard van der Hoff 53a327b4d5 Require that service_name be a byte string
it is only ever a bytes now, so let's enforce that.
2019-01-22 20:35:12 +00:00
Richard van der Hoff fe212bbe4a Kill off matrix_federation_endpoint
this thing is now redundant.
2019-01-22 20:34:35 +00:00
Richard van der Hoff 7871146667 Make MatrixFederationClient use MatrixFederationAgent
... instead of the matrix_federation_endpoint
2019-01-22 20:34:35 +00:00
Richard van der Hoff 44be7513bf MatrixFederationAgent
Pull the magic that is currently in matrix_federation_endpoint and friends into
an agent-like thing
2019-01-22 20:34:35 +00:00
Richard van der Hoff 33a55289cb
Refactor and bugfix for resove_service (#4427) 2019-01-22 10:59:27 +00:00
Richard van der Hoff de6888e7ce Remove redundant WrappedConnection (#4409)
* Remove redundant WrappedConnection

The matrix federation client uses an HTTP connection pool, which times out its
idle HTTP connections, so there is no need for any of this business.
2019-01-18 23:07:38 +11:00
Richard van der Hoff 9feb5d0b71
sign_request -> build_auth_headers (#4408)
Just got very confused about the fact that the headers are only an output, not
an input.
2019-01-17 12:40:09 +00:00
Erik Johnston 34ea14139d Fixup docstrings for matrixfederationclient 2019-01-09 09:25:59 +00:00
Erik Johnston d422570e9b Use RequestSendFailed when fail to parse content type headers 2019-01-08 14:22:18 +00:00
Erik Johnston b970cb0e96 Refactor request sending to have better excpetions (#4358)
* Correctly retry and back off if we get a HTTPerror response

* Refactor request sending to have better excpetions

MatrixFederationHttpClient blindly reraised exceptions to the caller
without differentiating "expected" failures (e.g. connection timeouts
etc) versus more severe problems (e.g. programming errors).

This commit adds a RequestSendFailed exception that is raised when
"expected" failures happen, allowing the TransactionQueue to log them as
warnings while allowing us to log other exceptions as actual exceptions.
2019-01-08 11:04:28 +00:00
Amber Brown ea6abf6724
Fix IP URL previews on Python 3 (#4215) 2018-12-22 01:56:13 +11:00
Richard van der Hoff 0a1fc52971 fix parse_string docstring 2018-11-08 11:12:29 +00:00
Amber Brown b3708830b8
Fix URL preview bugs (type error when loading cache from db, content-type including quotes) (#4157) 2018-11-08 01:37:43 +11:00
Travis Ralston 43c3f0b02f
Merge pull request #3969 from turt2live/travis/fix-federated-group-requests
Handle HttpResponseException more safely for federated groups
2018-10-23 10:41:04 -06:00
Amber Brown b69216f768
Make the metrics less racy (#4061) 2018-10-19 21:45:45 +11:00
Richard van der Hoff b8a5b0097c
Various cleanups in the federation client code (#4031)
- Improve logging: log things in the right order, include destination and txids
  in all log lines, don't log successful responses twice

- Fix the docstring on TransportLayerClient.send_transaction

- Don't use treq.request, which is overcomplicated for our purposes: just use a
  twisted.web.client.Agent.

- simplify the logic for setting up the bodyProducer

- fix bytes/str confusions
2018-10-16 10:44:49 +01:00
Travis Ralston e3586f7c06 Merge branch 'develop' into travis/fix-federated-group-requests 2018-10-12 14:49:58 -06:00
Erik Johnston 334e075dd8 Fix error when logging incomplete requests
If a connection is lost before a request is read from Request, Twisted
sets `method` (and `uri`) attributes to dummy values. These dummy values
have incorrect types (i.e. they're not bytes), and so things like
`__repr__` would raise an exception.

To fix this we had a helper method to return the method with a
consistent type.
2018-10-02 12:06:22 +01:00
Travis Ralston 219606a6ed Fix exception documentation in matrixfederationclient.py 2018-09-26 13:26:27 -06:00
Richard van der Hoff 94f7befc31
Merge pull request #3925 from matrix-org/erikj/fix_producers_unregistered
Fix spurious exceptions when client closes conncetion
2018-09-25 11:52:06 +01:00
Erik Johnston b28a7ed503 Fix spurious exceptions when client closes conncetion
If a HTTP handler throws an exception while processing a request we
automatically write a JSON error response. If the handler had already
started writing a response twisted throws an exception.

We should check for this case and simple abort the connection if there
was an error after the response had started being written.
2018-09-20 13:44:20 +01:00
Amber Brown 1f3f5fcf52
Fix client IPs being broken on Python 3 (#3908) 2018-09-20 20:14:34 +10:00
Amber Brown aeca5a5ed5
Add a regression test for logging on failed connections (#3912) 2018-09-20 16:28:18 +10:00
Erik Johnston a334e1cace Update to use new timeout function everywhere.
The existing deferred timeout helper function (and the one into twisted)
suffer from a bug when a deferred's canceller throws an exception, #3842.

The new helper function doesn't suffer from this problem.
2018-09-19 10:39:40 +01:00
Amber Brown 47c02e6332
Merge pull request #3909 from turt2live/travis/fix-logging-1
Fix matrixfederationclient.py logging: Destination is a string
2018-09-19 18:14:47 +10:00
Amber Brown 3d6b24fb1b
Merge pull request #3907 from matrix-org/rav/set_sni_to_server_name
Set SNI to the server_name, not whatever was in the SRV record
2018-09-19 17:59:33 +10:00
Travis Ralston 35aec19f0a Destination is a string 2018-09-18 15:29:30 -06:00
Richard van der Hoff 31c15dcb80
Refactor matrixfederationclient to fix logging (#3906)
We want to wait until we have read the response body before we log the request
as complete, otherwise a confusing thing happens where the request appears to
have completed, but we later fail it.

To do this, we factor the salient details of a request out to a separate
object, which can then keep track of the txn_id, so that it can be logged.
2018-09-18 18:17:15 +01:00
Richard van der Hoff b3097396e7 Set SNI to the server_name, not whatever was in the SRV record
Fixes #3843
2018-09-18 17:01:12 +01:00
Richard van der Hoff ac80cb08fe Fix more b'abcd' noise in metrics 2018-09-17 17:16:50 +01:00
Richard van der Hoff f00a9d2636 Fix some b'abcd' noise in logs and metrics
Python 3 compatibility: make sure that we decode some byte sequences before we
use them to create log lines and metrics labels.
2018-09-17 16:15:42 +01:00
Erik Johnston fcfe7a850d Add an awful secondary timeout to fix wedged requests
This is an attempt to mitigate #3842 by adding yet-another-timeout
2018-09-14 19:23:07 +01:00
Erik Johnston 3e6e94fe9f
Merge pull request #3872 from matrix-org/hawkowl/timeouts-2
timeouts 2: electric boogaloo
2018-09-14 16:58:44 +01:00
Amber Brown bc9af88a2d fix 2018-09-15 00:26:00 +10:00
Erik Johnston 9e2f9a7b57 Measure outbound requests 2018-09-14 15:11:26 +01:00
Amber Brown c971aa7b9d fix 2018-09-14 03:57:02 +10:00
Amber Brown 8f08d848f5 fix 2018-09-14 03:53:56 +10:00
Amber Brown 7c33ab76da redact better 2018-09-14 03:45:34 +10:00
Amber Brown 63755fa4c2 we do that higher up 2018-09-14 03:21:47 +10:00
Amber Brown 73884ebac5 Merge remote-tracking branch 'origin/develop' into hawkowl/timeouts-2 2018-09-14 03:11:25 +10:00
Amber Brown 7c27c4d51c
merge (#3576) 2018-09-14 03:11:11 +10:00
Amber Brown 1c3f4d9ca5 buffer? 2018-09-14 03:09:13 +10:00
Amber Brown bfa0b759e0
Attempt to figure out what's going on with timeouts (#3857) 2018-09-14 00:15:51 +10:00
Erik Johnston 8c5b84441b Log outbound requests when we retry 2018-09-12 16:22:14 +01:00
Erik Johnston 4084a774a8 Timeout reading body for outbound HTTP requests 2018-09-12 10:10:20 +01:00
Amber Brown 2d2828dcbc
Port http/ to Python 3 (#3771) 2018-09-06 00:10:47 +10:00
Erik Johnston 1d67b13674 Fix bug when federation_domain_whitelist is an emtpy list
Outbound federation were incorrectly allowed when the config option was
set to an empty list
2018-08-24 12:13:12 +01:00
Richard van der Hoff a52f276990 Bugfixes
--------
 
 - Fix bug in v0.33.3rc1 which caused infinite loops and OOMs
 ([\#3723](https://github.com/matrix-org/synapse/issues/3723))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEIQBQJ4l+yK4dlKkFIwi0edOSShEFAlt6/yMACgkQIwi0edOS
 ShFrzAgAhLA74fClydGLfhPNCXIw12Amv2ASA6+r0ukqjYQ/9MZ0CRsL+PMZJTk6
 Z75PK3TkX38zSpPaUx9HdMQO/yE+rpI99ZxlBM+8jzsf1jyF8ljqxSfixXUlaXX9
 698PQQ3f+IDoSs59NKrHvoWhUThaBpf/1GwRxG5mUcm2oKJItNMHk/uLurNCaIIZ
 YSwWcxvZdluOMX0WrlF/S524DqtgULcRf3E9x3mumobkVGxI3UH73hXgGUcPi7XV
 ttLuy7fVbV2Cun/wvjBqD7kc8LNbKwSaT9y+/xS0D5QYjph4rpQiTG7S5rtPLPMU
 Z2lL1C5ggBCkD43g3orDvehbNWCL5g==
 =0vEe
 -----END PGP SIGNATURE-----

Merge tag 'v0.33.3rc2' into develop

Bugfixes
--------

- Fix bug in v0.33.3rc1 which caused infinite loops and OOMs
([\#3723](https://github.com/matrix-org/synapse/issues/3723))
2018-08-21 20:30:09 +01:00
Erik Johnston cd6937fb26 Fix typo 2018-08-21 16:28:10 +01:00
Richard van der Hoff be6527325a Fix exceptions when a connection is closed before we read the headers
This fixes bugs introduced in #3700, by making sure that we behave sanely
when an incoming connection is closed before the headers are read.
2018-08-20 18:21:10 +01:00
Erik Johnston b01a755498 Make the in flight requests metrics thread safe 2018-08-20 17:27:52 +01:00
Amber Brown 324525f40c
Port over enough to get some sytests running on Python 3 (#3668) 2018-08-20 23:54:49 +10:00
Richard van der Hoff 63260397c6
Merge pull request #3701 from matrix-org/rav/use_producer_for_responses
Use a producer to stream back responses
2018-08-17 14:58:45 +01:00
Richard van der Hoff afcd655ab6 Use a producer to stream back responses
The problem with dumping all of the json response into the Request object at
once is that doing so starts the timeout for the next request to be received:
so if it takes longer than 60s to stream back the response to the client, the
client never gets it.

The correct solution is to use a Producer; then the timeout is only started
once all of the content is sent over the TCP connection.
2018-08-15 15:04:16 +01:00
Richard van der Hoff eaaa2248ff Refactor request logging code
This commit moves a bunch of the logic for deciding when to log the receipt and
completion of HTTP requests into SynapseRequest, rather than in the request
handling wrappers.

Advantages of this are:
 * we get logs for *all* requests (including OPTIONS and HEADs), rather than
   just those that end up hitting handlers we've remembered to decorate
   correctly.

 * when a request handler wires up a Producer (as the media stuff does
   currently, and as other things will do soon), we log at the point that all
   of the traffic has been sent to the client.
2018-08-15 13:47:52 +01:00
Amber Brown b37c472419
Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678) 2018-08-10 23:50:21 +10:00
Richard van der Hoff 3c0213a217
Merge pull request #3439 from vojeroen/send_sni_for_federation_requests
send SNI for federation requests
2018-08-10 12:23:54 +01:00
Richard van der Hoff f900d50824 include known room versions in outgoing make_joins 2018-08-06 13:45:37 +01:00
Richard van der Hoff 01e93f48ed Kill off MatrixCodeMessageException
This code brings the SimpleHttpClient into line with the
MatrixFederationHttpClient by having it raise HttpResponseExceptions when a
request fails (rather than trying to parse for matrix errors and maybe raising
MatrixCodeMessageException).

Then, whenever we were checking for MatrixCodeMessageException and turning them
into SynapseErrors, we now need to check for HttpResponseExceptions and call
to_synapse_error.
2018-08-01 16:02:46 +01:00
Richard van der Hoff fa7dc889f1 Be more careful which errors we send back over the C-S API
We really shouldn't be sending all CodeMessageExceptions back over the C-S API;
it will include things like 401s which we shouldn't proxy.

That means that we need to explicitly turn a few HttpResponseExceptions into
SynapseErrors in the federation layer.

The effect of the latter is that the matrix errcode will get passed through
correctly to calling clients, which might help with some of the random
M_UNKNOWN errors when trying to join rooms.
2018-08-01 16:02:38 +01:00
Amber Brown da7785147d
Python 3: Convert some unicode/bytes uses (#3569) 2018-08-02 00:54:06 +10:00
Richard van der Hoff 7041cd872b
Merge branch 'develop' into send_sni_for_federation_requests 2018-07-27 09:17:11 +01:00
Amber Brown a97c845271
Move v1-only APIs into their own module & isolate deprecated ones (#3460) 2018-07-19 20:03:33 +10:00
Erik Johnston 5bd0a47fcd pep8 2018-07-18 14:19:00 +01:00
Erik Johnston e45a46b6e4 Add response code to response timer metrics 2018-07-18 13:59:36 +01:00
Amber Brown bc006b3c9d
Refactor REST API tests to use explicit reactors (#3351) 2018-07-17 20:43:18 +10:00
Krombel 78a9ddcf9a rerun isort with latest version 2018-07-16 14:23:25 +02:00
Krombel 4a27000548 check isort by travis 2018-07-16 13:57:33 +02:00
Jeroen 505530f36a Merge remote-tracking branch 'upstream/develop' into send_sni_for_federation_requests
# Conflicts:
#	synapse/crypto/context_factory.py
2018-07-14 20:24:46 +02:00
Krombel 3366b9c534 rename assert_params_in_request to assert_params_in_dict
the method "assert_params_in_request" does handle dicts and not
requests. A request body has to be parsed to json before this method
can be used
2018-07-13 21:53:01 +02:00
Richard van der Hoff 55370331da
Refactor logcontext resource usage tracking (#3501)
Factor out the resource usage tracking out to a separate object, which can be
passed around and copied independently of the logcontext itself.
2018-07-10 13:56:07 +01:00
Jeroen b5e157d895 Merge branch 'develop' into send_sni_for_federation_requests
# Conflicts:
#	synapse/http/endpoint.py
2018-07-09 08:51:11 +02:00
Amber Brown 49af402019 run isort 2018-07-09 16:09:20 +10:00
Richard van der Hoff 546bc9e28b More server_name validation
We need to do a bit more validation when we get a server name, but don't want
to be re-doing it all over the shop, so factor out a separate
parse_and_validate_server_name, and do the extra validation.

Also, use it to verify the server name in the config file.
2018-07-04 18:59:51 +01:00
Richard van der Hoff 508196e08a
Reject invalid server names (#3480)
Make sure that server_names used in auth headers are sane, and reject them with
a sensible error code, before they disappear off into the depths of the system.
2018-07-03 14:36:14 +01:00
Matthew Hodgson fc4f8f33be replace invalid utf8 with \ufffd 2018-07-02 11:33:02 +01:00
Matthew Hodgson 1c867f5391 a fix which doesn't NPE everywhere 2018-07-01 11:56:33 +01:00
Matthew Hodgson f131bf8d3e don't mix unicode strings with utf8-in-byte-strings
otherwise we explode with:

```
Traceback (most recent call last):
  File /usr/lib/python2.7/logging/handlers.py, line 78, in emit
    logging.FileHandler.emit(self, record)
  File /usr/lib/python2.7/logging/__init__.py, line 950, in emit
    StreamHandler.emit(self, record)
  File /usr/lib/python2.7/logging/__init__.py, line 887, in emit
    self.handleError(record)
  File /usr/lib/python2.7/logging/__init__.py, line 810, in handleError
    None, sys.stderr)
  File /usr/lib/python2.7/traceback.py, line 124, in print_exception
    _print(file, 'Traceback (most recent call last):')
  File /usr/lib/python2.7/traceback.py, line 13, in _print
    file.write(str+terminator)
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_io.py, line 170, in write
    self.log.emit(self.level, format=u{log_io}, log_io=line)
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 144, in emit
    self.observer(event)
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 136, in __call__
    errorLogger = self._errorLoggerForObserver(brokenObserver)
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 156, in _errorLoggerForObserver
    if obs is not observer
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 81, in __init__
    self.log = Logger(observer=self)
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 64, in __init__
    namespace = self._namespaceFromCallingContext()
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 42, in _namespaceFromCallingContext
    return currentframe(2).f_globals[__name__]
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/compat.py, line 93, in currentframe
    for x in range(n + 1):
RuntimeError: maximum recursion depth exceeded while calling a Python object
Logged from file site.py, line 129
  File /usr/lib/python2.7/logging/__init__.py, line 859, in emit
    msg = self.format(record)
  File /usr/lib/python2.7/logging/__init__.py, line 732, in format
    return fmt.format(record)
  File /usr/lib/python2.7/logging/__init__.py, line 471, in format
    record.message = record.getMessage()
  File /usr/lib/python2.7/logging/__init__.py, line 335, in getMessage
    msg = msg % self.args
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 4: ordinal not in range(128)
Logged from file site.py, line 129
```

...where the logger apparently recurses whilst trying to log the error, hitting the
maximum recursion depth and killing everything badly.
2018-07-01 05:08:58 +01:00
Amber Brown 6350bf925e
Attempt to be more performant on PyPy (#3462) 2018-06-28 14:49:57 +01:00
Jeroen b7f34ee348 allow self-signed certificates 2018-06-26 20:41:05 +02:00
Erik Johnston 947fea67cb Need to pass reactor to endpoint fac 2018-06-25 15:22:57 +01:00
Amber Brown 07cad26d65
Remove all global reactor imports & pass it around explicitly (#3424) 2018-06-25 14:08:28 +01:00
Jeroen 07b4f88de9 formatting changes for pep8 2018-06-25 12:31:16 +02:00
Jeroen 3d605853c8 send SNI for federation requests 2018-06-24 22:38:43 +02:00
Richard van der Hoff 70e6501913
Merge pull request #3419 from matrix-org/rav/events_per_request
Log number of events fetched from DB
2018-06-22 11:17:56 +01:00
Amber Brown 77ac14b960
Pass around the reactor explicitly (#3385) 2018-06-22 09:37:10 +01:00
Richard van der Hoff b088aafcae Log number of events fetched from DB
When we finish processing a request, log the number of events we fetched from
the database to handle it.

[I'm trying to figure out which requests are responsible for large amounts of
event cache churn. It may turn out to be more helpful to add counts to the
prometheus per-request/block metrics, but that is an extension to this code
anyway.]
2018-06-21 06:15:03 +01:00
Amber Brown f6c4d74f96
Fix inflight requests metric (incorrect name & traceback) (#3413) 2018-06-20 11:18:57 +01:00
David Baker 187a546bff
Merge pull request #3276 from matrix-org/dbkr/unbind
Remove email addresses / phone numbers from ID servers when they're removed from synapse
2018-06-11 16:02:00 +01:00
Michael Telatynski 330432031b
redact_uri in two missed log paths 2018-06-06 10:25:48 +01:00
David Baker bf54c1cf6c pep8 2018-06-06 10:15:33 +01:00
David Baker 3e4bc4488c More doc fixes 2018-06-06 09:44:10 +01:00
Michael Telatynski e6cbf47773
factor out uri redaction into a method on http 2018-06-05 18:31:40 +01:00
David Baker 607bd27c83 fix pep8 2018-06-05 18:10:35 +01:00
David Baker d62162bbec doc fixes 2018-06-05 18:09:13 +01:00
David Baker c5930d513a Docstring 2018-06-04 12:05:58 +01:00
Michael Telatynski 09503126df
Strip `access_token` from outgoing requests using existing regex 2018-06-02 23:25:13 +01:00
Amber Brown 3ef5cd74a6 update to more consistently use seconds in any metrics or logging 2018-05-28 19:39:27 +10:00
Amber Brown a2eb5db4a0 update metrics to be in seconds 2018-05-28 19:10:27 +10:00
Amber Brown 754826a830 Merge remote-tracking branch 'origin/develop' into 3218-official-prom 2018-05-28 18:57:23 +10:00
Amber Brown c4fb15a06c
Merge pull request #3246 from NotAFile/py3-repr-string
use repr, not str
2018-05-24 13:00:20 -05:00
David Baker 77a23e2e05 Merge remote-tracking branch 'origin/develop' into dbkr/unbind 2018-05-24 16:20:53 +01:00
Erik Johnston 46345187cc
Merge pull request #3243 from NotAFile/py3-six-3
Replace some more comparisons with six
2018-05-24 16:08:57 +01:00
Amber Brown e987079037 fixes 2018-05-23 13:03:51 -05:00
David Baker 2c7866d664 Hit the 3pid unbind endpoint on deactivation 2018-05-23 14:38:56 +01:00
Amber Brown 53cc2cde1f cleanup 2018-05-22 17:32:57 -05:00
Amber Brown 071206304d cleanup pep8 errors 2018-05-22 16:54:22 -05:00
Amber Brown 8f5a688d42 cleanups, self-registration 2018-05-22 10:56:03 -05:00
Amber Brown a8990fa2ec Merge remote-tracking branch 'origin/develop' into 3218-official-prom 2018-05-22 10:50:26 -05:00
Erik Johnston c435b0b441 Don't store context 2018-05-22 09:34:26 +01:00
Erik Johnston fb2806b186 Move in_flight_requests_count to be a callback metric 2018-05-22 09:31:53 +01:00
Amber Brown fcc525b0b7 rest of the changes 2018-05-21 19:48:57 -05:00
Amber Brown df9f72d9e5 replacing portions 2018-05-21 19:47:37 -05:00
Erik Johnston dfa70adc33 Add in flight request metrics
This tracks CPU and DB usage while requests are in flight, rather than
when we write the response.
2018-05-21 16:23:06 +01:00
Adrian Tschira d9fe2b2d9d Replace some more comparisons with six
plus a bonus b"" string I missed last time

Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:56:31 +02:00
Adrian Tschira b932b4ea25 use repr, not str
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:28:42 +02:00
Richard van der Hoff 8030a825c8
Merge pull request #3213 from matrix-org/rav/consent_handler
ConsentResource to gather policy consent from users
2018-05-16 07:19:18 +01:00
Richard van der Hoff 47815edcfa ConsentResource to gather policy consent from users
Hopefully there are enough comments and docs in this that it makes sense on its
own.
2018-05-15 15:11:59 +01:00
Richard van der Hoff 318711e139 Set Server header in SynapseRequest
(instead of everywhere that writes a response. Or rather, the subset of places
which write responses where we haven't forgotten it).

This also means that we don't have to have the mysterious version_string
attribute in anything with a request handler.

Unfortunately it does mean that we have to pass the version string wherever we
instantiate a SynapseSite, which has been c&ped 150 times, but that is code
that ought to be cleaned up anyway really.
2018-05-10 18:50:27 +01:00
Richard van der Hoff 645cb4bf06 Remove redundant request_handler decorator
This is needless complexity; we might as well use the wrapper directly.

Also rename wrap_request_handler->wrap_json_request_handler.
2018-05-10 12:19:53 +01:00
Richard van der Hoff 09f570b935 Factor wrap_request_handler_with_logging out of wrap_request_handler
... so that it can be used on non-JSON endpoints
2018-05-10 12:19:52 +01:00
Richard van der Hoff 9589a1925e Remove include_metrics param
The metrics are now available via the request, so this is redundant and can go
away at last.
2018-05-10 12:19:52 +01:00
Richard van der Hoff 49e5a613f1 Move outgoing_responses_counter handling to RequestMetrics
it's much neater there.
2018-05-10 12:19:52 +01:00
Richard van der Hoff b8700dd7d0 Bump requests_counter in wrapped_request_handler
less magic
2018-05-10 12:19:52 +01:00
Richard van der Hoff c6f730282c Move RequestMetrics handling into SynapseRequest.processing()
It fits quite nicely here, and opens the path to getting rid of the
"include_metrics" mess.
2018-05-10 12:19:51 +01:00
Richard van der Hoff 09b29f9c4a Make RequestMetrics take a raw time rather than a clock
... which is going to make it easier to move around.
2018-05-10 12:18:52 +01:00
Richard van der Hoff 8460e48d06 Move request_id management into SynapseRequest 2018-05-10 11:48:17 +01:00
Richard van der Hoff 18e144fe08 Move RequestsMetrics to its own file
This is useful in its own right, because server.py is full of stuff; but more
importantly, I want to do some refactoring that will cause a circular reference
as it is.
2018-05-09 19:55:03 +01:00
Richard van der Hoff 2117f84323
Merge pull request #3182 from Half-Shot/hs/fix-twisted-shutdown
Fix 'Unhandled Error' logs with Twisted 18.4
2018-05-03 12:40:11 +01:00
Will Hunt 2e7a94c36b Don't abortConnection() if the transport connection has already closed. 2018-05-03 12:31:47 +01:00
Richard van der Hoff 1dfd650348 add missing param to cancelled_to_request_timed_out_error
This gets two arguments, not one.
2018-05-02 22:42:36 +01:00
Richard van der Hoff 049b0b5af2
Merge pull request #3154 from NotAFile/py3-stringio
Replace stringIO imports with six
2018-04-30 00:59:04 +01:00
Richard van der Hoff dbf6f28d64
Merge pull request #3155 from NotAFile/py3-bytes-1
more bytes strings
2018-04-30 00:38:21 +01:00
Richard van der Hoff 9e2601f830
Merge pull request #3108 from NotAFile/py3-six-urlparse
Use six.moves.urlparse
2018-04-30 00:33:05 +01:00
Adrian Tschira e9143b6593 more bytes strings
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-29 00:13:57 +02:00
Adrian Tschira 4f2f5171b7 replace stringIO imports 2018-04-28 13:46:23 +02:00
Richard van der Hoff 9d2c1b8429 Backport deferred.addTimeout
Twisted 16.0 doesn't have addTimeout, so let's backport it.
2018-04-27 12:52:30 +01:00
Richard van der Hoff 1ea904b9f0 Use deferred.addTimeout instead of time_bound_deferred
This doesn't feel like a wheel we need to reinvent.
2018-04-23 00:53:18 +01:00
Adrian Tschira 2a3c33ff03 Use six.moves.urlparse
The imports were shuffled around a bunch in py3

Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 21:22:43 +02:00
Richard van der Hoff 87478c5a60
Merge pull request #3061 from NotAFile/add-some-byte-strings
Add b prefixes to some strings that are bytes in py3
2018-04-09 23:54:05 +01:00
Richard van der Hoff 664adb4236
Merge pull request #3016 from silkeh/improve-service-lookups
Improve handling of SRV records for federation connections
2018-04-09 23:40:06 +01:00
Richard van der Hoff 13decdbf96 Revert "Merge pull request #3066 from matrix-org/rav/remove_redundant_metrics"
We aren't ready to release this yet, so I'm reverting it for now.

This reverts commit d1679a4ed7, reversing
changes made to e089100c62.
2018-04-09 12:59:12 +01:00
Erik Johnston 7b824f1475 Add response size metrics 2018-04-06 13:20:11 +01:00
Krombel 1d71f484d4 use PUT instead of POST for federating groups/m.join_policy 2018-04-06 12:54:09 +02:00
Richard van der Hoff 518f6de088 Remove redundant metrics which were deprecated in 0.27.0. 2018-04-04 19:46:28 +01:00
Adrian Tschira 6168351877 Add b prefixes to some strings that are bytes in py3
This has no effect on python2

Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-04 13:48:51 +02:00
Silke 72251d1b97 Remove address resolution of hosts in SRV records
Signed-off-by: Silke Hofstra <silke@slxh.eu>
2018-04-04 12:26:50 +02:00
Erik Johnston 1c41b05c8c Add Cache-Control headers to all JSON APIs
It is especially important that sync requests don't get cached, as if a
sync returns the same token given then the client will call sync with
the same parameters again. If the previous response was cached it will
get reused, resulting in the client tight looping making the same
request and never making any progress.

In general, clients will expect to get up to date data when requesting
APIs, and so its safer to do a blanket no cache policy than only
whitelisting APIs that we know will break things if they get cached.
2018-03-21 17:46:26 +00:00
Erik Johnston fa72803490 Merge branch 'master' of github.com:matrix-org/synapse into develop 2018-03-19 11:41:01 +00:00
Erik Johnston a8ce159be4 Replace some ujson with simplejson to make it work 2018-03-16 00:27:09 +00:00
Erik Johnston 926ba76e23 Replace ujson with simplejson 2018-03-15 23:43:31 +00:00
Richard van der Hoff 58dd148c4f Add some docstrings to help figure this out 2018-03-09 18:05:41 +00:00
Richard van der Hoff 88541f9009 Add a metric which increments when a request is received
It's useful to know when there are peaks in incoming requests - which isn't
quite the same as there being peaks in outgoing responses, due to the time
taken to handle requests.
2018-03-09 16:30:26 +00:00
Richard van der Hoff dbe80a286b refactor JsonResource
rephrase the OPTIONS and unrecognised request handling so that they look
similar to the common flow.
2018-03-09 16:22:16 +00:00
Richard van der Hoff 10b34dbb9a
Merge pull request #2858 from matrix-org/rav/purge_updates
delete_local_events for purge_room_history
2018-02-09 14:11:00 +00:00
Richard van der Hoff 74fcbf741b delete_local_events for purge_history
Add a flag which makes the purger delete local events
2018-02-09 13:07:41 +00:00
Richard van der Hoff 78e7e05188
Merge pull request #2838 from matrix-org/rav/fix_logging_on_dns_fail
Remove spurious log argument
2018-01-31 22:18:46 +01:00
Richard van der Hoff 421d68ca8c
Merge pull request #2817 from matrix-org/rav/http_conn_pool
Use a connection pool for the SimpleHttpClient
2018-01-31 22:14:22 +01:00
Richard van der Hoff af19f5e9aa Remove spurious log argument
... which would cause scary-looking and unhelpful errors in the log on dns fail
2018-01-30 17:52:03 +00:00
Richard van der Hoff 02ba118f81 Increase http conn pool size 2018-01-29 14:30:15 +00:00
Matthew Hodgson ab9f844aaf
Add federation_domain_whitelist option (#2820)
Add federation_domain_whitelist

gives a way to restrict which domains your HS is allowed to federate with.
useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network
2018-01-22 19:11:18 +01:00
Richard van der Hoff 2c8526cac7 Use a connection pool for the SimpleHttpClient
In particular I hope this will help the pusher, which makes many requests to
sygnal, and is currently negotiating SSL for each one.
2018-01-20 00:55:44 +00:00
Richard van der Hoff e8f7541d3f Merge remote-tracking branch 'origin/develop' into rav/track_db_scheduling 2018-01-17 14:01:57 +00:00
Richard van der Hoff fb6563b4be
Merge pull request #2793 from matrix-org/rav/db_txn_time_in_millis
Track db txn time in millisecs
2018-01-17 13:52:42 +00:00
Richard van der Hoff 936482d507 Fix 'NoneType' object has no attribute 'writeHeaders'
Avoid throwing a (harmless) exception when we try to write an error response to
an http request where the client has disconnected.

This comes up as a CRITICAL error in the logs which tends to mislead people
into thinking there's an actual problem
2018-01-16 17:58:16 +00:00
Richard van der Hoff 3d12d97415 Track DB scheduling delay per-request
For each request, track the amount of time spent waiting for a db
connection. This entails adding it to the LoggingContext and we may as well add
metrics for it while we are passing.
2018-01-16 17:23:32 +00:00
Richard van der Hoff 6324b65f08 Track db txn time in millisecs
... to reduce the amount of floating-point foo we do.
2018-01-16 15:53:18 +00:00
Richard van der Hoff 39f4e29d01 Reorganise request and block metrics
In order to circumvent the number of duplicate foo:count metrics increasing
without bounds, it's time for a rearrangement.

The following are all deprecated, and replaced with synapse_util_metrics_block_count:
  synapse_util_metrics_block_timer:count
  synapse_util_metrics_block_ru_utime:count
  synapse_util_metrics_block_ru_stime:count
  synapse_util_metrics_block_db_txn_count:count
  synapse_util_metrics_block_db_txn_duration:count

The following are all deprecated, and replaced with synapse_http_server_response_count:
   synapse_http_server_requests
   synapse_http_server_response_time:count
   synapse_http_server_response_ru_utime:count
   synapse_http_server_response_ru_stime:count
   synapse_http_server_response_db_txn_count:count
   synapse_http_server_response_db_txn_duration:count

The following are renamed (the old metrics are kept for now, but deprecated):

  synapse_util_metrics_block_timer:total ->
     synapse_util_metrics_block_time_seconds

  synapse_util_metrics_block_ru_utime:total ->
     synapse_util_metrics_block_ru_utime_seconds

  synapse_util_metrics_block_ru_stime:total ->
     synapse_util_metrics_block_ru_stime_seconds

  synapse_util_metrics_block_db_txn_count:total ->
     synapse_util_metrics_block_db_txn_count

  synapse_util_metrics_block_db_txn_duration:total ->
     synapse_util_metrics_block_db_txn_duration_seconds

  synapse_http_server_response_time:total ->
     synapse_http_server_response_time_seconds

  synapse_http_server_response_ru_utime:total ->
     synapse_http_server_response_ru_utime_seconds

  synapse_http_server_response_ru_stime:total ->
     synapse_http_server_response_ru_stime_seconds

   synapse_http_server_response_db_txn_count:total ->
      synapse_http_server_response_db_txn_count

   synapse_http_server_response_db_txn_duration:total
      synapse_http_server_response_db_txn_duration_seconds
2018-01-15 17:09:44 +00:00
Richard van der Hoff d90e8ea444 Update http request metrics before calling servlet
Make sure that we set the servlet name in the metrics object *before* calling
the servlet, in case the servlet throws an exception.
2018-01-09 18:27:35 +00:00
Richard van der Hoff df6c72ede3
Merge pull request #2711 from matrix-org/rav/fix_dns_errhandler
Fix error handling on dns lookup
2017-11-27 12:19:18 +00:00
Richard van der Hoff 6be01f599b Improve tracebacks on exceptions
Use failure.Failure to recover our failure, which will give us a useful
stacktrace, unlike the rethrown exception.
2017-11-27 12:05:58 +00:00
Richard van der Hoff 8b38096a89 Fix error handling on dns lookup
pass the right arguments to the errback handler

Fixes "TypeError('eb() takes exactly 2 arguments (1 given)',)"
2017-11-24 16:47:48 +00:00
Richard van der Hoff f90649eb2b Fix 500 on invalid utf-8 in request
If somebody sends us a request where the the body is invalid utf-8, we should
return a 400 rather than a 500. (json.loads throws a UnicodeError in this
situation)

We might as well catch all Exceptions here: it seems very unlikely that we
would get a request that *isn't caused by invalid json.
2017-11-10 09:15:39 +00:00
Richard van der Hoff fcdfc911ee Add a hook for custom rest endpoints
Let the user specify custom modules which can be used for implementing extra
endpoints.
2017-11-02 14:36:55 +00:00
Richard van der Hoff 8b56977b6f Merge pull request #2586 from matrix-org/rav/frontend_proxy_auth_header
Front-end proxy: pass through auth header
2017-10-27 11:01:50 +01:00
Richard van der Hoff 173567a7f2 Docstring for post_urlencoded_get_json 2017-10-27 10:59:50 +01:00
Richard van der Hoff 0d8e3ad48b Fix logcontext leaks in httpclient
`preserve_context_over_fn` is borked
2017-10-26 18:17:10 +01:00
Richard van der Hoff 12ef02dc3d SimpleHTTPClient: add support for headers
Sometimes we need to pass headers into these methods
2017-10-26 17:59:50 +01:00
Richard van der Hoff eaaabc6c4f replace 'except:' with 'except Exception:'
what could possibly go wrong
2017-10-23 15:52:32 +01:00
David Baker 27955056e0 Merge branch 'develop' into erikj/groups_merged 2017-10-02 16:20:41 +01:00
Richard van der Hoff e43de3ae4b Improve logging of failures in matrixfederationclient
* don't log exception types twice
* not all exceptions have a meaningful 'message'. Use the repr rather than
  attempting to build a string ourselves.
2017-09-28 15:38:09 +01:00
Richard van der Hoff 75e67b9ee4 Handle SERVFAILs when doing AAAA lookups for federation (#2477)
... to cope with people with broken dnssec setups, mostly
2017-09-28 15:24:00 +01:00
Richard van der Hoff f65e31d22f Do an AAAA lookup on SRV record targets (#2462)
Support SRV records which point at AAAA records, as well as A records.

Fixes https://github.com/matrix-org/synapse/issues/2405
2017-09-22 20:26:47 +01:00
Erik Johnston e884ff31d8 Add DELETE 2017-07-18 16:41:44 +01:00
Erik Johnston 05c13f6c22 Add 'args' param to post_json 2017-07-18 16:40:21 +01:00
Erik Johnston b8ca494ee9 Initial group server implementation 2017-07-10 15:44:15 +01:00
Krombel 1ff419d343 allow Authorization header which handling got implemented in #1098
Signed-off-by: Matthias Kesler <krombel@krombel.de>
2017-06-16 11:21:14 +02:00
Kegan Dougal d0debb2116 Remember how twisted works 2017-05-05 11:00:21 +01:00
Kegan Dougal c2ddd773bc Include the clock 2017-05-05 10:52:46 +01:00
Kegan Dougal 7dd3bf5e24 Rewrite SimpleHttpClient.request to include timeouts
Fixes #2191
2017-05-05 10:49:19 +01:00
David Baker 482a2ad122 No need for the exception variable 2017-05-03 11:02:59 +01:00
David Baker c0380402bc List caught expection types 2017-05-03 10:56:22 +01:00
David Baker 5fd12dce01 Remove debugging 2017-04-26 12:36:26 +01:00