Commit Graph

234 Commits

Author SHA1 Message Date
Andrew Morgan cf301e37d8 Add workaround note 2019-03-13 11:14:43 +00:00
Andrew Morgan 0ea8582f8b Cleaner way of implementing trailing slashes 2019-03-12 14:11:11 +00:00
Andrew Morgan 64ff11019e Retry certain federation requests on 404 2019-03-08 18:22:47 +00:00
Richard van der Hoff c88bc53903
Fix TaskStopped exceptions when outbound requests time out (#4690) 2019-02-20 11:35:52 +00:00
Richard van der Hoff 5d27730a73
Move ClientTLSOptionsFactory init out of refresh_certificates (#4611)
It's nothing to do with refreshing the certificates. No idea why it was here.
2019-02-11 18:03:30 +00:00
Richard van der Hoff f0ba34f581 Fix noisy "twisted.internet.task.TaskStopped" errors in logs
Fixes #4003
2019-02-01 12:22:57 +00:00
Richard van der Hoff 8520bc3109
Fix Host header sent by MatrixFederationAgent (#4468)
Move the Host header logic down here so that (a) it is used if we reuse the
agent elsewhere, and (b) we can mess about with it with .well-known.
2019-01-25 12:38:16 +00:00
Richard van der Hoff 7871146667 Make MatrixFederationClient use MatrixFederationAgent
... instead of the matrix_federation_endpoint
2019-01-22 20:34:35 +00:00
Richard van der Hoff de6888e7ce Remove redundant WrappedConnection (#4409)
* Remove redundant WrappedConnection

The matrix federation client uses an HTTP connection pool, which times out its
idle HTTP connections, so there is no need for any of this business.
2019-01-18 23:07:38 +11:00
Richard van der Hoff 9feb5d0b71
sign_request -> build_auth_headers (#4408)
Just got very confused about the fact that the headers are only an output, not
an input.
2019-01-17 12:40:09 +00:00
Erik Johnston 34ea14139d Fixup docstrings for matrixfederationclient 2019-01-09 09:25:59 +00:00
Erik Johnston d422570e9b Use RequestSendFailed when fail to parse content type headers 2019-01-08 14:22:18 +00:00
Erik Johnston b970cb0e96 Refactor request sending to have better excpetions (#4358)
* Correctly retry and back off if we get a HTTPerror response

* Refactor request sending to have better excpetions

MatrixFederationHttpClient blindly reraised exceptions to the caller
without differentiating "expected" failures (e.g. connection timeouts
etc) versus more severe problems (e.g. programming errors).

This commit adds a RequestSendFailed exception that is raised when
"expected" failures happen, allowing the TransactionQueue to log them as
warnings while allowing us to log other exceptions as actual exceptions.
2019-01-08 11:04:28 +00:00
Travis Ralston 43c3f0b02f
Merge pull request #3969 from turt2live/travis/fix-federated-group-requests
Handle HttpResponseException more safely for federated groups
2018-10-23 10:41:04 -06:00
Richard van der Hoff b8a5b0097c
Various cleanups in the federation client code (#4031)
- Improve logging: log things in the right order, include destination and txids
  in all log lines, don't log successful responses twice

- Fix the docstring on TransportLayerClient.send_transaction

- Don't use treq.request, which is overcomplicated for our purposes: just use a
  twisted.web.client.Agent.

- simplify the logic for setting up the bodyProducer

- fix bytes/str confusions
2018-10-16 10:44:49 +01:00
Travis Ralston 219606a6ed Fix exception documentation in matrixfederationclient.py 2018-09-26 13:26:27 -06:00
Erik Johnston a334e1cace Update to use new timeout function everywhere.
The existing deferred timeout helper function (and the one into twisted)
suffer from a bug when a deferred's canceller throws an exception, #3842.

The new helper function doesn't suffer from this problem.
2018-09-19 10:39:40 +01:00
Travis Ralston 35aec19f0a Destination is a string 2018-09-18 15:29:30 -06:00
Richard van der Hoff 31c15dcb80
Refactor matrixfederationclient to fix logging (#3906)
We want to wait until we have read the response body before we log the request
as complete, otherwise a confusing thing happens where the request appears to
have completed, but we later fail it.

To do this, we factor the salient details of a request out to a separate
object, which can then keep track of the txn_id, so that it can be logged.
2018-09-18 18:17:15 +01:00
Richard van der Hoff f00a9d2636 Fix some b'abcd' noise in logs and metrics
Python 3 compatibility: make sure that we decode some byte sequences before we
use them to create log lines and metrics labels.
2018-09-17 16:15:42 +01:00
Erik Johnston fcfe7a850d Add an awful secondary timeout to fix wedged requests
This is an attempt to mitigate #3842 by adding yet-another-timeout
2018-09-14 19:23:07 +01:00
Erik Johnston 3e6e94fe9f
Merge pull request #3872 from matrix-org/hawkowl/timeouts-2
timeouts 2: electric boogaloo
2018-09-14 16:58:44 +01:00
Erik Johnston 9e2f9a7b57 Measure outbound requests 2018-09-14 15:11:26 +01:00
Amber Brown 63755fa4c2 we do that higher up 2018-09-14 03:21:47 +10:00
Amber Brown 1c3f4d9ca5 buffer? 2018-09-14 03:09:13 +10:00
Amber Brown bfa0b759e0
Attempt to figure out what's going on with timeouts (#3857) 2018-09-14 00:15:51 +10:00
Erik Johnston 8c5b84441b Log outbound requests when we retry 2018-09-12 16:22:14 +01:00
Erik Johnston 4084a774a8 Timeout reading body for outbound HTTP requests 2018-09-12 10:10:20 +01:00
Amber Brown 2d2828dcbc
Port http/ to Python 3 (#3771) 2018-09-06 00:10:47 +10:00
Erik Johnston 1d67b13674 Fix bug when federation_domain_whitelist is an emtpy list
Outbound federation were incorrectly allowed when the config option was
set to an empty list
2018-08-24 12:13:12 +01:00
Amber Brown b37c472419
Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678) 2018-08-10 23:50:21 +10:00
Richard van der Hoff 3c0213a217
Merge pull request #3439 from vojeroen/send_sni_for_federation_requests
send SNI for federation requests
2018-08-10 12:23:54 +01:00
Richard van der Hoff f900d50824 include known room versions in outgoing make_joins 2018-08-06 13:45:37 +01:00
Jeroen 505530f36a Merge remote-tracking branch 'upstream/develop' into send_sni_for_federation_requests
# Conflicts:
#	synapse/crypto/context_factory.py
2018-07-14 20:24:46 +02:00
Jeroen b5e157d895 Merge branch 'develop' into send_sni_for_federation_requests
# Conflicts:
#	synapse/http/endpoint.py
2018-07-09 08:51:11 +02:00
Amber Brown 49af402019 run isort 2018-07-09 16:09:20 +10:00
Amber Brown 6350bf925e
Attempt to be more performant on PyPy (#3462) 2018-06-28 14:49:57 +01:00
Jeroen 07b4f88de9 formatting changes for pep8 2018-06-25 12:31:16 +02:00
Jeroen 3d605853c8 send SNI for federation requests 2018-06-24 22:38:43 +02:00
Amber Brown 77ac14b960
Pass around the reactor explicitly (#3385) 2018-06-22 09:37:10 +01:00
David Baker 187a546bff
Merge pull request #3276 from matrix-org/dbkr/unbind
Remove email addresses / phone numbers from ID servers when they're removed from synapse
2018-06-11 16:02:00 +01:00
David Baker bf54c1cf6c pep8 2018-06-06 10:15:33 +01:00
David Baker 3e4bc4488c More doc fixes 2018-06-06 09:44:10 +01:00
David Baker 607bd27c83 fix pep8 2018-06-05 18:10:35 +01:00
David Baker d62162bbec doc fixes 2018-06-05 18:09:13 +01:00
David Baker c5930d513a Docstring 2018-06-04 12:05:58 +01:00
Amber Brown 754826a830 Merge remote-tracking branch 'origin/develop' into 3218-official-prom 2018-05-28 18:57:23 +10:00
David Baker 77a23e2e05 Merge remote-tracking branch 'origin/develop' into dbkr/unbind 2018-05-24 16:20:53 +01:00
David Baker 2c7866d664 Hit the 3pid unbind endpoint on deactivation 2018-05-23 14:38:56 +01:00
Amber Brown 53cc2cde1f cleanup 2018-05-22 17:32:57 -05:00
Amber Brown df9f72d9e5 replacing portions 2018-05-21 19:47:37 -05:00
Adrian Tschira d9fe2b2d9d Replace some more comparisons with six
plus a bonus b"" string I missed last time

Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:56:31 +02:00
Richard van der Hoff 9e2601f830
Merge pull request #3108 from NotAFile/py3-six-urlparse
Use six.moves.urlparse
2018-04-30 00:33:05 +01:00
Richard van der Hoff 9d2c1b8429 Backport deferred.addTimeout
Twisted 16.0 doesn't have addTimeout, so let's backport it.
2018-04-27 12:52:30 +01:00
Richard van der Hoff 1ea904b9f0 Use deferred.addTimeout instead of time_bound_deferred
This doesn't feel like a wheel we need to reinvent.
2018-04-23 00:53:18 +01:00
Adrian Tschira 2a3c33ff03 Use six.moves.urlparse
The imports were shuffled around a bunch in py3

Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 21:22:43 +02:00
Krombel 1d71f484d4 use PUT instead of POST for federating groups/m.join_policy 2018-04-06 12:54:09 +02:00
Matthew Hodgson ab9f844aaf
Add federation_domain_whitelist option (#2820)
Add federation_domain_whitelist

gives a way to restrict which domains your HS is allowed to federate with.
useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network
2018-01-22 19:11:18 +01:00
Richard van der Hoff eaaabc6c4f replace 'except:' with 'except Exception:'
what could possibly go wrong
2017-10-23 15:52:32 +01:00
David Baker 27955056e0 Merge branch 'develop' into erikj/groups_merged 2017-10-02 16:20:41 +01:00
Richard van der Hoff e43de3ae4b Improve logging of failures in matrixfederationclient
* don't log exception types twice
* not all exceptions have a meaningful 'message'. Use the repr rather than
  attempting to build a string ourselves.
2017-09-28 15:38:09 +01:00
Erik Johnston e884ff31d8 Add DELETE 2017-07-18 16:41:44 +01:00
Erik Johnston 05c13f6c22 Add 'args' param to post_json 2017-07-18 16:40:21 +01:00
Richard van der Hoff 4903ccf159 Fix some lies, and other clarifications, in docstrings
The documentation on get_json has been wrong ever since the very first commit
to synapse...
2017-04-21 01:31:09 +01:00
Richard van der Hoff a380f041c2 try not to drop context after federation requests
preserve_context_over_fn uses a ContextPreservingDeferred, which only restores
context for the duration of its callbacks, which isn't really correct, and
means that subsequent operations in the same request can end up without their
logcontexts.
2017-03-23 22:36:21 +00:00
Richard van der Hoff 5a16cb4bf0 Ignore backoff history for invites, aliases, and roomdirs
Add a param to the federation client which lets us ignore historical backoff
data for federation queries, and set it for a handful of operations.
2017-03-23 12:23:22 +00:00
Richard van der Hoff 4bd597d9fc push federation retry limiter down to matrixfederationclient
rather than having to instrument everywhere we make a federation call,
make the MatrixFederationHttpClient manage the retry limiter.
2017-03-23 09:28:46 +00:00
Richard van der Hoff ad8a26e361 MatrixFederationHttpClient: clean up
rename _create_request to _request, and push ascii-encoding of `destination`
and `path` down into it
2017-03-23 00:27:04 +00:00
Richard van der Hoff 170ccc9de5 Fix routing loop when fetching remote media
When we proxy a media request to a remote server, add a query-param, which will
tell the remote server to 404 if it doesn't recognise the server_name.

This should fix a routing loop where the server keeps forwarding back to
itself.

Also improves the error handling on remote media fetches, so that we don't
always return a rather obscure 502.
2017-03-13 16:30:36 +00:00
Erik Johnston b4bc6fef5b Respect long_retries param and default to off 2016-12-29 00:58:34 +00:00
Erik Johnston b7336ff32d Clean up 2016-12-29 00:09:33 +00:00
Erik Johnston 5b6672c66d Wrap connections in an N minute timeout to ensure they get reaped correctly 2016-12-29 00:06:53 +00:00
Richard van der Hoff b5b3a7e867 More intelligent Content-Type parsing
Content-Type is allowed to contain options (`; charset=utf-8`, for
instance). We should allow that.
2016-11-30 15:07:32 +00:00
Mark Haines 949c2c5435 Add a timeout parameter for end2end key queries.
Add a timeout parameter for controlling how long synapse will wait
for responses from remote servers. For servers that fail include how
they failed to make it easier to debug.

Fetch keys from different servers in parallel rather than in series.

Set the default timeout to 10s.
2016-09-12 18:17:09 +01:00
Erik Johnston ca8abfbf30 Clean up TransactionQueue 2016-08-10 16:24:16 +01:00
Daniel Wagner-Hall d83d004ccd Fix flake8 warnings for new flake8 2016-02-02 17:18:50 +00:00
Matthew Hodgson 6c28ac260c copyrights 2016-01-07 04:26:29 +00:00
Erik Johnston 0eabfa55f6 Fix typo 2015-11-20 17:17:58 +00:00
Erik Johnston 6408541075 Don't limit connections to perspective servers 2015-11-20 17:15:44 +00:00
Erik Johnston 2f2bbb4d06 Use min and not max to set an upper bound on retry interval 2015-11-20 09:34:58 +00:00
Erik Johnston cbf3cd6151 Add comment 2015-11-17 18:29:29 +00:00
Erik Johnston cf4ef5f3c7 Only retry federation requests for a long time for background requests 2015-11-17 18:26:50 +00:00
Erik Johnston afdfd12bdf Implement required method 'resumeProducing' 2015-11-17 16:57:06 +00:00
Erik Johnston bceec65913 Slightly more aggressive retry timers at HTTP level 2015-11-17 15:10:05 +00:00
Erik Johnston eacb068ac2 Retry dead servers a lot less often 2015-11-02 16:56:30 +00:00
Daniel Wagner-Hall 81a93ddcc8 Allow configuration to ignore invalid SSL certs
This will be useful for sytest, and sytest only, hence the aggressive
config key name.
2015-09-09 12:02:07 +01:00
Mark Haines 78323ccdb3 Remove syutil dependency in favour of smaller single-purpose libraries 2015-08-24 16:17:38 +01:00
Erik Johnston adbd720fab PEP8 2015-08-13 11:47:38 +01:00
Mark Haines 998a72d4d9 Merge branch 'develop' into markjh/twisted-15
Conflicts:
	synapse/http/matrixfederationclient.py
2015-08-12 17:21:14 +01:00
Erik Johnston 653533a3da Fix log context when sending requests 2015-06-19 11:46:49 +01:00
Erik Johnston 18968efa0a Remove stale debug lines 2015-06-19 10:18:02 +01:00
Erik Johnston 9d112f4440 Add IDs to outbound transactions 2015-06-19 10:13:03 +01:00
Mark Haines 90abdaf3bc Use Twisted-15.2.1, Use Agent.usingEndpointFactory rather than implement our own Agent 2015-06-01 10:51:50 +01:00
Erik Johnston 79e37a7ecb Correctly pass connection pool parameter 2015-05-28 16:48:53 +01:00
Erik Johnston 11f51e6ded Up maxPersistentPerHost count 2015-05-28 15:45:46 +01:00
Erik Johnston 291e942332 Use connection pool for federation connections 2015-05-28 15:43:21 +01:00
Erik Johnston 284f55a7fb Add doc strings 2015-05-22 15:18:04 +01:00
Erik Johnston 5b1631a4a9 Add a timeout param to get_event 2015-05-19 14:53:32 +01:00
Erik Johnston 476899295f Change the way we do logging contexts so that they survive divergences 2015-05-08 16:32:18 +01:00
Paul "LeoNerd" Evans 128cf2daf7 Appease pep8 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans b0cf867319 Use _ instead of . as a metric namespacing separator, for Prometheus 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans f9478e475b Rename Metrics' "keys" to "labels" 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans 7d72e44eb9 Add vector counters to HTTP clients and servers; count the requests by method and responses by method and response code 2015-03-12 16:24:50 +00:00
Erik Johnston 7025781df8 Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.7.1 2015-02-18 17:37:43 +00:00
Erik Johnston 5358966a87 Use git aware version string in User-Agent and Server headers 2015-02-18 16:52:04 +00:00
Erik Johnston 7e9d59f3b4 Don't convert DNSLookupError to a 4xx SynapseError 2015-02-18 10:58:13 +00:00
Erik Johnston 65ca713ff5 Add .__name__ after type(e) 2015-02-18 10:51:32 +00:00
Erik Johnston 5e24471469 Fix up ResponseNeverReceived to str 2015-02-18 10:50:10 +00:00
Erik Johnston 0db52d43fa strings.join() expects iterable of strings 2015-02-18 10:41:46 +00:00
Erik Johnston 0647e27a41 Remove unused import 2015-02-17 15:19:54 +00:00
Erik Johnston 4de93001bf Make matrixfederationclient log more nicely 2015-02-17 15:12:06 +00:00
Erik Johnston c52e8d395b Merge pull request #61 from matrix-org/timeout-federation-requests
Timeout federation requests
2015-02-11 17:10:33 +00:00
Erik Johnston 4ebbaf0d43 Blunty replace json with simplejson 2015-02-11 14:23:10 +00:00
Erik Johnston fb233dc40b Merge branch 'develop' of github.com:matrix-org/synapse into timeout-federation-requests 2015-02-11 10:33:19 +00:00
Erik Johnston 05b961d7e3 PEP8 2015-02-11 10:28:46 +00:00
Erik Johnston dcf52469e8 Move time_bound_deferred into Clock 2015-02-11 10:25:06 +00:00
Erik Johnston 0e6b3e4e40 Time out HTTP federation requests 2015-02-10 18:17:27 +00:00
Erik Johnston 9f2573eea1 Return body of response in HttpResponseException 2015-02-06 10:55:01 +00:00
Erik Johnston ae46f10fc5 Apply sanity to the transport client interface. Convert 'make_join' and 'send_join' to accept iterables of destinations 2015-02-04 16:28:12 +00:00
Erik Johnston 5a3a15f5c1 Make post_json(...) actually send data. 2015-01-29 13:58:22 +00:00
Erik Johnston c183cec8f6 Add post_json(...) method to federation client 2015-01-29 13:44:52 +00:00
Mark Haines 5fed042640 Finish renaming "context" to "room_id" in federation codebase 2015-01-16 19:01:03 +00:00
Mark Haines adb04b1e57 Update copyright notices 2015-01-06 13:21:39 +00:00
Mark Haines 24b5d01853 Include version in User-Agent and Server headers 2014-12-22 10:16:02 +00:00
Mark Haines d80d505b1f Limit the size of images that are thumbnailed serverside. Limit the size of file that a server will download from a remote server 2014-12-11 14:19:32 +00:00
Mark Haines 4f37c0ea9d Merge branch 'develop' into media_repository 2014-12-10 16:55:06 +00:00
Matthew Hodgson 0f4dcab238 turn back on per-request transaction retries, so that every time we try to hit a dead server we actually end up hammering 5 times :| 2014-12-10 10:28:27 +00:00
Erik Johnston b8d30899b1 Code style. 2014-12-10 10:16:09 +00:00
Matthew Hodgson faf12b64f8 add errbacks to enqueue_pdu deferreds; change logging for failed federation sends to warn rather than exception 2014-12-10 00:12:51 +00:00
Matthew Hodgson 2b1acb7671 squidge to 79 columns as per pep8 2014-12-10 00:03:55 +00:00
Matthew Hodgson aed62a3583 track replication destination health, and perform exponential back-off when sending transactions. does *not* yet retry transactions, but drops them on the floor if waiting for a server to recover. 2014-12-07 02:26:07 +00:00
Mark Haines c01fd5573c Implement download support for media_repository 2014-12-04 14:22:31 +00:00
Mark Haines 610c2ea131 Fix pep8 and pyflakes warnings 2014-11-20 18:00:10 +00:00
David Baker e377d33652 Separate out the matrix http client completely because just about all of its code it now separate from the simple case we need for standard HTTP(S) 2014-11-20 17:41:56 +00:00