1c262431f9
Fix handling of connection timeouts in outgoing http requests ( #8400 )
...
* Remove `on_timeout_cancel` from `timeout_deferred`
The `on_timeout_cancel` param to `timeout_deferred` wasn't always called on a
timeout (in particular if the canceller raised an exception), so it was
unreliable. It was also only used in one place, and to be honest it's easier to
do what it does a different way.
* Fix handling of connection timeouts in outgoing http requests
Turns out that if we get a timeout during connection, then a different
exception is raised, which wasn't always handled correctly.
To fix it, catch the exception in SimpleHttpClient and turn it into a
RequestTimedOutError (which is already a documented exception).
Also add a description to RequestTimedOutError so that we can see which stage
it failed at.
* Fix incorrect handling of timeouts reading federation responses
This was trapping the wrong sort of TimeoutError, so was never being hit.
The effect was relatively minor, but we should fix this so that it does the
expected thing.
* Fix inconsistent handling of `timeout` param between methods
`get_json`, `put_json` and `delete_json` were applying a different timeout to
the response body to `post_json`; bring them in line and test.
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Erik Johnston <erik@matrix.org>
2020-09-29 10:29:21 +01:00
11c9e17738
Add type annotations to SimpleHttpClient ( #8372 )
2020-09-24 15:47:20 +01:00
c619253db8
Stop sub-classing object ( #8249 )
2020-09-04 06:54:56 -04:00
eebf52be06
Be stricter about JSON that is accepted by Synapse ( #8106 )
2020-08-19 07:26:03 -04:00
a0f574f3c2
Reduce INFO logging ( #8050 )
...
c.f. #8021
A lot of the code here is to change the `Completed 200 OK` logging to include the request URI so that we can drop the `Sending request...` log line.
Some notes:
1. We won't log retries, which may be confusing considering the time taken log line includes retries and sleeps.
2. The `_send_request_with_optional_trailing_slash` will always be logged *without* the forward slash, even if it succeeded only with the forward slash.
2020-08-11 18:10:07 +01:00
88a3ff12f0
Convert the SimpleHttpClient to async. ( #8016 )
2020-08-04 07:22:04 -04:00
a53e0160a2
Ensure the msg property of HttpResponseException is a string. ( #7979 )
2020-07-29 13:56:06 -04:00
35450519de
Ensure that calls to `json.dumps` are compatible with the standard library json. ( #7836 )
2020-07-15 13:40:54 -04:00
f13061d515
Fix client reader sharding tests ( #7853 )
...
* Fix client reader sharding tests
* Newsfile
* Fix typing
* Update changelog.d/7853.misc
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Move mocking of http_client to tests
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2020-07-15 15:27:35 +01:00
66a4af8d96
Do not use canonicaljson to magically handle decoding bytes from JSON. ( #7802 )
2020-07-10 14:30:08 -04:00
a3f11567d9
Replace all remaining six usage with native Python 3 equivalents ( #7704 )
2020-06-16 08:51:47 -04:00
7cb8b4bc67
Allow configuration of Synapse's cache without using synctl or environment variables ( #6391 )
2020-05-11 18:45:23 +01:00
616af44137
Implement OpenID Connect-based login ( #7256 )
2020-05-08 08:30:40 -04:00
c37db0211e
Share SSL contexts for non-federation requests ( #7094 )
...
Extends #5794 etc to the SimpleHttpClient so that it also applies to non-federation requests.
Fixes #7092 .
2020-03-17 21:32:25 +00:00
1cb84c6486
Support for routing outbound HTTP requests via a proxy ( #6239 )
...
The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy.
The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`.
The proxy will then be used for
* push
* url previews
* phone-home stats
* recaptcha validation
* CAS auth validation
It will *not* be used for:
* Application Services
* Identity servers
* Outbound federation
* In worker configurations, connections from workers to masters
Fixes #4198 .
2019-11-01 14:07:44 +00:00
54fef094b3
Remove usage of deprecated logger.warn method from codebase ( #6271 )
...
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
2019-10-31 10:23:24 +00:00
f3451118a6
Edit SimpleHttpClient to reference that header keys can be passed as str or bytes ( #6077 )
2019-09-27 17:59:18 +01:00
f7c873a643
Trace how long it takes for the send trasaction to complete, including retrys ( #5986 )
2019-09-05 17:44:55 +01:00
36f34e6f3d
Remove unused methods from c/s api v1 in register.py ( #5963 )
...
These methods were part of the v1 C/S API. Remove them as they are no longer used by any code paths.
2019-09-02 18:29:21 +01:00
4806651744
Replace returnValue with return ( #5736 )
2019-07-23 23:00:55 +10:00
463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ ( #5606 )
2019-07-04 00:07:04 +10:00
32e7c9e7f2
Run Black. ( #5482 )
2019-06-20 19:32:02 +10:00
95d38afe96
Don't log exception when failing to fetch remote content.
...
In particular, let's not log stack traces when we stop processing
becuase the response body was too large.
2019-06-07 12:39:10 +01:00
5a4b328f52
Add ability to blacklist ip ranges for federation traffic ( #5043 )
2019-05-13 19:05:06 +01:00
2f48c4e1ae
URL preview blacklisting fixes ( #5155 )
...
Prevents a SynapseError being raised inside of a IResolutionReceiver and instead opts to just return 0 results. This thus means that we have to lump a failed lookup and a blacklisted lookup together with the same error message, but the substitute should be generic enough to cover both cases.
2019-05-10 10:32:44 -07:00
c88bc53903
Fix TaskStopped exceptions when outbound requests time out ( #4690 )
2019-02-20 11:35:52 +00:00
f4697b5ec1
Fix UnboundLocalError in post_urlencoded_get_json ( #4460 )
...
This could cause exceptions if the id server returned 4xx responses.
2019-01-24 13:38:29 +00:00
ea6abf6724
Fix IP URL previews on Python 3 ( #4215 )
2018-12-22 01:56:13 +11:00
a334e1cace
Update to use new timeout function everywhere.
...
The existing deferred timeout helper function (and the one into twisted)
suffer from a bug when a deferred's canceller throws an exception, #3842 .
The new helper function doesn't suffer from this problem.
2018-09-19 10:39:40 +01:00
bc9af88a2d
fix
2018-09-15 00:26:00 +10:00
7c27c4d51c
merge ( #3576 )
2018-09-14 03:11:11 +10:00
2d2828dcbc
Port http/ to Python 3 ( #3771 )
2018-09-06 00:10:47 +10:00
b37c472419
Rename async to async_helpers because `async` is a keyword on Python 3.7 ( #3678 )
2018-08-10 23:50:21 +10:00
01e93f48ed
Kill off MatrixCodeMessageException
...
This code brings the SimpleHttpClient into line with the
MatrixFederationHttpClient by having it raise HttpResponseExceptions when a
request fails (rather than trying to parse for matrix errors and maybe raising
MatrixCodeMessageException).
Then, whenever we were checking for MatrixCodeMessageException and turning them
into SynapseErrors, we now need to check for HttpResponseExceptions and call
to_synapse_error.
2018-08-01 16:02:46 +01:00
a97c845271
Move v1-only APIs into their own module & isolate deprecated ones ( #3460 )
2018-07-19 20:03:33 +10:00
49af402019
run isort
2018-07-09 16:09:20 +10:00
6350bf925e
Attempt to be more performant on PyPy ( #3462 )
2018-06-28 14:49:57 +01:00
77ac14b960
Pass around the reactor explicitly ( #3385 )
2018-06-22 09:37:10 +01:00
330432031b
redact_uri in two missed log paths
2018-06-06 10:25:48 +01:00
e6cbf47773
factor out uri redaction into a method on http
2018-06-05 18:31:40 +01:00
09503126df
Strip `access_token` from outgoing requests using existing regex
2018-06-02 23:25:13 +01:00
53cc2cde1f
cleanup
2018-05-22 17:32:57 -05:00
df9f72d9e5
replacing portions
2018-05-21 19:47:37 -05:00
4f2f5171b7
replace stringIO imports
2018-04-28 13:46:23 +02:00
9d2c1b8429
Backport deferred.addTimeout
...
Twisted 16.0 doesn't have addTimeout, so let's backport it.
2018-04-27 12:52:30 +01:00
1ea904b9f0
Use deferred.addTimeout instead of time_bound_deferred
...
This doesn't feel like a wheel we need to reinvent.
2018-04-23 00:53:18 +01:00
02ba118f81
Increase http conn pool size
2018-01-29 14:30:15 +00:00
2c8526cac7
Use a connection pool for the SimpleHttpClient
...
In particular I hope this will help the pusher, which makes many requests to
sygnal, and is currently negotiating SSL for each one.
2018-01-20 00:55:44 +00:00
8b56977b6f
Merge pull request #2586 from matrix-org/rav/frontend_proxy_auth_header
...
Front-end proxy: pass through auth header
2017-10-27 11:01:50 +01:00
173567a7f2
Docstring for post_urlencoded_get_json
2017-10-27 10:59:50 +01:00
0d8e3ad48b
Fix logcontext leaks in httpclient
...
`preserve_context_over_fn` is borked
2017-10-26 18:17:10 +01:00
12ef02dc3d
SimpleHTTPClient: add support for headers
...
Sometimes we need to pass headers into these methods
2017-10-26 17:59:50 +01:00
d0debb2116
Remember how twisted works
2017-05-05 11:00:21 +01:00
c2ddd773bc
Include the clock
2017-05-05 10:52:46 +01:00
7dd3bf5e24
Rewrite SimpleHttpClient.request to include timeouts
...
Fixes #2191
2017-05-05 10:49:19 +01:00
482a2ad122
No need for the exception variable
2017-05-03 11:02:59 +01:00
c0380402bc
List caught expection types
2017-05-03 10:56:22 +01:00
5fd12dce01
Remove debugging
2017-04-26 12:36:26 +01:00
c366276056
Fix get_json
2017-04-26 10:07:01 +01:00
1a9255c12e
Use CodeMessageException subclass instead
...
Parse json errors from get_json client methods and throw special
errors.
2017-04-25 19:30:55 +01:00
a46982cee9
Need the HTTP status code
2017-04-21 16:20:12 +01:00
70caf49914
Do the same for get_json
2017-04-21 16:09:03 +01:00
a1595cec78
Don't error for 3xx responses
2017-04-21 11:51:17 +01:00
a90a0f5c8a
Propagate errors sensibly from proxied IS requests
...
When we're proxying Matrix endpoints, parse out Matrix error
responses and turn them into SynapseErrors so they can be
propagated sensibly upstream.
2017-04-21 11:32:48 +01:00
0648e76979
Remove spurious newline
...
Apparently I just removed the spaces instead...
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-12 18:41:30 +01:00
d3bd94805f
Fixup for #1689 and #1690
...
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-12 16:32:47 +01:00
6e18805ac2
IPv6 support for client.py
...
This is an (untested) general sketch of how to use wrapClientTLS to implement TLS over IPv6, as well as faster connections over IPv4.
2016-12-11 11:10:32 +01:00
255c229f23
Work around TLS bug in twisted
...
Wrap up twisted's FileBodyProducer to work around
https://twistedmatrix.com/trac/ticket/8473 . Hopefully this fixes
https://matrix.org/jira/browse/SYN-700 .
2016-06-15 10:39:08 +01:00
eb79110beb
Clean up the blacklist/whitelist handling.
...
Always set the config key with an empty list, even if a list isn't specified.
This means that the codepaths are the same for both the empty list and
for a missing key. Since the behaviour is the same for both cases this
makes the code somewhat easier to reason about.
2016-05-16 13:03:59 +01:00
792def4928
add a url_preview_ip_range_whitelist config param so we can whitelist the matrix.org IP space
2016-05-01 12:44:24 +01:00
f505575f69
Make InsecureInterceptableContextFactory work with SpiderEndpoint
2016-04-19 16:08:14 +01:00
83b2f83da0
actually throw meaningful errors
2016-04-08 21:36:59 +01:00
dafef5a688
Add url_preview_enabled config option to turn on/off preview_url endpoint. defaults to off.
...
Add url_preview_ip_range_blacklist to let admins specify internal IP ranges that must not be spidered.
Add url_preview_url_blacklist to let admins specify URL patterns that must not be spidered.
Implement a custom SpiderEndpoint and associated support classes to implement url_preview_ip_range_blacklist
Add commentary and generally address PR feedback
2016-04-08 18:37:15 +01:00
8b98a7e8c3
pep8
2016-04-03 12:56:29 +01:00
7426c86eb8
add a persistent cache of URL lookups, and fix up the in-memory one to work
2016-04-03 00:31:57 +01:00
d1b154a10f
support gzip compression, and don't pass through error msgs
2016-04-02 03:06:39 +01:00
683e564815
handle spidered relative images correctly
2016-03-31 23:52:58 +01:00
0d3d7de6fc
sync in changes from matrixfederationclient
2016-03-31 12:42:27 +01:00
a8a5dd3b44
handle requests with missing content-length headers (e.g. YouTube)
2016-03-31 01:55:21 +01:00
721b2bfa85
implement redirects
2016-03-29 03:32:52 +01:00
64b4aead15
make it work
2016-03-29 03:13:25 +01:00
dd4287ca5d
make it build
2016-03-29 02:07:57 +01:00
d9d48aad2d
Merge branch 'develop' into matthew/preview_urls
2016-03-27 22:54:42 +01:00
9a3c80a348
pep8
2016-03-08 10:09:07 +00:00
7bcee4733a
Encode unicode objects given to post_urlencode* otherwise urllib.urlencode chokes.
2016-03-08 10:04:38 +00:00
7dd0c1730a
initial WIP of a tentative preview_url endpoint - incomplete, untested, experimental, etc. just putting it here for safekeeping for now
2016-01-24 18:47:27 -05:00
6c28ac260c
copyrights
2016-01-07 04:26:29 +00:00
b8dd5b1a2d
Verify third party ID server certificates
2015-10-16 14:54:54 +01:00
a80ef851f7
Fix previous merge to s/version_string/user_agent/
2015-10-10 12:35:39 +01:00
625e13bfde
Add get_raw method to SimpleHttpClient, use this in CAS auth rather than requests
2015-10-10 10:54:19 +01:00
492beb62a8
Use space not dash as delimiter
2015-10-06 09:53:33 -05:00
e0b466bcfd
Use space not dash as delimiter
2015-10-06 09:32:26 -05:00
b28c7da0a4
Preserve version string in user agent
2015-10-05 20:49:39 -05:00
8fc52bc56a
Allow synapse's useragent to be customized
...
This will allow me to write tests which verify which server made HTTP
requests in a federation context.
2015-10-02 17:13:51 -05:00
3bcbabc9fb
Rename context factory
...
Mjark is officially no fun.
2015-09-15 15:46:22 +01:00
6485f03d91
Fix random formatting
2015-09-09 13:05:00 +01:00
81a93ddcc8
Allow configuration to ignore invalid SSL certs
...
This will be useful for sytest, and sytest only, hence the aggressive
config key name.
2015-09-09 12:02:07 +01:00
78323ccdb3
Remove syutil dependency in favour of smaller single-purpose libraries
2015-08-24 16:17:38 +01:00
653533a3da
Fix log context when sending requests
2015-06-19 11:46:49 +01:00
83f119a84a
Log requests and responses sent via http.client
2015-06-15 13:14:12 +01:00
Richard van der Hoff