Merge remote-tracking branch 'origin/develop' into erikj/destination_retry_cache

This commit is contained in:
Richard van der Hoff 2018-09-25 12:03:54 +01:00
commit a1cd37390f
51 changed files with 128 additions and 62 deletions

View File

@ -1,3 +1,77 @@
Synapse 0.33.5.1 (2018-09-25)
=============================
Internal Changes
----------------
- Fix incompatibility with older Twisted version in tests. Thanks
@OlegGirko! ([\#3940](https://github.com/matrix-org/synapse/issues/3940))
Synapse 0.33.5 (2018-09-24)
===========================
No significant changes.
Synapse 0.33.5rc1 (2018-09-17)
==============================
Features
--------
- Python 3.5 and 3.6 support is now in beta. ([\#3576](https://github.com/matrix-org/synapse/issues/3576))
- Implement `event_format` filter param in `/sync` ([\#3790](https://github.com/matrix-org/synapse/issues/3790))
- Add synapse_admin_mau:registered_reserved_users metric to expose number of real reaserved users ([\#3846](https://github.com/matrix-org/synapse/issues/3846))
Bugfixes
--------
- Remove connection ID for replication prometheus metrics, as it creates a large number of new series. ([\#3788](https://github.com/matrix-org/synapse/issues/3788))
- guest users should not be part of mau total ([\#3800](https://github.com/matrix-org/synapse/issues/3800))
- Bump dependency on pyopenssl 16.x, to avoid incompatibility with recent Twisted. ([\#3804](https://github.com/matrix-org/synapse/issues/3804))
- Fix existing room tags not coming down sync when joining a room ([\#3810](https://github.com/matrix-org/synapse/issues/3810))
- Fix jwt import check ([\#3824](https://github.com/matrix-org/synapse/issues/3824))
- fix VOIP crashes under Python 3 (#3821) ([\#3835](https://github.com/matrix-org/synapse/issues/3835))
- Fix manhole so that it works with latest openssh clients ([\#3841](https://github.com/matrix-org/synapse/issues/3841))
- Fix outbound requests occasionally wedging, which can result in federation breaking between servers. ([\#3845](https://github.com/matrix-org/synapse/issues/3845))
- Show heroes if room name/canonical alias has been deleted ([\#3851](https://github.com/matrix-org/synapse/issues/3851))
- Fix handling of redacted events from federation ([\#3859](https://github.com/matrix-org/synapse/issues/3859))
- ([\#3874](https://github.com/matrix-org/synapse/issues/3874))
- Mitigate outbound federation randomly becoming wedged ([\#3875](https://github.com/matrix-org/synapse/issues/3875))
Internal Changes
----------------
- CircleCI tests now run on the potential merge of a PR. ([\#3704](https://github.com/matrix-org/synapse/issues/3704))
- http/ is now ported to Python 3. ([\#3771](https://github.com/matrix-org/synapse/issues/3771))
- Improve human readable error messages for threepid registration/account update ([\#3789](https://github.com/matrix-org/synapse/issues/3789))
- Make /sync slightly faster by avoiding needless copies ([\#3795](https://github.com/matrix-org/synapse/issues/3795))
- handlers/ is now ported to Python 3. ([\#3803](https://github.com/matrix-org/synapse/issues/3803))
- Limit the number of PDUs/EDUs per federation transaction ([\#3805](https://github.com/matrix-org/synapse/issues/3805))
- Only start postgres instance for postgres tests on Travis CI ([\#3806](https://github.com/matrix-org/synapse/issues/3806))
- tests/ is now ported to Python 3. ([\#3808](https://github.com/matrix-org/synapse/issues/3808))
- crypto/ is now ported to Python 3. ([\#3822](https://github.com/matrix-org/synapse/issues/3822))
- rest/ is now ported to Python 3. ([\#3823](https://github.com/matrix-org/synapse/issues/3823))
- add some logging for the keyring queue ([\#3826](https://github.com/matrix-org/synapse/issues/3826))
- speed up lazy loading by 2-3x ([\#3827](https://github.com/matrix-org/synapse/issues/3827))
- Improved Dockerfile to remove build requirements after building reducing the image size. ([\#3834](https://github.com/matrix-org/synapse/issues/3834))
- Disable lazy loading for incremental syncs for now ([\#3840](https://github.com/matrix-org/synapse/issues/3840))
- federation/ is now ported to Python 3. ([\#3847](https://github.com/matrix-org/synapse/issues/3847))
- Log when we retry outbound requests ([\#3853](https://github.com/matrix-org/synapse/issues/3853))
- Removed some excess logging messages. ([\#3855](https://github.com/matrix-org/synapse/issues/3855))
- Speed up purge history for rooms that have been previously purged ([\#3856](https://github.com/matrix-org/synapse/issues/3856))
- Refactor some HTTP timeout code. ([\#3857](https://github.com/matrix-org/synapse/issues/3857))
- Fix running merged builds on CircleCI ([\#3858](https://github.com/matrix-org/synapse/issues/3858))
- Fix typo in replication stream exception. ([\#3860](https://github.com/matrix-org/synapse/issues/3860))
- Add in flight real time metrics for Measure blocks ([\#3871](https://github.com/matrix-org/synapse/issues/3871))
- Disable buffering and automatic retrying in treq requests to prevent timeouts. ([\#3872](https://github.com/matrix-org/synapse/issues/3872))
- mention jemalloc in the README ([\#3877](https://github.com/matrix-org/synapse/issues/3877))
- Remove unmaintained "nuke-room-from-db.sh" script ([\#3888](https://github.com/matrix-org/synapse/issues/3888))
Synapse 0.33.4 (2018-09-07) Synapse 0.33.4 (2018-09-07)
=========================== ===========================

View File

@ -1 +0,0 @@
Python 3.5+ is now supported.

1
changelog.d/3578.bugfix Normal file
View File

@ -0,0 +1 @@
Fix problem when playing media from Chrome using direct URL (thanks @remjey!)

View File

@ -1 +0,0 @@
CircleCI tests now run on the potential merge of a PR.

View File

@ -1 +0,0 @@
http/ is now ported to Python 3.

View File

@ -1 +0,0 @@
Remove connection ID for replication prometheus metrics, as it creates a large number of new series.

View File

@ -1 +0,0 @@
Improve human readable error messages for threepid registration/account update

View File

@ -1 +0,0 @@
Implement `event_format` filter param in `/sync`

View File

@ -1 +0,0 @@
Make /sync slightly faster by avoiding needless copies

View File

@ -1 +0,0 @@
guest users should not be part of mau total

View File

@ -1 +0,0 @@
handlers/ is now ported to Python 3.

View File

@ -1 +0,0 @@
Bump dependency on pyopenssl 16.x, to avoid incompatibility with recent Twisted.

View File

@ -1 +0,0 @@
Limit the number of PDUs/EDUs per federation transaction

View File

@ -1 +0,0 @@
Only start postgres instance for postgres tests on Travis CI

View File

@ -1 +0,0 @@
tests/ is now ported to Python 3.

View File

@ -1 +0,0 @@
Fix existing room tags not coming down sync when joining a room

View File

@ -1 +0,0 @@
crypto/ is now ported to Python 3.

View File

@ -1 +0,0 @@
rest/ is now ported to Python 3.

View File

@ -1 +0,0 @@
Fix jwt import check

View File

@ -1 +0,0 @@
add some logging for the keyring queue

View File

@ -1 +0,0 @@
speed up lazy loading by 2-3x

View File

@ -1 +0,0 @@
Improved Dockerfile to remove build requirements after building reducing the image size.

View File

@ -1 +0,0 @@
fix VOIP crashes under Python 3 (#3821)

View File

@ -1 +0,0 @@
Disable lazy loading for incremental syncs for now

View File

@ -1 +0,0 @@
Fix manhole so that it works with latest openssh clients

View File

@ -1 +0,0 @@
Fix outbound requests occasionally wedging, which can result in federation breaking between servers.

View File

@ -1 +0,0 @@
Add synapse_admin_mau:registered_reserved_users metric to expose number of real reaserved users

View File

@ -1 +0,0 @@
federation/ is now ported to Python 3.

View File

@ -1 +0,0 @@
Show heroes if room name/canonical alias has been deleted

View File

@ -1 +0,0 @@
Log when we retry outbound requests

View File

@ -1 +0,0 @@
Removed some excess logging messages.

View File

@ -1 +0,0 @@
Speed up purge history for rooms that have been previously purged

View File

@ -1 +0,0 @@
Refactor some HTTP timeout code.

View File

@ -1 +0,0 @@
Fix running merged builds on CircleCI

View File

@ -1 +0,0 @@
Fix handling of redacted events from federation

View File

@ -1 +0,0 @@
Fix typo in replication stream exception.

View File

@ -1 +0,0 @@
Add in flight real time metrics for Measure blocks

View File

@ -1 +0,0 @@
Disable buffering and automatic retrying in treq requests to prevent timeouts.

View File

@ -1 +0,0 @@
Mitigate outbound federation randomly becoming wedged

View File

@ -1 +0,0 @@
mention jemalloc in the README

View File

@ -1 +0,0 @@
Remove unmaintained "nuke-room-from-db.sh" script

1
changelog.d/3925.misc Normal file
View File

@ -0,0 +1 @@
Fix spurious exceptions when remote http client closes conncetion

1
changelog.d/3936.bugfix Normal file
View File

@ -0,0 +1 @@
Fix out-of-bounds error when LLing yourself

1
changelog.d/3947.misc Normal file
View File

@ -0,0 +1 @@
Require attrs 16.0.0 or later

View File

@ -27,4 +27,4 @@ try:
except ImportError: except ImportError:
pass pass
__version__ = "0.33.4" __version__ = "0.33.5.1"

View File

@ -713,10 +713,6 @@ class SyncHandler(object):
) )
] ]
# always make sure we LL ourselves so we know we're in the room
# (if we are), to fix https://github.com/vector-im/riot-web/issues/7209
types.append((EventTypes.Member, sync_config.user.to_string()))
# only apply the filtering to room members # only apply the filtering to room members
filtered_types = [EventTypes.Member] filtered_types = [EventTypes.Member]
@ -726,6 +722,13 @@ class SyncHandler(object):
} }
if full_state: if full_state:
if lazy_load_members:
# always make sure we LL ourselves so we know we're in the room
# (if we are) to fix https://github.com/vector-im/riot-web/issues/7209
# We only need apply this on full state syncs given we disabled
# LL for incr syncs in #3840.
types.append((EventTypes.Member, sync_config.user.to_string()))
if batch: if batch:
current_state_ids = yield self.store.get_state_ids_for_event( current_state_ids = yield self.store.get_state_ids_for_event(
batch.events[-1].event_id, types=types, batch.events[-1].event_id, types=types,
@ -794,7 +797,7 @@ class SyncHandler(object):
else: else:
state_ids = {} state_ids = {}
if lazy_load_members: if lazy_load_members:
if types: if types and batch.events:
# We're returning an incremental sync, with no # We're returning an incremental sync, with no
# "gap" since the previous sync, so normally there would be # "gap" since the previous sync, so normally there would be
# no state to return. # no state to return.

View File

@ -84,10 +84,21 @@ def wrap_json_request_handler(h):
logger.info( logger.info(
"%s SynapseError: %s - %s", request, code, e.msg "%s SynapseError: %s - %s", request, code, e.msg
) )
respond_with_json(
request, code, e.error_dict(), send_cors=True, # Only respond with an error response if we haven't already started
pretty_print=_request_user_agent_is_curl(request), # writing, otherwise lets just kill the connection
) if request.startedWriting:
if request.transport:
try:
request.transport.abortConnection()
except Exception:
# abortConnection throws if the connection is already closed
pass
else:
respond_with_json(
request, code, e.error_dict(), send_cors=True,
pretty_print=_request_user_agent_is_curl(request),
)
except Exception: except Exception:
# failure.Failure() fishes the original Failure out # failure.Failure() fishes the original Failure out
@ -100,16 +111,26 @@ def wrap_json_request_handler(h):
request, request,
f.getTraceback().rstrip(), f.getTraceback().rstrip(),
) )
respond_with_json( # Only respond with an error response if we haven't already started
request, # writing, otherwise lets just kill the connection
500, if request.startedWriting:
{ if request.transport:
"error": "Internal server error", try:
"errcode": Codes.UNKNOWN, request.transport.abortConnection()
}, except Exception:
send_cors=True, # abortConnection throws if the connection is already closed
pretty_print=_request_user_agent_is_curl(request), pass
) else:
respond_with_json(
request,
500,
{
"error": "Internal server error",
"errcode": Codes.UNKNOWN,
},
send_cors=True,
pretty_print=_request_user_agent_is_curl(request),
)
return wrap_async_request_handler(wrapped_request_handler) return wrap_async_request_handler(wrapped_request_handler)

View File

@ -58,7 +58,9 @@ REQUIREMENTS = {
"phonenumbers>=8.2.0": ["phonenumbers"], "phonenumbers>=8.2.0": ["phonenumbers"],
"six": ["six"], "six": ["six"],
"prometheus_client": ["prometheus_client"], "prometheus_client": ["prometheus_client"],
"attrs": ["attr"],
# we use attr.s(slots), which arrived in 16.0.0
"attrs>=16.0.0": ["attr>=16.0.0"],
"netaddr>=0.7.18": ["netaddr"], "netaddr>=0.7.18": ["netaddr"],
} }

View File

@ -52,6 +52,7 @@ class DownloadResource(Resource):
b" script-src 'none';" b" script-src 'none';"
b" plugin-types application/pdf;" b" plugin-types application/pdf;"
b" style-src 'unsafe-inline';" b" style-src 'unsafe-inline';"
b" media-src 'self';"
b" object-src 'self';" b" object-src 'self';"
) )
server_name, media_id, name = parse_media_id(request) server_name, media_id, name = parse_media_id(request)

View File

@ -54,7 +54,7 @@ class FederationClientTests(HomeserverTestCase):
def test_client_never_connect(self): def test_client_never_connect(self):
""" """
If the HTTP request is not connected and is timed out, it'll give a If the HTTP request is not connected and is timed out, it'll give a
ConnectingCancelledError. ConnectingCancelledError or TimeoutError.
""" """
d = self.cl.get_json("testserv:8008", "foo/bar", timeout=10000) d = self.cl.get_json("testserv:8008", "foo/bar", timeout=10000)
@ -76,7 +76,7 @@ class FederationClientTests(HomeserverTestCase):
self.reactor.advance(10.5) self.reactor.advance(10.5)
f = self.failureResultOf(d) f = self.failureResultOf(d)
self.assertIsInstance(f.value, ConnectingCancelledError) self.assertIsInstance(f.value, (ConnectingCancelledError, TimeoutError))
def test_client_connect_no_response(self): def test_client_connect_no_response(self):
""" """