4a15a3e4d5
Include eventid in log lines when processing incoming federation transactions ( #3959 )
...
when processing incoming transactions, it can be hard to see what's going on,
because we process a bunch of stuff in parallel, and because we may end up
recursively working our way through a chain of three or four events.
This commit creates a way to use logcontexts to add the relevant event ids to
the log lines.
2018-09-27 11:25:34 +01:00
ae6ad4cf41
docstrings and unittests for storage.state ( #3958 )
...
I spent ages trying to figure out how I was going mad...
2018-09-27 11:22:25 +01:00
2c695fd1aa
Merge pull request #3963 from matrix-org/rav/get_state_for_room_docstring
...
fix docstring for FederationClient.get_state_for_room
2018-09-27 18:06:38 +10:00
82fa31799c
Remove debugging statement
2018-09-26 14:01:02 -06:00
2a7b3439de
Changelog
2018-09-26 13:51:34 -06:00
b4c3bc1734
Handle HttpResponseException more safely for federated groups
2018-09-26 13:48:04 -06:00
219606a6ed
Fix exception documentation in matrixfederationclient.py
2018-09-26 13:26:27 -06:00
e70b4ce069
Logging improvements
...
Some logging tweaks to help with debugging incoming federation transactions
2018-09-26 17:36:14 +01:00
3c37c7e45b
changelog
2018-09-26 17:01:30 +01:00
a87d419a85
Run notify_app_services as a bg process
...
This ensures that its resource usage metrics get recorded somewhere rather than
getting lost.
(It also fixes an error when called from a nested logging context which
completes before the bg process)
2018-09-26 17:00:40 +01:00
0c4a99ea2d
changelog
2018-09-26 16:54:54 +01:00
9453c65948
remove spurious federation checks on localhost
...
There's really no point in checking for destinations called "localhost" because
there is nothing stopping people creating other DNS entries which point to
127.0.0.1. The right fix for this is
https://github.com/matrix-org/synapse/issues/3953 .
Blocking localhost, on the other hand, means that you get a surprise when
trying to connect a test server on localhost to an existing server (with a
'normal' server_name).
2018-09-26 16:53:52 +01:00
ab59f3d8da
changelog
2018-09-26 16:52:24 +01:00
607eec0456
fix docstring for FederationClient.get_state_for_room
...
trivial fixes for docstring
2018-09-26 16:52:24 +01:00
d514608b5c
towncrier
2018-09-26 16:19:53 +01:00
28781b65e7
fix #3854
2018-09-26 16:16:41 +01:00
d4e0861ff9
Reduce the load on our CI ( #3957 )
...
* changelog
* reduce circleci config
* plus a handy script
* fix regex
2018-09-27 00:23:21 +10:00
a5e70b31a1
changelog
2018-09-26 14:41:14 +01:00
8afddf7afe
Fix error handling for missing auth_event
...
When we were authorizing an event, if there was no `m.room.create` in its
auth_events, we would raise a SynapseError with a cryptic message, which then
meant that we would bail out of processing any incoming events, rather than
storing a rejection for the faulty event and moving on.
We should treat the absent event the same as any other auth failure, by
raising an AuthError, so that the event is marked as rejected.
2018-09-26 14:40:16 +01:00
bf01efb864
Merge branch 'develop' into rav/hacky_cache_factor_fix
2018-09-26 13:24:07 +01:00
8834396b8a
Actuall set cache factors in workers
2018-09-26 13:23:02 +01:00
4e8276a34a
Merge pull request #3956 from matrix-org/rav/fix_expiring_cache_len
...
Fix ExpiringCache.__len__ to be accurate
2018-09-26 13:10:13 +01:00
5b4028fa78
Merge branch 'rav/fix_expiring_cache_len' into erikj/destination_retry_cache
2018-09-26 12:55:53 +01:00
7ee94fc1ba
Log which cache is throwing exceptions
2018-09-26 12:43:08 +01:00
66a1d57adb
Merge pull request #3948 from matrix-org/rav/no_symlink_synctl
...
Move synctl into top dir to avoid a symlink
2018-09-26 21:41:58 +10:00
c2185f14d7
Merge pull request #3924 from matrix-org/rav/clean_up_on_receive_pdu
...
Comments and interface cleanup for on_receive_pdu
2018-09-26 21:41:26 +10:00
8ac9fa7375
changelog
2018-09-26 12:35:10 +01:00
3baf6e1667
Fix ExpiringCache.__len__ to be accurate
...
It used to try and produce an estimate, which was sometimes negative.
This caused metrics to be sad, so lets always just calculate it from
scratch.
(This appears to have been a longstanding bug, but one which has been made more
of a problem by #3932 and #3933 ).
(This was originally done by Erik as part of #3933 . I'm cherry-picking it
because really it's a fix in its own right)
2018-09-26 12:32:29 +01:00
f65163627f
Merge pull request #3911 from matrix-org/jcgruenhage/docker-support-python3
...
make python 3 work in the docker container
2018-09-25 15:18:09 +01:00
df55a943ca
Update Dockerfile
2018-09-25 14:33:38 +02:00
e7fa032126
Update .dockerignore
2018-09-25 14:33:19 +02:00
0649306fde
Update grafana dashboard
2018-09-25 13:29:28 +01:00
a1cd37390f
Merge remote-tracking branch 'origin/develop' into erikj/destination_retry_cache
2018-09-25 12:03:54 +01:00
4c3e7eeec5
Merge pull request #3932 from matrix-org/erikj/auto_start_expiring_caches
...
Fix some instances of ExpiringCache not expiring cache items
2018-09-25 12:02:57 +01:00
6cf261930a
added "media-src: 'self'" to CSP for resources ( #3578 )
...
Synapse doesn’t allow for media resources to be played directly from
Chrome. It is a problem for users on other networks (e.g. IRC)
communicating with Matrix users through a gateway. The gateway sends
them the raw URL for the resource when a Matrix user uploads a video
and the video cannot be played directly in Chrome using that URL.
Chrome argues it is not authorized to play the video because of the
Content Security Policy. Chrome checks for the "media-src" policy which
is missing, and defauts to the "default-src" policy which is "none".
As Synapse already sends "object-src: 'self'" I thought it wouldn’t be
a problem to add "media-src: 'self'" to the CSP to fix this problem.
2018-09-25 11:55:02 +01:00
94f7befc31
Merge pull request #3925 from matrix-org/erikj/fix_producers_unregistered
...
Fix spurious exceptions when client closes conncetion
2018-09-25 11:52:06 +01:00
bd469adaa9
changelog
2018-09-25 11:22:17 +01:00
c53336986d
Move synctl into top dir to avoid a symlink
...
symlinks apparently break setuptools on python3 and alpine
(https://bugs.python.org/issue31940 ), so let's stop using a symlink and just
use the file directly.
2018-09-25 11:19:27 +01:00
e4e96486a9
Merge pull request #3947 from matrix-org/rav/attr_version
...
We require attrs 16.0.0
2018-09-25 11:16:42 +01:00
45a1053d44
changelog
2018-09-25 10:45:34 +01:00
a9d84f4e44
We require attrs 16.0.0
...
Ref: https://github.com/matrix-org/synapse/issues/3945
2018-09-25 10:43:39 +01:00
787d22ed6c
Only lazy load self-members on initial sync
...
Given we have disabled lazy loading for incr syncs in #3840 , we can make self-LL more efficient by only doing it on initial sync. Also adds a bounds check for if/when we change our mind, so that we don't try to include LL members on sync responses with no timeline.
2018-09-25 00:49:26 +01:00
fbe5ba25f6
Merge branch 'master' into develop
2018-09-25 03:10:01 +10:00
5121ae97f5
Internal Changes
...
----------------
- Fix incompatibility with older Twisted version in tests. Thanks
@OlegGirko!
([\#3940](https://github.com/matrix-org/synapse/issues/3940 ))
-----BEGIN PGP SIGNATURE-----
iQEzBAABCAAdFiEEIQBQJ4l+yK4dlKkFIwi0edOSShEFAlupGfUACgkQIwi0edOS
ShH/rgf5AdLpXHVmWN1YiKuwwgiK9i8/41pqcj5ZiVn+UQjzUIBNc0ERNf2gfoRU
iwECt3rDMatB2Xxd4VFNoTn6wFtuqNV4ke7jeghMU2KxAzmwX8F6b7qJWz9sywyO
7cOtByqVAKN2v8qLDI+wcGuBkxkFHMF8iKBQApkDn1gqvvqHs/yk9FprhLJOYvaz
iOvrgm4T3XLiMOqDmVH1byemynLQCagol/8rn63NnnDVaoHd9Cgar0qRMta2MYBL
xXjOJ2HS35Y7ZuIPhRAmOVhs6DXmjBnEgfxeDV+GAF1Co8qEH5Jbfq/vu/2gpKKf
TsA5HG3mDGBit67X9evU9SN5WFlcQw==
=CZ+X
-----END PGP SIGNATURE-----
Merge tag 'v0.33.5.1'
Internal Changes
----------------
- Fix incompatibility with older Twisted version in tests. Thanks
@OlegGirko!
([\#3940](https://github.com/matrix-org/synapse/issues/3940 ))
2018-09-25 03:09:30 +10:00
fc691ca97c
changelog
2018-09-25 02:54:59 +10:00
6b6cb32297
bump version
2018-09-25 02:54:34 +10:00
e37c221b97
changelog for 3940
2018-09-25 02:51:55 +10:00
7d3f639844
Fix compatibility issue with older Twisted in tests.
...
Older Twisted (18.4.0) returns TimeoutError instead of
ConnectingCancelledError when connection times out.
This change allows tests to be compatible with this behaviour.
Signed-off-by: Oleg Girko <ol@infoserver.lv>
2018-09-25 02:51:18 +10:00
04eed80a73
Merge branch 'master' into develop
2018-09-24 23:42:25 +10:00
829213523e
Features
...
--------
- Python 3.5 and 3.6 support is now in beta.
([\#3576](https://github.com/matrix-org/synapse/issues/3576 ))
- Implement `event_format` filter param in `/sync`
([\#3790](https://github.com/matrix-org/synapse/issues/3790 ))
- Add synapse_admin_mau:registered_reserved_users metric to expose
number of real reaserved users
([\#3846](https://github.com/matrix-org/synapse/issues/3846 ))
Bugfixes
--------
- Remove connection ID for replication prometheus metrics, as it creates
a large number of new series.
([\#3788](https://github.com/matrix-org/synapse/issues/3788 ))
- guest users should not be part of mau total
([\#3800](https://github.com/matrix-org/synapse/issues/3800 ))
- Bump dependency on pyopenssl 16.x, to avoid incompatibility with
recent Twisted.
([\#3804](https://github.com/matrix-org/synapse/issues/3804 ))
- Fix existing room tags not coming down sync when joining a room
([\#3810](https://github.com/matrix-org/synapse/issues/3810 ))
- Fix jwt import check
([\#3824](https://github.com/matrix-org/synapse/issues/3824 ))
- fix VOIP crashes under Python 3 (#3821 )
([\#3835](https://github.com/matrix-org/synapse/issues/3835 ))
- Fix manhole so that it works with latest openssh clients
([\#3841](https://github.com/matrix-org/synapse/issues/3841 ))
- Fix outbound requests occasionally wedging, which can result in
federation breaking between servers.
([\#3845](https://github.com/matrix-org/synapse/issues/3845 ))
- Show heroes if room name/canonical alias has been deleted
([\#3851](https://github.com/matrix-org/synapse/issues/3851 ))
- Fix handling of redacted events from federation
([\#3859](https://github.com/matrix-org/synapse/issues/3859 ))
- ([\#3874](https://github.com/matrix-org/synapse/issues/3874 ))
- Mitigate outbound federation randomly becoming wedged
([\#3875](https://github.com/matrix-org/synapse/issues/3875 ))
Internal Changes
----------------
- CircleCI tests now run on the potential merge of a PR.
([\#3704](https://github.com/matrix-org/synapse/issues/3704 ))
- http/ is now ported to Python 3.
([\#3771](https://github.com/matrix-org/synapse/issues/3771 ))
- Improve human readable error messages for threepid
registration/account update
([\#3789](https://github.com/matrix-org/synapse/issues/3789 ))
- Make /sync slightly faster by avoiding needless copies
([\#3795](https://github.com/matrix-org/synapse/issues/3795 ))
- handlers/ is now ported to Python 3.
([\#3803](https://github.com/matrix-org/synapse/issues/3803 ))
- Limit the number of PDUs/EDUs per federation transaction
([\#3805](https://github.com/matrix-org/synapse/issues/3805 ))
- Only start postgres instance for postgres tests on Travis CI
([\#3806](https://github.com/matrix-org/synapse/issues/3806 ))
- tests/ is now ported to Python 3.
([\#3808](https://github.com/matrix-org/synapse/issues/3808 ))
- crypto/ is now ported to Python 3.
([\#3822](https://github.com/matrix-org/synapse/issues/3822 ))
- rest/ is now ported to Python 3.
([\#3823](https://github.com/matrix-org/synapse/issues/3823 ))
- add some logging for the keyring queue
([\#3826](https://github.com/matrix-org/synapse/issues/3826 ))
- speed up lazy loading by 2-3x
([\#3827](https://github.com/matrix-org/synapse/issues/3827 ))
- Improved Dockerfile to remove build requirements after building
reducing the image size.
([\#3834](https://github.com/matrix-org/synapse/issues/3834 ))
- Disable lazy loading for incremental syncs for now
([\#3840](https://github.com/matrix-org/synapse/issues/3840 ))
- federation/ is now ported to Python 3.
([\#3847](https://github.com/matrix-org/synapse/issues/3847 ))
- Log when we retry outbound requests
([\#3853](https://github.com/matrix-org/synapse/issues/3853 ))
- Removed some excess logging messages.
([\#3855](https://github.com/matrix-org/synapse/issues/3855 ))
- Speed up purge history for rooms that have been previously purged
([\#3856](https://github.com/matrix-org/synapse/issues/3856 ))
- Refactor some HTTP timeout code.
([\#3857](https://github.com/matrix-org/synapse/issues/3857 ))
- Fix running merged builds on CircleCI
([\#3858](https://github.com/matrix-org/synapse/issues/3858 ))
- Fix typo in replication stream exception.
([\#3860](https://github.com/matrix-org/synapse/issues/3860 ))
- Add in flight real time metrics for Measure blocks
([\#3871](https://github.com/matrix-org/synapse/issues/3871 ))
- Disable buffering and automatic retrying in treq requests to prevent
timeouts. ([\#3872](https://github.com/matrix-org/synapse/issues/3872 ))
- mention jemalloc in the README
([\#3877](https://github.com/matrix-org/synapse/issues/3877 ))
- Remove unmaintained "nuke-room-from-db.sh" script
([\#3888](https://github.com/matrix-org/synapse/issues/3888 ))
-----BEGIN PGP SIGNATURE-----
iQEzBAABCAAdFiEEIQBQJ4l+yK4dlKkFIwi0edOSShEFAluo6WIACgkQIwi0edOS
ShHFIwf/ZURGbih+6q4jy8voc4wUbgtuNGabmFLQeXGhIXIOYltvKOrB8wcCEi/4
jODztPqewikZoI7NFA4M44GToQKacuIEOYK/M513l3t/mEpBQYDdvLGDccM+6XGc
RcrWqQEfJYkiNyWNmLx5TJ4bYaNelrpfUyNbbONph0wLO7o6f7ANoGyqqVKm/7pD
ufUwwBLqzd2U2vuvNCbQLSW0LkognP91vYytHMN5NTkoAR4BSYfDxQCShUSoNSlJ
098CEYjXZaDcY9ypGA6wlflbNfl3NC2KcTajLOCHdKe2U+FbEMc/9Ou690GbDRJ9
6L+Bonfwx0tsF04LHsteTdyTIOVp+Q==
=Dv5r
-----END PGP SIGNATURE-----
Merge tag 'v0.33.5'
Features
--------
- Python 3.5 and 3.6 support is now in beta.
([\#3576](https://github.com/matrix-org/synapse/issues/3576 ))
- Implement `event_format` filter param in `/sync`
([\#3790](https://github.com/matrix-org/synapse/issues/3790 ))
- Add synapse_admin_mau:registered_reserved_users metric to expose
number of real reaserved users
([\#3846](https://github.com/matrix-org/synapse/issues/3846 ))
Bugfixes
--------
- Remove connection ID for replication prometheus metrics, as it creates
a large number of new series.
([\#3788](https://github.com/matrix-org/synapse/issues/3788 ))
- guest users should not be part of mau total
([\#3800](https://github.com/matrix-org/synapse/issues/3800 ))
- Bump dependency on pyopenssl 16.x, to avoid incompatibility with
recent Twisted.
([\#3804](https://github.com/matrix-org/synapse/issues/3804 ))
- Fix existing room tags not coming down sync when joining a room
([\#3810](https://github.com/matrix-org/synapse/issues/3810 ))
- Fix jwt import check
([\#3824](https://github.com/matrix-org/synapse/issues/3824 ))
- fix VOIP crashes under Python 3 (#3821 )
([\#3835](https://github.com/matrix-org/synapse/issues/3835 ))
- Fix manhole so that it works with latest openssh clients
([\#3841](https://github.com/matrix-org/synapse/issues/3841 ))
- Fix outbound requests occasionally wedging, which can result in
federation breaking between servers.
([\#3845](https://github.com/matrix-org/synapse/issues/3845 ))
- Show heroes if room name/canonical alias has been deleted
([\#3851](https://github.com/matrix-org/synapse/issues/3851 ))
- Fix handling of redacted events from federation
([\#3859](https://github.com/matrix-org/synapse/issues/3859 ))
- ([\#3874](https://github.com/matrix-org/synapse/issues/3874 ))
- Mitigate outbound federation randomly becoming wedged
([\#3875](https://github.com/matrix-org/synapse/issues/3875 ))
Internal Changes
----------------
- CircleCI tests now run on the potential merge of a PR.
([\#3704](https://github.com/matrix-org/synapse/issues/3704 ))
- http/ is now ported to Python 3.
([\#3771](https://github.com/matrix-org/synapse/issues/3771 ))
- Improve human readable error messages for threepid
registration/account update
([\#3789](https://github.com/matrix-org/synapse/issues/3789 ))
- Make /sync slightly faster by avoiding needless copies
([\#3795](https://github.com/matrix-org/synapse/issues/3795 ))
- handlers/ is now ported to Python 3.
([\#3803](https://github.com/matrix-org/synapse/issues/3803 ))
- Limit the number of PDUs/EDUs per federation transaction
([\#3805](https://github.com/matrix-org/synapse/issues/3805 ))
- Only start postgres instance for postgres tests on Travis CI
([\#3806](https://github.com/matrix-org/synapse/issues/3806 ))
- tests/ is now ported to Python 3.
([\#3808](https://github.com/matrix-org/synapse/issues/3808 ))
- crypto/ is now ported to Python 3.
([\#3822](https://github.com/matrix-org/synapse/issues/3822 ))
- rest/ is now ported to Python 3.
([\#3823](https://github.com/matrix-org/synapse/issues/3823 ))
- add some logging for the keyring queue
([\#3826](https://github.com/matrix-org/synapse/issues/3826 ))
- speed up lazy loading by 2-3x
([\#3827](https://github.com/matrix-org/synapse/issues/3827 ))
- Improved Dockerfile to remove build requirements after building
reducing the image size.
([\#3834](https://github.com/matrix-org/synapse/issues/3834 ))
- Disable lazy loading for incremental syncs for now
([\#3840](https://github.com/matrix-org/synapse/issues/3840 ))
- federation/ is now ported to Python 3.
([\#3847](https://github.com/matrix-org/synapse/issues/3847 ))
- Log when we retry outbound requests
([\#3853](https://github.com/matrix-org/synapse/issues/3853 ))
- Removed some excess logging messages.
([\#3855](https://github.com/matrix-org/synapse/issues/3855 ))
- Speed up purge history for rooms that have been previously purged
([\#3856](https://github.com/matrix-org/synapse/issues/3856 ))
- Refactor some HTTP timeout code.
([\#3857](https://github.com/matrix-org/synapse/issues/3857 ))
- Fix running merged builds on CircleCI
([\#3858](https://github.com/matrix-org/synapse/issues/3858 ))
- Fix typo in replication stream exception.
([\#3860](https://github.com/matrix-org/synapse/issues/3860 ))
- Add in flight real time metrics for Measure blocks
([\#3871](https://github.com/matrix-org/synapse/issues/3871 ))
- Disable buffering and automatic retrying in treq requests to prevent
timeouts. ([\#3872](https://github.com/matrix-org/synapse/issues/3872 ))
- mention jemalloc in the README
([\#3877](https://github.com/matrix-org/synapse/issues/3877 ))
- Remove unmaintained "nuke-room-from-db.sh" script
([\#3888](https://github.com/matrix-org/synapse/issues/3888 ))
2018-09-24 23:41:35 +10:00
Richard van der Hoff