Mirrors
/
synapse-old
mirror of https://github.com/matrix-org/synapse.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
13,693 Commits 842 Branches 671 Tags 438 MiB
Commit Graph

9705 Commits

Author SHA1 Message Date
Erik Johnston 1371d5b798 Don't log stack traces for HTTP error responses 2019-01-08 12:28:30 +00:00
Richard van der Hoff ba85786d71 0.34.1rc1 2019-01-08 11:15:34 +00:00
Erik Johnston b970cb0e96 Refactor request sending to have better excpetions (#4358) 
* Correctly retry and back off if we get a HTTPerror response

* Refactor request sending to have better excpetions

MatrixFederationHttpClient blindly reraised exceptions to the caller
without differentiating "expected" failures (e.g. connection timeouts
etc) versus more severe problems (e.g. programming errors).

This commit adds a RequestSendFailed exception that is raised when
"expected" failures happen, allowing the TransactionQueue to log them as
warnings while allowing us to log other exceptions as actual exceptions.
 2019-01-08 11:04:28 +00:00
Amber Brown 1dcb086f33
Fix synapse.config.__main__ on python 3 (#4356) 2019-01-08 10:03:09 +11:00
Travis Ralston bc1fa8cd01 Add GET account data routes (#4303) 
As per https://github.com/matrix-org/matrix-doc/issues/1339
 2019-01-07 10:27:54 +00:00
Richard van der Hoff b7c0218812
Check jinja version for consent resource (#4327) 
* Raise a ConfigError if an invalid resource is specified

* Require Jinja 2.9 for the consent resource

* changelog
 2019-01-07 10:14:31 +00:00
Michael Telatynski a27e501b09 fix the check for whether `is_url` to match all the other ones in codebase (#3405) 
Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>
 2019-01-06 21:43:25 +00:00
Matthew Hodgson 27128145e6 fix NPE in /messages by checking if all events were filtered out (#4330) 2019-01-02 23:37:39 +00:00
Neil Johnson 84b6fae1f5 Ensure synchrotrons can access is_support_user in the storage layer 2019-01-02 10:19:59 +00:00
Amber Brown d7843f47b6
Remove v1 only REST APIs now we don't ship matrix console (#4334) 2018-12-29 23:12:30 +11:00
Will Hunt 0708f437cc Log roomid along with Unknown room (#4297) 2018-12-24 10:49:42 +01:00
Richard van der Hoff cd018e3543
Merge pull request #4307 from matrix-org/erikj/v2_rooms 
Add v2 room version
 2018-12-24 10:48:59 +01:00
Richard van der Hoff f85676cc93
Return well_known in /login response (#4319) 
... as per MSC1730.
 2018-12-24 10:44:33 +01:00
Richard van der Hoff 9c2af7b2c5 Add a script to generate a clean config file (#4315) 2018-12-22 02:04:57 +11:00
Richard van der Hoff f3561f8d86 Fix indentation in default config (#4313) 
These settings are not supposed to be under 'listeners'.
 2018-12-22 02:01:43 +11:00
Amber Brown ea6abf6724
Fix IP URL previews on Python 3 (#4215) 2018-12-22 01:56:13 +11:00
Amber Brown c26f49a664
Make the dependencies more like a standard Python project and hook up the optional dependencies to setuptools (#4298) 2018-12-22 01:37:26 +11:00
Richard van der Hoff 0b26feb422 Merge branch 'master' into develop 2018-12-20 16:48:47 +00:00
Richard van der Hoff 6dd7271243 Prepare 0.34 release 2018-12-20 12:07:15 +00:00
Richard van der Hoff d3c9c562c6 Debian packaging via dh_virtualenv 2018-12-20 11:35:23 +00:00
Richard van der Hoff 1a6d5bfa08 Debian packaging via dh_virtualenv (#4285) 2018-12-20 11:33:29 +00:00
David Baker 1c0051114a Add 'sandbox' to CSP for media repo (#4284) 
* Add 'sandbox' to the CSP for media repo

* Changelog
 2018-12-20 11:09:18 +00:00
Nathan Pennie 81b513416e
Fixed line length 2018-12-19 14:12:33 -05:00
Nathan Pennie 668e6625b0
Fix the variable names used for account_data 2018-12-19 06:53:02 -05:00
Erik Johnston df89f8afb8 Add v2 room version 2018-12-18 18:10:37 +00:00
Neil Johnson 7e22cd90f5
ensure can report mau stats when hs.config.mau_stats_only is set (#4305) 
* ensure can report mau stats when hs.config.mau_stats_only is set
 2018-12-18 14:36:11 +00:00
Neil Johnson d2f7c4e6b1
create support user (#4141) 
Allow for the creation of a support user.

A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.
 2018-12-14 18:20:59 +00:00
Nad Chishtie 128294d3e5 Improved welcome page (#4294) 
* Improved static/index.html look, feel and content.

* Added accessibility tags to in-line SVG.

* Grammar.
 2018-12-14 05:13:56 +11:00
Amber Brown c7144b105c Settings Fix deleting e2e room keys on xenial (#4295) 2018-12-14 05:13:33 +11:00
Amber Brown e93a0ebf50
Settings Fix deleting e2e room keys on xenial (#4295) 2018-12-14 05:10:31 +11:00
Nad Chishtie cef8ae272a Improved welcome page (#4294) 
* Improved static/index.html look, feel and content.

* Added accessibility tags to in-line SVG.

* Grammar.
 2018-12-14 02:42:51 +11:00
Richard van der Hoff f208f608cb Merge branch 'release-v0.34.0' into develop 2018-12-11 15:43:20 +00:00
Richard van der Hoff eb835bf65b 0.34.0rc2 2018-12-11 15:35:47 +00:00
Richard van der Hoff 188945713e
Merge pull request #4290 from matrix-org/rav/remove_webclient 
Stop installing Matrix Console by default
 2018-12-11 16:24:15 +01:00
Richard van der Hoff 95c1f6500b
Merge pull request #4289 from matrix-org/rav/welcome_page 
Add a welcome page to the static resources
 2018-12-11 16:24:06 +01:00
Richard van der Hoff f537432ef9 Add a welcome page to the static resources 
This is largely a precursor for the removal of the bundled webclient. The idea
is to present a page at / which reassures people that something is working, and
to give them some links for next steps.

The welcome page lives at `/_matrix/static/`, so is enabled alongside the other
`static` resources (which, in practice, means the client API is enabled). We'll
redirect to it from `/` if we have nothing better to display there.

It would be nice to have a way to disable it (in the same way that you might
disable the nginx welcome page), but I can't really think of a good way to do
that without a load of ickiness.

It's based on the work done by @krombel for #2601.
 2018-12-11 13:26:22 +00:00
Richard van der Hoff df96177ca7 Stop installing Matrix Console by default 
This is based on the work done by @krombel in #2601.
 2018-12-11 13:20:33 +00:00
David Baker 89ac2a5bdb Add 'sandbox' to CSP for media repo (#4284) 
* Add 'sandbox' to the CSP for media repo

* Changelog
 2018-12-11 04:05:02 +11:00
Will Hunt fee831c040 Move imports to one line 2018-12-10 13:52:33 +00:00
Will Hunt 466c1f3e01
Use `send_cors` 2018-12-10 13:11:37 +00:00
Will Hunt 91206e09f2 changelog & isort 2018-12-09 17:39:44 +00:00
Will Hunt dbf736ba66
Make /config more CORS-y 2018-12-09 13:27:22 +00:00
Richard van der Hoff 30da50a5b8
Initialise user displayname from SAML2 data (#4272) 
When we register a new user from SAML2 data, initialise their displayname
correctly.
 2018-12-07 14:44:46 +01:00
Richard van der Hoff c7401a697f
Implement SAML2 authentication (#4267) 
This implements both a SAML2 metadata endpoint (at
`/_matrix/saml2/metadata.xml`), and a SAML2 response receiver (at
`/_matrix/saml2/authn_response`). If the SAML2 response matches what's been
configured, we complete the SSO login flow by redirecting to the client url
(aka `RelayState` in SAML2 jargon) with a login token.

What we don't yet have is anything to build a SAML2 request and redirect the
user to the identity provider. That is left as an exercise for the reader.
 2018-12-07 13:11:11 +01:00
Richard van der Hoff c588b9b9e4
Factor SSO success handling out of CAS login (#4264) 
This is mostly factoring out the post-CAS-login code to somewhere we can reuse
it for other SSO flows, but it also fixes the userid mapping while we're at it.
 2018-12-07 13:10:07 +01:00
rkfg ae19a7db8c Prevent crash on pagination. 2018-12-06 13:32:05 +03:00
Richard van der Hoff b0c24a66ec Rip out half-implemented m.login.saml2 support (#4265) 
* Rip out half-implemented m.login.saml2 support

This was implemented in an odd way that left most of the work to the client, in
a way that I really didn't understand. It's going to be a pain to maintain, so
let's start by ripping it out.

* drop undocumented dependency on dateutil

It turns out we were relying on dateutil being pulled in transitively by
pysaml2. There's no need for that bloat.
 2018-12-06 19:44:38 +11:00
Richard van der Hoff 9a3e24a13d drop undocumented dependency on dateutil (#4266) 
It turns out we were relying on dateutil being pulled in transitively by
pysaml2. There's no need for that bloat.
 2018-12-06 04:52:42 +11:00
Richard van der Hoff e8d98466b0
Implement .well-known handling (#4262) 
Sometimes it's useful for synapse to generate its own .well-known file.
 2018-12-05 14:38:58 +01:00
Richard van der Hoff e55983defe Prepare 0.34.0rc1 2018-12-04 13:52:16 +00:00
Travis Ralston 1737753a62 Add an option to enable recording IPs for appservice users (#3831) 2018-12-04 12:44:41 +01:00
Amber Brown fd96dd75a3 Fix non-ASCII pushrules (#4248) 2018-12-04 12:44:02 +01:00
Travis Ralston 158ffb92f1 Add an option to disable search for homeservers which may not be interested in it (#4230) 
This is useful for homeservers not intended for users, such as bot-only homeservers or ones that only process IoT data.
 2018-12-04 12:01:02 +01:00
Ben Parsons b5ac0ffa0a add more detail to logging regarding "More than one row matched" error (#4234) 2018-12-04 11:57:39 +01:00
Richard van der Hoff ecc23188f4
Fix UnicodeDecodeError when postgres is not configured in english (#4253) 
This is a bit of a half-assed effort at fixing https://github.com/matrix-org/synapse/issues/4252. Fundamentally the right answer is to drop support for Python 2.
 2018-12-04 11:55:52 +01:00
Richard van der Hoff f144c0a210
Merge pull request #4244 from aaronraimist/drop-sent-txt 
Drop sent_transactions
 2018-12-04 11:41:28 +01:00
Richard van der Hoff a077e710a3
Merge pull request #4250 from matrix-org/hawkowl/pusher-remove-py3 
Fix removing pushers on python 3
 2018-12-04 11:22:46 +01:00
Richard van der Hoff 52e87fbfbe
Run the AS senders as background processes (#4189) 
This should fix some "Starting db connection from sentinel context" warnings,
and will mean we get metrics for these processes.
 2018-12-04 10:53:49 +01:00
Amber Brown d3c61ef906 fix type error 2018-12-03 22:27:41 +11:00
Richard van der Hoff c03324294d Workaround for non-ascii event ids (#4241) 
It turns out that we accept events with non-ascii IDs, which would later cause
an explosion during state res.

Fixes #4226
 2018-12-03 21:47:48 +11:00
Aaron Raimist 704c5298f0
Drop sent_transactions 
Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2018-12-01 23:07:35 -06:00
Neil Johnson 7039ece8fb Neilj/fix autojoin (#4223) 
* Fix auto join failures for servers that require user consent

* Fix auto join failures for servers that require user consent
 2018-11-28 22:24:57 +11:00
Neil Johnson f9b136a886 Neilj/fix mau initial reserved users (#4211) 
* fix transaction wrapping bug that caused get_user_id_by_threepid_txn to fail

* towncrier

* white space
 2018-11-28 20:33:41 +11:00
Richard van der Hoff 944d524f18 Support m.login.sso (#4220) 
* Clean up the CSS for the fallback login form

I was finding this hard to work with, so simplify a bunch of things. Each
flow is now a form inside a div of class login_flow.

The login_flow class now has a fixed width, as that looks much better than each
flow having a differnt width.

* Support m.login.sso

MSC1721 renames m.login.cas to m.login.sso. This implements the change
(retaining support for m.login.cas for older clients).

* changelog
 2018-11-27 18:51:52 +11:00
Richard van der Hoff 6c18cc4b50 Ignore __pycache__ directories in schema delta dir 
Now that we use py3, compiled python ends up in __pycache__ rather than *.pyc.
 2018-11-20 22:52:34 +00:00
Neil Johnson 78ba0e7ab8 Remove riot.im from the list of trusted Identity Servers in the default configuration (#4207) 2018-11-20 12:29:25 +01:00
Richard van der Hoff 416c671474
Merge pull request #4204 from matrix-org/rav/logcontext_leak_fixes 
Fix some logcontext leaks
 2018-11-20 12:19:19 +01:00
Amber Brown 31425d82a3 Merge remote-tracking branch 'origin/master' into develop 2018-11-19 12:55:25 -06:00
Amber Brown d102e19e47 version 2018-11-19 12:42:49 -06:00
Amber Brown 80cac86b2c
Fix fallback auth on Python 3 (#4197) 2018-11-19 12:27:33 -06:00
Richard van der Hoff f5faf6bc14 Fix logcontext leak in EmailPusher 2018-11-19 17:07:01 +00:00
Travis Ralston ab4526a153 Remove duplicate slashes in generated consent URLs 2018-11-15 20:41:53 -07:00
Amber Brown 8b1affe7d5
Fix Content-Disposition in media repository (#4176) 2018-11-15 15:55:58 -06:00
Travis Ralston 835779f7fb Add option to track MAU stats (but not limit people) (#3830) 2018-11-15 18:08:27 +00:00
Amber Brown df758e155d
Use <meta> tags to discover the per-page encoding of html previews (#4183) 2018-11-15 11:05:08 -06:00
Neil Johnson bf648c37e7 release 0.33.9rc1 2018-11-14 11:45:52 +00:00
Richard van der Hoff c1efcd7c6a Add a test for the public T&Cs form 2018-11-14 10:46:27 +00:00
Richard van der Hoff 83a5f459aa Fix an internal server error when viewing the public privacy policy 2018-11-14 10:21:07 +00:00
David Baker 0869566ad3
Merge pull request #4113 from matrix-org/dbkr/e2e_backup_versions_are_numbers 
Make e2e backup versions numeric in the DB
 2018-11-14 07:55:48 +00:00
David Baker bca3b91c2d Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backup_versions_are_numbers 2018-11-09 18:35:02 +00:00
Erik Johnston db5a1c059a
Merge pull request #4166 from matrix-org/erikj/drop_unknown_events 
Drop incoming events from federation for unknown rooms
 2018-11-09 17:59:34 +00:00
Erik Johnston dc59ad5334 Remove hack to support rejoining rooms 2018-11-09 14:58:09 +00:00
David Baker d44dea0223 pep8 2018-11-09 14:38:31 +00:00
David Baker 4f93abd62d add docs 2018-11-09 13:25:38 +00:00
Erik Johnston 30dd27afff Simplify to always drop events if server isn't in the room 2018-11-09 11:36:45 +00:00
Richard van der Hoff 3cecf5340d
Update synapse/federation/federation_server.py 
Co-Authored-By: erikjohnston <erikj@jki.re>
 2018-11-09 11:28:25 +00:00
Richard van der Hoff 9bce065a53
Update synapse/federation/federation_server.py 
Co-Authored-By: erikjohnston <erikj@jki.re>
 2018-11-09 11:28:22 +00:00
David Baker d3fa6194f7 Remove unnecessary str() 2018-11-09 11:11:31 +00:00
Erik Johnston 7b22421a7b
Merge pull request #4164 from matrix-org/erikj/fix_device_comparison 
Fix noop checks when updating device keys
 2018-11-08 14:37:20 +00:00
Erik Johnston 5ebed18692 Lets convert bytes to unicode instead 2018-11-08 12:33:13 +00:00
Erik Johnston b1a22b24ab Fix noop checks when updating device keys 
Clients often reupload their device keys (for some reason) so its
important for the server to check for no-ops before sending out device
list update notifications.

The check is broken in python 3 due to the fact comparing bytes and
unicode always fails, and that we write bytes to the DB but get unicode
when we read.
 2018-11-08 12:18:38 +00:00
Erik Johnston 9417986f77 Drop PDUs of unknown rooms 
When we receive events over federation we will need to know the room
version to be able to correctly handle them, e.g. once we start changing
event formats. Currently, we attempt to handle events in unknown rooms.
 2018-11-08 12:11:20 +00:00
Richard van der Hoff 0a1fc52971 fix parse_string docstring 2018-11-08 11:12:29 +00:00
hera 2b075fb03a Fix encoding error for consent form on python3 
The form was rendering this as "b'01234....'".

-- richvdh
 2018-11-08 11:05:39 +00:00
Amber Brown b3708830b8
Fix URL preview bugs (type error when loading cache from db, content-type including quotes) (#4157) 2018-11-08 01:37:43 +11:00
Travis Ralston 0f5e51f726 Add config variables for enabling terms auth and the policy name (#4142) 
So people can still collect consent the old way if they want to.
 2018-11-06 10:32:34 +00:00
Hubert Chathi f1087106cf
handle empty backups according to latest spec proposal (#4123) 
fixes #4056
 2018-11-05 17:59:29 -05:00
Amber Brown efdcbbe46b
Tests for user consent resource (#4140) 2018-11-06 05:53:44 +11:00
Erik Johnston bc80b3f454 Add helpers for getting prev and auth events (#4139) 
* Add helpers for getting prev and auth events

This is in preparation for allowing the event format to change between
room versions.
 2018-11-06 00:35:15 +11:00
Amber Brown 0467384d2f
Set the encoding to UTF8 in the default logconfig (#4138) 2018-11-03 02:28:07 +11:00
Erik Johnston 90d713b8c6
Merge pull request #4137 from matrix-org/erikj/clean_up_events 
Clean up event accesses and tests
 2018-11-02 14:12:49 +00:00
Erik Johnston b86d05a279 Clean up event accesses and tests 
This is in preparation to refactor FrozenEvent to support different
event formats for different room versions
 2018-11-02 13:44:14 +00:00
Amber Brown cb7a6b2379
Fix typing being reset causing infinite syncs (#4127) 2018-11-03 00:19:23 +11:00
Richard van der Hoff efb9343c8c
Merge pull request #4132 from matrix-org/rav/fix_device_list_locking 
Fix locked upsert on device_lists_remote_cache
 2018-11-02 10:50:53 +00:00
Richard van der Hoff 00f12e00f8
Merge pull request #4133 from matrix-org/travis/fix-terms-auth 
Fix logic error that prevented guests from seeing the privacy policy
 2018-11-02 10:50:43 +00:00
Erik Johnston b199534518
Merge pull request #4135 from matrix-org/erikj/fix_state_res_none 
Fix None exception in state res v2
 2018-11-02 10:45:57 +00:00
Richard van der Hoff 350f654e7b Add unique indexes to a couple of tables 
The indexes on device_lists_remote_extremeties can be unique, and they
therefore should, to ensure that the db remains consistent.
 2018-11-02 10:36:13 +00:00
Richard van der Hoff 50e328d1e7 Remove redundant database locks for device list updates 
We can rely on the application-level per-user linearizer.
 2018-11-02 10:36:13 +00:00
Erik Johnston 54aec35867 Fix None exception in state res v2 2018-11-02 10:29:19 +00:00
Travis Ralston 642505abc3 Fix logic error that prevented guests from seeing the privacy policy 2018-11-01 16:48:32 -06:00
Travis Ralston c68aab1536
Merge pull request #4004 from matrix-org/travis/login-terms 
Add m.login.terms to the registration flow
 2018-11-01 11:03:38 -06:00
Erik Johnston 1b21e771d0
Merge pull request #4128 from matrix-org/erikj/state_res_v2_version 
Add STATE_V2_TEST room version
 2018-11-01 13:17:57 +00:00
Erik Johnston b3dd6fa981 Add STATE_V2_TEST room version 2018-11-01 11:43:46 +00:00
Amber Brown 073d400b84 Merge branch 'master' into develop 2018-11-01 21:32:12 +11:00
Amber Brown aa98e38896 version 2018-11-01 21:28:35 +11:00
Travis Ralston a8d41c6aff Include a version query string arg for the consent route 2018-10-31 13:19:28 -06:00
Travis Ralston d1e7b9c44c Merge branch 'develop' into travis/login-terms 2018-10-31 13:15:14 -06:00
Richard van der Hoff 1729ba1650
Merge pull request #4101 from matrix-org/rav/aliases_for_upgrades 
Attempt to move room aliases on room upgrades
 2018-10-31 17:52:18 +00:00
Richard van der Hoff 0f8591a5a8 Avoid else clause on exception for clarity 2018-10-31 15:43:57 +00:00
Richard van der Hoff 94c7fadc98 Attempt to move room aliases on room upgrades 2018-10-31 15:43:57 +00:00
Richard van der Hoff 9b827c40ca
Log some bits about event creation (#4121) 
I found these helpful in debugging my room upgrade tests.
 2018-10-31 15:42:23 +00:00
Amber Brown 916efc8249
Remove fetching keys via the deprecated v1 kex method (#4120) 2018-10-31 23:14:39 +11:00
Amber Brown f79f454485
Remove deprecated v1 key exchange endpoint (#4119) 2018-10-31 22:29:02 +11:00
Amber Brown 67c1924899 version bump 2018-10-31 06:27:05 +11:00
Amber Brown 086e1a8f3e Fix search 500ing (#4122) 2018-10-31 06:26:46 +11:00
Amber Brown 3bade14ec0
Fix search 500ing (#4122) 2018-10-31 04:33:41 +11:00
Amber Brown 2e223a8c22
Remove the unused /pull federation API (#4118) 2018-10-31 04:24:59 +11:00
Erik Johnston 0794504bce
Merge pull request #4006 from matrix-org/erikj/purge_state_groups 
Delete unreferenced state groups during purge
 2018-10-30 16:58:22 +00:00
Amber Brown 0dce9e1379
Write some tests for the email pusher (#4095) 2018-10-30 23:55:43 +11:00
David Baker e0934acdbb Cast to int here too 2018-10-30 11:12:23 +00:00
David Baker 12941f5f8b Cast bacjup version to int when querying 2018-10-30 11:01:07 +00:00
David Baker 2f0f911c52 Convert version back to a string 2018-10-30 10:35:18 +00:00
David Baker 64fa557f80 Try & make it work on postgres 2018-10-30 09:51:04 +00:00
David Baker 563f9b61b1 Make e2e backup versions numeric in the DB 
We were doing max(version) which does not do what we wanted
on a column of type TEXT.
 2018-10-29 21:01:22 +00:00
Erik Johnston 88e5ffe6fe Deduplicate device updates sent over replication 
We currently send several kHz of device list updates over replication
occisonally, which often causes the replications streams to lag and then
get dropped.

A lot of those updates will actually be duplicates, since we don't send
e.g. device_ids across replication, so let's deduplicate it when we pull
them out of the database.
 2018-10-29 17:34:34 +00:00
Erik Johnston a163b748a5 Don't truncate command name in metrics 2018-10-29 17:34:21 +00:00
Erik Johnston ad88460e0d Move _find_unreferenced_groups 2018-10-29 14:24:19 +00:00
Erik Johnston 664b192a3b Fix set operations thinko 2018-10-29 14:21:43 +00:00
Erik Johnston f4f223aa44 Don't make temporary list 2018-10-29 14:01:49 +00:00
Erik Johnston b2399f6281 Make SQL a bit cleaner 2018-10-29 14:01:11 +00:00
Amber Brown 4cd1c9f2ff
Delete the disused & unspecced identicon functionality (#4106) 2018-10-29 23:57:24 +11:00
Richard van der Hoff 7fbfea062e
Merge pull request #4100 from matrix-org/rav/room_upgrade_avatar 
Remember to copy the avatar on room upgrades
 2018-10-29 12:49:21 +00:00
Richard van der Hoff 56ca578f77
Merge pull request #4099 from matrix-org/rav/upgrade_odd_pls 
Better handling of odd PLs during room upgrades
 2018-10-29 12:48:51 +00:00
Richard van der Hoff bf33eed609
Merge pull request #4091 from matrix-org/rav/room_version_upgrades 
Room version upgrade support
 2018-10-29 12:47:20 +00:00
Amber Brown c4b3698a80
Make the replication logger quieter (#4108) 2018-10-29 22:59:44 +11:00
Amber Brown 2b791865c4 version bump 2018-10-29 21:52:52 +11:00
Richard van der Hoff db24d7f15e Better handling of odd PLs during room upgrades 
Fixes handling of rooms where we have permission to send the tombstone, but not
other state. We need to (a) fail more gracefully when we can't send the PLs in
the old room, and (b) not set the PLs in the new room until we are done with
the other stuff.
 2018-10-27 00:54:26 +01:00
Richard van der Hoff 5caf79b312 Remember to copy the avatar on room upgrades 2018-10-26 23:56:40 +01:00
Richard van der Hoff 54bbe71867 optimise state copying 2018-10-26 22:51:34 +01:00
Richard van der Hoff 193cadc988 Address review comments 
Improve comments, get old room state from the context we already have
 2018-10-26 17:10:30 +01:00
Erik Johnston 03e634dad4 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups 2018-10-26 16:22:45 +01:00
Amber Brown 77d70a7646
Port register_new_matrix_user to Python 3 and add tests (#4085) 2018-10-26 22:05:22 +11:00
Richard van der Hoff e6babc27d5 restrict PLs in old room 2018-10-25 19:18:25 +01:00
Richard van der Hoff 3a263bf3ae copy state 2018-10-25 19:18:25 +01:00
Richard van der Hoff 1b9f253e20 preserve PLs 2018-10-25 19:10:24 +01:00
Richard van der Hoff 4cda300058 preserve room visibility 2018-10-25 19:10:24 +01:00
Richard van der Hoff 0f7d1c9906 Basic initial support for room upgrades 
Currently just creates a new, empty, room, and sends a tombstone in the old
room.
 2018-10-25 19:10:24 +01:00
Richard van der Hoff e1948175ee Allow power_level_content_override=None for _send_events_for_new_room 2018-10-25 19:10:24 +01:00
Richard van der Hoff 7f7b2cd3de Make room_member_handler a member of RoomCreationHandler 
... to save passing it into `_send_events_for_new_room`
 2018-10-25 19:10:18 +01:00
Richard van der Hoff 871c4abfec Factor _generate_room_id out of create_room 
we're going to need this for room upgrades.
 2018-10-25 18:23:09 +01:00
Erik Johnston cb53ce9d64
Refactor state group lookup to reduce DB hits (#4011) 
Currently when fetching state groups from the data store we make two
hits two the database: once for members and once for non-members (unless
request is filtered to one or the other). This adds needless load to the
datbase, so this PR refactors the lookup to make only a single database
hit.
 2018-10-25 17:49:55 +01:00
Erik Johnston c85e063302
Merge pull request #4051 from matrix-org/erikj/alias_disallow_list 
Add config option to control alias creation
 2018-10-25 17:04:59 +01:00
Neil Johnson 95ad128851
Merge pull request #4081 from matrix-org/neilj/fix_mau_init 
fix race condiftion in calling initialise_reserved_users
 2018-10-25 16:33:40 +01:00
Neil Johnson fcbd488e9a add new line 2018-10-25 16:13:43 +01:00
Erik Johnston b94a43d5b5 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/alias_disallow_list 2018-10-25 15:25:31 +01:00
Erik Johnston e5481b22aa Use allow/deny 2018-10-25 15:25:21 +01:00
Neil Johnson f8fe98812b improve comments 2018-10-25 14:58:59 +01:00
Neil Johnson f7f487e14c Merge branch 'develop' of github.com:matrix-org/synapse into matthew/autocreate_autojoin 2018-10-25 14:40:06 +01:00
Travis Ralston 54def42c19 Merge branch 'develop' into travis/login-terms 2018-10-24 13:22:59 -06:00
Neil Johnson ea69a84bbb fix style inconsistencies 2018-10-24 17:18:08 +01:00
Neil Johnson 663d9db8e7 commit transaction before closing 2018-10-24 17:17:30 +01:00
Neil Johnson 07126e43a4 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_mau_init 2018-10-24 16:25:39 +01:00
Neil Johnson 9ec2186586 isort 2018-10-24 16:09:21 +01:00
Neil Johnson 9532caf6ef remove trailing whiter space 2018-10-24 16:08:25 +01:00
Richard van der Hoff 94a49e0636
fix tuple 
Co-Authored-By: neilisfragile <neil@matrix.org>
 2018-10-24 14:39:23 +01:00
Richard van der Hoff 3ad359e5be Merge remote-tracking branch 'origin/develop' into rav/fix_event_filter_validation 2018-10-24 11:23:49 +01:00
Richard van der Hoff 7328039117
Merge pull request #4082 from matrix-org/rav/fix_pep8 
Fix a number of flake8 errors
 2018-10-24 11:23:35 +01:00
Erik Johnston 3904cbf307
Merge pull request #4040 from matrix-org/erikj/states_res_v2_rebase 
Add v2 state resolution algorithm
 2018-10-24 11:12:12 +01:00
Richard van der Hoff 7e07d25ed6 Allow backslashes in event field filters 
Fixes a bug introduced in https://github.com/matrix-org/synapse/pull/1783 which
meant that single backslashes were not allowed in event field filters.

The intention here is to allow single-backslashes, but disallow
double-backslashes.
 2018-10-24 11:11:24 +01:00
Richard van der Hoff ef771cc4c2 Fix a number of flake8 errors 
Broadly three things here:

* disable W504 which seems a bit whacko
* remove a bunch of `as e` expressions from exception handlers that don't use
  them
* use `r""` for strings which include backslashes

Also, we don't use pep8 any more, so we can get rid of the duplicate config
there.
 2018-10-24 10:39:03 +01:00
Erik Johnston 47a9ba435d Use match rather than search 2018-10-24 09:54:57 +01:00
Richard van der Hoff e0b9d5f0af
Merge pull request #4075 from matrix-org/rav/fix_pusher_logcontexts 
Clean up the way logcontexts and threads work in the pushers
 2018-10-24 09:53:57 +01:00
Erik Johnston dacbeb2e03 Comment 2018-10-24 09:47:49 +01:00
Erik Johnston 810715f79a Rename resolve_events_with_factory 2018-10-24 09:44:22 +01:00
Erik Johnston cb23aa4c42
Merge pull request #4063 from matrix-org/erikj/cleaup_alias_creation 
Clean up room alias creation
 2018-10-24 09:40:27 +01:00
Richard van der Hoff c573794b22 Fix start_pushers vs _start_pushers confusion 2018-10-24 09:24:55 +01:00
Richard van der Hoff e564306e31 sanity-check the is_processing flag 
... and rename it, for even more sanity
 2018-10-24 09:23:33 +01:00
Neil Johnson a67d8ace9b remove errant exception and style 2018-10-23 17:44:39 +01:00
Travis Ralston 43c3f0b02f
Merge pull request #3969 from turt2live/travis/fix-federated-group-requests 
Handle HttpResponseException more safely for federated groups
 2018-10-23 10:41:04 -06:00
Travis Ralston 3e704822be
Comments help 2018-10-23 10:25:31 -06:00
Neil Johnson 329d18b39c remove white space 2018-10-23 15:27:20 +01:00
Neil Johnson 6105c6101f fix race condiftion in calling initialise_reserved_users 2018-10-23 15:24:58 +01:00
Richard van der Hoff b3f6dddad2
Give some more things logcontexts (#4077) 2018-10-23 14:29:17 +01:00
Richard van der Hoff 5c445114d3
Correctly account for cpu usage by background threads (#4074) 
Wrap calls to deferToThread() in a thing which uses a child logcontext to
attribute CPU usage to the right request.

While we're in the area, remove the logcontext_tracer stuff, which is never
used, and afaik doesn't work.

Fixes #4064
 2018-10-23 13:12:32 +01:00
Richard van der Hoff 026cd91ac8 Run PusherPool.start as a background process 
We don't do anything with the result, so this is needed to give this code a
logcontext.
 2018-10-22 16:12:11 +01:00
Richard van der Hoff f749607c91 Make on_started synchronous too 
This brings it into line with on_new_notifications and on_new_receipts. It
requires a little bit of hoop-jumping in EmailPusher to load the throttle
params before the first loop.
 2018-10-22 16:12:11 +01:00
Richard van der Hoff e7a16c6210 Remove redundant run_as_background_process() from pusherpool 
`on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher`
now always return synchronously, so we can remove the `defer.gatherResults` on
their results, and the `run_as_background_process` wrappers can be removed too
because the PusherPool methods will now complete quickly enough.
 2018-10-22 16:12:11 +01:00
Richard van der Hoff c7273c11bc Give pushers their own background logcontext 
Each pusher has its own loop which runs for as long as it has work to do. This
should run in its own background thread with its own logcontext, as other
similar loops elsewhere in the system do - which means that CPU usage is
consistently attributed to that loop, rather than to whatever request happened
to start the loop.
 2018-10-22 16:12:11 +01:00
Richard van der Hoff 5110f4e425 move get_all_pushers call down 
simplifies the interface to _start_pushers
 2018-10-22 16:12:11 +01:00
Richard van der Hoff 04277d0ed8 Factor PusherPool._start_pusher out of _start_pushers 
... and use it from start_pusher_by_id. This mostly simplifies
start_pusher_by_id.
 2018-10-22 16:12:11 +01:00
Richard van der Hoff 3e8b02c939 Rename _refresh_pusher 
This is public (or at least, called from outside the class), so ought to have a
better name.
 2018-10-22 16:12:11 +01:00
Richard van der Hoff 7aea00069c
Merge pull request #4076 from matrix-org/rav/fix_init_logcontexts 
Run MAU queries as background processes
 2018-10-22 14:46:59 +01:00
Richard van der Hoff 911db96658
Merge pull request #4073 from matrix-org/rav/require_psutil 
Make psutil an explicit dependency
 2018-10-22 12:33:21 +01:00
Richard van der Hoff a6f421e812 Run MAU queries as background processes 
Fixes #3820
 2018-10-20 02:14:35 +01:00
Amber Brown e1728dfcbe
Make scripts/ and scripts-dev/ pass pyflakes (and the rest of the codebase on py3) (#4068) 2018-10-20 11:16:55 +11:00
Richard van der Hoff e5b52d0f94 Make psutil an explicit dependency 
As of #4027, we require psutil to be installed, so it should be in our
dependency list. We can also remove some of the conditional import code
introduced by #992.

Fixes #4062.
 2018-10-19 21:51:15 +01:00
Richard van der Hoff f62c597d14
Merge pull request #4069 from matrix-org/rav/fix_email_templates_4065 
Calculate absolute path for email templates
 2018-10-19 16:33:54 +01:00
Erik Johnston 67f7b9cb50 pep8 2018-10-19 16:06:59 +01:00
Erik Johnston 056f099126 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups 2018-10-19 15:48:59 +01:00
Erik Johnston 47a9da28ca Batch process handling state groups 2018-10-19 15:48:15 +01:00
Richard van der Hoff cc325c7069 Calculate absolute path for email templates 2018-10-19 14:01:59 +01:00
Amber Brown e404ba9aac
Fix manhole on py3 (pt 2) (#4067) 2018-10-19 22:26:00 +11:00
Amber Brown b69216f768
Make the metrics less racy (#4061) 2018-10-19 21:45:45 +11:00
Erik Johnston 9fafdfa97d Anchor returned regex to start and end of string 2018-10-19 10:22:45 +01:00
Erik Johnston 084046456e Add config option to control alias creation 2018-10-19 10:22:45 +01:00
Erik Johnston 74e7617083 Clean up room alias creation 2018-10-19 10:11:56 +01:00
Amber Brown a36b0ec195 make a bytestring 2018-10-19 09:24:00 +11:00
Travis Ralston 49a044aa5f Merge branch 'develop' into travis/login-terms 2018-10-18 09:57:58 -06:00
Richard van der Hoff c632bc8654 Merge branch 'master' into develop 2018-10-18 15:07:03 +01:00
Richard van der Hoff 926da4dda8 0.33.7 2018-10-18 14:57:32 +01:00
Richard van der Hoff c7d0f34a3c v0.33.7rc2 2018-10-17 17:40:19 +01:00
Richard van der Hoff 0fd2321629 Fix incorrect truncation in get_missing_events 
It's quite important that get_missing_events returns the *latest* events in the
room; however we were pulling event ids out of the database until we got *at
least* 10, and then taking the *earliest* of the results.

We also shouldn't really be relying on depth, and should be checking the
room_id.
 2018-10-17 17:35:26 +01:00
Richard van der Hoff c8f2c19991 Put the warning blob at the top of the file 2018-10-17 16:56:22 +01:00
Richard van der Hoff 1519572961 Ship the email templates as package_data 
move the example email templates into the synapse package so that they can be
used as package data, which should mean that all of the packaging mechanisms
(pip, docker, debian, arch, etc) should now come with the example templates.

In order to grandfather in people who relied on the templates being in the old
place, check for that situation and fall back to using the defaults if the
templates directory does not exist.
 2018-10-17 16:46:02 +01:00
Erik Johnston f6a0a02a62 Fix bug where we raised StopIteration in a generator 
This made python 3.7 unhappy
 2018-10-17 16:10:52 +01:00
Will Hunt d6a7797dd1 Fix roomlist since tokens on Python 3 (#4046) 
Thanks @Half-Shot !!!
 2018-10-17 23:04:55 +11:00
Neil Johnson c6584f4b5f clean up config error logic and imports 2018-10-17 11:36:41 +01:00
Richard van der Hoff fc0f13dd03 Fix incorrect truncation in get_missing_events 
It's quite important that get_missing_events returns the *latest* events in the
room; however we were pulling event ids out of the database until we got *at
least* 10, and then taking the *earliest* of the results.

We also shouldn't really be relying on depth, and should be checking the
room_id.
 2018-10-16 21:10:04 +01:00
Erik Johnston 4a28d3d36f Update event_auth table for rejected events 2018-10-16 16:28:42 +01:00
Erik Johnston 15133477ee Fix up use of resolve_events_with_factory 2018-10-16 16:28:42 +01:00
Erik Johnston 6bd856caa2 User event.sender rather than alias event.user_id 2018-10-16 16:16:13 +01:00
Erik Johnston e238013c44 Add v2 state res algorithm. 
We hook this up to the vdh test room version.
 2018-10-16 16:16:13 +01:00
Richard van der Hoff b8a5b0097c
Various cleanups in the federation client code (#4031) 
- Improve logging: log things in the right order, include destination and txids
  in all log lines, don't log successful responses twice

- Fix the docstring on TransportLayerClient.send_transaction

- Don't use treq.request, which is overcomplicated for our purposes: just use a
  twisted.web.client.Agent.

- simplify the logic for setting up the bodyProducer

- fix bytes/str confusions
 2018-10-16 10:44:49 +01:00
Travis Ralston a8ed93a4b5 pep8 2018-10-15 16:10:29 -06:00
Travis Ralston 442734ff9e Ensure the terms params are actually provided 2018-10-15 14:56:13 -06:00
Travis Ralston 762a0982aa Python is hard 2018-10-15 14:46:09 -06:00
Travis Ralston f293d124b6 Merge branch 'develop' into travis/login-terms 2018-10-15 14:44:32 -06:00
Amber Brown f726f2dc6c version bump 2018-10-15 22:21:45 +11:00
David Baker 03c11032c3
Merge pull request #4019 from matrix-org/dbkr/e2e_backups 
E2E backups
 2018-10-15 10:19:43 +01:00
Ivan Shapovalov 06bc8d2fe5 synapse/app: frontend_proxy.py: actually make workers work on py3 2018-10-14 20:08:39 +03:00
Ivan Shapovalov fb216a22db synapse/visibility.py: fix SyntaxError on py3.7 2018-10-14 20:08:17 +03:00
Neil Johnson 1ccafb0c5e no need to join room if creator 2018-10-13 21:14:21 +01:00
Travis Ralston dd99db846d Update login terms structure for the proposed language support 2018-10-12 18:03:27 -06:00
Travis Ralston 5119818e9d Rely on the lack of ?u to represent public access 
also general cleanup
 2018-10-12 18:03:17 -06:00
Travis Ralston 7ede650956 Merge branch 'develop' into travis/login-terms 2018-10-12 16:24:07 -06:00
Travis Ralston 164f8e4843 isort 2018-10-12 15:11:59 -06:00
Travis Ralston 7bb651de6a More sane handling of group errors and pep8 2018-10-12 14:53:30 -06:00
Travis Ralston e3586f7c06 Merge branch 'develop' into travis/fix-federated-group-requests 2018-10-12 14:49:58 -06:00
Neil Johnson a2bfb778c8 improve auto room join logic, comments and tests 2018-10-12 18:17:36 +01:00
David Baker a45f2c3a00 missed one 2018-10-12 14:33:55 +01:00
Amber Brown 381d2cfdf0
Make workers work on Py3 (#4027) 2018-10-13 00:14:08 +11:00
Erik Johnston 67a1e315cc Fix up comments 2018-10-12 13:49:48 +01:00
David Baker 8c0ff0287a Linting soothes the savage PEP8 monster 2018-10-12 13:47:43 +01:00
David Baker 306361b31b Misc PR feedback bits 2018-10-12 11:48:56 +01:00
David Baker bddfad253a Don't mangle exceptions 2018-10-12 11:48:02 +01:00
David Baker 86ef9760a7 Split /room_keys/version into 2 servlets 2018-10-12 11:35:08 +01:00
David Baker 83e72bb2f0 PR feedback pt. 1 2018-10-12 11:26:18 +01:00
Richard van der Hoff 8ddd0f273c Comments on get_all_new_events_stream 
just some docstrings to clarify the behaviour here
 2018-10-12 09:55:41 +01:00
Erik Johnston 7e561b5c1a Add description to counter metric 2018-10-10 11:41:15 +01:00
Erik Johnston 3cbe8331e6 Track number of non-empty sync responses instead 2018-10-10 11:23:17 +01:00
Erik Johnston 395276b405 Append _total to metric and fix up spelling 2018-10-10 09:24:39 +01:00
David Baker b8d9e108be Fix mergefail 2018-10-09 18:04:21 +01:00
Erik Johnston bdc27d6716 Add metric to count lazy member sync requests 2018-10-09 14:17:52 +01:00
David Baker d3464ce708 isort 2018-10-09 10:33:59 +01:00
Erik Johnston 9eb1a79100
Merge pull request #4008 from matrix-org/erikj/log_looping_exceptions 
Log looping call exceptions
 2018-10-09 10:12:10 +01:00
David Baker dc045ef202 Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backups 2018-10-09 10:05:02 +01:00
David Baker f4a4dbcad1 Apparently this blank line is Very Important 2018-10-09 09:47:04 +01:00
David Baker 0c905ee015 be python3 compatible 2018-10-09 09:39:13 +01:00
Erik Johnston 6982320572 Remove unnecessary extra function call layer 2018-10-08 14:06:19 +01:00
Richard van der Hoff 495975e231 Optimisation for filter_events_for_server 
We're better off hashing just the event_id than the whole ((type, state_key),
event_id) tuple - so use a dict instead of a set.

Also, iteritems > items.
 2018-10-08 13:46:52 +01:00
Erik Johnston 8a1817f0d2 Use errback pattern and catch async failures 2018-10-08 13:29:47 +01:00
David Baker 497444f1fd Don't reuse backup versions 
Since we don't actually delete the keys, just mark the versions
as deleted in the db rather than actually deleting them, then we
won't reuse versions.

Fixes https://github.com/vector-im/riot-web/issues/7448
 2018-10-05 15:08:36 +01:00
Erik Johnston f7199e8734 Log looping call exceptions 
If a looping call function errors, then it kills the loop entirely.
Currently it throws away the exception logs, so we should make it
actually log them.

Fixes #3929
 2018-10-05 11:24:12 +01:00
Neil Johnson ed82043efb
Merge branch 'develop' into matthew/autocreate_autojoin 2018-10-04 17:26:59 +01:00
Neil Johnson 2dadc092b8 move logic into register, fix room alias localpart bug, tests 2018-10-04 17:00:27 +01:00
Richard van der Hoff c6dbd216e6
Merge pull request #3995 from matrix-org/rav/no_deextrem_outliers 
Fix bug in forward_extremity update logic
 2018-10-04 16:27:05 +01:00
Erik Johnston 4917ff5523 Add state_group index to event_to_state_groups 
This is needed to efficiently check for unreferenced state groups during
purge.
 2018-10-04 16:03:08 +01:00
Erik Johnston 17d585753f Delete unreferened state groups during purge 2018-10-04 16:03:06 +01:00
Amber Brown d86794325f Merge branch 'master' into develop 2018-10-04 22:41:52 +10:00
Amber Brown dd59dfc51f full version 2018-10-04 22:37:55 +10:00
Travis Ralston 537d0b7b36 Use a flag rather than a new route for the public policy 
This also means that the template now has optional parameters, which will need to be documented somehow.
 2018-10-03 17:50:11 -06:00
Travis Ralston f9d34a763c Auto-consent to the privacy policy if the user registered with terms 2018-10-03 17:39:45 -06:00
Travis Ralston dfcad5fad5 Make the terms flow requried 2018-10-03 17:39:00 -06:00
Travis Ralston 3099d96dba Flesh out the fallback auth for terms 2018-10-03 17:39:00 -06:00
Travis Ralston 149c4f1765 Supply params for terms auth stage 
As per https://github.com/matrix-org/matrix-doc/pull/1692
 2018-10-03 15:57:42 -06:00
Travis Ralston fd99787162 Incorporate Dave's work for GDPR login flows 
As per https://github.com/vector-im/riot-web/issues/7168#issuecomment-419996117
 2018-10-03 15:57:42 -06:00
Richard van der Hoff a59d899668 Pin to prometheus_client<0.4 to avoid renaming all of our metrics 2018-10-03 17:20:15 +01:00
Erik Johnston 8935ec5a93
Merge pull request #3999 from matrix-org/erikj/fix_3pid_invite_rejetion 
Fix handling of rejected threepid invites
 2018-10-03 14:47:41 +01:00
Erik Johnston 81e2813948
Merge pull request #3996 from matrix-org/erikj/fix_bg_iteration 
Fix exception in background metrics collection
 2018-10-03 14:14:38 +01:00
Erik Johnston 52e6e815be Sanitise error messages when user doesn't have permission to invite 2018-10-03 14:13:07 +01:00
Erik Johnston 69e857853f Fix handling of rejected threepid invites 2018-10-03 11:57:30 +01:00
Erik Johnston 495a9d06bb Fix exception handling in fetching remote profiles 2018-10-03 11:34:30 +01:00
Erik Johnston 7c570bff74 Fix exception in background metrics collection 
We attempted to iterate through a list on a separate thread without
doing the necessary copying.
 2018-10-03 11:28:01 +01:00
Richard van der Hoff 9693625e55 actually exclude outliers 2018-10-03 10:19:41 +01:00
Richard van der Hoff 3e39783d5d remove debugging 2018-10-02 23:44:14 +01:00
Richard van der Hoff ae61ade891 Fix bug in forward_extremity update logic 
An event does not stop being a forward_extremity just because an outlier or
rejected event refers to it.
 2018-10-02 23:36:08 +01:00
Amber Brown da8f82008d version 2018-10-03 02:15:48 +10:00
Erik Johnston 7258d081a5 Fix bug when invalidating destination retry timings 2018-10-02 15:47:57 +01:00
Richard van der Hoff 2b8d28b095
Merge pull request #3960 from matrix-org/rav/fix_missing_create_event_error 
Fix error handling for missing auth_event
 2018-10-02 13:57:54 +01:00
Amber Brown 7232917f12
Disable frozen dicts by default (#3987) 2018-10-02 22:53:47 +10:00
Erik Johnston 334e075dd8 Fix error when logging incomplete requests 
If a connection is lost before a request is read from Request, Twisted
sets `method` (and `uri`) attributes to dummy values. These dummy values
have incorrect types (i.e. they're not bytes), and so things like
`__repr__` would raise an exception.

To fix this we had a helper method to return the method with a
consistent type.
 2018-10-02 12:06:22 +01:00
Richard van der Hoff 8c41b0ca66
Merge pull request #3989 from matrix-org/rav/better_stacktraces 
Avoid reraise, to improve stacktraces
 2018-10-02 10:47:38 +01:00
Erik Johnston bc29946809
Merge pull request #3986 from matrix-org/erikj/fix_sync_with_redacted_state 
Fix lazy loaded sync with rejected state events
 2018-10-02 11:38:35 +02:00
Richard van der Hoff 8174c6725b Avoid reraise, to improve stacktraces 2018-10-01 18:50:34 +01:00
Richard van der Hoff 5908a8f6f5 Merge branch 'develop' into rav/fix_missing_create_event_error 2018-10-01 17:16:20 +01:00
Richard van der Hoff b5b93f45d5
Merge pull request #3968 from matrix-org/rav/fix_federation_errors 
Fix exceptions when handling incoming transactions
 2018-10-01 15:54:24 +01:00
Amber Brown 6e05fd032c
Fix userconsent on Python 3 (#3938) 2018-10-02 00:11:58 +10:00
Erik Johnston 82f922b4af Fix lazy loaded sync with rejected state events 
In particular, we assume that the name and canonical alias events in
the state have not been rejected. In practice this may not be the case
(though we should probably think about fixing that) so lets ensure that
we gracefully handle that case, rather than 404'ing the sync request
like we do now.
 2018-10-01 14:19:40 +01:00
Erik Johnston 8f5c23d0cd
Merge pull request #3933 from matrix-org/erikj/destination_retry_cache 
Add a five minute cache to get_destination_retry_timings
 2018-10-01 14:20:30 +02:00
Erik Johnston 4f3e3ac192 Correctly match 'dict.pop' api 2018-10-01 12:25:27 +01:00
Erik Johnston 8ea887856c Don't update eviction metrics on explicit removal 2018-10-01 12:00:58 +01:00
Matthew Hodgson 23b6a0537f emit room aliases event 2018-09-29 02:19:37 +01:00
Matthew Hodgson 5b68f29f48 fix thinkos 2018-09-29 02:14:40 +01:00
Matthew Hodgson 8f646f2d04 fix UTs 2018-09-28 15:37:28 +01:00
Richard van der Hoff 3deaad2fb4
Merge pull request #3964 from matrix-org/rav/remove_localhost_checks 
remove spurious federation checks on localhost
 2018-09-28 13:35:47 +01:00
Richard van der Hoff 965154d60a Fix complete fail to do the right thing 2018-09-28 12:45:54 +01:00
Richard van der Hoff 19475cf337 Remove redundant call to start_get_pdu_cache 
I think this got forgotten in #3932. We were getting away with it because it
was the last call in this function.
 2018-09-28 12:01:23 +01:00
Richard van der Hoff 9c8cec5dab Merge remote-tracking branch 'origin/develop' into erikj/destination_retry_cache 2018-09-28 10:51:09 +01:00
Matthew Hodgson 07340cdaca untested stab at autocreating autojoin rooms 2018-09-28 01:42:53 +01:00
Richard van der Hoff f094f715cf Merge remote-tracking branch 'origin/develop' into rav/fix_federation_errors 2018-09-27 15:18:21 +01:00
Richard van der Hoff 36c62a67c4
Merge pull request #3794 from matrix-org/erikj/faster_typing 
Improve performance of getting typing updates for replication
 2018-09-27 15:14:26 +01:00
Richard van der Hoff e1e3e77bfd
Merge pull request #3967 from matrix-org/rav/federation_handler_cleanups 
Clarifications in FederationHandler
 2018-09-27 15:06:59 +01:00
Amber Brown 6de3884e5e
Merge pull request #3965 from matrix-org/rav/notify_app_services_bg_process 
Run notify_app_services as a bg process
 2018-09-27 23:40:30 +10:00
Amber Brown a512e637ac
Merge pull request #3970 from schnuffle/develop-py3 
Replaced all occurences of e.message with str(e)
 2018-09-27 23:39:45 +10:00
Amber Brown b3064532d0
Run our oldest supported configuration in CI (#3952) 2018-09-27 23:21:54 +10:00
Schnuffle dc5db01ff2 Replaced all occurences of e.message with str(e) 
Signed-off-by: Schnuffle  <schnuffle@github.com>
 2018-09-27 13:38:50 +02:00
Richard van der Hoff 51d33d5178
Merge pull request #3961 from matrix-org/neilj/lock_mau_upserts 
fix #3854 MAU transaction errors
 2018-09-27 12:31:37 +01:00
Richard van der Hoff 333bee27f5 Include event when resolving state for missing prevs 
If we have a forward extremity for a room as `E`, and you receive `A`, `B`,
s.t. `A -> B -> E`, and `B` also points to an unknown event `X`, then we need
to do state res between `X` and `E`.

When that happens, we need to make sure we include `X` in the state that goes
into the state res alg.

Fixes #3934.
 2018-09-27 11:37:39 +01:00
Richard van der Hoff bd61c82bdf Include state from remote servers in pdu handling 
If we've fetched state events from remote servers in order to resolve the state
for a new event, we need to actually pass those events into
resolve_events_with_factory (so that it can do the state res) and then persist
the ones we need - otherwise other bits of the codebase get confused about why
we have state groups pointing to non-existent events.
 2018-09-27 11:37:39 +01:00
Richard van der Hoff a215b698c4 Fix "unhashable type: 'list'" exception in federation handling 
get_state_groups returns a map from state_group_id to a list of FrozenEvents,
so was very much the wrong thing to be putting as one of the entries in the
list passed to resolve_events_with_factory (which expects maps from
(event_type, state_key) to event id).

We actually want get_state_groups_ids().values() rather than
get_state_groups().

This fixes the main problem in #3923, but there are other problems with this
bit of code which get discovered once you do so.
 2018-09-27 11:37:39 +01:00
Richard van der Hoff 28223841e0 more comments 2018-09-27 11:31:51 +01:00
Richard van der Hoff e3c159863d Clarifications in FederationHandler 
* add some comments on things that look a bit bogus
* rename this `state` variable to avoid confusion with the `state` used
  elsewhere in this function. (There was no actual conflict, but it was
  a confusing bit of spaghetti.)
 2018-09-27 11:31:51 +01:00
Richard van der Hoff 92abd3d6d6
Merge pull request #3966 from matrix-org/rav/rx_txn_logging_2 
Logging improvements
 2018-09-27 11:27:46 +01:00
Richard van der Hoff 4a15a3e4d5
Include eventid in log lines when processing incoming federation transactions (#3959) 
when processing incoming transactions, it can be hard to see what's going on,
because we process a bunch of stuff in parallel, and because we may end up
recursively working our way through a chain of three or four events.

This commit creates a way to use logcontexts to add the relevant event ids to
the log lines.
 2018-09-27 11:25:34 +01:00
Richard van der Hoff ae6ad4cf41
docstrings and unittests for storage.state (#3958) 
I spent ages trying to figure out how I was going mad...
 2018-09-27 11:22:25 +01:00
Travis Ralston 82fa31799c Remove debugging statement 2018-09-26 14:01:02 -06:00
Travis Ralston b4c3bc1734 Handle HttpResponseException more safely for federated groups 2018-09-26 13:48:04 -06:00
Travis Ralston 219606a6ed Fix exception documentation in matrixfederationclient.py 2018-09-26 13:26:27 -06:00
Richard van der Hoff e70b4ce069 Logging improvements 
Some logging tweaks to help with debugging incoming federation transactions
 2018-09-26 17:36:14 +01:00
Richard van der Hoff a87d419a85 Run notify_app_services as a bg process 
This ensures that its resource usage metrics get recorded somewhere rather than
getting lost.

(It also fixes an error when called from a nested logging context which
completes before the bg process)
 2018-09-26 17:00:40 +01:00
Richard van der Hoff 9453c65948 remove spurious federation checks on localhost 
There's really no point in checking for destinations called "localhost" because
there is nothing stopping people creating other DNS entries which point to
127.0.0.1. The right fix for this is
https://github.com/matrix-org/synapse/issues/3953.

Blocking localhost, on the other hand, means that you get a surprise when
trying to connect a test server on localhost to an existing server (with a
'normal' server_name).
 2018-09-26 16:53:52 +01:00
Richard van der Hoff 607eec0456 fix docstring for FederationClient.get_state_for_room 
trivial fixes for docstring
 2018-09-26 16:52:24 +01:00
Neil Johnson 28781b65e7 fix #3854 2018-09-26 16:16:41 +01:00
Richard van der Hoff 8afddf7afe Fix error handling for missing auth_event 
When we were authorizing an event, if there was no `m.room.create` in its
auth_events, we would raise a SynapseError with a cryptic message, which then
meant that we would bail out of processing any incoming events, rather than
storing a rejection for the faulty event and moving on.

We should treat the absent event the same as any other auth failure, by
raising an AuthError, so that the event is marked as rejected.
 2018-09-26 14:40:16 +01:00
Richard van der Hoff 4e8276a34a
Merge pull request #3956 from matrix-org/rav/fix_expiring_cache_len 
Fix ExpiringCache.__len__ to be accurate
 2018-09-26 13:10:13 +01:00
Richard van der Hoff 5b4028fa78 Merge branch 'rav/fix_expiring_cache_len' into erikj/destination_retry_cache 2018-09-26 12:55:53 +01:00
Richard van der Hoff 7ee94fc1ba Log which cache is throwing exceptions 2018-09-26 12:43:08 +01:00
Amber Brown 66a1d57adb
Merge pull request #3948 from matrix-org/rav/no_symlink_synctl 
Move synctl into top dir to avoid a symlink
 2018-09-26 21:41:58 +10:00
Amber Brown c2185f14d7
Merge pull request #3924 from matrix-org/rav/clean_up_on_receive_pdu 
Comments and interface cleanup for on_receive_pdu
 2018-09-26 21:41:26 +10:00
Erik Johnston 3baf6e1667 Fix ExpiringCache.__len__ to be accurate 
It used to try and produce an estimate, which was sometimes negative.
This caused metrics to be sad, so lets always just calculate it from
scratch.

(This appears to have been a longstanding bug, but one which has been made more
of a problem by #3932 and #3933).

(This was originally done by Erik as part of #3933. I'm cherry-picking it
because really it's a fix in its own right)
 2018-09-26 12:32:29 +01:00
Richard van der Hoff a1cd37390f Merge remote-tracking branch 'origin/develop' into erikj/destination_retry_cache 2018-09-25 12:03:54 +01:00
Richard van der Hoff 4c3e7eeec5
Merge pull request #3932 from matrix-org/erikj/auto_start_expiring_caches 
Fix some instances of ExpiringCache not expiring cache items
 2018-09-25 12:02:57 +01:00
Jérémy Farnaud 6cf261930a added "media-src: 'self'" to CSP for resources (#3578) 
Synapse doesn’t allow for media resources to be played directly from
Chrome. It is a problem for users on other networks (e.g. IRC)
communicating with Matrix users through a gateway. The gateway sends
them the raw URL for the resource when a Matrix user uploads a video
and the video cannot be played directly in Chrome using that URL.

Chrome argues it is not authorized to play the video because of the
Content Security Policy. Chrome checks for the "media-src" policy which
is missing, and defauts to the "default-src" policy which is "none".

As Synapse already sends "object-src: 'self'" I thought it wouldn’t be
a problem to add "media-src: 'self'" to the CSP to fix this problem.
 2018-09-25 11:55:02 +01:00
Richard van der Hoff 94f7befc31
Merge pull request #3925 from matrix-org/erikj/fix_producers_unregistered 
Fix spurious exceptions when client closes conncetion
 2018-09-25 11:52:06 +01:00
Richard van der Hoff c53336986d Move synctl into top dir to avoid a symlink 
symlinks apparently break setuptools on python3 and alpine
(https://bugs.python.org/issue31940), so let's stop using a symlink and just
use the file directly.
 2018-09-25 11:19:27 +01:00
Richard van der Hoff a9d84f4e44 We require attrs 16.0.0 
Ref: https://github.com/matrix-org/synapse/issues/3945
 2018-09-25 10:43:39 +01:00
Matthew Hodgson 787d22ed6c
Only lazy load self-members on initial sync 
Given we have disabled lazy loading for incr syncs in #3840, we can make self-LL more efficient by only doing it on initial sync.  Also adds a bounds check for if/when we change our mind, so that we don't try to include LL members on sync responses with no timeline.
 2018-09-25 00:49:26 +01:00
Amber Brown fbe5ba25f6 Merge branch 'master' into develop 2018-09-25 03:10:01 +10:00
Amber Brown 6b6cb32297 bump version 2018-09-25 02:54:34 +10:00
Amber Brown 04eed80a73 Merge branch 'master' into develop 2018-09-24 23:42:25 +10:00
Amber Brown e302f40e20 update version 2018-09-24 23:40:05 +10:00
Erik Johnston 19dc676d1a Fix ExpiringCache.__len__ to be accurate 
It used to try and produce an estimate, which was sometimes negative.
This caused metrics to be sad, so lets always just calculate it from
scratch.
 2018-09-21 16:25:42 +01:00
Erik Johnston fdd1a62e8d Add a five minute cache to get_destination_retry_timings 
Hopefully helps with #3931
 2018-09-21 14:56:12 +01:00
Erik Johnston 79eded1ae4 Make ExpiringCache slightly more performant 2018-09-21 14:52:21 +01:00
Erik Johnston 8601c24287 Fix some instances of ExpiringCache not expiring cache items 
ExpiringCache required that `start()` be called before it would actually
start expiring entries. A number of places didn't do that.

This PR removes `start` from ExpiringCache, and automatically starts
backround reaping process on creation instead.
 2018-09-21 14:19:46 +01:00
Erik Johnston ad53a5497d
Merge pull request #3927 from matrix-org/erikj/handle_background_errors 
Handle exceptions thrown by background tasks
 2018-09-21 09:26:30 +01:00
Matthew Hodgson a2ddaa90f2
Always LL ourselves if we're in a room to simplify clients (#3916) 
Should fix https://github.com/vector-im/riot-web/issues/7209
 2018-09-20 21:21:54 +01:00
Erik Johnston 94ae1dea3c Add missing logger 2018-09-20 17:05:34 +01:00
Erik Johnston 9ea408441f Handle exceptions thrown by background tasks 
Fixes #3921
 2018-09-20 16:15:21 +01:00
Erik Johnston b28a7ed503 Fix spurious exceptions when client closes conncetion 
If a HTTP handler throws an exception while processing a request we
automatically write a JSON error response. If the handler had already
started writing a response twisted throws an exception.

We should check for this case and simple abort the connection if there
was an error after the response had started being written.
 2018-09-20 13:44:20 +01:00
Neil Johnson 23b53b4ef8
Merge pull request #3868 from matrix-org/neilj/fix_room_invite_mail_links 
Neilj/fix room invite mail links
 2018-09-20 13:32:38 +01:00
Richard van der Hoff 703de4ec13 Comments and interface cleanup for on_receive_pdu 
Add some informative comments about what's going on here.

Also, `sent_to_us_directly` and `get_missing` were doing the same thing (apart
from in `_handle_queued_pdus`, which looks like a bug), so let's get rid of
`get_missing` and use `sent_to_us_directly` consistently.
 2018-09-20 13:06:55 +01:00
Amber Brown 1f3f5fcf52
Fix client IPs being broken on Python 3 (#3908) 2018-09-20 20:14:34 +10:00
Erik Johnston 3fd68d533b
Merge pull request #3914 from matrix-org/erikj/remove_retry_cache 
Remove get_destination_retry_timings cache
 2018-09-20 10:54:49 +01:00
Amber Brown aeca5a5ed5
Add a regression test for logging on failed connections (#3912) 2018-09-20 16:28:18 +10:00
Richard van der Hoff 642199570c
Improve the logging when handling a federation transaction (#3904) 
Let's try to rationalise the logging that happens when we are processing an
incoming transaction, to make it easier to figure out what is going wrong when
they take ages. In particular:

- make everything start with a [room_id event_id] prefix
- make sure we log a warning when catching exceptions rather than just turning
  them into other, more cryptic, exceptions.
 2018-09-19 17:28:18 +01:00
Erik Johnston ce846bb620 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/faster_typing 2018-09-19 15:08:36 +01:00
Erik Johnston bbab6ebfd9 Fix up changelog and remove spurious comment 2018-09-19 14:45:14 +01:00
Erik Johnston 392a54128c pep8 2018-09-19 14:37:49 +01:00
Erik Johnston b9158ac2bf Remove get_destination_retry_timings cache 
Currently we rely on the master to invalidate this cache promptly.
However, after having moved most federation endpoints off of master this
no longer happens, causing outbound fedeariont to get blackholed.

Fixes #3798
 2018-09-19 14:22:57 +01:00
Erik Johnston 80d2d50f47 Fixup 2018-09-19 11:19:47 +01:00
Erik Johnston 9407bcf37a Replace custom DeferredTimeoutError with defer.TimeoutError 2018-09-19 11:07:29 +01:00
Erik Johnston 6c48aa0256 Run canceller first to allow it to generate correct error 2018-09-19 11:07:27 +01:00
Erik Johnston a334e1cace Update to use new timeout function everywhere. 
The existing deferred timeout helper function (and the one into twisted)
suffer from a bug when a deferred's canceller throws an exception, #3842.

The new helper function doesn't suffer from this problem.
 2018-09-19 10:39:40 +01:00
Amber Brown 47c02e6332
Merge pull request #3909 from turt2live/travis/fix-logging-1 
Fix matrixfederationclient.py logging: Destination is a string
 2018-09-19 18:14:47 +10:00
Amber Brown 3d6b24fb1b
Merge pull request #3907 from matrix-org/rav/set_sni_to_server_name 
Set SNI to the server_name, not whatever was in the SRV record
 2018-09-19 17:59:33 +10:00
Amber Brown f773ecbd61
Merge pull request #3903 from matrix-org/rav/increase_get_missing_events_timeout 
Bump timeout on get_missing_events request
 2018-09-19 17:57:48 +10:00
Travis Ralston 35aec19f0a Destination is a string 2018-09-18 15:29:30 -06:00
Richard van der Hoff 38ead946a9 Merge remote-tracking branch 'origin/develop' into neilj/fix_room_invite_mail_links 2018-09-18 19:02:45 +01:00
Richard van der Hoff a219ce8726
Use directory server for room joins (#3899) 
When we do a join, always try the server we used for the alias lookup first.

Fixes #2418
 2018-09-18 18:27:37 +01:00
Richard van der Hoff 31c15dcb80
Refactor matrixfederationclient to fix logging (#3906) 
We want to wait until we have read the response body before we log the request
as complete, otherwise a confusing thing happens where the request appears to
have completed, but we later fail it.

To do this, we factor the salient details of a request out to a separate
object, which can then keep track of the txn_id, so that it can be logged.
 2018-09-18 18:17:15 +01:00
Amber Brown c600886d47
Merge pull request #3894 from matrix-org/hs/phone_home_py_version 
Add python_version phone home stat
 2018-09-19 02:40:04 +10:00
Richard van der Hoff b3097396e7 Set SNI to the server_name, not whatever was in the SRV record 
Fixes #3843
 2018-09-18 17:01:12 +01:00
Richard van der Hoff 550007cb0e Bump timeout on get_missing_events request 2018-09-18 15:02:51 +01:00
Richard van der Hoff 286d6930b7
Merge pull request #3879 from matrix-org/matthew/fix-autojoin 
don't ratelimit autojoins
 2018-09-18 13:07:01 +01:00
Richard van der Hoff 1e09a1d48a
Merge pull request #3889 from matrix-org/rav/404_on_remove_unknown_alias 
Return a 404 when deleting unknown room alias
 2018-09-18 12:59:30 +01:00
Will Hunt 5baa087312
typo 2018-09-17 17:37:56 +01:00
Will Hunt b58714789f
make pip happy? 2018-09-17 17:35:54 +01:00
Richard van der Hoff ac80cb08fe Fix more b'abcd' noise in metrics 2018-09-17 17:16:50 +01:00
Will Hunt 9a1cceeca9
Use a string for versions 2018-09-17 17:09:06 +01:00
Richard van der Hoff f75b9961c6 Reinstate missing null check 2018-09-17 16:52:02 +01:00
Will Hunt 2b39494cd5
Add python_version phone home stat 2018-09-17 16:35:18 +01:00
Richard van der Hoff f00a9d2636 Fix some b'abcd' noise in logs and metrics 
Python 3 compatibility: make sure that we decode some byte sequences before we
use them to create log lines and metrics labels.
 2018-09-17 16:15:42 +01:00
Amber Brown fe88907d04 version 2018-09-17 22:33:22 +10:00
Richard van der Hoff 85a43f4167 Return a 404 when deleting unknown room alias 
As per https://github.com/matrix-org/matrix-doc/issues/1675

Fixes https://github.com/matrix-org/synapse/issues/2782
 2018-09-17 13:19:00 +01:00
Matthew Hodgson d42d79e3c3 don't ratelimit autojoins 2018-09-15 22:27:41 +01:00
Erik Johnston 24efb2a70d Fix timeout function 
Turns out deferred.cancel sometimes throws, so we do that last to ensure
that we always do resolve the new deferred.
 2018-09-15 11:38:39 +01:00
Erik Johnston fcfe7a850d Add an awful secondary timeout to fix wedged requests 
This is an attempt to mitigate #3842 by adding yet-another-timeout
 2018-09-14 19:23:07 +01:00
Matthew Hodgson 024be6cf18
don't filter membership events based on history visibility (#3874) 
don't filter membership events based on history visibility
as we will already have filtered the messages in the timeline, and state events
are always visible.

and because @erikjohnston said so.
 2018-09-14 18:12:52 +01:00
Erik Johnston 3e6e94fe9f
Merge pull request #3872 from matrix-org/hawkowl/timeouts-2 
timeouts 2: electric boogaloo
 2018-09-14 16:58:44 +01:00
Amber Brown bc9af88a2d fix 2018-09-15 00:26:00 +10:00
Erik Johnston d0f6c1ce21 Remove spurious comment 2018-09-14 15:12:36 +01:00
Erik Johnston 9e2f9a7b57 Measure outbound requests 2018-09-14 15:11:26 +01:00
Erik Johnston 0a81038ea0 Add in flight real time metrics for Measure blocks 2018-09-14 15:08:37 +01:00
Neil Johnson 7de1989ea2 fix link for case that config.email_riot_base_url is set 2018-09-13 22:43:50 +01:00
Amber Brown c971aa7b9d fix 2018-09-14 03:57:02 +10:00
Amber Brown 8f08d848f5 fix 2018-09-14 03:53:56 +10:00
Travis Ralston f1a7264663
Fix minor typo in exception 2018-09-13 11:51:12 -06:00
Amber Brown 7c33ab76da redact better 2018-09-14 03:45:34 +10:00
Amber Brown 63755fa4c2 we do that higher up 2018-09-14 03:21:47 +10:00
Amber Brown 73884ebac5 Merge remote-tracking branch 'origin/develop' into hawkowl/timeouts-2 2018-09-14 03:11:25 +10:00
Amber Brown 7c27c4d51c
merge (#3576) 2018-09-14 03:11:11 +10:00
Amber Brown 1c3f4d9ca5 buffer? 2018-09-14 03:09:13 +10:00
David Baker bc74925c5b WIP e2e key backups 
Continues from uhoreg's branch

This just fixed the errcode on /room_keys/version if no backup and
updates the schema delta to be on the latest so it gets run
 2018-09-13 17:02:59 +01:00
Erik Johnston 6c0f8d9d50
Merge pull request #3856 from matrix-org/erikj/speed_up_purge 
Make purge history slightly faster
 2018-09-13 16:14:46 +01:00
Erik Johnston ed5331a627 comment 2018-09-13 16:10:56 +01:00
Erik Johnston 89a76d1889 Fix handling of redacted events from federation 
If we receive an event that doesn't pass their content hash check (e.g.
due to already being redacted) then we hit a bug which causes an
exception to be raised, which then promplty stops the event (and
request) from being processed.

This effects all sorts of federation APIs, including joining rooms with
a redacted state event.
 2018-09-13 15:44:12 +01:00
Amber Brown bfa0b759e0
Attempt to figure out what's going on with timeouts (#3857) 2018-09-14 00:15:51 +10:00
Erik Johnston 9cbd0094f0 pep8 2018-09-13 15:15:35 +01:00
Erik Johnston 9dbe38ea7d Create indices after insertion 2018-09-13 15:05:52 +01:00
Erik Johnston 93139a1fb8 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/speed_up_purge 2018-09-13 12:57:09 +01:00
Erik Johnston c857f5ef9b Make purge history slightly faster 
Don't pull out events that are outliers and won't be deleted, as nothing
should happen to them.
 2018-09-13 12:48:10 +01:00
Amber Brown b7d2fb5eb9
Remove some superfluous logging (#3855) 2018-09-13 19:59:32 +10:00
Neil Johnson f30a303590
Merge pull request #3846 from matrix-org/neilj/expose-registered-users 
expose number of real reserved users
 2018-09-12 17:14:04 +01:00
Matthew Hodgson 2ac1abbc7e
show heroes if a room has a 'deleted' name/canonical_alias (#3851) 2018-09-12 17:11:05 +01:00
Erik Johnston fa0d464fa4
Merge pull request #3853 from matrix-org/erikj/log_outbound_each_time 
Log outbound requests when we retry
 2018-09-12 16:55:40 +01:00
Matthew Hodgson 0403cf0783 argh pep8 2018-09-12 16:54:28 +01:00
Matthew Hodgson 0e200e366d correctly log gappy sync metrics 2018-09-12 16:47:20 +01:00
Matthew Hodgson 11bfc2af1c fix logline 2018-09-12 16:45:42 +01:00
Neil Johnson 8decd6233d improve naming 2018-09-12 16:22:15 +01:00
Erik Johnston 8c5b84441b Log outbound requests when we retry 2018-09-12 16:22:14 +01:00
Erik Johnston 54f8616d2c
Merge pull request #3841 from matrix-org/erikj/manhole_key_length 
Change the manhole SSH key to have more bits
 2018-09-12 14:33:39 +01:00
Amber Brown 7ca097f77e
Port federation/ to py3 (#3847) 2018-09-12 23:23:32 +10:00
Neil Johnson 0ddf486724 expose number of real reserved users 2018-09-12 11:58:52 +01:00
Amber Brown 546aee7e52
Merge pull request #3835 from krombel/fix_3821 
fix VOIP crashes under Python 3
 2018-09-12 20:44:18 +10:00
Amber Brown 33716c4aea
Merge pull request #3826 from matrix-org/rav/logging_for_keyring 
add some logging for the keyring queue
 2018-09-12 20:43:47 +10:00
Amber Brown bc635026c5
Merge pull request #3824 from matrix-org/rav/fix_jwt_import 
Fix jwt import check
 2018-09-12 20:41:57 +10:00
Amber Brown 02aa41809b
Port rest/ to Python 3 (#3823) 2018-09-12 20:41:31 +10:00
Amber Brown 8fd93b5eea
Port crypto/ to Python 3 (#3822) 2018-09-12 20:16:31 +10:00
Erik Johnston 4084a774a8 Timeout reading body for outbound HTTP requests 2018-09-12 10:10:20 +01:00
Matthew Hodgson b041115415
Speed up lazy loading (#3827) 
* speed up room summaries by pulling their data from room_memberships rather than room state
* disable LL for incr syncs, and log incr sync stats  (#3840)
 2018-09-12 00:50:39 +01:00
Erik Johnston 9e05c8d309 Change the manhole SSH key to have more bits 
Newer versions of openssh client refuse to connect to the old key due to
its length.
 2018-09-11 10:42:10 +01:00
Krombel 7bc22539ff fix VOIP crashes under Python 3 (#3821) 2018-09-10 14:30:08 +02:00
Amber Brown 9a5ea511b5
Merge pull request #3810 from matrix-org/erikj/send_tags_down_sync_on_join 
Send existing room tags down sync on join
 2018-09-07 23:28:42 +10:00
Richard van der Hoff b8ad756bd0 Fix jwt import check 
This handy code attempted to check that we could import jwt, but utterly failed
to check it was the right jwt.

Fixes https://github.com/matrix-org/synapse/issues/3793
 2018-09-07 14:20:54 +01:00
Amber Brown 771d213ac5 Merge branch 'master' into develop 2018-09-07 21:45:38 +10:00
Amber Brown 6febd8e8f7 version 2018-09-07 21:40:57 +10:00
Richard van der Hoff cd7ef43872 clearer logging when things fail, too 2018-09-06 23:56:47 +01:00
Richard van der Hoff 806964b5de add some logging for the keyring queue 
why is it so damn slow?
 2018-09-06 18:51:06 +01:00
Neil Johnson c5440b2ca0
Merge pull request #3800 from matrix-org/neilj/remove-guests-from-mau-count 
guest users should not be part of mau total
 2018-09-06 17:45:55 +01:00
Neil Johnson 84a750e0c3 ensure guests never enter mau list 2018-09-06 17:22:53 +01:00
Erik Johnston f60c9e2a01 Don't send empty tags list down sync 2018-09-06 17:01:41 +01:00
Erik Johnston 7baf66ef5d Send existing room tags down sync on join 
When a user joined a room any existing tags were not sent down the sync
stream. Ordinarily this isn't a problem because the user needs to be in
the room to have set tags in it, however synapse will sometimes add tags
for a user to a room, e.g. for server notices, which need to come down
sync.
 2018-09-06 16:46:51 +01:00
Hubert Chathi 3801b8aa03 try to make flake8 and isort happy 2018-09-06 11:35:19 -04:00
Amber Brown 654324eded
Merge pull request #3805 from matrix-org/erikj/limit_transaction_pdus_edus 
Limit the number of PDUs/EDUs per fedreation transaction
 2018-09-07 01:33:31 +10:00
Amber Brown 70fd75cd1d
Merge pull request #3788 from matrix-org/erikj/remove_conn_id 
Remove conn_id from repl prometheus metrics
 2018-09-07 00:48:19 +10:00
Amber Brown ee7c8bd2b5
Merge pull request #3795 from matrix-org/erikj/faster_sync_state 
User iter* during sync state calculations
 2018-09-07 00:24:43 +10:00
Erik Johnston 6707a3212c Limit the number of PDUs/EDUs per fedreation transaction 2018-09-06 15:23:55 +01:00
Amber Brown 135f3b4390
Merge pull request #3804 from matrix-org/rav/fix_openssl_dep 
bump dep on pyopenssl to 16.x
 2018-09-07 00:23:39 +10:00
Amber Brown 2608ebc04c
Port handlers/ to Python 3 (#3803) 2018-09-07 00:22:23 +10:00
Erik Johnston 417e7077aa Bump version and changelog 2018-09-06 14:08:55 +01:00
Erik Johnston d64b24dfe6 Synapse 0.33.3.1 (2018-09-06) 
=============================
 
 SECURITY FIXES
 --------------
 
 - Fix an issue where event signatures were not always correctly validated ([\#3796](https://github.com/matrix-org/synapse/issues/3796))
 - Fix an issue where server_acls could be circumvented for incoming events ([\#3796](https://github.com/matrix-org/synapse/issues/3796))
 
 Internal Changes
 ----------------
 
 - Unignore synctl in .dockerignore to fix docker builds ([\#3802](https://github.com/matrix-org/synapse/issues/3802))
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2
 
 iQEcBAABCAAGBQJbkPLrAAoJEIofk9V1tejV7K4IAItYIX98DKN9x3FNs9Hd69Pw
 mZtqmuJ12YAyiVZoR7IJ5GfyctHCYUUXcqmNR1O+2/IRvezFnU6ZTcPW3OfNfnuD
 vlnMiK53F21T96ul3Wu47Z2wyO+WkoKeXdvlqt3Wa8HpnPbU6y5CSac2vK57ppTU
 DcMZKXaDae6vA7bsjryYgTRRplH6eFWZEexjbuZudbDvqkySi2zGUTs6SUesMC5B
 FRPgI6p6sCeQgGMJY+d1i+ZdzWkZhH5OLH2icf2MI1hnhSnRSJ/scSGm5OYDCwYy
 03hOy0ZVMr8SB5/j0TjSaFXonKsmPKqxnU2g7crJmvFP4BfJrEbym0y9YhYXnVQ=
 =8PDU
 -----END PGP SIGNATURE-----

Merge tag 'v0.33.3.1' into release-v0.33.4

Synapse 0.33.3.1 (2018-09-06)
=============================

SECURITY FIXES
--------------

- Fix an issue where event signatures were not always correctly validated ([\#3796](https://github.com/matrix-org/synapse/issues/3796))
- Fix an issue where server_acls could be circumvented for incoming events ([\#3796](https://github.com/matrix-org/synapse/issues/3796))

Internal Changes
----------------

- Unignore synctl in .dockerignore to fix docker builds ([\#3802](https://github.com/matrix-org/synapse/issues/3802))
 2018-09-06 14:08:33 +01:00
Richard van der Hoff 4f8baab0c4 Merge branch 'master' into develop 2018-09-06 13:05:22 +01:00
Richard van der Hoff 625542878d bump dep on pyopenssl to 16.x 2018-09-06 12:53:15 +01:00
Richard van der Hoff 80189ed27c prepare v0.33.3.1 2018-09-06 10:26:23 +01:00
Neil Johnson 61b05727fa guest users should not be part of mau total 2018-09-05 22:30:36 +01:00
Richard van der Hoff dfba1d843d
Merge pull request #3790 from matrix-org/rav/respect_event_format_in_filter 
Implement 'event_format' filter param in /sync
 2018-09-05 16:24:14 +01:00
Erik Johnston 7419764351 User iter* during sync state calculations 2018-09-05 16:19:50 +01:00
Amber Brown 2d2828dcbc
Port http/ to Python 3 (#3771) 2018-09-06 00:10:47 +10:00
Richard van der Hoff c127c8d042 Fix origin handling for pushed transactions 
Use the actual origin for push transactions, rather than whatever the remote
server claimed.
 2018-09-05 13:08:07 +01:00
Richard van der Hoff 804dd41e18 Check that signatures on events are valid 
We should check that both the sender's server, and the server which created the
event_id (which may be different from whatever the remote server has told us
the origin is), have signed the event.
 2018-09-05 13:08:07 +01:00
Erik Johnston 5f02017aea Improve performance of getting typing updates for replication 
Fetching the list of all new typing notifications involved iterating
over all rooms and comparing their serial. Lets move to using a stream
change cache, like we do for other streams.
 2018-09-05 10:20:40 +01:00
Richard van der Hoff 87c18d12ee Implement 'event_format' filter param in /sync 
This has been specced and part-implemented; let's implement it for /sync (but
no other endpoints yet :/).
 2018-09-04 15:20:09 +01:00
Neil Johnson a6cf7d9d9a
Merge pull request #3789 from matrix-org/neilj/improve_threepid_error_strings 
improve human readable error messages
 2018-09-04 13:16:00 +00:00
Amber Brown 7e9ced4178 version and towncrier 2018-09-04 21:12:04 +10:00
Neil Johnson bae37cd811 improve human readable error message 2018-09-04 12:07:00 +01:00
Neil Johnson c42f7fd7b9 improve human readable error messages 2018-09-04 12:03:17 +01:00
Erik Johnston 3e242dc149 Remove conn_id 2018-09-04 11:45:52 +01:00
Erik Johnston b13836da7f Remove conn_id from repl prometheus metrics 
`conn_id` gets set to a random string, and so we end up filling up
prometheus with tonnes of data series, which is bad.
 2018-09-03 17:22:49 +01:00
Erik Johnston 567363e497
Merge pull request #3737 from matrix-org/erikj/remove_redundant_state_func 
Remove unnecessary resolve_events_with_state_map
 2018-09-03 16:19:41 +01:00
Neil Johnson 301cb60d0b assert rather than warn 2018-08-31 17:29:35 +01:00
Neil Johnson 0b01281e77 move threepid checker to config, add missing yields 2018-08-31 17:11:11 +01:00
Neil Johnson e8e540630e fix reference to is_threepid_reserved 2018-08-31 16:09:15 +01:00
Neil Johnson 09f3cf1a7e ensure post registration auth checks do not fail erroneously 2018-08-31 15:42:51 +01:00
Neil Johnson ea068d6f3c fix bug where preserved threepid user comes to sign up and server is mau blocked 2018-08-31 10:49:14 +01:00
Amber Brown 14e4d4f4bf
Port storage/ to Python 3 (#3725) 2018-08-31 00:19:58 +10:00
Richard van der Hoff 475253a88e
Merge pull request #3764 from matrix-org/rav/close_db_conn_after_init 
Make sure that we close db connections opened during init
 2018-08-30 10:47:27 +01:00
Krombel 79a8a347a6 fix #3445 
itervalues(d) calls d.itervalues() [PY2] and d.values() [PY3]
but SortedDict only implements d.values()
 2018-08-29 16:28:25 +02:00
Matthew Hodgson b1580f50fe
don't return non-LL-member state in incremental sync state blocks (#3760) 
don't return non-LL-member state in incremental sync state blocks
 2018-08-28 23:25:58 +01:00
Richard van der Hoff 32eb1dedd2 use abc.abstractproperty 
This gives clearer messages when someone gets it wrong
 2018-08-28 17:10:43 +01:00
Richard van der Hoff 0b07f02e19 Make sure that we close db connections opened during init 
We should explicitly close any db connections we open, because failing to do so
can block other transactions as per
https://github.com/matrix-org/synapse/issues/3682.

Let's also try to factor out some of the boilerplate by having server classes
define their datastore class rather than duplicating the whole of `setup`.
 2018-08-28 13:39:49 +01:00
Hubert Chathi 16a31c6fce update to newer Synapse APIs 2018-08-24 22:51:25 -04:00
Erik Johnston 9db2476991 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_contact 2018-08-24 17:00:37 +01:00
Erik Johnston 05077e06fa Change admin_uri to admin_contact in config and errors 2018-08-24 16:51:27 +01:00
Hubert Chathi 83caead95a
Merge branch 'develop' into e2e_backups 2018-08-24 11:44:26 -04:00
Erik Johnston 01a5a8b9e3 Fix checking if service notice room is already tagged 
This manifested in synapse repeatedly setting the tag for the room
 2018-08-24 16:22:37 +01:00
Erik Johnston cdd24449ee Ensure we wake up /sync when we add tag to notice room 2018-08-24 14:50:03 +01:00
Erik Johnston 14d49c51db Make content of tag an empty object rather than null 2018-08-24 14:44:16 +01:00
Erik Johnston 84b4e76fed
Merge pull request #3754 from matrix-org/erikj/fix_whitelist 
Allow federation_domain_whitelist to be emtpy list
 2018-08-24 12:23:39 +01:00
Erik Johnston 1d67b13674 Fix bug when federation_domain_whitelist is an emtpy list 
Outbound federation were incorrectly allowed when the config option was
set to an empty list
 2018-08-24 12:13:12 +01:00
Erik Johnston 92d50e3c2a
Merge pull request #3753 from matrix-org/erikj/fix_no_server_noticse 
Fix bug where we broke sync when using limit_usage_by_mau
 2018-08-24 11:56:08 +01:00
Richard van der Hoff e94cdbaecf
Merge pull request #3751 from matrix-org/rav/twisted_17 
Pin to twisted 17.1 or later
 2018-08-24 11:55:52 +01:00
Erik Johnston 7f147d623b Fix bug where we broke sync when using limit_usage_by_mau 
We assumed that we always had service notices configured, but that is
not always true
 2018-08-24 11:33:50 +01:00
Richard van der Hoff f584d6108f Pin to twisted 17.1 or later 
Fixes https://github.com/matrix-org/synapse/issues/3741.
 2018-08-24 10:02:31 +01:00
Erik Johnston cd77270a66 Implement trail users 2018-08-23 19:17:19 +01:00
Erik Johnston 7e6e588e60 Fix bug where we resent "limit exceeded" server notices 
This was due to a bug where we mutated a cached event's contents
 2018-08-23 16:21:20 +01:00
Erik Johnston 7a0da69eee Add missing yield 2018-08-23 10:28:12 +01:00
Erik Johnston fd2dbf1836 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-22 17:06:10 +01:00
Erik Johnston 9643a6f7f2 Update notice format 2018-08-22 17:00:29 +01:00
Erik Johnston 5c261107c9 Remove unnecessary resolve_events_with_state_map 
We only ever used the synchronous resolve_events_with_state_map in one
place, which is trivial to replace with the async version.
 2018-08-22 15:41:15 +01:00
Richard van der Hoff c7181dcc6c Merge branch 'master' into develop 2018-08-22 14:37:11 +01:00
Richard van der Hoff 74854a9719 Use recaptcha_ajax.js directly from Google 
This was originally done in commit c75b71a397,
but got reverted on this branch due to the PR (#3677) being based on the wrong
branch.

We're ready to merge this to master now, so let's make it match
release-v0.33.3.
 2018-08-22 14:30:49 +01:00
Richard van der Hoff 48fec67536 Features 
--------
 
 - Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! ([\#3439](https://github.com/matrix-org/synapse/issues/3439))
 - Add /_media/r0/config ([\#3184](https://github.com/matrix-org/synapse/issues/3184))
 - speed up /members API and add `at` and `membership` params as per MSC1227 ([\#3568](https://github.com/matrix-org/synapse/issues/3568))
 - implement `summary` block in /sync response as per MSC688 ([\#3574](https://github.com/matrix-org/synapse/issues/3574))
 - Add lazy-loading support to /messages as per MSC1227 ([\#3589](https://github.com/matrix-org/synapse/issues/3589))
 - Add ability to limit number of monthly active users on the server ([\#3633](https://github.com/matrix-org/synapse/issues/3633))
 - Support more federation endpoints on workers ([\#3653](https://github.com/matrix-org/synapse/issues/3653))
 - Basic support for room versioning ([\#3654](https://github.com/matrix-org/synapse/issues/3654))
 - Ability to disable client/server Synapse via conf toggle ([\#3655](https://github.com/matrix-org/synapse/issues/3655))
 - Ability to whitelist specific threepids against monthly active user limiting ([\#3662](https://github.com/matrix-org/synapse/issues/3662))
 - Add some metrics for the appservice and federation event sending loops ([\#3664](https://github.com/matrix-org/synapse/issues/3664))
 - Where server is disabled, block ability for locked out users to read new messages ([\#3670](https://github.com/matrix-org/synapse/issues/3670))
 - set admin uri via config, to be used in error messages where the user should contact the administrator ([\#3687](https://github.com/matrix-org/synapse/issues/3687))
 - Synapse's presence functionality can now be disabled with the "use_presence" configuration option. ([\#3694](https://github.com/matrix-org/synapse/issues/3694))
 - For resource limit blocked users, prevent writing into rooms ([\#3708](https://github.com/matrix-org/synapse/issues/3708))
 
 Bugfixes
 --------
 
 - Fix occasional glitches in the synapse_event_persisted_position metric ([\#3658](https://github.com/matrix-org/synapse/issues/3658))
 - Fix bug on deleting 3pid when using identity servers that don't support unbind API ([\#3661](https://github.com/matrix-org/synapse/issues/3661))
 - Make the tests pass on Twisted < 18.7.0 ([\#3676](https://github.com/matrix-org/synapse/issues/3676))
 - Don’t ship recaptcha_ajax.js, use it directly from Google ([\#3677](https://github.com/matrix-org/synapse/issues/3677))
 - Fixes test_reap_monthly_active_users so it passes under postgres ([\#3681](https://github.com/matrix-org/synapse/issues/3681))
 - Fix mau blocking calulation bug on login ([\#3689](https://github.com/matrix-org/synapse/issues/3689))
 - Fix missing yield in synapse.storage.monthly_active_users.initialise_reserved_users ([\#3692](https://github.com/matrix-org/synapse/issues/3692))
 - Improve HTTP request logging to include all requests ([\#3700](https://github.com/matrix-org/synapse/issues/3700))
 - Avoid timing out requests while we are streaming back the response ([\#3701](https://github.com/matrix-org/synapse/issues/3701))
 - Support more federation endpoints on workers ([\#3705](https://github.com/matrix-org/synapse/issues/3705), [\#3713](https://github.com/matrix-org/synapse/issues/3713))
 - Fix "Starting db txn 'get_all_updated_receipts' from sentinel context" warning ([\#3710](https://github.com/matrix-org/synapse/issues/3710))
 - Fix bug where `state_cache` cache factor ignored environment variables ([\#3719](https://github.com/matrix-org/synapse/issues/3719))
 - Fix bug in v0.33.3rc1 which caused infinite loops and OOMs ([\#3723](https://github.com/matrix-org/synapse/issues/3723))
 - Fix bug introduced in v0.33.3rc1 which made the ToS give a 500 error ([\#3732](https://github.com/matrix-org/synapse/issues/3732))
 
 Deprecations and Removals
 -------------------------
 
 - The Shared-Secret registration method of the legacy v1/register REST endpoint has been removed. For a replacement, please see [the admin/register API documentation](https://github.com/matrix-org/synapse/blob/master/docs/admin_api/register_api.rst). ([\#3703](https://github.com/matrix-org/synapse/issues/3703))
 
 Internal Changes
 ----------------
 
 - The test suite now can run under PostgreSQL. ([\#3423](https://github.com/matrix-org/synapse/issues/3423))
 - Refactor HTTP replication endpoints to reduce code duplication ([\#3632](https://github.com/matrix-org/synapse/issues/3632))
 - Tests now correctly execute on Python 3. ([\#3647](https://github.com/matrix-org/synapse/issues/3647))
 - Sytests can now be run inside a Docker container. ([\#3660](https://github.com/matrix-org/synapse/issues/3660))
 - Port over enough to Python 3 to allow the sytests to start. ([\#3668](https://github.com/matrix-org/synapse/issues/3668))
 - Update docker base image from alpine 3.7 to 3.8. ([\#3669](https://github.com/matrix-org/synapse/issues/3669))
 - Rename synapse.util.async to synapse.util.async_helpers to mitigate async becoming a keyword on Python 3.7. ([\#3678](https://github.com/matrix-org/synapse/issues/3678))
 - Synapse's tests are now formatted with the black autoformatter. ([\#3679](https://github.com/matrix-org/synapse/issues/3679))
 - Implemented a new testing base class to reduce test boilerplate. ([\#3684](https://github.com/matrix-org/synapse/issues/3684))
 - Rename MAU prometheus metrics ([\#3690](https://github.com/matrix-org/synapse/issues/3690))
 - add new error type ResourceLimit ([\#3707](https://github.com/matrix-org/synapse/issues/3707))
 - Logcontexts for replication command handlers ([\#3709](https://github.com/matrix-org/synapse/issues/3709))
 - Update admin register API documentation to reference a real user ID. ([\#3712](https://github.com/matrix-org/synapse/issues/3712))
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1
 
 iQEcBAABAgAGBQJbfWKDAAoJEIofk9V1tejVHOMH/RmP+axF2K+Socnnb5Fa6RTo
 pDJXVwsvNQxcwzZIJXh2TRilVr88jx7A1PHUde0TmkHRmis/bDzZxa59QRA4rFEy
 fLYMx9Tyt0RzAJrEd0p7jbIkJxxPm5PbMWfCgVfzg44q0kepRW6ZVl1+kFHDrsCo
 ocE03WdLe2sRHSudaVtBLOVMA1lDE22v40/7xiu9IbUOnF+WZ3cjXYX+SNr5eb6g
 7YGbXOk4/E8XoUnpzH5oEnNtiDK5piRrr7o1282kecY20ZMNzKgseBHU6nYMv5ZO
 TUpCPwbukmzeGHITupY1RkzpmBbEIJTCnmLP8fiUw+RWznYyFC/aRFMiWTzle3A=
 =ngmn
 -----END PGP SIGNATURE-----

Merge tag 'v0.33.3'

Features
--------

- Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! ([\#3439](https://github.com/matrix-org/synapse/issues/3439))
- Add /_media/r0/config ([\#3184](https://github.com/matrix-org/synapse/issues/3184))
- speed up /members API and add `at` and `membership` params as per MSC1227 ([\#3568](https://github.com/matrix-org/synapse/issues/3568))
- implement `summary` block in /sync response as per MSC688 ([\#3574](https://github.com/matrix-org/synapse/issues/3574))
- Add lazy-loading support to /messages as per MSC1227 ([\#3589](https://github.com/matrix-org/synapse/issues/3589))
- Add ability to limit number of monthly active users on the server ([\#3633](https://github.com/matrix-org/synapse/issues/3633))
- Support more federation endpoints on workers ([\#3653](https://github.com/matrix-org/synapse/issues/3653))
- Basic support for room versioning ([\#3654](https://github.com/matrix-org/synapse/issues/3654))
- Ability to disable client/server Synapse via conf toggle ([\#3655](https://github.com/matrix-org/synapse/issues/3655))
- Ability to whitelist specific threepids against monthly active user limiting ([\#3662](https://github.com/matrix-org/synapse/issues/3662))
- Add some metrics for the appservice and federation event sending loops ([\#3664](https://github.com/matrix-org/synapse/issues/3664))
- Where server is disabled, block ability for locked out users to read new messages ([\#3670](https://github.com/matrix-org/synapse/issues/3670))
- set admin uri via config, to be used in error messages where the user should contact the administrator ([\#3687](https://github.com/matrix-org/synapse/issues/3687))
- Synapse's presence functionality can now be disabled with the "use_presence" configuration option. ([\#3694](https://github.com/matrix-org/synapse/issues/3694))
- For resource limit blocked users, prevent writing into rooms ([\#3708](https://github.com/matrix-org/synapse/issues/3708))

Bugfixes
--------

- Fix occasional glitches in the synapse_event_persisted_position metric ([\#3658](https://github.com/matrix-org/synapse/issues/3658))
- Fix bug on deleting 3pid when using identity servers that don't support unbind API ([\#3661](https://github.com/matrix-org/synapse/issues/3661))
- Make the tests pass on Twisted < 18.7.0 ([\#3676](https://github.com/matrix-org/synapse/issues/3676))
- Don’t ship recaptcha_ajax.js, use it directly from Google ([\#3677](https://github.com/matrix-org/synapse/issues/3677))
- Fixes test_reap_monthly_active_users so it passes under postgres ([\#3681](https://github.com/matrix-org/synapse/issues/3681))
- Fix mau blocking calulation bug on login ([\#3689](https://github.com/matrix-org/synapse/issues/3689))
- Fix missing yield in synapse.storage.monthly_active_users.initialise_reserved_users ([\#3692](https://github.com/matrix-org/synapse/issues/3692))
- Improve HTTP request logging to include all requests ([\#3700](https://github.com/matrix-org/synapse/issues/3700))
- Avoid timing out requests while we are streaming back the response ([\#3701](https://github.com/matrix-org/synapse/issues/3701))
- Support more federation endpoints on workers ([\#3705](https://github.com/matrix-org/synapse/issues/3705), [\#3713](https://github.com/matrix-org/synapse/issues/3713))
- Fix "Starting db txn 'get_all_updated_receipts' from sentinel context" warning ([\#3710](https://github.com/matrix-org/synapse/issues/3710))
- Fix bug where `state_cache` cache factor ignored environment variables ([\#3719](https://github.com/matrix-org/synapse/issues/3719))
- Fix bug in v0.33.3rc1 which caused infinite loops and OOMs ([\#3723](https://github.com/matrix-org/synapse/issues/3723))
- Fix bug introduced in v0.33.3rc1 which made the ToS give a 500 error ([\#3732](https://github.com/matrix-org/synapse/issues/3732))

Deprecations and Removals
-------------------------

- The Shared-Secret registration method of the legacy v1/register REST endpoint has been removed. For a replacement, please see [the admin/register API documentation](https://github.com/matrix-org/synapse/blob/master/docs/admin_api/register_api.rst). ([\#3703](https://github.com/matrix-org/synapse/issues/3703))

Internal Changes
----------------

- The test suite now can run under PostgreSQL. ([\#3423](https://github.com/matrix-org/synapse/issues/3423))
- Refactor HTTP replication endpoints to reduce code duplication ([\#3632](https://github.com/matrix-org/synapse/issues/3632))
- Tests now correctly execute on Python 3. ([\#3647](https://github.com/matrix-org/synapse/issues/3647))
- Sytests can now be run inside a Docker container. ([\#3660](https://github.com/matrix-org/synapse/issues/3660))
- Port over enough to Python 3 to allow the sytests to start. ([\#3668](https://github.com/matrix-org/synapse/issues/3668))
- Update docker base image from alpine 3.7 to 3.8. ([\#3669](https://github.com/matrix-org/synapse/issues/3669))
- Rename synapse.util.async to synapse.util.async_helpers to mitigate async becoming a keyword on Python 3.7. ([\#3678](https://github.com/matrix-org/synapse/issues/3678))
- Synapse's tests are now formatted with the black autoformatter. ([\#3679](https://github.com/matrix-org/synapse/issues/3679))
- Implemented a new testing base class to reduce test boilerplate. ([\#3684](https://github.com/matrix-org/synapse/issues/3684))
- Rename MAU prometheus metrics ([\#3690](https://github.com/matrix-org/synapse/issues/3690))
- add new error type ResourceLimit ([\#3707](https://github.com/matrix-org/synapse/issues/3707))
- Logcontexts for replication command handlers ([\#3709](https://github.com/matrix-org/synapse/issues/3709))
- Update admin register API documentation to reference a real user ID. ([\#3712](https://github.com/matrix-org/synapse/issues/3712))
 2018-08-22 14:28:55 +01:00
Richard van der Hoff 4e5a4549b6 bump version to 0.33.3 2018-08-22 14:07:10 +01:00
Erik Johnston db10f553ba
Merge pull request #3724 from Half-Shot/hs/guest-fetch-event 
Allow guests to use /rooms/:roomId/event/:eventId
 2018-08-22 13:41:08 +01:00
Erik Johnston 764030cf63
Merge pull request #3659 from matrix-org/erikj/split_profiles 
Allow profile updates to happen on workers
 2018-08-22 11:35:55 +01:00
Erik Johnston 8432e2ebd7 Rename WorkerProfileHandler to BaseProfileHandler 2018-08-22 10:13:40 +01:00
Erik Johnston a81f140880 Add assert to ensure handler is only run on master 2018-08-22 10:11:21 +01:00
Erik Johnston 47b25ba5f3 Remove redundant vars 2018-08-22 10:09:05 +01:00
Erik Johnston 3bf8bab8f9
Merge pull request #3673 from matrix-org/erikj/refactor_state_handler 
Refactor state module to support multiple room versions
 2018-08-22 10:04:55 +01:00
Richard van der Hoff a4cf660a32
Merge pull request #3735 from matrix-org/travis/federation-spelling 
limt -> limit
 2018-08-22 09:34:21 +01:00
Richard van der Hoff 0d568ff403 Merge remote-tracking branch 'origin/release-v0.33.3' into develop 2018-08-22 09:15:44 +01:00
Matthew Hodgson bb81e78ec6
Split the state_group_cache in two (#3726) 
Splits the state_group_cache in two.

One half contains normal state events; the other contains member events.

The idea is that the lazyloading common case of: "I want a subset of member events plus all of the other state" can be accomplished efficiently by splitting the cache into two, and asking for "all events" from the non-members cache, and "just these keys" from the members cache.  This means we can avoid having to make DictionaryCache aware of these sort of complicated queries, whilst letting LL requests benefit from the caching.

Previously we were unable to sensibly use the caching and had to pull all state from the DB irrespective of the filtering, which made things slow.  Hopefully fixes https://github.com/matrix-org/synapse/issues/3720.
 2018-08-22 00:56:37 +02:00
Richard van der Hoff f7bf181a90 fix another consent encoding fail 2018-08-21 23:14:25 +01:00
Richard van der Hoff f7baff6f7b Fix 500 error from /consent form 
Fixes #3731
 2018-08-21 22:47:07 +01:00
Richard van der Hoff a52f276990 Bugfixes 
--------
 
 - Fix bug in v0.33.3rc1 which caused infinite loops and OOMs
 ([\#3723](https://github.com/matrix-org/synapse/issues/3723))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEIQBQJ4l+yK4dlKkFIwi0edOSShEFAlt6/yMACgkQIwi0edOS
 ShFrzAgAhLA74fClydGLfhPNCXIw12Amv2ASA6+r0ukqjYQ/9MZ0CRsL+PMZJTk6
 Z75PK3TkX38zSpPaUx9HdMQO/yE+rpI99ZxlBM+8jzsf1jyF8ljqxSfixXUlaXX9
 698PQQ3f+IDoSs59NKrHvoWhUThaBpf/1GwRxG5mUcm2oKJItNMHk/uLurNCaIIZ
 YSwWcxvZdluOMX0WrlF/S524DqtgULcRf3E9x3mumobkVGxI3UH73hXgGUcPi7XV
 ttLuy7fVbV2Cun/wvjBqD7kc8LNbKwSaT9y+/xS0D5QYjph4rpQiTG7S5rtPLPMU
 Z2lL1C5ggBCkD43g3orDvehbNWCL5g==
 =0vEe
 -----END PGP SIGNATURE-----

Merge tag 'v0.33.3rc2' into develop

Bugfixes
--------

- Fix bug in v0.33.3rc1 which caused infinite loops and OOMs
([\#3723](https://github.com/matrix-org/synapse/issues/3723))
 2018-08-21 20:30:09 +01:00
Hubert Chathi 42a394caa2 allow session_data to be any JSON instead of just a string 2018-08-21 14:51:34 -04:00
Erik Johnston 46c832eaac
Merge pull request #3727 from matrix-org/erikj/dont_error_on_missing_keys 
Don't log exceptions when failing to fetch server keys
 2018-08-21 17:07:20 +01:00
Erik Johnston cd6937fb26 Fix typo 2018-08-21 16:28:10 +01:00
Hubert Chathi 8550a7e9c2 allow auth_data to be any JSON instead of a string 2018-08-21 10:38:00 -04:00
Erik Johnston c2c153dd3b Log more detail when we fail to authenticate request 2018-08-21 11:42:49 +01:00
Erik Johnston 808d8e06aa Don't log exceptions when failing to fetch server keys 
Not being able to resolve or connect to remote servers is an expected
error, so we shouldn't log at ERROR with stacktraces.
 2018-08-21 11:19:26 +01:00
Erik Johnston 3f6762f0bb isort 2018-08-21 09:38:38 +01:00
Amber Brown 3b5b64ac99 changelog 2018-08-21 03:48:55 +10:00
Will Hunt f89f6b7c09 Allow guests to access /rooms/:roomId/event/:eventId 2018-08-20 18:25:54 +01:00
Richard van der Hoff be6527325a Fix exceptions when a connection is closed before we read the headers 
This fixes bugs introduced in #3700, by making sure that we behave sanely
when an incoming connection is closed before the headers are read.
 2018-08-20 18:21:10 +01:00
Richard van der Hoff 55e6bdf287 Robustness fix for logcontext filter 
Make the logcontext filter not explode if it somehow ends up with a logcontext
of None, since that infinite-loops the whole logging system.
 2018-08-20 18:20:07 +01:00
Erik Johnston b01a755498 Make the in flight requests metrics thread safe 2018-08-20 17:27:52 +01:00
Erik Johnston 1058d14127 Make the in flight background process metrics thread safe 2018-08-20 17:27:24 +01:00
Amber Brown 9a2f960736 version 2018-08-21 00:00:19 +10:00
Amber Brown 324525f40c
Port over enough to get some sytests running on Python 3 (#3668) 2018-08-20 23:54:49 +10:00
Erik Johnston 4d664278af Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler 2018-08-20 14:49:43 +01:00
Erik Johnston 8dee601054 Remove redundant room_version checks 2018-08-20 14:48:53 +01:00
Erik Johnston e21c368b8b Revert spurious change 2018-08-20 13:54:51 +01:00
Erik Johnston f2a48d87df Use get_cache_factor_for function for `state_cache` 
This allows the cache factor for `state_cache` to be individually
specified in the enviroment
 2018-08-20 13:01:46 +01:00
Erik Johnston 2aa7cc6a46
Merge pull request #3713 from matrix-org/erikj/fixup_fed_logging 
Fix logging bug in EDU handling over replication
 2018-08-20 10:51:45 +01:00
Neil Johnson e07970165f rename error code 2018-08-18 14:39:45 +01:00
Neil Johnson c5171bf171 special case server_notices_mxid 2018-08-18 12:33:07 +01:00
Neil Johnson ba1fbf7d5b special case server_notices_mxid 2018-08-18 12:31:08 +01:00
Richard van der Hoff 3cef867cc1
Merge pull request #3709 from matrix-org/rav/logcontext_for_replication_commands 
Logcontexts for replication command handlers
 2018-08-17 16:22:07 +01:00
Richard van der Hoff c144252a8c
Merge pull request #3710 from matrix-org/rav/logcontext_for_pusher_updates 
Fix logcontexts for running pushers
 2018-08-17 16:21:49 +01:00
Amber Brown c334ca67bb
Integrate presence from hotfixes (#3694) 2018-08-18 01:08:45 +10:00
Amber Brown 04f5d2db62
Remove v1/register's broken shared secret functionality (#3703) 2018-08-18 00:55:01 +10:00
Erik Johnston ab822a2d1f Add some fixmes 2018-08-17 15:31:50 +01:00
Erik Johnston 91cdb6de08 Call UserDirectoryHandler methods directly 
Turns out that the user directory handling is fairly racey as a bunch
of stuff assumes that the processing happens on master, which it doesn't
when there is a synapse.app.user_dir worker. So lets just call the
function directly until we actually get round to fixing it, since it
doesn't make the situation any worse.
 2018-08-17 15:26:13 +01:00
Neil Johnson d49b77404b clean up, no functional changes 2018-08-17 15:21:34 +01:00
Richard van der Hoff 63260397c6
Merge pull request #3701 from matrix-org/rav/use_producer_for_responses 
Use a producer to stream back responses
 2018-08-17 14:58:45 +01:00
Richard van der Hoff 3f8709ffe4
Merge pull request #3700 from matrix-org/rav/wait_for_producers 
Refactor request logging code
 2018-08-17 14:57:45 +01:00
Neil Johnson 3ee57bdcbb Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-17 14:34:10 +01:00
Erik Johnston 782689bd40 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles 2018-08-17 14:15:48 +01:00
Erik Johnston ca87ad1def Split ProfileHandler into master and worker 2018-08-17 14:15:14 +01:00
Neil Johnson b5f638f1f4 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-17 14:04:15 +01:00
Neil Johnson 9fd161c6fb Merge branch 'neilj/limit_exceeded_error' of github.com:matrix-org/synapse into neilj/limit_exceeded_error 2018-08-17 13:58:40 +01:00
Neil Johnson 0195dfbf52 server limits config docs 2018-08-17 13:58:25 +01:00
Neil Johnson 69c49d3fa3
Merge branch 'develop' into neilj/limit_exceeded_error 2018-08-17 12:44:26 +00:00
Erik Johnston 38f708a2bb Remote profile cache should remain in master worker 2018-08-17 11:37:42 +01:00
Erik Johnston 3b2dcfff78 Fix logging bug in EDU handling over replication 2018-08-17 11:11:06 +01:00
Neil Johnson 521d369e7a remove errant yield 2018-08-17 10:12:11 +01:00
Richard van der Hoff 0e8d78f6aa Logcontexts for replication command handlers 
Run the handlers for replication commands as background processes. This should
improve the visibility in our metrics, and reduce the number of "running db
transaction from sentinel context" warnings.

Ideally it means converting the things that fire off deferreds into the night
into things that actually return a Deferred when they are done. I've made a bit
of a stab at this, but it will probably be leaky.
 2018-08-17 00:43:43 +01:00
Richard van der Hoff 66f7dc8c87 Fix logcontexts for running pushers 
First of all, avoid resetting the logcontext before running the pushers, to fix
the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning.

Instead, give them their own "background process" logcontexts.
 2018-08-17 00:32:39 +01:00
Neil Johnson bcfeb44afe call reap on start up and fix under reaping bug 2018-08-16 22:55:32 +01:00
Neil Johnson 372bf073c1 block event creation and room creation on hitting resource limits 2018-08-16 21:25:16 +01:00
Neil Johnson 13ad9930c8 add new error type ResourceLimit 2018-08-16 18:02:02 +01:00
Neil Johnson 51b17ec566 flake8 2018-08-16 17:32:22 +01:00
Neil Johnson 3c1080b6e4 refactor for readability, and reuse caching for setting tags 2018-08-16 17:02:04 +01:00
Neil Johnson eff3ae3b9a add room tagging 2018-08-16 15:48:34 +01:00
Erik Johnston 7d5b1a60a3 Fix inbound federation on reader worker 
Inbound federation requires calculating push, which in turn relies on
having access to account data.
 2018-08-16 15:32:20 +01:00
Neil Johnson bfb6c58624 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-16 15:10:16 +01:00
Neil Johnson a675f9c556 check for room state before deciding on action 2018-08-16 14:53:35 +01:00
Will Hunt c151b32b1d Add GET media/v1/config (#3184) 2018-08-16 14:23:38 +01:00
Matthew Hodgson 762a758fea lazyload aware /messages (#3589) 2018-08-16 14:22:47 +01:00
Neil Johnson 25d2b5d55f Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-16 11:37:17 +01:00
Neil Johnson df1e4f259f WIP impl commiting to get feedback 2018-08-16 11:10:53 +01:00
Neil Johnson c055c91655 fix case where empty string state check is evaulated as False 2018-08-16 11:10:19 +01:00
Matthew Hodgson 3f543dc021 initial cut at a room summary API (#3574) 2018-08-16 09:46:50 +01:00
Neil Johnson 8cfad2e686 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-15 17:20:38 +01:00
Neil Johnson 81d727efa9
Merge pull request #3689 from matrix-org/neilj/fix_off_by_1+maus 
Fix Mau off by one errors
 2018-08-15 16:19:41 +00:00
Matthew Hodgson 2f78f432c4 speed up /members and add at= and membership params (#3568) 2018-08-15 16:35:22 +01:00
Neil Johnson fc5d937550 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-15 16:31:40 +01:00
Neil Johnson 86a00e05e1 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus 2018-08-15 16:27:08 +01:00