Commit Graph

1348 Commits

Author SHA1 Message Date
Erik Johnston e79ba9eb34 Fix tests 2019-01-24 09:28:16 +00:00
Richard van der Hoff 6b574f3df7 fix python2 test failure 2019-01-23 11:25:36 +00:00
Jason Robinson 6f680241bd Fix flake8 issues
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-01-23 10:53:48 +02:00
Jason Robinson 1838ef1ac3 Fix openid tests after rebase
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-01-23 10:38:13 +02:00
Jason Robinson a47fac9af6 Fix sorting of imports in tests. Remove an unnecessary mock
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-01-23 10:32:41 +02:00
Jason Robinson 5336e49b39 Add tests for the openid lister for SynapseHomeServer
Check all possible variants of openid and federation listener on/off
possibilities.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-01-23 10:32:41 +02:00
Jason Robinson 6d25599098 Add tests for the openid lister for FederationReaderServer
Check all possible variants of openid and federation listener on/off
possibilities.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-01-23 10:32:41 +02:00
Jason Robinson ab97b6e33c Fix a test docstring in frontend proxy tests
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2019-01-23 10:32:41 +02:00
Richard van der Hoff d02c4532c0 Add a test for MatrixFederationAgent 2019-01-22 20:35:12 +00:00
Richard van der Hoff 7021784d46 put resolve_service in an object
this makes it easier to stub things out for tests.
2019-01-22 20:35:12 +00:00
Richard van der Hoff 53a327b4d5 Require that service_name be a byte string
it is only ever a bytes now, so let's enforce that.
2019-01-22 20:35:12 +00:00
Richard van der Hoff 7871146667 Make MatrixFederationClient use MatrixFederationAgent
... instead of the matrix_federation_endpoint
2019-01-22 20:34:35 +00:00
Neil Johnson d619b113ed Fix None guard in config.server.is_threepid_reserved 2019-01-22 16:52:29 +00:00
Richard van der Hoff 33a55289cb
Refactor and bugfix for resove_service (#4427) 2019-01-22 10:59:27 +00:00
Amber Brown 23b0813599
Require ECDH key exchange & remove dh_params (#4429)
* remove dh_params and set better cipher string
2019-01-22 21:58:50 +11:00
Richard van der Hoff de6888e7ce Remove redundant WrappedConnection (#4409)
* Remove redundant WrappedConnection

The matrix federation client uses an HTTP connection pool, which times out its
idle HTTP connections, so there is no need for any of this business.
2019-01-18 23:07:38 +11:00
Richard van der Hoff 676cf2ee26
Fix incorrect logcontexts after a Deferred was cancelled (#4407) 2019-01-17 14:00:23 +00:00
Amber Brown 522dada206 Merge remote-tracking branch 'origin/master' into develop 2019-01-11 02:22:48 +11:00
Richard van der Hoff e0910d0145 Merge branch rav/macaroon_key_fix_0.34 into rav/macaroon_key_fix_0.34.1
Fixes #4371
2019-01-10 14:12:50 +00:00
Richard van der Hoff aa70d24125 Merge branch 'rav/macaroon_key_fix' into rav/macaroon_key_fix_0.34 2019-01-10 12:58:33 +00:00
Richard van der Hoff 566947ff34 Skip macaroon check for access tokens in the db 2019-01-10 12:57:21 +00:00
Amber Brown 7960c26fda
Fix adding new rows instead of updating them if one of the key values is a NULL in upserts. (#4369) 2019-01-09 22:26:25 +11:00
Erik Johnston b970cb0e96 Refactor request sending to have better excpetions (#4358)
* Correctly retry and back off if we get a HTTPerror response

* Refactor request sending to have better excpetions

MatrixFederationHttpClient blindly reraised exceptions to the caller
without differentiating "expected" failures (e.g. connection timeouts
etc) versus more severe problems (e.g. programming errors).

This commit adds a RequestSendFailed exception that is raised when
"expected" failures happen, allowing the TransactionQueue to log them as
warnings while allowing us to log other exceptions as actual exceptions.
2019-01-08 11:04:28 +00:00
Amber Brown d7843f47b6
Remove v1 only REST APIs now we don't ship matrix console (#4334) 2018-12-29 23:12:30 +11:00
Richard van der Hoff f85676cc93
Return well_known in /login response (#4319)
... as per MSC1730.
2018-12-24 10:44:33 +01:00
Richard van der Hoff 3355c1a9ec
Merge pull request #4317 from de-vri-es/test-metric-prometheus-0.5
Fix test_metrics.py compatibility prometheus_client 0.5
2018-12-23 00:14:47 +01:00
Amber Brown ea6abf6724
Fix IP URL previews on Python 3 (#4215) 2018-12-22 01:56:13 +11:00
Maarten de Vries 48b7ff7a35 Fix test_metrics.py compatibility prometheus_client 0.5
prometheus_client 0.5 has a named-tuple Sample type with more member
than the old plain tuple had. This commit makes sure the unit test
detects this and changes the way it reads the sample.

Signed-off-by: Maarten de Vries <maarten@de-vri.es>
2018-12-21 01:53:57 +01:00
Neil Johnson d2f7c4e6b1
create support user (#4141)
Allow for the creation of a support user.

A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.
2018-12-14 18:20:59 +00:00
Richard van der Hoff 30da50a5b8
Initialise user displayname from SAML2 data (#4272)
When we register a new user from SAML2 data, initialise their displayname
correctly.
2018-12-07 14:44:46 +01:00
Richard van der Hoff c7401a697f
Implement SAML2 authentication (#4267)
This implements both a SAML2 metadata endpoint (at
`/_matrix/saml2/metadata.xml`), and a SAML2 response receiver (at
`/_matrix/saml2/authn_response`). If the SAML2 response matches what's been
configured, we complete the SSO login flow by redirecting to the client url
(aka `RelayState` in SAML2 jargon) with a login token.

What we don't yet have is anything to build a SAML2 request and redirect the
user to the identity provider. That is left as an exercise for the reader.
2018-12-07 13:11:11 +01:00
Richard van der Hoff c588b9b9e4
Factor SSO success handling out of CAS login (#4264)
This is mostly factoring out the post-CAS-login code to somewhere we can reuse
it for other SSO flows, but it also fixes the userid mapping while we're at it.
2018-12-07 13:10:07 +01:00
Richard van der Hoff b0c24a66ec Rip out half-implemented m.login.saml2 support (#4265)
* Rip out half-implemented m.login.saml2 support

This was implemented in an odd way that left most of the work to the client, in
a way that I really didn't understand. It's going to be a pain to maintain, so
let's start by ripping it out.

* drop undocumented dependency on dateutil

It turns out we were relying on dateutil being pulled in transitively by
pysaml2. There's no need for that bloat.
2018-12-06 19:44:38 +11:00
Richard van der Hoff e8d98466b0
Implement .well-known handling (#4262)
Sometimes it's useful for synapse to generate its own .well-known file.
2018-12-05 14:38:58 +01:00
Richard van der Hoff 48972ce9d1
Patch defer.inlineCallbacks to check logcontexts in tests (#4205) 2018-12-04 11:30:32 +01:00
Neil Johnson 7039ece8fb Neilj/fix autojoin (#4223)
* Fix auto join failures for servers that require user consent

* Fix auto join failures for servers that require user consent
2018-11-28 22:24:57 +11:00
Richard van der Hoff a44c0a096f Check logcontexts before and after each test (#4190)
* Add better diagnostics to flakey keyring test

* fix interpolation fail

* Check logcontexts before and after each test

* update changelog

* update changelog
2018-11-27 13:47:18 +11:00
Richard van der Hoff 80527b568d Fix more logcontext leaks in tests (#4209) 2018-11-27 13:01:04 +11:00
Richard van der Hoff de8772a655 Do a GC after each test to fix logcontext leaks (#4227)
* Some words about garbage collections and logcontexts

* Do a GC after each test to fix logcontext leaks

This feels like an awful hack, but...

* changelog
2018-11-27 13:00:33 +11:00
Richard van der Hoff 416c671474
Merge pull request #4204 from matrix-org/rav/logcontext_leak_fixes
Fix some logcontext leaks
2018-11-20 12:19:19 +01:00
Amber Brown 80cac86b2c
Fix fallback auth on Python 3 (#4197) 2018-11-19 12:27:33 -06:00
Richard van der Hoff 828f18bd8b Fix logcontext leak in test_url_preview 2018-11-19 17:07:01 +00:00
Richard van der Hoff a267c2e3ed Fix logcontext leak in http pusher test 2018-11-19 17:07:01 +00:00
Richard van der Hoff 884a561447 Fix some tests which leaked logcontexts 2018-11-19 17:07:01 +00:00
Travis Ralston 3da9781c98 Fix the terms UI auth tests
By setting the config value directly, we skip the block that adds the slash automatically for us.
2018-11-15 23:00:28 -07:00
Amber Brown 8b1affe7d5
Fix Content-Disposition in media repository (#4176) 2018-11-15 15:55:58 -06:00
Travis Ralston 835779f7fb Add option to track MAU stats (but not limit people) (#3830) 2018-11-15 18:08:27 +00:00
Amber Brown df758e155d
Use <meta> tags to discover the per-page encoding of html previews (#4183) 2018-11-15 11:05:08 -06:00
Richard van der Hoff c1efcd7c6a Add a test for the public T&Cs form 2018-11-14 10:46:27 +00:00
Erik Johnston abaa93c158 Add test to assert set_e2e_device_keys correctly returns False on no-op 2018-11-08 14:06:44 +00:00
Amber Brown b3708830b8
Fix URL preview bugs (type error when loading cache from db, content-type including quotes) (#4157) 2018-11-08 01:37:43 +11:00
Amber Brown e62f7f17b3
Remove some boilerplate in tests (#4156) 2018-11-07 03:00:00 +11:00
Travis Ralston 0f5e51f726 Add config variables for enabling terms auth and the policy name (#4142)
So people can still collect consent the old way if they want to.
2018-11-06 10:32:34 +00:00
Hubert Chathi f1087106cf
handle empty backups according to latest spec proposal (#4123)
fixes #4056
2018-11-05 17:59:29 -05:00
Amber Brown efdcbbe46b
Tests for user consent resource (#4140) 2018-11-06 05:53:44 +11:00
Amber Brown 5a63589e80
Add some tests for the HTTP pusher (#4149) 2018-11-06 05:53:24 +11:00
Erik Johnston bc80b3f454 Add helpers for getting prev and auth events (#4139)
* Add helpers for getting prev and auth events

This is in preparation for allowing the event format to change between
room versions.
2018-11-06 00:35:15 +11:00
Erik Johnston 90d713b8c6
Merge pull request #4137 from matrix-org/erikj/clean_up_events
Clean up event accesses and tests
2018-11-02 14:12:49 +00:00
Erik Johnston b86d05a279 Clean up event accesses and tests
This is in preparation to refactor FrozenEvent to support different
event formats for different room versions
2018-11-02 13:44:14 +00:00
Amber Brown cb7a6b2379
Fix typing being reset causing infinite syncs (#4127) 2018-11-03 00:19:23 +11:00
Erik Johnston b199534518
Merge pull request #4135 from matrix-org/erikj/fix_state_res_none
Fix None exception in state res v2
2018-11-02 10:45:57 +00:00
Erik Johnston 54aec35867 Fix None exception in state res v2 2018-11-02 10:29:19 +00:00
Travis Ralston a8c9faa9a2 The tests also need a version parameter 2018-10-31 13:28:08 -06:00
Travis Ralston d1e7b9c44c Merge branch 'develop' into travis/login-terms 2018-10-31 13:15:14 -06:00
Amber Brown 3bade14ec0
Fix search 500ing (#4122) 2018-10-31 04:33:41 +11:00
Amber Brown 0dce9e1379
Write some tests for the email pusher (#4095) 2018-10-30 23:55:43 +11:00
Richard van der Hoff bf33eed609
Merge pull request #4091 from matrix-org/rav/room_version_upgrades
Room version upgrade support
2018-10-29 12:47:20 +00:00
Amber Brown 77d70a7646
Port register_new_matrix_user to Python 3 and add tests (#4085) 2018-10-26 22:05:22 +11:00
Richard van der Hoff 474810d9d5 fix broken test
This test stubbed out some stuff in a very weird way. I have no idea why. It broke.
2018-10-25 23:15:03 +01:00
Erik Johnston cb53ce9d64
Refactor state group lookup to reduce DB hits (#4011)
Currently when fetching state groups from the data store we make two
hits two the database: once for members and once for non-members (unless
request is filtered to one or the other). This adds needless load to the
datbase, so this PR refactors the lookup to make only a single database
hit.
2018-10-25 17:49:55 +01:00
Erik Johnston c85e063302
Merge pull request #4051 from matrix-org/erikj/alias_disallow_list
Add config option to control alias creation
2018-10-25 17:04:59 +01:00
Neil Johnson 95ad128851
Merge pull request #4081 from matrix-org/neilj/fix_mau_init
fix race condiftion in calling initialise_reserved_users
2018-10-25 16:33:40 +01:00
Erik Johnston b94a43d5b5 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/alias_disallow_list 2018-10-25 15:25:31 +01:00
Erik Johnston e5481b22aa Use allow/deny 2018-10-25 15:25:21 +01:00
Neil Johnson f7f487e14c Merge branch 'develop' of github.com:matrix-org/synapse into matthew/autocreate_autojoin 2018-10-25 14:40:06 +01:00
Travis Ralston a5468eaadf pep8 2018-10-24 13:54:38 -06:00
Travis Ralston 81880beff4 It helps to import things 2018-10-24 13:32:13 -06:00
Travis Ralston 4acb6fe8a3 Move test to where the other integration tests are 2018-10-24 13:24:24 -06:00
Travis Ralston 9283987f7e Fix test
Debug tests

Try printing the channel

fix

Import and use six

Remove debugging

Disable captcha

Add some mocks

Define the URL

Fix the clock?

Less rendering?

use the other render

Complete the dummy auth stage

Fix last stage of the test

Remove mocks we don't need
2018-10-24 13:23:08 -06:00
Travis Ralston 54def42c19 Merge branch 'develop' into travis/login-terms 2018-10-24 13:22:59 -06:00
Neil Johnson ea69a84bbb fix style inconsistencies 2018-10-24 17:18:08 +01:00
Neil Johnson 07126e43a4 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_mau_init 2018-10-24 16:25:39 +01:00
Richard van der Hoff 3ad359e5be Merge remote-tracking branch 'origin/develop' into rav/fix_event_filter_validation 2018-10-24 11:23:49 +01:00
Richard van der Hoff 7328039117
Merge pull request #4082 from matrix-org/rav/fix_pep8
Fix a number of flake8 errors
2018-10-24 11:23:35 +01:00
Erik Johnston 3904cbf307
Merge pull request #4040 from matrix-org/erikj/states_res_v2_rebase
Add v2 state resolution algorithm
2018-10-24 11:12:12 +01:00
Richard van der Hoff 7e07d25ed6 Allow backslashes in event field filters
Fixes a bug introduced in https://github.com/matrix-org/synapse/pull/1783 which
meant that single backslashes were not allowed in event field filters.

The intention here is to allow single-backslashes, but disallow
double-backslashes.
2018-10-24 11:11:24 +01:00
Richard van der Hoff ef771cc4c2 Fix a number of flake8 errors
Broadly three things here:

* disable W504 which seems a bit whacko
* remove a bunch of `as e` expressions from exception handlers that don't use
  them
* use `r""` for strings which include backslashes

Also, we don't use pep8 any more, so we can get rid of the duplicate config
there.
2018-10-24 10:39:03 +01:00
Erik Johnston b313b9b009 isort 2018-10-24 10:02:41 +01:00
Erik Johnston 810715f79a Rename resolve_events_with_factory 2018-10-24 09:44:22 +01:00
Neil Johnson 6105c6101f fix race condiftion in calling initialise_reserved_users 2018-10-23 15:24:58 +01:00
Erik Johnston 3c580c2b47 Add tests for alias creation rules 2018-10-19 10:22:45 +01:00
Erik Johnston 1b4bf232b9 Add tests for config generation 2018-10-19 10:22:45 +01:00
Travis Ralston dba84fa69c Fix terms UI auth test 2018-10-18 12:45:21 -06:00
Travis Ralston 88c5ffec33 Test for terms UI auth 2018-10-18 12:35:30 -06:00
Will Hunt d6a7797dd1 Fix roomlist since tokens on Python 3 (#4046)
Thanks @Half-Shot !!!
2018-10-17 23:04:55 +11:00
Erik Johnston 947c7443eb Add some state res v2 tests 2018-10-16 16:28:39 +01:00
Neil Johnson a2bfb778c8 improve auto room join logic, comments and tests 2018-10-12 18:17:36 +01:00
David Baker dc045ef202 Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backups 2018-10-09 10:05:02 +01:00
Neil Johnson ed82043efb
Merge branch 'develop' into matthew/autocreate_autojoin 2018-10-04 17:26:59 +01:00
Neil Johnson 2dadc092b8 move logic into register, fix room alias localpart bug, tests 2018-10-04 17:00:27 +01:00
Erik Johnston 9c834b8ee9 Add tests 2018-10-02 16:22:39 +01:00
Amber Brown 7232917f12
Disable frozen dicts by default (#3987) 2018-10-02 22:53:47 +10:00
Richard van der Hoff b5b93f45d5
Merge pull request #3968 from matrix-org/rav/fix_federation_errors
Fix exceptions when handling incoming transactions
2018-10-01 15:54:24 +01:00
Amber Brown 6e05fd032c
Fix userconsent on Python 3 (#3938) 2018-10-02 00:11:58 +10:00
Richard van der Hoff f094f715cf Merge remote-tracking branch 'origin/develop' into rav/fix_federation_errors 2018-09-27 15:18:21 +01:00
Amber Brown a512e637ac
Merge pull request #3970 from schnuffle/develop-py3
Replaced all occurences of e.message with str(e)
2018-09-27 23:39:45 +10:00
Richard van der Hoff 484a9b8c81 Remove redundant, failing, test
This test didn't do what it claimed to do, and what it claimed to do was the
same as test_cant_hide_direct_ancestors anyway.

This stuff is tested by sytest anyway.
2018-09-27 13:11:23 +01:00
Schnuffle dc5db01ff2 Replaced all occurences of e.message with str(e)
Signed-off-by: Schnuffle  <schnuffle@github.com>
2018-09-27 13:38:50 +02:00
Richard van der Hoff 4a15a3e4d5
Include eventid in log lines when processing incoming federation transactions (#3959)
when processing incoming transactions, it can be hard to see what's going on,
because we process a bunch of stuff in parallel, and because we may end up
recursively working our way through a chain of three or four events.

This commit creates a way to use logcontexts to add the relevant event ids to
the log lines.
2018-09-27 11:25:34 +01:00
Richard van der Hoff ae6ad4cf41
docstrings and unittests for storage.state (#3958)
I spent ages trying to figure out how I was going mad...
2018-09-27 11:22:25 +01:00
Richard van der Hoff 4c3e7eeec5
Merge pull request #3932 from matrix-org/erikj/auto_start_expiring_caches
Fix some instances of ExpiringCache not expiring cache items
2018-09-25 12:02:57 +01:00
Amber Brown fbe5ba25f6 Merge branch 'master' into develop 2018-09-25 03:10:01 +10:00
Oleg Girko 7d3f639844 Fix compatibility issue with older Twisted in tests.
Older Twisted (18.4.0) returns TimeoutError instead of
ConnectingCancelledError when connection times out.
This change allows tests to be compatible with this behaviour.

Signed-off-by: Oleg Girko <ol@infoserver.lv>
2018-09-25 02:51:18 +10:00
Erik Johnston 8601c24287 Fix some instances of ExpiringCache not expiring cache items
ExpiringCache required that `start()` be called before it would actually
start expiring entries. A number of places didn't do that.

This PR removes `start` from ExpiringCache, and automatically starts
backround reaping process on creation instead.
2018-09-21 14:19:46 +01:00
Amber Brown 1f3f5fcf52
Fix client IPs being broken on Python 3 (#3908) 2018-09-20 20:14:34 +10:00
Amber Brown aeca5a5ed5
Add a regression test for logging on failed connections (#3912) 2018-09-20 16:28:18 +10:00
Richard van der Hoff 31c15dcb80
Refactor matrixfederationclient to fix logging (#3906)
We want to wait until we have read the response body before we log the request
as complete, otherwise a confusing thing happens where the request appears to
have completed, but we later fail it.

To do this, we factor the salient details of a request out to a separate
object, which can then keep track of the txn_id, so that it can be logged.
2018-09-18 18:17:15 +01:00
Erik Johnston f6e82dcddb Tests 2018-09-14 15:08:37 +01:00
Amber Brown bfa0b759e0
Attempt to figure out what's going on with timeouts (#3857) 2018-09-14 00:15:51 +10:00
Neil Johnson 8decd6233d improve naming 2018-09-12 16:22:15 +01:00
Neil Johnson 0ddf486724 expose number of real reserved users 2018-09-12 11:58:52 +01:00
Amber Brown 52ec6e9dfa
Port tests/ to Python 3 (#3808) 2018-09-07 02:58:18 +10:00
Neil Johnson c5440b2ca0
Merge pull request #3800 from matrix-org/neilj/remove-guests-from-mau-count
guest users should not be part of mau total
2018-09-06 17:45:55 +01:00
Neil Johnson 84a750e0c3 ensure guests never enter mau list 2018-09-06 17:22:53 +01:00
Hubert Chathi 3801b8aa03 try to make flake8 and isort happy 2018-09-06 11:35:19 -04:00
Richard van der Hoff 4f8baab0c4 Merge branch 'master' into develop 2018-09-06 13:05:22 +01:00
Neil Johnson 61b05727fa guest users should not be part of mau total 2018-09-05 22:30:36 +01:00
Richard van der Hoff c127c8d042 Fix origin handling for pushed transactions
Use the actual origin for push transactions, rather than whatever the remote
server claimed.
2018-09-05 13:08:07 +01:00
Amber Brown 77055dba92
Fix tests on postgresql (#3740) 2018-09-04 02:21:48 +10:00
Neil Johnson 0b01281e77 move threepid checker to config, add missing yields 2018-08-31 17:11:11 +01:00
Neil Johnson ea068d6f3c fix bug where preserved threepid user comes to sign up and server is mau blocked 2018-08-31 10:49:14 +01:00
Amber Brown 14e4d4f4bf
Port storage/ to Python 3 (#3725) 2018-08-31 00:19:58 +10:00
Richard van der Hoff 414fa36f3e Fix up tests 2018-08-28 17:21:05 +01:00
Hubert Chathi 16a31c6fce update to newer Synapse APIs 2018-08-24 22:51:25 -04:00
Erik Johnston 9db2476991 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_contact 2018-08-24 17:00:37 +01:00
Erik Johnston 05077e06fa Change admin_uri to admin_contact in config and errors 2018-08-24 16:51:27 +01:00
Hubert Chathi 83caead95a
Merge branch 'develop' into e2e_backups 2018-08-24 11:44:26 -04:00
Erik Johnston 01a5a8b9e3 Fix checking if service notice room is already tagged
This manifested in synapse repeatedly setting the tag for the room
2018-08-24 16:22:37 +01:00
Erik Johnston f2cbbda956 Unit tests 2018-08-23 19:17:19 +01:00
Erik Johnston cd77270a66 Implement trail users 2018-08-23 19:17:19 +01:00
Erik Johnston 7e6e588e60 Fix bug where we resent "limit exceeded" server notices
This was due to a bug where we mutated a cached event's contents
2018-08-23 16:21:20 +01:00
Erik Johnston c5842dff1a Actually run the tests 2018-08-23 10:35:54 +01:00
Erik Johnston fd2dbf1836 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-22 17:06:10 +01:00
Erik Johnston 9643a6f7f2 Update notice format 2018-08-22 17:00:29 +01:00
Erik Johnston 764030cf63
Merge pull request #3659 from matrix-org/erikj/split_profiles
Allow profile updates to happen on workers
2018-08-22 11:35:55 +01:00
Erik Johnston 3bf8bab8f9
Merge pull request #3673 from matrix-org/erikj/refactor_state_handler
Refactor state module to support multiple room versions
2018-08-22 10:04:55 +01:00
Matthew Hodgson bb81e78ec6
Split the state_group_cache in two (#3726)
Splits the state_group_cache in two.

One half contains normal state events; the other contains member events.

The idea is that the lazyloading common case of: "I want a subset of member events plus all of the other state" can be accomplished efficiently by splitting the cache into two, and asking for "all events" from the non-members cache, and "just these keys" from the members cache.  This means we can avoid having to make DictionaryCache aware of these sort of complicated queries, whilst letting LL requests benefit from the caching.

Previously we were unable to sensibly use the caching and had to pull all state from the DB irrespective of the filtering, which made things slow.  Hopefully fixes https://github.com/matrix-org/synapse/issues/3720.
2018-08-22 00:56:37 +02:00
Erik Johnston 4d664278af Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler 2018-08-20 14:49:43 +01:00
Neil Johnson e07970165f rename error code 2018-08-18 14:39:45 +01:00
Neil Johnson ba1fbf7d5b special case server_notices_mxid 2018-08-18 12:31:08 +01:00
Amber Brown c334ca67bb
Integrate presence from hotfixes (#3694) 2018-08-18 01:08:45 +10:00
Neil Johnson d49b77404b clean up, no functional changes 2018-08-17 15:21:34 +01:00
Richard van der Hoff 63260397c6
Merge pull request #3701 from matrix-org/rav/use_producer_for_responses
Use a producer to stream back responses
2018-08-17 14:58:45 +01:00
Neil Johnson 3ee57bdcbb Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-17 14:34:10 +01:00
Erik Johnston 782689bd40 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles 2018-08-17 14:15:48 +01:00
Erik Johnston ca87ad1def Split ProfileHandler into master and worker 2018-08-17 14:15:14 +01:00
Neil Johnson b5f638f1f4 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-17 14:04:15 +01:00
Neil Johnson 69c49d3fa3
Merge branch 'develop' into neilj/limit_exceeded_error 2018-08-17 12:44:26 +00:00
Neil Johnson bcfeb44afe call reap on start up and fix under reaping bug 2018-08-16 22:55:32 +01:00
Neil Johnson 13ad9930c8 add new error type ResourceLimit 2018-08-16 18:02:02 +01:00
Neil Johnson 3c1080b6e4 refactor for readability, and reuse caching for setting tags 2018-08-16 17:02:04 +01:00
Neil Johnson a675f9c556 check for room state before deciding on action 2018-08-16 14:53:35 +01:00
Neil Johnson 8cfad2e686 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-15 17:20:38 +01:00
Neil Johnson 81d727efa9
Merge pull request #3689 from matrix-org/neilj/fix_off_by_1+maus
Fix Mau off by one errors
2018-08-15 16:19:41 +00:00
Matthew Hodgson 2f78f432c4 speed up /members and add at= and membership params (#3568) 2018-08-15 16:35:22 +01:00
Neil Johnson fc5d937550 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/server_notices_on_blocking 2018-08-15 16:31:40 +01:00
Neil Johnson 86a00e05e1 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus 2018-08-15 16:27:08 +01:00
Amber Brown a87af25fbb Fix the tests 2018-08-15 15:12:23 +01:00
Neil Johnson 75c663c7b9 update error codes 2018-08-15 11:27:48 +01:00
Neil Johnson b8429c7c81 update error codes for resource limiting 2018-08-15 10:19:48 +01:00
Neil Johnson ab035bdeac replace admin_email with admin_uri for greater flexibility 2018-08-15 10:16:41 +01:00
Neil Johnson 19b433e3f4 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/admin_email 2018-08-14 17:44:46 +01:00
Neil Johnson b586b8b986 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus 2018-08-14 17:43:22 +01:00
Neil Johnson 7277216d01 fix setup_test_homeserver to be postgres compatible 2018-08-14 17:14:39 +01:00
Neil Johnson 1522ed9c07 in case max_mau is less than I think 2018-08-14 16:52:30 +01:00
Neil Johnson 414d54b61a
Merge pull request #3670 from matrix-org/neilj/mau_sync_block
Block ability to read via sync if mau limit exceeded
2018-08-14 15:21:31 +00:00
Neil Johnson 06b331ff40 fix off by 1 errors 2018-08-14 15:28:15 +01:00
Neil Johnson 8f9a7eb58d support admin_email config and pass through into blocking errors, return AuthError in all cases 2018-08-14 15:11:54 +01:00
Neil Johnson ed4bc3d2fc fix off by 1s on mau 2018-08-14 15:04:48 +01:00
Neil Johnson 9b5bf3d858 Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus 2018-08-14 14:51:38 +01:00
Neil Johnson e25d87d97b Merge branch 'neilj/mau_sync_block' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus 2018-08-14 14:32:18 +01:00
Amber Brown bdfbd934d6
Implement a new test baseclass to cut down on boilerplate (#3684) 2018-08-14 20:53:43 +10:00
Neil Johnson f4b49152e2 support admin_email config and pass through into blocking errors, return AuthError in all cases 2018-08-13 21:09:47 +01:00
Neil Johnson ce7de9ae6b Revert "support admin_email config and pass through into blocking errors, return AuthError in all cases"
This reverts commit 0d43f991a1.
2018-08-13 18:06:18 +01:00
Neil Johnson 0d43f991a1 support admin_email config and pass through into blocking errors, return AuthError in all cases 2018-08-13 18:00:23 +01:00
Amber Brown 99dd975dae
Run tests under PostgreSQL (#3423) 2018-08-13 16:47:46 +10:00
Matthew Hodgson 66a4ca1d28 404 nicely if you try to interact with a missing current version 2018-08-12 19:14:31 -04:00
Matthew Hodgson edc427a351 flake8 2018-08-12 19:14:31 -04:00
Matthew Hodgson fe87890b18 implement remaining tests and make them work 2018-08-12 19:14:31 -04:00
Matthew Hodgson f6a3067868 linting 2018-08-12 19:14:31 -04:00
Matthew Hodgson 15d513f16f fix idiocies and so make tests pass 2018-08-12 19:14:31 -04:00
Matthew Hodgson 174be586e5 first cut at a UT 2018-08-12 19:14:31 -04:00
Neil Johnson 6c6aba76e1 implementation of server notices to alert on hitting resource limits 2018-08-10 15:12:59 +01:00
Amber Brown 807449d8f2 fix up a forced long line 2018-08-11 00:01:43 +10:00
black 8b3d9b6b19 Run black. 2018-08-10 23:54:09 +10:00
Amber Brown b37c472419
Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678) 2018-08-10 23:50:21 +10:00
Richard van der Hoff 3c0213a217
Merge pull request #3439 from vojeroen/send_sni_for_federation_requests
send SNI for federation requests
2018-08-10 12:23:54 +01:00
Neil Johnson 885ea9c602 rename _user_last_seen_monthly_active 2018-08-09 18:02:12 +01:00
Neil Johnson 04df714259 fix imports 2018-08-09 17:41:52 +01:00
Neil Johnson 09cf130898 only block on sync where user is not part of the mau cohort 2018-08-09 17:39:12 +01:00
Erik Johnston 3e19beb941 Fix tests 2018-08-09 14:58:49 +01:00
Neil Johnson 69ce057ea6 block sync if auth checks fail 2018-08-09 12:26:27 +01:00
Neil Johnson 3dce9050cf Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block 2018-08-09 11:42:02 +01:00
Neil Johnson 0ad98e38d0
Merge pull request #3655 from matrix-org/neilj/disable_hs
Flag to disable HS without disabling federation
2018-08-09 10:41:43 +00:00
Neil Johnson a5ef110749 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block 2018-08-09 11:40:37 +01:00
Amber Brown 2511f3f8a0
Test fixes for Python 3 (#3647) 2018-08-09 12:22:01 +10:00
Neil Johnson e92fb00f32 sync auth blocking 2018-08-08 17:54:49 +01:00
Neil Johnson 5298d79fb5
Merge branch 'develop' into neilj/disable_hs 2018-08-08 16:13:03 +00:00
Richard van der Hoff 8521ae13e3
Merge pull request #3654 from matrix-org/rav/room_versions
Support for room versioning
2018-08-08 17:10:53 +01:00
Neil Johnson d2f3ef98ac
Merge branch 'develop' into neilj/disable_hs 2018-08-08 15:55:47 +00:00
Neil Johnson 7f3d897e7a mock config.max_mau_value 2018-08-08 11:46:23 +01:00
Neil Johnson e8eba2b4e3 implement reserved users for mau limits 2018-08-07 17:49:43 +01:00
Neil Johnson fbe255f9a4 add default mau_limits_reserved_threepids 2018-08-06 23:24:54 +01:00
Neil Johnson 42c6823827 disable HS from config 2018-08-04 22:07:04 +01:00
Neil Johnson d08296f9f2 remove unused import 2018-08-03 23:00:16 +01:00
Neil Johnson 886be75ad1 bug fixes 2018-08-03 22:29:03 +01:00
Neil Johnson e10830e976 wip commit - tests failing 2018-08-03 17:55:50 +01:00
Richard van der Hoff 0ca459ea33 Basic support for room versioning
This is the first tranche of support for room versioning. It includes:
 * setting the default room version in the config file
 * new room_version param on the createRoom API
 * storing the version of newly-created rooms in the m.room.create event
 * fishing the version of existing rooms out of the m.room.create event
2018-08-03 16:08:32 +01:00
Neil Johnson 5593ff6773 fix (lots of) py3 test failures 2018-08-03 14:59:17 +01:00
Neil Johnson b2aab04d2c fix py3 test failure 2018-08-03 14:12:56 +01:00
Neil Johnson 950807d93a fix caching and tests 2018-08-03 13:49:53 +01:00
Neil Johnson 74b1d46ad9 do mau checks based on monthly_active_users table 2018-08-02 16:57:35 +01:00
Neil Johnson 9180061b49 remove unused count_monthly_users 2018-08-02 15:55:29 +01:00
Neil Johnson c4ffbecb68 fix test, update constructor call 2018-08-02 13:51:05 +01:00
Neil Johnson 00f99f74b1 insertion into monthly_active_users 2018-08-02 13:47:19 +01:00
Neil Johnson 165e067033 Revert "change monthly_active_users table to be a single column"
This reverts commit ec716a35b2.
2018-08-02 10:59:58 +01:00
Neil Johnson ec716a35b2 change monthly_active_users table to be a single column 2018-08-01 17:54:37 +01:00
Neil Johnson d766f26de9 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker 2018-08-01 17:49:41 +01:00
Neil Johnson 085435e13a
Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits
Initial impl of capping MAU
2018-08-01 15:58:45 +00:00
Amber Brown da7785147d
Python 3: Convert some unicode/bytes uses (#3569) 2018-08-02 00:54:06 +10:00
Neil Johnson c480c4c962 fix isort 2018-08-01 14:25:58 +01:00
Neil Johnson 6eed16d8a2 fix test for py3 2018-08-01 14:02:10 +01:00
Neil Johnson 303f1c851f Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits 2018-08-01 13:42:50 +01:00
Neil Johnson 4e6e00152c fix known broken test 2018-08-01 11:48:37 +01:00
Richard van der Hoff cab782c17e
Merge pull request #3384 from matrix-org/rav/rewrite_cachedlist_decorator
Rewrite cache list decorator
2018-08-01 10:28:56 +01:00
Neil Johnson 7931393495 make count_monthly_users async synapse/handlers/auth.py 2018-08-01 10:21:56 +01:00
Neil Johnson 6ef983ce5c api into monthly_active_users table 2018-07-31 16:36:24 +01:00
Neil Johnson df2235e7fa coding style 2018-07-31 13:16:20 +01:00
Neil Johnson 7d05406a07 fix user_ips counting 2018-07-31 12:03:23 +01:00
Travis Ralston e908b86832 Remove pdu_failures from transactions
The field is never read from, and all the opportunities given to populate it are not utilized. It should be very safe to remove this.
2018-07-30 16:28:47 -06:00
Neil Johnson cefac79c10 monthly_active_tests 2018-07-30 22:08:09 +01:00
Neil Johnson 251e6c1210 limit register and sign in on number of monthly users 2018-07-30 15:55:57 +01:00
Richard van der Hoff a8cbce0ced fix invalidation 2018-07-27 16:17:17 +01:00
Richard van der Hoff f102c05856 Rewrite cache list decorator
Because it was complicated and annoyed me. I suspect this will be more
efficient too.
2018-07-27 13:47:04 +01:00
Richard van der Hoff 7041cd872b
Merge branch 'develop' into send_sni_for_federation_requests 2018-07-27 09:17:11 +01:00
Matthew Hodgson 2565804030 Merge branch 'develop' into matthew/filter_members 2018-07-25 17:27:49 +01:00
Matthew Hodgson 0620d27f4d flake8 2018-07-25 17:21:17 +01:00
Matthew Hodgson 0a7ee0ab8b add tests for _get_some_state_from_cache 2018-07-25 16:33:52 +01:00
Richard van der Hoff f59be4eb0e Fix unit tests
on_notifier_poke no longer runs synchonously, so we have to do a different hack
to make sure that the replication data has been sent. Let's actually listen for
its arrival.
2018-07-25 10:30:36 +01:00
Matthew Hodgson cd241d6bda incorporate more review 2018-07-24 12:39:40 +01:00