Commit Graph

2963 Commits

Author SHA1 Message Date
Erik Johnston 664b192a3b Fix set operations thinko 2018-10-29 14:21:43 +00:00
Erik Johnston f4f223aa44 Don't make temporary list 2018-10-29 14:01:49 +00:00
Erik Johnston b2399f6281 Make SQL a bit cleaner 2018-10-29 14:01:11 +00:00
Erik Johnston 03e634dad4 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups 2018-10-26 16:22:45 +01:00
Richard van der Hoff 4cda300058 preserve room visibility 2018-10-25 19:10:24 +01:00
Erik Johnston cb53ce9d64
Refactor state group lookup to reduce DB hits (#4011)
Currently when fetching state groups from the data store we make two
hits two the database: once for members and once for non-members (unless
request is filtered to one or the other). This adds needless load to the
datbase, so this PR refactors the lookup to make only a single database
hit.
2018-10-25 17:49:55 +01:00
Neil Johnson 95ad128851
Merge pull request #4081 from matrix-org/neilj/fix_mau_init
fix race condiftion in calling initialise_reserved_users
2018-10-25 16:33:40 +01:00
Neil Johnson fcbd488e9a add new line 2018-10-25 16:13:43 +01:00
Neil Johnson f8fe98812b improve comments 2018-10-25 14:58:59 +01:00
Neil Johnson f7f487e14c Merge branch 'develop' of github.com:matrix-org/synapse into matthew/autocreate_autojoin 2018-10-25 14:40:06 +01:00
Neil Johnson ea69a84bbb fix style inconsistencies 2018-10-24 17:18:08 +01:00
Neil Johnson 07126e43a4 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_mau_init 2018-10-24 16:25:39 +01:00
Richard van der Hoff 7328039117
Merge pull request #4082 from matrix-org/rav/fix_pep8
Fix a number of flake8 errors
2018-10-24 11:23:35 +01:00
Erik Johnston 3904cbf307
Merge pull request #4040 from matrix-org/erikj/states_res_v2_rebase
Add v2 state resolution algorithm
2018-10-24 11:12:12 +01:00
Richard van der Hoff ef771cc4c2 Fix a number of flake8 errors
Broadly three things here:

* disable W504 which seems a bit whacko
* remove a bunch of `as e` expressions from exception handlers that don't use
  them
* use `r""` for strings which include backslashes

Also, we don't use pep8 any more, so we can get rid of the duplicate config
there.
2018-10-24 10:39:03 +01:00
Neil Johnson 329d18b39c remove white space 2018-10-23 15:27:20 +01:00
Neil Johnson 6105c6101f fix race condiftion in calling initialise_reserved_users 2018-10-23 15:24:58 +01:00
Amber Brown e1728dfcbe
Make scripts/ and scripts-dev/ pass pyflakes (and the rest of the codebase on py3) (#4068) 2018-10-20 11:16:55 +11:00
Erik Johnston 67f7b9cb50 pep8 2018-10-19 16:06:59 +01:00
Erik Johnston 056f099126 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups 2018-10-19 15:48:59 +01:00
Erik Johnston 47a9da28ca Batch process handling state groups 2018-10-19 15:48:15 +01:00
Richard van der Hoff fc0f13dd03 Fix incorrect truncation in get_missing_events
It's quite important that get_missing_events returns the *latest* events in the
room; however we were pulling event ids out of the database until we got *at
least* 10, and then taking the *earliest* of the results.

We also shouldn't really be relying on depth, and should be checking the
room_id.
2018-10-16 21:10:04 +01:00
Erik Johnston 4a28d3d36f Update event_auth table for rejected events 2018-10-16 16:28:42 +01:00
Erik Johnston e238013c44 Add v2 state res algorithm.
We hook this up to the vdh test room version.
2018-10-16 16:16:13 +01:00
David Baker 03c11032c3
Merge pull request #4019 from matrix-org/dbkr/e2e_backups
E2E backups
2018-10-15 10:19:43 +01:00
Erik Johnston 67a1e315cc Fix up comments 2018-10-12 13:49:48 +01:00
David Baker 306361b31b Misc PR feedback bits 2018-10-12 11:48:56 +01:00
Richard van der Hoff 8ddd0f273c Comments on get_all_new_events_stream
just some docstrings to clarify the behaviour here
2018-10-12 09:55:41 +01:00
David Baker dc045ef202 Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backups 2018-10-09 10:05:02 +01:00
David Baker 497444f1fd Don't reuse backup versions
Since we don't actually delete the keys, just mark the versions
as deleted in the db rather than actually deleting them, then we
won't reuse versions.

Fixes https://github.com/vector-im/riot-web/issues/7448
2018-10-05 15:08:36 +01:00
Neil Johnson ed82043efb
Merge branch 'develop' into matthew/autocreate_autojoin 2018-10-04 17:26:59 +01:00
Erik Johnston 4917ff5523 Add state_group index to event_to_state_groups
This is needed to efficiently check for unreferenced state groups during
purge.
2018-10-04 16:03:08 +01:00
Erik Johnston 17d585753f Delete unreferened state groups during purge 2018-10-04 16:03:06 +01:00
Richard van der Hoff 9693625e55 actually exclude outliers 2018-10-03 10:19:41 +01:00
Richard van der Hoff 3e39783d5d remove debugging 2018-10-02 23:44:14 +01:00
Richard van der Hoff ae61ade891 Fix bug in forward_extremity update logic
An event does not stop being a forward_extremity just because an outlier or
rejected event refers to it.
2018-10-02 23:36:08 +01:00
Erik Johnston 7258d081a5 Fix bug when invalidating destination retry timings 2018-10-02 15:47:57 +01:00
Matthew Hodgson 5b68f29f48 fix thinkos 2018-09-29 02:14:40 +01:00
Richard van der Hoff 9c8cec5dab Merge remote-tracking branch 'origin/develop' into erikj/destination_retry_cache 2018-09-28 10:51:09 +01:00
Richard van der Hoff 51d33d5178
Merge pull request #3961 from matrix-org/neilj/lock_mau_upserts
fix #3854 MAU transaction errors
2018-09-27 12:31:37 +01:00
Richard van der Hoff ae6ad4cf41
docstrings and unittests for storage.state (#3958)
I spent ages trying to figure out how I was going mad...
2018-09-27 11:22:25 +01:00
Neil Johnson 28781b65e7 fix #3854 2018-09-26 16:16:41 +01:00
Erik Johnston fdd1a62e8d Add a five minute cache to get_destination_retry_timings
Hopefully helps with #3931
2018-09-21 14:56:12 +01:00
Amber Brown 1f3f5fcf52
Fix client IPs being broken on Python 3 (#3908) 2018-09-20 20:14:34 +10:00
Erik Johnston bbab6ebfd9 Fix up changelog and remove spurious comment 2018-09-19 14:45:14 +01:00
Erik Johnston 392a54128c pep8 2018-09-19 14:37:49 +01:00
Erik Johnston b9158ac2bf Remove get_destination_retry_timings cache
Currently we rely on the master to invalidate this cache promptly.
However, after having moved most federation endpoints off of master this
no longer happens, causing outbound fedeariont to get blackholed.

Fixes #3798
2018-09-19 14:22:57 +01:00
Richard van der Hoff 85a43f4167 Return a 404 when deleting unknown room alias
As per https://github.com/matrix-org/matrix-doc/issues/1675

Fixes https://github.com/matrix-org/synapse/issues/2782
2018-09-17 13:19:00 +01:00
David Baker bc74925c5b WIP e2e key backups
Continues from uhoreg's branch

This just fixed the errcode on /room_keys/version if no backup and
updates the schema delta to be on the latest so it gets run
2018-09-13 17:02:59 +01:00
Erik Johnston ed5331a627 comment 2018-09-13 16:10:56 +01:00
Erik Johnston 9cbd0094f0 pep8 2018-09-13 15:15:35 +01:00
Erik Johnston 9dbe38ea7d Create indices after insertion 2018-09-13 15:05:52 +01:00
Erik Johnston c857f5ef9b Make purge history slightly faster
Don't pull out events that are outliers and won't be deleted, as nothing
should happen to them.
2018-09-13 12:48:10 +01:00
Neil Johnson f30a303590
Merge pull request #3846 from matrix-org/neilj/expose-registered-users
expose number of real reserved users
2018-09-12 17:14:04 +01:00
Neil Johnson 8decd6233d improve naming 2018-09-12 16:22:15 +01:00
Neil Johnson 0ddf486724 expose number of real reserved users 2018-09-12 11:58:52 +01:00
Amber Brown 33716c4aea
Merge pull request #3826 from matrix-org/rav/logging_for_keyring
add some logging for the keyring queue
2018-09-12 20:43:47 +10:00
Matthew Hodgson b041115415
Speed up lazy loading (#3827)
* speed up room summaries by pulling their data from room_memberships rather than room state
* disable LL for incr syncs, and log incr sync stats  (#3840)
2018-09-12 00:50:39 +01:00
Richard van der Hoff cd7ef43872 clearer logging when things fail, too 2018-09-06 23:56:47 +01:00
Neil Johnson 84a750e0c3 ensure guests never enter mau list 2018-09-06 17:22:53 +01:00
Hubert Chathi 3801b8aa03 try to make flake8 and isort happy 2018-09-06 11:35:19 -04:00
Neil Johnson 61b05727fa guest users should not be part of mau total 2018-09-05 22:30:36 +01:00
Neil Johnson 0b01281e77 move threepid checker to config, add missing yields 2018-08-31 17:11:11 +01:00
Neil Johnson 09f3cf1a7e ensure post registration auth checks do not fail erroneously 2018-08-31 15:42:51 +01:00
Amber Brown 14e4d4f4bf
Port storage/ to Python 3 (#3725) 2018-08-31 00:19:58 +10:00
Hubert Chathi 83caead95a
Merge branch 'develop' into e2e_backups 2018-08-24 11:44:26 -04:00
Erik Johnston cd77270a66 Implement trail users 2018-08-23 19:17:19 +01:00
Erik Johnston 7a0da69eee Add missing yield 2018-08-23 10:28:12 +01:00
Erik Johnston 764030cf63
Merge pull request #3659 from matrix-org/erikj/split_profiles
Allow profile updates to happen on workers
2018-08-22 11:35:55 +01:00
Erik Johnston 3bf8bab8f9
Merge pull request #3673 from matrix-org/erikj/refactor_state_handler
Refactor state module to support multiple room versions
2018-08-22 10:04:55 +01:00
Matthew Hodgson bb81e78ec6
Split the state_group_cache in two (#3726)
Splits the state_group_cache in two.

One half contains normal state events; the other contains member events.

The idea is that the lazyloading common case of: "I want a subset of member events plus all of the other state" can be accomplished efficiently by splitting the cache into two, and asking for "all events" from the non-members cache, and "just these keys" from the members cache.  This means we can avoid having to make DictionaryCache aware of these sort of complicated queries, whilst letting LL requests benefit from the caching.

Previously we were unable to sensibly use the caching and had to pull all state from the DB irrespective of the filtering, which made things slow.  Hopefully fixes https://github.com/matrix-org/synapse/issues/3720.
2018-08-22 00:56:37 +02:00
Hubert Chathi 42a394caa2 allow session_data to be any JSON instead of just a string 2018-08-21 14:51:34 -04:00
Hubert Chathi 8550a7e9c2 allow auth_data to be any JSON instead of a string 2018-08-21 10:38:00 -04:00
Erik Johnston 4d664278af Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler 2018-08-20 14:49:43 +01:00
Erik Johnston 782689bd40 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles 2018-08-17 14:15:48 +01:00
Erik Johnston 38f708a2bb Remote profile cache should remain in master worker 2018-08-17 11:37:42 +01:00
Neil Johnson bcfeb44afe call reap on start up and fix under reaping bug 2018-08-16 22:55:32 +01:00
Matthew Hodgson 3f543dc021 initial cut at a room summary API (#3574) 2018-08-16 09:46:50 +01:00
Matthew Hodgson 2f78f432c4 speed up /members and add at= and membership params (#3568) 2018-08-15 16:35:22 +01:00
Erik Johnston ef184caf30 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation 2018-08-15 14:25:46 +01:00
Neil Johnson 9ecbaf8ba8 adding missing yield 2018-08-14 16:55:28 +01:00
Neil Johnson 414d54b61a
Merge pull request #3670 from matrix-org/neilj/mau_sync_block
Block ability to read via sync if mau limit exceeded
2018-08-14 15:21:31 +00:00
Amber Brown 591bf87c6a Merge remote-tracking branch 'origin/develop' into neilj/fix_reap_users_in_postgres 2018-08-14 20:56:23 +10:00
Amber Brown 99dd975dae
Run tests under PostgreSQL (#3423) 2018-08-13 16:47:46 +10:00
Matthew Hodgson f0cede5556 missing import 2018-08-12 19:14:31 -04:00
Matthew Hodgson 66a4ca1d28 404 nicely if you try to interact with a missing current version 2018-08-12 19:14:31 -04:00
Matthew Hodgson edc427a351 flake8 2018-08-12 19:14:31 -04:00
Matthew Hodgson fe87890b18 implement remaining tests and make them work 2018-08-12 19:14:31 -04:00
Matthew Hodgson 15d513f16f fix idiocies and so make tests pass 2018-08-12 19:14:31 -04:00
Matthew Hodgson b5eee511c7 don't needlessly return user_id 2018-08-12 19:14:31 -04:00
Matthew Hodgson 982edca380 fix flakes 2018-08-12 19:14:31 -04:00
Matthew Hodgson 234611f347 fix typos 2018-08-12 19:14:31 -04:00
Matthew Hodgson 9f0791b7bd add a tonne of docstring; make upload_room_keys properly assert version 2018-08-12 19:14:31 -04:00
Matthew Hodgson 8d14598e90 add storage docstring; remove unused set_e2e_room_keys 2018-08-12 19:14:31 -04:00
Matthew Hodgson cac0253799 rename room_key_version table correctly, and fix opt args 2018-08-12 19:14:31 -04:00
Matthew Hodgson 0abb205b47 blindly incorporate PR review - needs testing & fixing 2018-08-12 19:14:31 -04:00
Matthew Hodgson 69e51c7ba4 make /room_keys/version work 2018-08-12 19:14:31 -04:00
Matthew Hodgson 8ae64b270f implement /room_keys/version too (untested) 2018-08-12 19:14:31 -04:00
Matthew Hodgson 6b8c07abc2 make it work and fix pep8 2018-08-12 19:13:09 -04:00
Matthew Hodgson 0bc4627a73 interim WIP checkin; doesn't build yet 2018-08-12 18:23:10 -04:00
Matthew Hodgson 53ace904b2 total WIP skeleton for /room_keys API 2018-08-12 18:23:10 -04:00
Neil Johnson 31fa743567 fix sqlite/postgres incompatibility in reap_monthly_active_users 2018-08-11 22:38:34 +01:00
Amber Brown b37c472419
Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678) 2018-08-10 23:50:21 +10:00
Neil Johnson 885ea9c602 rename _user_last_seen_monthly_active 2018-08-09 18:02:12 +01:00
Erik Johnston ce6db0e547 Choose state algorithm based on room version 2018-08-09 14:58:47 +01:00
Erik Johnston 484a0ebdfc Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles 2018-08-09 10:16:29 +01:00
Erik Johnston 5785b93711 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation 2018-08-09 10:16:16 +01:00
Richard van der Hoff 8521ae13e3
Merge pull request #3654 from matrix-org/rav/room_versions
Support for room versioning
2018-08-08 17:10:53 +01:00
Neil Johnson be59910b93 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/reserved_users 2018-08-08 13:45:21 +01:00
Neil Johnson 990fe9fc23
Merge pull request #3633 from matrix-org/neilj/mau_tracker
API for monthly_active_users table
2018-08-08 12:44:15 +00:00
Neil Johnson 312ae74746 typos 2018-08-08 13:33:16 +01:00
Neil Johnson ef3589063a prevent total number of reserved users being too large 2018-08-07 17:59:27 +01:00
Neil Johnson e8eba2b4e3 implement reserved users for mau limits 2018-08-07 17:49:43 +01:00
Erik Johnston cd9765805e Allow ratelimiting on workers 2018-08-07 10:50:28 +01:00
Erik Johnston 495cb100d1 Allow profile changes to happen on workers 2018-08-07 10:50:26 +01:00
Richard van der Hoff ca9bc1f4fe Fix occasional glitches in the synapse_event_persisted_position metric
Every so often this metric glitched to a negative number. I'm assuming it was
due to backfilled events.
2018-08-07 10:00:03 +01:00
Neil Johnson a74b25faaa WIP building out mau reserved users 2018-08-06 23:25:25 +01:00
Neil Johnson e54794f5b6 Fix postgres compatibility bug 2018-08-06 21:55:54 +01:00
Neil Johnson 16d78be315 make use of _simple_select_one_onecol, improved comments 2018-08-06 17:51:15 +01:00
Erik Johnston 96a9a29645 Pull in necessary stores in federation_reader 2018-08-06 15:23:57 +01:00
Erik Johnston 62ace05c45 Move necessary storage functions to worker classes 2018-08-06 15:23:38 +01:00
Neil Johnson 886be75ad1 bug fixes 2018-08-03 22:29:03 +01:00
Neil Johnson e10830e976 wip commit - tests failing 2018-08-03 17:55:50 +01:00
Richard van der Hoff 0ca459ea33 Basic support for room versioning
This is the first tranche of support for room versioning. It includes:
 * setting the default room version in the config file
 * new room_version param on the createRoom API
 * storing the version of newly-created rooms in the m.room.create event
 * fishing the version of existing rooms out of the m.room.create event
2018-08-03 16:08:32 +01:00
Neil Johnson 950807d93a fix caching and tests 2018-08-03 13:49:53 +01:00
Neil Johnson 897c51d274 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker 2018-08-03 13:40:47 +01:00
Neil Johnson 5e2f7b8084 typo 2018-08-02 22:47:48 +01:00
Neil Johnson 4ecb4bdac9 wip attempt at caching 2018-08-02 22:41:05 +01:00
Neil Johnson 74b1d46ad9 do mau checks based on monthly_active_users table 2018-08-02 16:57:35 +01:00
Neil Johnson 9180061b49 remove unused count_monthly_users 2018-08-02 15:55:29 +01:00
Richard van der Hoff 704c3e6239 Merge branch 'master' into develop 2018-08-02 15:43:30 +01:00
Richard van der Hoff 14fa9d4d92 Avoid extra db lookups
Since we're about to look up the events themselves anyway, we can skip the
extra db queries here.
2018-08-02 13:55:51 +01:00
Richard van der Hoff 0a65450d04 Validation for events/rooms in fed requests
When we get a federation request which refers to an event id, make sure that
said event is in the room the caller claims it is in.

(patch supplied by @turt2live)
2018-08-02 13:48:40 +01:00
Neil Johnson 00f99f74b1 insertion into monthly_active_users 2018-08-02 13:47:19 +01:00
Neil Johnson 4a6725d9d1 Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/mau_tracker 2018-08-02 11:04:18 +01:00
Neil Johnson 165e067033 Revert "change monthly_active_users table to be a single column"
This reverts commit ec716a35b2.
2018-08-02 10:59:58 +01:00
Erik Johnston 40c1c59cf4
Merge pull request #3621 from matrix-org/erikj/split_fed_store
Split out DB writes in federation handler
2018-08-02 10:41:42 +01:00
Neil Johnson 08281fe6b7 self.db_conn unused 2018-08-01 23:26:24 +01:00
Neil Johnson c21d82bab3 normalise reaping query 2018-08-01 23:24:38 +01:00
Neil Johnson ec716a35b2 change monthly_active_users table to be a single column 2018-08-01 17:54:37 +01:00
Neil Johnson d766f26de9 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker 2018-08-01 17:49:41 +01:00
Neil Johnson 085435e13a
Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits
Initial impl of capping MAU
2018-08-01 15:58:45 +00:00
Neil Johnson 7ff44d9215 improve clarity 2018-08-01 16:17:00 +01:00
Amber Brown da7785147d
Python 3: Convert some unicode/bytes uses (#3569) 2018-08-02 00:54:06 +10:00
Neil Johnson 303f1c851f Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits 2018-08-01 13:42:50 +01:00
Erik Johnston a6d7b74915 update docs 2018-08-01 13:39:14 +01:00
Neil Johnson 4e5ac901dd clean up 2018-08-01 12:03:57 +01:00
Neil Johnson f9f5559971 fix comment 2018-08-01 12:03:42 +01:00
Neil Johnson 6023cdd227 remove errant print 2018-08-01 10:27:17 +01:00
Neil Johnson 7931393495 make count_monthly_users async synapse/handlers/auth.py 2018-08-01 10:21:56 +01:00
Neil Johnson 6ef983ce5c api into monthly_active_users table 2018-07-31 16:36:24 +01:00
Richard van der Hoff 5de936caa1
Merge pull request #3612 from matrix-org/rav/store_heirarchy
Make EventStore inherit from EventFederationStore
2018-07-31 13:44:04 +01:00
Neil Johnson df2235e7fa coding style 2018-07-31 13:16:20 +01:00
Richard van der Hoff 0bc9b9e397 reinstate explicit include of EventsWorkerStore 2018-07-31 13:11:04 +01:00
Neil Johnson fef7e58ac6 actually close conn 2018-07-30 22:29:44 +01:00
Neil Johnson 9b13817e06 factor out metrics from __init__ to app/homeserver 2018-07-30 22:07:07 +01:00
Neil Johnson 251e6c1210 limit register and sign in on number of monthly users 2018-07-30 15:55:57 +01:00
Erik Johnston 143f1a2532 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store 2018-07-30 09:56:18 +01:00
Matthew Hodgson e9b2d047f6
make /context lazyload & filter aware (#3567)
make /context lazyload & filter aware.
2018-07-27 15:12:50 +01:00
Richard van der Hoff 65c8dee900
Merge pull request #3614 from matrix-org/rav/stop_populating_event_content
Stop populating events.content
2018-07-26 22:54:08 +01:00
Richard van der Hoff 9e68b1bd2d
Merge pull request #3613 from matrix-org/rav/stop_using_event_edges_room_id
Remove some redundant joins on event_edges.room_id
2018-07-26 22:31:01 +01:00
Richard van der Hoff ef9d51b081
Merge pull request #3610 from matrix-org/rav/fix_looping_calls
Fix some looping_call calls which were broken in #3604
2018-07-26 14:55:57 +01:00
Richard van der Hoff 51d7df1915 Create the column nullable
There's no real point in ever making the column non-nullable, and doing so
breaks the sytests.
2018-07-26 14:54:04 +01:00
Richard van der Hoff 5c1d301fd9 Stop populating events.content
This field is no longer read from, so we should stop populating it. Once we're
happy that this doesn't break everything, and a rollback is unlikely, we can
think about dropping the column.
2018-07-26 14:43:02 +01:00
Richard van der Hoff bd4b25f4d0 Remove some redundant joins on event_edges.room_id
We've long passed the point where it's possible to have the same event_id in
different tables, so these join conditions are redundant: we can just join on
event_id.

event_edges is of non-trivial size, and the room_id column is wasteful, so
let's stop reading from it. In future, we can stop writing to it, and then drop
it.
2018-07-26 13:19:08 +01:00
Richard van der Hoff 1b4d73fa52 comment on event_edges 2018-07-26 12:53:51 +01:00
Richard van der Hoff 21e878ebb6 Make EventStore inherit from EventFederationStore
(since it uses methods therein)

Turns out that we had a bunch of things which were incorrectly importing
EventWorkerStore from events.py rather than events_worker.py, which broke once
I removed the import into events.py.
2018-07-26 12:48:51 +01:00
Richard van der Hoff 03751a6420 Fix some looping_call calls which were broken in #3604
It turns out that looping_call does check the deferred returned by its
callback, and (at least in the case of client_ips), we were relying on this,
and I broke it in #3604.

Update run_as_background_process to return the deferred, and make sure we
return it to clock.looping_call.
2018-07-26 11:48:08 +01:00
Matthew Hodgson bc7944e6d2 switch missing_types to be a bool 2018-07-25 23:36:31 +01:00
Matthew Hodgson 2565804030 Merge branch 'develop' into matthew/filter_members 2018-07-25 17:27:49 +01:00
Matthew Hodgson 7d9fb88617 incorporate more review. 2018-07-25 16:33:50 +01:00
Erik Johnston 78a691d005 Split out DB writes in federation handler
This will allow us to easily add an internal replication API to proxy
these reqeusts to master, so that we can move federation APIs to
workers.
2018-07-25 16:22:56 +01:00
Erik Johnston 3849f7f69f
Merge pull request #3603 from matrix-org/erikj/handle_outliers
Correctly handle outliers during persist events
2018-07-25 13:24:04 +01:00
Richard van der Hoff cee1ae1b72
Merge pull request #3606 from matrix-org/rav/logcontext_fixes_once_more
Fix another logcontext leak in _persist_events
2018-07-25 11:56:00 +01:00
Richard van der Hoff 32b30e15f2
Merge pull request #3607 from matrix-org/rav/fix_persist_events_integrity_error
Fix occasional 'tuple index out of range' error
2018-07-25 11:55:45 +01:00
Erik Johnston 7780a7b47c Actually fix it by adding continue 2018-07-25 11:13:20 +01:00
Richard van der Hoff 1be94440d3 Fix occasional 'tuple index out of range' error
This fixes a bug in _delete_existing_rows_txn which was introduced in #3435
(though it's been on matrix-org-hotfixes for *years*). This code is only called
when there is some sort of conflict the first time we try to persist an event,
so it only happens rarely. Still, the exceptions are annoying.
2018-07-25 11:05:58 +01:00
Richard van der Hoff 07defd5fe6 Fix another logcontext leak in _persist_events
We need to run the errback in the sentinel context to avoid losing our own
context.

Also: add logging to runInteraction to help identify where "Starting db
connection from sentinel context" warnings are coming from
2018-07-25 10:53:23 +01:00
Erik Johnston a297ff2b16 Fix typo in conditional 2018-07-25 09:48:01 +01:00
Richard van der Hoff 371da42ae4 Wrap a number of things that run in the background
This will reduce the number of "Starting db connection from sentinel context"
warnings, and will help with our metrics.
2018-07-25 09:41:12 +01:00
Erik Johnston ec56121b0d Correctly handle outliers during persist events
We incorrectly asserted that all contexts must have a non None state
group without consider outliers. This would usually be fine as the
assertion would never be hit, as there is a shortcut during persistence
if the forward extremities don't change.

However, if the outlier is being persisted with non-outlier events, the
function would be called and the assertion would be hit.

Fixes #3601
2018-07-25 09:35:02 +01:00
Matthew Hodgson cb5c37a57c handle the edge case for _get_some_state_from_cache where types is [] 2018-07-24 20:34:45 +01:00
Erik Johnston 3188973857 Pull out did_forget to worker store 2018-07-24 16:49:14 +01:00
Erik Johnston 60a1d147a7
Merge pull request #3595 from matrix-org/erikj/use_deltas
Use deltas to calculate current state deltas
2018-07-24 15:41:18 +01:00
Erik Johnston 709c309b0e Expand on docstring comment about return value 2018-07-24 15:12:50 +01:00
Erik Johnston 8f65ab98d2 Remove unnecessary iteritems 2018-07-24 15:08:01 +01:00
Erik Johnston ed0dd68731 Fixup comment (and indent) 2018-07-24 14:31:38 +01:00
Erik Johnston 811ac73a42 Don't fetch state from the database unless needed 2018-07-24 14:18:23 +01:00
Erik Johnston a79410e7b8 Have _get_new_state_after_events return delta
If we have a delta from the existing to new current state, then we can
reuse that rather than manually working it out by fetching both lots of
state.
2018-07-24 14:14:17 +01:00
Richard van der Hoff 93b0722c50
Merge pull request #3583 from matrix-org/rav/remove_who_forgot_in_room
Remove redundant checks on room forgottenness
2018-07-24 14:11:11 +01:00
Matthew Hodgson 454f59b7ad Merge branch 'develop' into matthew/filter_members 2018-07-24 14:03:37 +01:00
Erik Johnston 223341205e Don't require to_delete to have event_ids 2018-07-24 14:02:40 +01:00
Matthew Hodgson e22700c3dd consider non-filter_type types as wildcards, thus missing from the state-group-cache 2018-07-24 13:59:07 +01:00
Matthew Hodgson d19fba3655 Merge branch 'develop' into matthew/filter_members 2018-07-24 12:39:54 +01:00
Matthew Hodgson cd241d6bda incorporate more review 2018-07-24 12:39:40 +01:00
Richard van der Hoff 30bfed5aa5 Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room 2018-07-24 11:46:09 +01:00
Erik Johnston 97acd385a3 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/speed_up_calculate_state_delta 2018-07-24 11:32:13 +01:00
Erik Johnston 0fa73e4a63 Remove unnecessary if 2018-07-24 11:19:23 +01:00
Erik Johnston ff5426f6b8 Speed up _calculate_state_delta 2018-07-24 10:55:11 +01:00
Richard van der Hoff a678145010
Merge branch 'develop' into rav/logcontext_fixes 2018-07-24 10:43:30 +01:00