Commit Graph

55 Commits

Author SHA1 Message Date
Patrick Cloke d5e73cb6aa
Define StateMap as immutable and add a MutableStateMap type. (#8183) 2020-08-28 07:28:53 -04:00
Patrick Cloke 5758dcf30c
Add type hints for state. (#8140) 2020-08-24 14:25:27 -04:00
Patrick Cloke f40645e60b
Convert events worker database to async/await. (#8071) 2020-08-18 16:20:49 -04:00
Erik Johnston a7bdf98d01
Rename database classes to make some sense (#8033) 2020-08-05 21:38:57 +01:00
Patrick Cloke b975fa2e99
Convert state resolution to async/await (#7942) 2020-07-24 10:59:51 -04:00
Erik Johnston 831b31e563
Add another yield point to state res v2 (#7746) 2020-06-26 10:44:52 +01:00
Erik Johnston 0e0a2817a2
Yield during large v2 state res. (#7735)
State res v2 across large data sets can be very CPU intensive, and if
all the relevant events are in the cache the algorithm will run from
start to finish within a single reactor tick. This can result in
blocking the reactor tick for several seconds, which can have major
repercussions on other requests.

To fix this we simply add the occaisonal `sleep(0)` during iterations to
yield execution until the next reactor tick. The aim is to only do this
for large data sets so that we don't impact otherwise quick resolutions.=
2020-06-24 18:48:18 +01:00
Erik Johnston 91e886d615
Speed up state res v2 across large state differences. (#7725) 2020-06-19 13:56:35 +01:00
Patrick Cloke bd6dc17221
Replace iteritems/itervalues/iterkeys with native versions. (#7692) 2020-06-15 07:03:36 -04:00
Amber Brown 7cb8b4bc67
Allow configuration of Synapse's cache without using synctl or environment variables (#6391) 2020-05-11 18:45:23 +01:00
Erik Johnston 4a17a647a9
Improve get auth chain difference algorithm. (#7095)
It was originally implemented by pulling the full auth chain of all
state sets out of the database and doing set comparison. However, that
can take a lot work if the state and auth chains are large.

Instead, lets try and fetch the auth chains at the same time and
calculate the difference on the fly, allowing us to bail early if all
the auth chains converge. Assuming that the auth chains do converge more
often than not, this should improve performance. Hopefully.
2020-03-18 16:46:41 +00:00
Patrick Cloke 509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
Erik Johnston 2b37eabca1
Reduce auth chains fetched during v2 state res. (#6952)
The state res v2 algorithm only cares about the difference between auth
chains, so we can pass in the known common state to the `get_auth_chain`
storage function so that it can ignore those events.
2020-02-19 15:04:47 +00:00
Richard van der Hoff d7bf793cc1 s/get_room_version/get_room_version_id/
... to make way for a forthcoming get_room_version which returns a RoomVersion
object.
2020-01-31 10:06:21 +00:00
Richard van der Hoff a8ce7aeb43
Pass room version object into event_auth.check and check_redaction (#6788)
These are easier to work with than the strings and we normally have one around.

This fixes `FederationHander._persist_auth_tree` which was passing a
RoomVersion object into event_auth.check instead of a string.
2020-01-28 14:18:29 +00:00
Erik Johnston d386f2f339
Add StateMap type alias (#6715) 2020-01-16 13:31:22 +00:00
Richard van der Hoff dc96943d51 Merge branch 'master' into develop 2019-12-31 11:01:06 +00:00
Richard van der Hoff 92eac974b9
Hacks to work around #6605 (#6608)
When we have an event which refers to non-existent auth_events, ignore said events rather than exploding in a ball of fire.

Fixes #6605.
2019-12-31 10:41:44 +00:00
Richard van der Hoff 83895316d4 sanity-checking for events used in state res (#6531)
When we perform state resolution, check that all of the events involved are in
the right room.
2019-12-16 13:53:53 +00:00
Richard van der Hoff 1da15f05f5
sanity-checking for events used in state res (#6531)
When we perform state resolution, check that all of the events involved are in
the right room.
2019-12-13 12:55:32 +00:00
Andrew Morgan fc316a4894
Prevent redacted events from appearing in message search (#6377) 2019-12-11 13:39:47 +00:00
Richard van der Hoff 807ec3bd99
Fix bug which caused rejected events to be stored with the wrong room state (#6320)
Fixes a bug where rejected events were persisted with the wrong state group.

Also fixes an occasional internal-server-error when receiving events over
federation which are rejected and (possibly because they are
backwards-extremities) have no prev_group.

Fixes #6289.
2019-11-06 10:01:39 +00:00
Richard van der Hoff 4086002827
Improve documentation for EventContext fields (#6319) 2019-11-05 13:23:25 +00:00
Erik Johnston 69f0054ce6 Port to use state storage 2019-10-30 14:46:54 +00:00
Richard van der Hoff 66537e10ce
add some metrics on the federation sender (#6160) 2019-10-03 17:47:20 +01:00
Andrew Morgan 2a44782666
Remove double return statements (#5962)
Remove all the "double return" statements which were a result of us removing all the instances of

```
defer.returnValue(...)
return
```

statements when we switched to python3 fully.
2019-09-03 11:42:45 +01:00
Amber Brown 4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00
Amber Brown 463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ (#5606) 2019-07-04 00:07:04 +10:00
Erik Johnston 45f28a9d2f Merge branch 'develop' of github.com:matrix-org/synapse into erikj/histogram_extremities 2019-06-20 11:59:14 +01:00
Amber Brown 32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Erik Johnston 65787b0f7c Add descriptions and remove redundant set(..) 2019-06-19 11:49:39 +01:00
Erik Johnston 16a3124b76 Only count non-cache state resolution 2019-06-18 13:02:06 +01:00
Erik Johnston c9385dd238 Use consistent buckets 2019-06-18 12:43:41 +01:00
Erik Johnston ff88d36dcb Add metric fo number of state groups in resolution 2019-06-17 17:02:53 +01:00
Andrew Morgan 4a4d5c4fd6
Fix grammar and document get_current_users_in_room (#4998) 2019-04-03 14:32:20 +01:00
Richard van der Hoff 54a87a7b08
Collect room-version variations into one place (#4969)
Collect all the things that make room-versions different to one another into
one place, so that it's easier to define new room versions.
2019-04-01 10:24:38 +01:00
Erik Johnston a1b0e1879b Enable room version v3 2019-01-29 23:09:10 +00:00
Erik Johnston 47e2dd1994 Drop vdh support 2019-01-29 21:24:34 +00:00
Erik Johnston ae2a957dba Pass through room version to event auth 2019-01-25 18:31:41 +00:00
Richard van der Hoff cd018e3543
Merge pull request #4307 from matrix-org/erikj/v2_rooms
Add v2 room version
2018-12-24 10:48:59 +01:00
Erik Johnston df89f8afb8 Add v2 room version 2018-12-18 18:10:37 +00:00
Richard van der Hoff c03324294d Workaround for non-ascii event ids (#4241)
It turns out that we accept events with non-ascii IDs, which would later cause
an explosion during state res.

Fixes #4226
2018-12-03 21:47:48 +11:00
Erik Johnston bc80b3f454 Add helpers for getting prev and auth events (#4139)
* Add helpers for getting prev and auth events

This is in preparation for allowing the event format to change between
room versions.
2018-11-06 00:35:15 +11:00
Erik Johnston 54aec35867 Fix None exception in state res v2 2018-11-02 10:29:19 +00:00
Erik Johnston b3dd6fa981 Add STATE_V2_TEST room version 2018-11-01 11:43:46 +00:00
Erik Johnston 810715f79a Rename resolve_events_with_factory 2018-10-24 09:44:22 +01:00
Erik Johnston e238013c44 Add v2 state res algorithm.
We hook this up to the vdh test room version.
2018-10-16 16:16:13 +01:00
Richard van der Hoff e70b4ce069 Logging improvements
Some logging tweaks to help with debugging incoming federation transactions
2018-09-26 17:36:14 +01:00
Erik Johnston 8601c24287 Fix some instances of ExpiringCache not expiring cache items
ExpiringCache required that `start()` be called before it would actually
start expiring entries. A number of places didn't do that.

This PR removes `start` from ExpiringCache, and automatically starts
backround reaping process on creation instead.
2018-09-21 14:19:46 +01:00
Erik Johnston 5c261107c9 Remove unnecessary resolve_events_with_state_map
We only ever used the synchronous resolve_events_with_state_map in one
place, which is trivial to replace with the async version.
2018-08-22 15:41:15 +01:00