6cfee09be9
Make __init__ consitstent across Store heirarchy
...
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
2017-11-13 10:46:07 +00:00
eaaabc6c4f
replace 'except:' with 'except Exception:'
...
what could possibly go wrong
2017-10-23 15:52:32 +01:00
2e9f5ea31a
Fix logcontext handling for persist_events
...
* don't use preserve_context_over_deferred, which is known broken.
* remove a redundant preserve_fn.
* add/improve some comments
2017-10-17 10:59:30 +01:00
11d62f43c9
Invalidate cache
2017-10-03 14:12:28 +01:00
6ff14ddd2e
Make into list
2017-06-29 15:47:37 +01:00
5946aa0877
Prefill forward extrems and event to state groups
2017-06-29 15:38:48 +01:00
dfeca6cf40
Merge pull request #2286 from matrix-org/erikj/split_out_user_dir
...
Split out user directory to a separate process
2017-06-16 13:01:19 +01:00
6aa5bc8635
Initial worker impl
2017-06-16 11:47:11 +01:00
052c5d19d5
Merge pull request #2281 from matrix-org/erikj/phone_home_stats
...
Fix phone home stats
2017-06-15 12:46:23 +01:00
4b461a6931
Add some more stats
2017-06-15 09:39:39 +01:00
617304b2cf
Fix phone home stats
2017-06-14 19:47:15 +01:00
d53fe399eb
Add cache for is_host_joined
2017-06-13 09:56:18 +01:00
c62c480dc6
Merge pull request #2259 from matrix-org/erikj/fix_state_woes
...
Fix bug where state_group tables got corrupted
2017-06-07 17:51:25 +01:00
197bd126f0
Fix bug where state_group tables got corrupted
...
This is due to the fact that we prefilled caches using txn.call_after,
which always gets called including on error.
We fix this by making txn.call_after only fire when a transaction
completes successfully, which is what we want most of the time anyway.
2017-06-07 17:39:36 +01:00
4abcff0177
Fix typo
2017-05-31 16:22:36 +01:00
304880d185
Add stream change cache
2017-05-31 15:46:36 +01:00
04095f7581
Add clobbered event_id
2017-05-30 14:53:01 +01:00
a584a81b3e
Add current_state_delta_stream table
2017-05-30 14:44:09 +01:00
c049472b8a
Only store event_auth for state events
2017-05-24 15:23:31 +01:00
b8492b6c2f
Merge pull request #2224 from matrix-org/erikj/prefill_state
...
Prefill state caches
2017-05-16 15:50:11 +01:00
331570ea6f
Remove spurious merge artifacts
2017-05-16 15:33:07 +01:00
871605f4e2
Comments
2017-05-15 15:11:30 +01:00
e0d2f6d5b0
Add more granular event send metrics
2017-05-15 15:11:30 +01:00
bfbc907cec
Prefill state caches
2017-05-15 15:11:13 +01:00
34194aaff7
Don't create event_search index on sqlite
...
... because the table is virtual
2017-05-11 12:46:55 +01:00
114f290947
Add more logging for purging
...
Log the number of events we will be deleting at info.
2017-05-11 12:08:47 +01:00
baafb85ba4
Add an index to event_search
...
- to make the purge API quicker
2017-05-11 12:05:22 +01:00
dc026bb16f
Tidy purge code and add some comments
...
Try to make this clearer with more comments and some variable renames
2017-05-11 10:56:12 +01:00
8e345ce465
Don't de-delta state groups we're about to delete
2017-05-10 18:44:22 +01:00
b64d312421
add some logging to purge_history
2017-05-10 18:44:22 +01:00
093f7e47cc
Expand docstring a bit
2017-05-08 16:14:46 +01:00
ffad4fe35b
Don't update event cache hit ratio from get_joined_users
...
Otherwise the hit ration of plain get_events gets completely skewed by
calls to get_joined_users* functions.
2017-05-08 16:06:17 +01:00
587f07543f
Revert "Prefill state caches"
2017-05-04 15:07:27 +01:00
1827057acc
Comments
2017-05-03 09:56:05 +01:00
8346e6e696
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/prefill_state
2017-05-03 09:46:40 +01:00
3e5a62ecd8
Add more granular event send metrics
2017-05-02 14:23:26 +01:00
a2c89a225c
Prefill state caches
2017-05-02 10:40:31 +01:00
6a70647d45
Correct logic in is_event_after
2017-04-13 13:46:17 +01:00
b9557064bf
Simplify is_event_after logic
2017-04-12 14:36:20 +01:00
7f94709066
travis flake8..
2017-04-11 18:35:45 +01:00
73880268ef
Refactor event ordering check to events store
2017-04-11 17:34:09 +01:00
24d35ab47b
Add new storage functions for new replication
...
The new replication protocol will keep all the streams separate, rather
than muxing multiple streams into one.
2017-03-30 11:48:35 +01:00
bac9bf1b12
Typo
2017-03-27 18:02:17 +01:00
d82c42837f
Short circuit if all new events have same state group
2017-03-27 18:00:47 +01:00
e71940aa64
Use iter(items|values)
2017-03-24 10:57:02 +00:00
00957d1aa4
User Cursor.__iter__ instead of fetchall
...
This prevents unnecessary construction of lists
2017-03-23 17:53:49 +00:00
37a187bfab
Merge pull request #2033 from matrix-org/erikj/repl_speed
...
Don't send the full event json over replication
2017-03-21 13:11:15 +00:00
61f471f779
Don't send the full event json over replication
2017-03-17 15:50:01 +00:00
0c01f829ae
Avoid resetting state on rejected events
...
When we get a rejected event, give it the same state_group as its prev_event,
rather than no state_group at all.
This should fix https://github.com/matrix-org/synapse/issues/1935 .
2017-03-17 15:06:08 +00:00
5068fb16a5
Refactoring and cleanups
...
A few non-functional changes:
* A bunch of docstrings to document types
* Split `EventsStore._persist_events_txn` up a bit. Hopefully it's a bit more
readable.
* Rephrase `EventFederationStore._update_min_depth_for_room_txn` to avoid
mind-bending conditional.
* Rephrase rejected/outlier conditional in `_update_outliers_txn` to avoid
mind-bending conditional.
2017-03-17 15:06:07 +00:00
bb256ac96f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/public_list_speed
2017-03-14 11:35:05 +00:00
cc7a294e2e
Fix current_state_events table to not lie
...
If we try and persist two state events that have the same ancestor we
calculate the wrong current state when persisting those events.
2017-03-14 10:57:43 +00:00
8ffbe43ba1
Get current state by using current_state_events table
2017-03-10 17:39:35 +00:00
a41dce8f8a
Remove needless check
2017-02-27 18:54:43 +00:00
c0d6045776
It should be all
2017-02-27 18:45:24 +00:00
49f4bc4709
Don't fetch current state in common case
...
Currently we fetch the list of current state events whenever we send
something in a room. This is overkill for the common case of persisting
a simple chain of non-state events, so lets handle that case specially.
2017-02-27 18:33:41 +00:00
7723b4caa4
Ignore new rejected events when working out forward extremeties.
2017-02-08 14:48:06 +00:00
ebfaff84c9
Merge pull request #1870 from matrix-org/erikj/cache_get_all_new_events
...
Add a small cache get_all_new_events
2017-02-01 13:22:02 +00:00
368c88c487
Add a small cache get_all_new_events
2017-02-01 10:50:44 +00:00
458b6f4733
Only invalidate membership caches based on the cache stream
...
Before we completely invalidated get_users_in_room whenever we updated
any current_state_events table. This was way too aggressive.
2017-01-31 16:09:03 +00:00
4b3403ca9b
Stream cache invalidations for room membership storage functions
2017-01-30 17:28:22 +00:00
fdf2a31a51
Typo
2017-01-23 16:14:14 +00:00
c77b24c092
Refactor to calculate state delta outside transaction
2017-01-23 14:51:33 +00:00
5d2134d485
Comments
2017-01-20 17:13:24 +00:00
a55fa2047f
Insert delta of current_state_events to be more efficient
2017-01-20 17:10:18 +00:00
d0897dead5
Spelling
2017-01-20 15:05:11 +00:00
567aa35b67
Update all call sites after rename
2017-01-20 14:40:31 +00:00
f2f40e64a9
Comments
2017-01-20 14:38:13 +00:00
4c6a31cd6e
Calculate the forward extremeties once
2017-01-20 14:28:53 +00:00
09eb08f910
Derive current_state_events from state groups
2017-01-20 11:52:51 +00:00
4ec1cf49e2
Lower loading events log to DEBUG
2017-01-17 17:28:32 +00:00
dd3df11c55
More logging for the linearizer and for get_events
2017-01-05 12:32:47 +00:00
f9d156d270
New Flake8 fixes
2016-11-15 11:22:29 +00:00
816988baaa
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_auth
2016-10-17 11:10:37 +01:00
2869a29fd7
Drop some unused indices
2016-10-17 11:08:19 +01:00
9040c9ffa1
Fix background reindex of origin_server_ts
...
The storage function `_get_events_txn` was removed everywhere except
from this background reindex. The function was removed due to it being
(almost) completely unused while also being large and complex.
Therefore, instead of resurrecting `_get_events_txn` we manually
reimplement the bits that are needed directly.
2016-09-27 11:23:49 +01:00
0b32bb20bb
Index contains_url for file search queries
2016-09-12 16:57:05 +01:00
70332a12dd
Take value in a better way
2016-09-05 14:57:14 +01:00
485d999c8a
Correctly delete old state groups in purge history API
2016-09-05 14:49:08 +01:00
598317927c
Limit the length of state chains
2016-09-02 10:41:38 +01:00
1bb8ec296d
Generate state group ids in state layer
2016-08-31 10:09:46 +01:00
5dc2a702cf
Make _state_groups_id_gen a normal IdGenerator
2016-08-30 16:55:11 +01:00
bed10f9880
Use state handler instead of get_users_in_room/get_joined_hosts
2016-08-26 14:54:30 +01:00
9219139351
Preserve some logcontexts
2016-08-24 11:58:40 +01:00
37adde32dc
Move defer.returnValue out of Measure
2016-08-19 18:23:44 +01:00
47dd8f02a1
Measure _get_event_from_row
2016-08-19 18:23:44 +01:00
8a57cc3123
Add missing database corruption recovery case
...
Signed-off-by: Benjamin Saunders <ben.e.saunders@gmail.com>
2016-08-14 11:50:22 -07:00
a8b946decb
Raise 404 when couldn't find event
2016-08-05 15:31:02 +01:00
f0fa66f495
Delete more tables
2016-08-05 10:40:08 +01:00
a2b7102eea
Tweak integrity error recovery to work as intended
2016-08-04 20:38:08 -07:00
8ad8490cff
Fix typo
2016-08-04 15:21:29 +01:00
59fa91fe88
Retry event persistence on IntegrityError
...
Due to a bug in the porting script some backfilled events were not
correctly persisted, causing irrecoverable IntegrityErrors on future
attempts to persist those events.
This commit adds a retry mechanism invoked upon IntegrityError,
where when retried the tables are purged for all references to the
events being persisted.
2016-08-04 15:02:15 +01:00
80ad710217
Remove other bit of deduplication
2016-08-03 13:25:59 +01:00
a8a32d2714
Ensure we only persist an event once at a time
2016-08-03 11:23:39 +01:00
a6f06ce3e2
Fix how push_actions are redacted.
2016-07-26 11:05:39 +01:00
efeb6176c1
Don't add rejected events if we've seen them befrore. Add some comments to explain what the code is doing mechanically
2016-07-26 10:49:52 +01:00
1b3c3e6d68
Only update the events and event_json tables for rejected events
2016-07-25 18:44:30 +01:00
8f7f4cb92b
Don't add the events to forward extremities if the event is rejected
2016-07-25 17:13:37 +01:00
2623cec874
Don't add rejections to the state_group, persist all rejections
2016-07-25 16:12:16 +01:00
1e2a7f18a1
Merge pull request #922 from matrix-org/erikj/file_api2
...
Feature: Add filter to /messages. Add 'contains_url' to filter.
2016-07-20 10:40:48 +01:00
Richard van der Hoff