126b9bf96f
Log in the correct places
2018-03-01 12:05:33 +00:00
157298f986
Don't do preserve_fn for every request
2018-03-01 11:59:45 +00:00
89f90d808a
Add some logging
2018-03-01 11:59:16 +00:00
8ded8ba2c7
Make repl send_event idempotent and retry on timeouts
...
If we treated timeouts as failures on the worker we would attempt to
clean up e.g. push actions while the master might still process the
event.
2018-03-01 11:20:34 +00:00
f381d63813
Check event auth on the worker
2018-03-01 10:18:37 +00:00
6b8604239f
Correctly send ratelimit and extra_users params
2018-03-01 10:08:39 +00:00
f756f961ea
Fixup comments
2018-03-01 10:05:27 +00:00
28e973ac11
Calculate push actions on worker
2018-02-28 18:02:30 +00:00
493e25d554
Move storage functions for push calculations
...
This will allow push actions for an event to be calculated on workers.
2018-02-27 13:58:16 +00:00
3594dbc6dc
Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidation
...
Fix missing invalidations for receipt storage
2018-02-27 11:34:26 +00:00
2311189ee4
Merge pull request #2903 from matrix-org/erikj/split_roommember_store
...
Split out RoomMemberStore
2018-02-27 11:32:10 +00:00
c57607874c
Merge pull request #2901 from matrix-org/erikj/split_as_stores
...
Split AS stores
2018-02-27 10:07:07 +00:00
8956f0147a
Add comment
2018-02-27 10:06:51 +00:00
e5b4a208ce
Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions
...
Batch inserts into event_push_actions_staging
2018-02-26 14:45:40 +00:00
45b5fe9122
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/handle_unpersisted_events_push
2018-02-26 13:49:24 +00:00
d62ce972f8
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_roommember_store
2018-02-23 11:46:24 +00:00
6ae9a3d2a6
Update copyright
2018-02-23 11:44:49 +00:00
a90c60912f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_event_push_actions
2018-02-23 11:26:31 +00:00
50e8657867
Merge pull request #2902 from matrix-org/erikj/split_events_store
...
Split out get_events and co into a worker store
2018-02-23 11:23:52 +00:00
1cf9e071dd
Merge pull request #2899 from matrix-org/erikj/split_pushers
...
Split PusherStore
2018-02-23 11:23:35 +00:00
d0957753bf
Merge pull request #2898 from matrix-org/erikj/split_push_rules_store
...
Split PushRulesStore
2018-02-23 11:23:23 +00:00
199dba6c15
Merge pull request #2897 from matrix-org/erikj/split_account_data
...
Split AccountDataStore and TagStore
2018-02-23 11:23:11 +00:00
70349872c2
Update copyright
2018-02-23 11:14:35 +00:00
eba93b05bf
Split EventsWorkerStore into separate file
2018-02-23 11:01:21 +00:00
bf8a36e080
Update copyright
2018-02-23 10:52:10 +00:00
5d0f665848
Remove redundant clock
2018-02-23 10:49:58 +00:00
3bd760628b
_event_persist_queue shouldn't be in worker store
2018-02-23 10:49:18 +00:00
eb9b5eec81
Update copyright
2018-02-23 10:42:39 +00:00
c2ecfcc3a4
Update copyright
2018-02-23 10:41:34 +00:00
7e6cf89dc2
Update copyright
2018-02-23 10:39:19 +00:00
26d37f7a63
Update copyright
2018-02-23 10:33:55 +00:00
bb73f55fc6
Use absolute imports
2018-02-23 10:31:16 +00:00
faeb369f15
Fix missing invalidations for receipt storage
2018-02-21 15:19:54 +00:00
3dec9c66b3
Split out RoomMemberStore
2018-02-21 12:07:26 +00:00
46244b2759
Split AS stores
2018-02-21 11:49:34 +00:00
27b094f382
Split out get_events and co into a worker store
2018-02-21 11:41:48 +00:00
573712da6b
Update comments
2018-02-21 11:29:49 +00:00
c96d547f4d
Actually use new param
2018-02-21 11:03:42 +00:00
d15d237b0d
Split out EventPushActionWorkerStore
2018-02-21 11:01:13 +00:00
27939cbb0e
Merge pull request #2893 from matrix-org/erikj/delete_from_staging_fed
...
Delete from push_actions_staging in federation too
2018-02-21 11:00:06 +00:00
6f72765371
Split PusherStore
2018-02-21 10:54:21 +00:00
cbaad969f9
Split PushRulesStore
2018-02-21 10:43:31 +00:00
ca9b9d9703
Split AccountDataStore and TagStore
2018-02-21 10:15:04 +00:00
8fbb4d0d19
Raise exception in abstract method
2018-02-20 17:59:23 +00:00
95e4cffd85
Fix comment
2018-02-20 17:58:40 +00:00
e316bbb4c0
Use abstract base class to access stream IDs
2018-02-20 17:43:57 +00:00
f5ac4dc2d4
Split ReceiptsStore
2018-02-20 16:28:28 +00:00
24087bffa9
Ensure all push actions are deleted from staging
2018-02-20 12:34:31 +00:00
ad0ccf15ea
Refactor _set_push_actions_for_event_and_users_txn to use events_and_contexts
2018-02-20 12:34:28 +00:00
d874d4f2d7
Delete from push_actions_staging in federation too
2018-02-20 11:37:52 +00:00
6ff8c87484
Batch inserts into event_push_actions_staging
2018-02-20 11:33:07 +00:00
324c3e9399
Merge pull request #2868 from matrix-org/erikj/refactor_media_storage
...
Make store_file use store_into_file
2018-02-20 11:31:24 +00:00
3acd616979
Update pynacl dependency to 1.2.1 or higher
...
Signed-off-by: Pascal Bach <pascal.bach@nextrem.ch>
2018-02-19 10:45:22 +01:00
d1a3325f99
(Really) fix tablescan of event_push_actions on purge
...
commit 278d21b5
2018-02-16 14:02:31 +00:00
6af025d3c4
Fix typo of double is_highlight
2018-02-16 11:35:31 +00:00
012e8e142a
Comments
2018-02-16 11:35:01 +00:00
b96278d6fe
Ensure that we delete staging push actions on errors
2018-02-15 15:47:06 +00:00
4810f7effd
Remove context.push_actions
2018-02-15 15:47:06 +00:00
c714c61853
Update event_push_actions table from staging table
2018-02-15 15:47:06 +00:00
acac21248c
Store push actions in staging area
2018-02-15 15:47:04 +00:00
106906a65e
Don't serialize current state over replication
2018-02-15 13:53:18 +00:00
5fb347fc41
Merge pull request #2872 from matrix-org/erikj/event_worker_dont_log
...
Don't log errors propogated from send_event
2018-02-15 12:31:49 +00:00
fd1601c596
Fix state group storage bug in workers
...
We needed to move `_count_state_group_hops_txn` to the
StateGroupWorkerStore.
2018-02-15 11:04:32 +00:00
ef344b10e5
Don't log errors propogated from send_event
2018-02-15 11:03:49 +00:00
b8d821aa68
Merge pull request #2867 from matrix-org/rav/rework_purge
...
purge_history cleanups
2018-02-15 09:49:07 +00:00
92c52df702
Make store_file use store_into_file
2018-02-14 17:55:18 +00:00
d28ec43e15
Merge pull request #2769 from matrix-org/matthew/hit_the_gin
...
switch back from GIST to GIN indexes
2018-02-14 16:59:03 +00:00
39bf47319f
purge_history: fix sqlite syntax error
...
apparently sqlite insists on indexes being named
2018-02-14 16:42:19 +00:00
ac27f6a35e
purge_history: handle sqlite asshattery
...
apparently creating a temporary table commits the transaction. because that's a
useful thing.
2018-02-14 16:41:12 +00:00
5978dccff0
remove overzealous exception handling
2018-02-14 15:54:09 +00:00
278d21b5e4
purge_history: fix index use
...
event_push_actions doesn't have an index on event_id, so we need to specify
room_id.
2018-02-14 15:44:51 +00:00
5fcbf1e07c
Rework event purge to use a temporary table
...
... which should speed things up by reducing the amount of data being shuffled
across the connection
2018-02-14 11:02:22 +00:00
c0c9327fe0
Merge pull request #2854 from matrix-org/erikj/event_create_worker
...
Create a worker for event creation
2018-02-13 18:07:10 +00:00
d627174da2
Fix log message in purge_history
...
(we don't just remove remote events)
2018-02-13 16:51:21 +00:00
ddb6a79b68
Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin
2018-02-13 16:45:36 +00:00
0b27ae8dc3
move search reindex to schema 47
...
We're up to schema v47 on develop now, so this will have to go in there to have
an effect.
This might cause an error if somebody has already run it in the v46 guise, and
runs it again in the v47 guise, because it will cause a duplicate entry in the
bbackground_updates table. On the other hand, the entry is removed once it is
complete, and it is unlikely that anyone other than matrix.org has run it on
v46. The update itself is harmless to re-run because it deliberately copes with
the index already existing.
2018-02-13 16:44:46 +00:00
4a6d551704
GIN reindex: Fix syntax errors, improve exception handling
2018-02-13 16:44:46 +00:00
bfdf7b9237
Merge pull request #2864 from matrix-org/rav/persist_event_caching
...
Use StateResolutionHandler to resolve state in persist_events
2018-02-13 14:45:57 +00:00
630caf8a70
style nit
2018-02-13 14:29:22 +00:00
8fd1a32456
Fix typos in purge api & doc
...
* It's supposed to be purge_local_events, not ..._history
* Fix the doc to have valid json
2018-02-13 13:09:39 +00:00
4d09366656
Merge pull request #2695 from okurz/feature/allow_recent_pysaml
...
Allow use of higher versions of saml2
2018-02-13 12:24:08 +00:00
a9b712e9dc
Merge branch 'develop' into matthew/gin_work_mem
2018-02-13 12:16:01 +00:00
1026690cd2
Merge pull request #2857 from matrix-org/erikj/upload_store
...
Tell storage providers about new file so they can upload
2018-02-12 13:52:58 +00:00
10b34dbb9a
Merge pull request #2858 from matrix-org/rav/purge_updates
...
delete_local_events for purge_room_history
2018-02-09 14:11:00 +00:00
39a6b35496
purge: move room_depth update to end
...
... to avoid locking the table for too long
2018-02-09 13:07:41 +00:00
74fcbf741b
delete_local_events for purge_history
...
Add a flag which makes the purger delete local events
2018-02-09 13:07:41 +00:00
e571aef06d
purge: Move cache invalidation to more appropriate place
...
it was a bit of a non-sequitur there
2018-02-09 13:07:41 +00:00
61ffaa8137
bump purge logging to info
...
this thing takes ages and the only sign of any progress is the logs, so having
some logs is useful.
2018-02-09 13:07:41 +00:00
671540dccf
rename delete_old_state -> purge_history
...
(beacause it deletes more than state)
2018-02-09 13:07:41 +00:00
5fa571a91b
Tell storage providers about new file so they can upload
2018-02-07 13:35:08 +00:00
50fe92cd26
Move presence handling into handle_new_client_event
...
As we want to have it run on the main synapse instance
2018-02-07 10:34:09 +00:00
8ec2e638be
Add event_creator worker
2018-02-07 10:32:32 +00:00
24dd73028a
Add replication http endpoint for event sending
2018-02-07 10:32:32 +00:00
e3624fad5f
Remove pointless ratelimit check
...
The intention was for the check to be called as early as possible in the
request, but actually was called just before the main ratelimit check,
so was fairly pointless.
2018-02-07 10:30:25 +00:00
617199d73d
Merge pull request #2847 from matrix-org/erikj/separate_event_creation
...
Split event creation into a separate handler
2018-02-06 17:01:17 +00:00
3e1e69ccaf
Update copyright
2018-02-06 16:40:38 +00:00
770b2252ca
s/_create_new_client_event/create_new_client_event/
2018-02-06 16:40:30 +00:00
3d33eef6fc
Store state groups separately from events ( #2784 )
...
* Split state group persist into seperate storage func
* Add per database engine code for state group id gen
* Move store_state_group to StateReadStore
This allows other workers to use it, and so resolve state.
* Hook up store_state_group
* Fix tests
* Rename _store_mult_state_groups_txn
* Rename StateGroupReadStore
* Remove redundant _have_persisted_state_group_txn
* Update comments
* Comment compute_event_context
* Set start val for state_group_id_seq
... otherwise we try to recreate old state groups
* Update comments
* Don't store state for outliers
* Update comment
* Update docstring as state groups are ints
2018-02-06 14:31:24 +00:00
b31bf0bb51
Merge pull request #2849 from matrix-org/rav/clean_up_state_delta
...
Remove redundant return value from _calculate_state_delta
2018-02-05 17:42:20 +01:00
ebfe64e3d6
Use StateResolutionHandler to resolve state in persist events
...
... and thus benefit (hopefully) from its cache.
2018-02-05 16:23:26 +00:00
225dc3b4cb
Flatten _get_new_state_after_events
...
rejig the if statements to simplify the logic and reduce indentation
2018-02-05 16:23:25 +00:00
9fcbbe8e7d
Check that events being persisted have state_group
2018-02-05 16:23:25 +00:00
447aed42d2
Add event_map param to resolve_state_groups
2018-02-05 16:23:25 +00:00
ee6fb4cf85
Remove redundant return value from _calculate_state_delta
...
we already have the state from _get_new_state_after_events, so returning it
from _calculate_state_delta is just confusing.
2018-02-05 16:23:20 +00:00
3c7b480ba3
Factor out common code for search insert
...
we can reuse the same code as is used for event insert, for doing the
background index population.
2018-02-05 16:12:14 +00:00
3fa362502c
Update places where we create events
2018-02-05 16:01:48 +00:00
5ff3d23564
Split event creation into a separate handler
2018-02-05 16:01:48 +00:00
c46e75d3d8
Move store_event_search_txn to SearchStore
...
... as a precursor to making event storing and doing the bg update share some
code.
2018-02-05 15:43:22 +00:00
db91e72ade
Merge pull request #2844 from matrix-org/rav/evicted_metrics
...
montoring metrics for number of cache evictions
2018-02-05 16:34:36 +01:00
bc496df192
report metrics on number of cache evictions
2018-02-05 15:34:01 +00:00
1f881e0746
Merge pull request #2791 from matrix-org/erikj/media_storage_refactor
...
Ensure media is in local cache before thumbnailing
2018-02-05 11:28:52 +00:00
80b8a28100
Factor out common code for search insert
...
we can reuse the same code as is used for event insert, for doing the
background index population.
2018-02-04 00:23:06 +00:00
bd25f9cf36
Clean up work_mem handling
...
Add some comments and improve exception handling when twiddling work_mem for
the search update
2018-02-03 23:05:41 +00:00
4eeae7ad65
Move store_event_search_txn to SearchStore
...
... as a precursor to making event storing and doing the bg update share some
code.
2018-02-03 22:59:45 +00:00
bb9f0f3cdb
Merge branch 'develop' into matthew/gin_work_mem
2018-02-03 22:40:28 +00:00
6b02fc80d1
Reinstate event_search_postgres_gist handler
...
People may have queued updates for this, so we can't just delete it.
2018-02-02 14:32:51 +00:00
9c9356512e
Merge pull request #2845 from matrix-org/rav/urlcache_error_handling
...
Handle url_previews with no content-type
2018-02-02 15:27:52 +01:00
18eae413af
Merge pull request #2842 from matrix-org/rav/state_resolution_handler
...
Factor out resolve_state_groups to a separate handler
2018-02-02 15:27:35 +01:00
78d6ddba86
Merge pull request #2841 from matrix-org/rav/refactor_calc_state_delta
...
factor _get_new_state_after_events out of _calculate_state_delta
2018-02-02 15:27:15 +01:00
9dcd667ac2
Merge pull request #2836 from matrix-org/rav/resolve_state_events_docstring
...
Docstring fixes
2018-02-02 15:26:49 +01:00
6e87b34f7b
Merge branch 'develop' into travis/admin-list-media
2018-02-01 18:05:47 -07:00
d5352cbba8
Handle url_previews with no content-type
...
avoid failing with an exception if the remote server doesn't give us a
Content-Type header.
Also, clean up the exception handling a bit.
2018-02-02 00:53:46 +00:00
14737ba495
doc arg types for _seperate
2018-02-01 12:41:34 +00:00
e15d4ea248
More docstring fixes
...
Fix a couple of errors in docstrings
2018-02-01 12:41:34 +00:00
a18828c129
Fix docstring for StateHandler.resolve_state_groups
...
The return type was a complete lie, so fix it
2018-02-01 12:41:34 +00:00
6da4c4d3bd
Factor out resolve_state_groups to a separate handler
...
We extract the storage-independent bits of the state group resolution out to a
separate functiom, and stick it in a new handler, in preparation for its use
from the storage layer.
2018-02-01 12:40:04 +00:00
0cbda53819
Rename resolve_state_groups -> resolve_state_groups_for_events
...
(to make way for a method that actually just does the state group resolution)
2018-02-01 12:40:00 +00:00
77c0629ebc
Merge pull request #2837 from matrix-org/rav/fix_quarantine_media
...
Fix sql error in quarantine_media
2018-02-01 11:45:58 +01:00
e16e45b1b4
pep8
...
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-31 15:30:38 -07:00
e1e4ec9f9d
factor _get_new_state_after_events out of _calculate_state_delta
...
This reduces the scope of a bunch of variables
2018-01-31 21:32:09 +00:00
78e7e05188
Merge pull request #2838 from matrix-org/rav/fix_logging_on_dns_fail
...
Remove spurious log argument
2018-01-31 22:18:46 +01:00
ad48dfe73d
Merge pull request #2835 from matrix-org/rav/remove_event_type_param
...
Remove unused "event_type" param on state.get_current_state_ids
2018-01-31 22:17:57 +01:00
518a74586c
Merge pull request #2834 from matrix-org/rav/better_persist_event_exception_handling
...
Improve exception handling in persist_event
2018-01-31 22:17:38 +01:00
d1fe4db882
Merge pull request #2833 from matrix-org/rav/pusher_hacks
...
Logging and metrics for the http pusher
2018-01-31 22:16:59 +01:00
421d68ca8c
Merge pull request #2817 from matrix-org/rav/http_conn_pool
...
Use a connection pool for the SimpleHttpClient
2018-01-31 22:14:22 +01:00
63c4383927
Documentation and naming
...
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-31 08:07:52 -07:00
af19f5e9aa
Remove spurious log argument
...
... which would cause scary-looking and unhelpful errors in the log on dns fail
2018-01-30 17:52:03 +00:00
773f0eed1e
Fix sql error in quarantine_media
2018-01-30 15:02:51 +00:00
adfc0c9539
docstring for get_current_state_ids
2018-01-29 17:39:55 +00:00
d413a2ba98
Remove unused "event_type" param on state.get_current_state_ids
...
this param doesn't seem to be used, and is a bit pointless anyway because it
can easily be replicated by the caller. It is also horrible, because it changes
the return type of the method.
2018-01-29 17:06:57 +00:00
b387ee17b6
Improve exception handling in persist_event
...
1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that
a Failure object will be constructed using the current exception state,
*including* its stack trace - so the stack trace is saved in the Failure,
leading to better exception reports.
2. Set `consumeErrors=True` on the ObservableDeferred, because we know that
there will always be at least one observer - which avoids a spurious "CRITICAL:
unhandled exception in Deferred" error in the logs
2018-01-29 17:05:33 +00:00
03dd745fe2
Better logging when pushes fail
2018-01-29 15:49:06 +00:00
e051abd20b
add appid/device_display_name to to pusher logging
2018-01-29 15:04:16 +00:00
02ba118f81
Increase http conn pool size
2018-01-29 14:30:15 +00:00
46022025ea
Fix SQL for user search
...
fix some syntax errors for user search when search_all_users is enabled
fixes #2801 , hopefully
2018-01-27 17:21:57 +00:00
4b0f06e99c
Merge pull request #2830 from matrix-org/rav/factor_out_get_conn
...
Factor out get_db_conn to HomeServer base class
2018-01-27 17:56:25 +01:00
e98f0f9112
Merge pull request #2827 from matrix-org/fix_server_500_on_public_rooms_call_when_no_rooms_exist
...
Fix server 500 on public rooms call when no rooms exist
2018-01-26 20:22:40 +00:00
25adde9a04
Factor out get_db_conn to HomeServer base class
...
This function is identical to all subclasses, so we may as well push it up to
the base class to reduce duplication (and make use of it in the tests)
2018-01-26 00:56:49 +00:00
73560237d6
add white space line
2018-01-26 00:15:10 +00:00
86c4f49a31
rather than try reconstruct the results object, better to guard against the xrange step argument being 0
2018-01-26 00:12:02 +00:00
Erik Johnston