112c2253e2
pep8
2018-04-06 15:43:27 +01:00
6850f8aea3
Get group_info from existing call to check_group_is_ours
2018-04-06 15:43:27 +01:00
cd087a265d
Don't use redundant inlineCallbacks
2018-04-06 15:43:27 +01:00
87c864b698
join_rule -> join_policy
2018-04-06 15:43:27 +01:00
ae85c7804e
is_joinable -> join_rule
2018-04-06 15:43:27 +01:00
f8d1917fce
Fix federation client `set_group_joinable` typo
2018-04-06 15:43:27 +01:00
6eb3aa94b6
Factor out add_user from accept_invite and join_group
2018-04-06 15:43:27 +01:00
edb45aae38
pep8
2018-04-06 15:43:27 +01:00
b370fe61c0
Implement group join API
2018-04-06 15:43:27 +01:00
7b824f1475
Add response size metrics
2018-04-06 13:20:11 +01:00
1d71f484d4
use PUT instead of POST for federating groups/m.join_policy
2018-04-06 12:54:09 +02:00
15e8ed874f
more verbosity in synctl
2018-04-06 09:28:36 +01:00
c7f0969731
Merge pull request #2986 from jplatte/join_reponse_room_id
...
Add room_id to the response of `rooms/{roomId}/join`
2018-04-05 17:29:06 +01:00
3449da3bc7
Merge pull request #3068 from matrix-org/rav/fix_cache_invalidation
...
Improve database cache performance
2018-04-05 17:21:44 +01:00
d1679a4ed7
Merge pull request #3066 from matrix-org/rav/remove_redundant_metrics
...
Remove redundant metrics which were deprecated in 0.27.0.
2018-04-05 17:21:18 +01:00
01afc563c3
Fix overzealous cache invalidation
...
Fixes an issue where a cache invalidation would invalidate *all* pending
entries, rather than just the entry that we intended to invalidate.
2018-04-05 16:24:04 +01:00
e089100c62
Merge pull request #3045 from matrix-org/dbkr/group_joinable
...
Add joinability for groups
2018-04-05 15:57:49 +01:00
68b0ee4e8d
Merge pull request #3041 from matrix-org/r30_stats
...
R30 stats
2018-04-05 15:37:37 +01:00
22284a6f65
Merge pull request #3060 from matrix-org/rav/kill_event_content
...
Remove uses of events.content
2018-04-05 15:02:17 +01:00
917380e89d
NON NULL -> NOT NULL
2018-04-05 14:32:12 +01:00
104c0bc1d5
Use "/settings/" (plural)
2018-04-05 14:07:16 +01:00
700e5e7198
Use DEFAULT join_policy of "invite" in db
2018-04-05 14:01:17 +01:00
b214a04ffc
Document set_group_join_policy
2018-04-05 13:29:16 +01:00
0e5f479fc0
Review comments
...
Use iteritems over item to loop over dict
formatting
2018-04-05 12:16:46 +01:00
518f6de088
Remove redundant metrics which were deprecated in 0.27.0.
2018-04-04 19:46:28 +01:00
e4570c53dd
phone home cache size configurations
2018-04-04 16:46:58 +01:00
6168351877
Add b prefixes to some strings that are bytes in py3
...
This has no effect on python2
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-04 13:48:51 +02:00
f92963f5db
Revert "improve mxid check performance"
2018-04-04 12:08:29 +01:00
72251d1b97
Remove address resolution of hosts in SRV records
...
Signed-off-by: Silke Hofstra <silke@slxh.eu>
2018-04-04 12:26:50 +02:00
725a72ec5a
Merge pull request #3000 from NotAFile/change-except-style
...
Replace old style error catching with 'as' keyword
2018-04-04 10:45:22 +01:00
a89f9f830c
Merge pull request #3044 from matrix-org/michaelk/performance_stats
...
Add basic performance statistics to phone home
2018-04-04 10:37:25 +01:00
39ce38b024
Merge pull request #3053 from NotAFile/speedup-mxid-check
...
improve mxid check performance
2018-04-04 10:19:52 +01:00
a9a74101a4
Document the behaviour of ResponseCache
...
it looks like everything that uses ResponseCache expects to have to
`make_deferred_yieldable` its results. It's debatable whether that is the best
approach, but let's document it for now to avoid further confusion.
2018-04-04 09:06:22 +01:00
eb8d8d6f57
Use join_policy API instead of joinable
...
The API is now under
/groups/$group_id/setting/m.join_policy
and expects a JSON blob of the shape
```json
{
"m.join_policy": {
"type": "invite"
}
}
```
where "invite" could alternatively be "open".
2018-04-03 16:16:40 +01:00
8da39ad98f
Merge pull request #3049 from matrix-org/rav/use_staticjson
...
Use static JSONEncoders
2018-04-03 15:18:32 +01:00
3ee4ad09eb
Fix json encoding bug in replication
...
json encoders have an encode method, not a dumps method.
2018-04-03 15:09:48 +01:00
11597ddea5
improve mxid check performance ~4x
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-03-31 02:00:22 +02:00
2fe3f848b9
Remove uses of events.content
2018-03-29 23:17:12 +01:00
05630758f2
Use static JSONEncoders
...
using json.dumps with custom options requires us to create a new JSONEncoder on
each call. It's more efficient to create one upfront and reuse it.
2018-03-29 23:13:33 +01:00
fcfe7f6ad3
Use simplejson throughout
...
Let's use simplejson rather than json, for consistency.
2018-03-29 22:45:52 +01:00
b4e37c6f50
pep8
2018-03-29 17:27:39 +01:00
9ee44a372d
Remove need for sqlite specific query
2018-03-29 16:45:34 +01:00
88cc9cc69e
Merge pull request #3043 from matrix-org/erikj/measure_state_group_creation
...
Measure time it takes to calculate state group ID
2018-03-28 17:40:14 +01:00
dc7c020b33
fix pep8 errors
2018-03-28 17:25:15 +01:00
c5de6987c2
This should probably be a PUT
2018-03-28 16:44:11 +01:00
241e4e8687
remove twisted deferral cruft
2018-03-28 16:25:53 +01:00
929b34963d
OK, smallint it is then
2018-03-28 14:53:55 +01:00
9a0db062af
Merge pull request #3034 from matrix-org/rav/fix_key_claim_errors
...
Fix error when claiming e2e keys from offline servers
2018-03-28 14:50:38 +01:00
a838444a70
Grr. Copy the definition from is_admin
2018-03-28 14:50:30 +01:00
4262aba17b
bump schema version
2018-03-28 14:40:03 +01:00
86932be2cb
Support multi client R30 for psql
2018-03-28 14:36:53 +01:00
32260baa41
pep8
2018-03-28 14:29:42 +01:00
33f6195d9a
Handle review comments
2018-03-28 14:25:25 +01:00
a164270833
Make column definition that works on both dbs
2018-03-28 14:23:00 +01:00
352e1ff9ed
Add schema delta file
2018-03-28 14:07:57 +01:00
79452edeee
Add joinability for groups
...
Adds API to set the 'joinable' flag, and corresponding flag in the
table.
2018-03-28 14:03:37 +01:00
6152e253d8
Merge branch 'develop' of into allow_auto_join_rooms
2018-03-28 14:45:28 +02:00
792d340572
rename stat to future proof
2018-03-28 12:25:02 +01:00
4ceaa7433a
As daemonizing will make a new process, defer call to init.
2018-03-28 12:19:01 +01:00
788e69098c
Add user_ips last seen index
2018-03-28 12:03:13 +01:00
0f890f477e
No need to cast in count_daily_users
2018-03-28 11:49:57 +01:00
545001b9e4
Fix search_user_dir multiple sqlite versions do different things
2018-03-28 11:19:45 +01:00
01ccc9e6f2
Measure time it takes to calculate state group ID
2018-03-28 11:03:52 +01:00
a32d2548d9
query and call for r30 stats
2018-03-28 10:39:13 +01:00
9187e0762f
count_daily_users failed if db was sqlite due to type failure - presumably this prevcented all sqlite homeservers reporting home
2018-03-28 10:02:32 +01:00
f879127aaa
Merge pull request #3029 from matrix-org/erikj/linearize_generate_user_id
...
Linearize calls to _generate_user_id
2018-03-28 10:00:31 +01:00
e6d87c93f3
Merge pull request #3030 from matrix-org/erikj/no_ujson
...
Remove last usage of ujson
2018-03-28 10:00:06 +01:00
004cc8a328
Merge pull request #3033 from matrix-org/erikj/calculate_state_metrics
...
Add counter metrics for calculating state delta
2018-03-28 09:59:42 +01:00
ef520d8d0e
Include coarse CPU and Memory use in stats callbacks.
...
This requires the psutil module, and is still opt-in based on the report_stats
config option.
2018-03-27 17:56:03 +01:00
a134c572a6
Stringify exceptions for keys/{query,claim}
...
Make sure we stringify any exceptions we return from keys/query and keys/claim,
to avoid a 'not JSON serializable' error later
Fixes #3010
2018-03-27 17:15:06 +01:00
c2a5cf2fe3
factor out exception handling for keys/claim and keys/query
...
this stuff is badly c&p'ed
2018-03-27 17:11:23 +01:00
800cfd5774
Comment
2018-03-27 13:30:39 +01:00
152c2ac19e
Fix indent
2018-03-27 13:13:46 +01:00
e70287cff3
Comment
2018-03-27 13:13:38 +01:00
03a26e28d9
Merge pull request #3017 from matrix-org/erikj/add_cache_control_headers
...
Add Cache-Control headers to all JSON APIs
2018-03-27 13:10:38 +01:00
3e0c0660b3
Also do check inside linearizer
2018-03-27 13:01:34 +01:00
3f49e131d9
Add counter metrics for calculating state delta
...
This will allow us to measure how often we calculate state deltas in
event persistence that we would have been able to calculate at the same
time we calculated the state for the event.
2018-03-27 10:57:35 +01:00
9b8c0fb162
Merge branch 'master' of github.com:matrix-org/synapse into develop
2018-03-26 21:41:55 +01:00
a9d7d98d3f
Bum version and changelog
2018-03-26 16:36:53 +01:00
bdbb1eec65
Merge branch 'erikj/simplejson_replication' of github.com:matrix-org/synapse into release-v0.27.0
2018-03-26 16:35:55 +01:00
9187862002
Bump version and changelog
2018-03-26 16:20:24 +01:00
51406dab96
version bump
2018-03-26 14:48:19 +01:00
fecb45e0c3
Remove last usage of ujson
2018-03-26 13:32:29 +01:00
44cd6e1358
PEP8
2018-03-26 12:06:48 +01:00
8d6dc106d1
Don't use _cursor_to_dict in find_next_generated_user_id_localpart
2018-03-26 12:02:44 +01:00
a052aa42e7
Linearize calls to _generate_user_id
2018-03-26 12:02:20 +01:00
8cbbfaefc1
404 correctly on missing paths via NoResource
...
fixes https://github.com/matrix-org/synapse/issues/2043 and https://github.com/matrix-org/synapse/issues/2029
2018-03-23 10:32:50 +00:00
84b5cc69f5
Merge pull request #3006 from matrix-org/erikj/state_iter
...
Use .iter* to avoid copies in StateHandler
2018-03-22 11:51:13 +00:00
fde8e8f09f
Fix s/iteriterms/itervalues
2018-03-22 11:42:16 +00:00
eb9fc021e3
Merge branch 'release-v0.27.0' of github.com:matrix-org/synapse into develop
2018-03-22 10:19:53 +00:00
1c41b05c8c
Add Cache-Control headers to all JSON APIs
...
It is especially important that sync requests don't get cached, as if a
sync returns the same token given then the client will call sync with
the same parameters again. If the previous response was cached it will
get reused, resulting in the client tight looping making the same
request and never making any progress.
In general, clients will expect to get up to date data when requesting
APIs, and so its safer to do a blanket no cache policy than only
whitelisting APIs that we know will break things if they get cached.
2018-03-21 17:46:26 +00:00
e66fbcbb02
fix merge conflicts
2018-03-20 14:25:31 +00:00
9aa5a0af51
Explicitly use simplejson
2018-03-20 09:58:13 +00:00
610accbb7f
Fix replication after switch to simplejson
...
Turns out that simplejson serialises namedtuple's as dictionaries rather
than tuples by default.
2018-03-19 16:12:48 +00:00
c384705ee8
Update __init__.py
...
bump version
2018-03-19 15:11:58 +00:00
3f961e638a
Merge pull request #3005 from matrix-org/erikj/fix_cache_size
...
Fix bug where state cache used lots of memory
2018-03-19 11:44:26 +00:00
fa72803490
Merge branch 'master' of github.com:matrix-org/synapse into develop
2018-03-19 11:41:01 +00:00
9a0d783c11
Add comments
2018-03-19 11:35:53 +00:00
38f952b9bc
spell out not to massively increase bcrypt rounds
2018-03-19 09:27:36 +00:00
a8ce159be4
Replace some ujson with simplejson to make it work
2018-03-16 00:27:09 +00:00
5b631ff41a
Remove wrong comment
2018-03-16 00:07:08 +00:00
ba48755d56
Bump version and changelog
2018-03-15 23:57:26 +00:00
926ba76e23
Replace ujson with simplejson
2018-03-15 23:43:31 +00:00
5a6e54264d
Make 'unexpected logging context' into warnings
...
I think we've now fixed enough of these that the rest can be logged at
warning.
2018-03-15 18:40:38 +00:00
9cf519769b
Use .iter* to avoid copies in StateHandler
2018-03-15 17:50:26 +00:00
7c7706f42b
Fix bug where state cache used lots of memory
...
The state cache bases its size on the sum of the size of entries. The
size of the entry is calculated once on insertion, so it is important
that the size of entries does not change.
The DictionaryCache modified the entries size, which caused the state
cache to incorrectly think it was smaller than it actually was.
2018-03-15 15:46:54 +00:00
2cc9f76bc3
replace old style error catching with 'as' keyword
...
This is both easier to read and compatible with python3 (not that that
matters)
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-03-15 16:11:17 +01:00
ddb00efc1d
Bump version number
2018-03-15 14:41:30 +00:00
91ea0202e6
move handling of auto_join_rooms to RegisterHandler
...
Currently the handling of auto_join_rooms only works when a user
registers itself via public register api. Registrations via
registration_shared_secret and ModuleApi do not work
This auto_joins the users in the registration handler which enables
the auto join feature for all 3 registration paths.
This is related to issue #2725
Signed-Off-by: Matthias Kesler <krombel@krombel.de>
2018-03-14 16:45:37 +01:00
4f28018c83
Register membership/state servlets in event_creator
2018-03-14 14:30:06 +00:00
57db62e554
Merge pull request #2992 from matrix-org/erikj/implement_member_workre
...
Implement RoomMemberWorkerHandler
2018-03-14 14:29:33 +00:00
0011ede3b0
Fix imports
2018-03-14 14:19:23 +00:00
62ad701326
s/join/joined/ in notify_user_membership_change
2018-03-14 14:17:43 +00:00
3f0f06cb31
Split RoomMemberWorkerHandler to separate file
2018-03-14 11:41:45 +00:00
3e839e0548
Merge pull request #2989 from matrix-org/erikj/profile_cache_master
...
Only update remote profile cache on master
2018-03-14 09:42:27 +00:00
ebd0127999
Merge pull request #2988 from matrix-org/erikj/split_profile_store
...
Split up ProfileStore
2018-03-14 09:41:06 +00:00
cfe75a9fb6
Merge pull request #2991 from matrix-org/erikj/fixup_rm
...
_remote_join and co take a requester
2018-03-14 09:40:57 +00:00
f51565e023
Merge pull request #2993 from matrix-org/erikj/is_blocked
...
Add is_blocked to worker store
2018-03-14 09:39:18 +00:00
d144ed6ffb
fix bug #2926 (loading all state for a given type from the DB if the state_key is None) ( #2990 )
...
Fixes a regression that had crept in where the caching layer upholds requests for loading state which is filtered by type (but not by state_key), but the DB layer itself would interpret a missing state_key as a request to filter by null state_key rather than returning all state_keys.
2018-03-13 22:36:04 +00:00
a08726fc42
Add is_blocked to worker store
2018-03-13 18:28:44 +00:00
b27320b550
Implement RoomMemberWorkerHandler
2018-03-13 18:26:00 +00:00
350331d466
_remote_join and co take a requester
2018-03-13 17:50:39 +00:00
1a69c6d590
Merge pull request #2987 from matrix-org/erikj/split_room_member_handler
...
Split RoomMemberHandler into base and master class
2018-03-13 17:40:00 +00:00
df8ff682a7
Only update remote profile cache on master
2018-03-13 17:38:21 +00:00
3518d0ea8f
Split up ProfileStore
2018-03-13 17:36:50 +00:00
d45a114824
Raise, don't return, exception
2018-03-13 17:24:34 +00:00
6dbebef141
Add missing param to docstrings
2018-03-13 17:15:32 +00:00
16adb11cc0
Correct import order
2018-03-13 16:57:07 +00:00
82f16faa78
Move user_*_room distributor stuff to master class
...
I added yields when calling user_left_room, but they shouldn't matter on
the master process as they always return None anyway.
2018-03-13 16:38:15 +00:00
b78717b87b
Split RoomMemberHandler into base and master class
...
The intention here is to split the class into the bits that can be done
on workers and the bits that have to be done on the master.
In future there will also be a class that can be run on the worker,
which will delegate work to the master when necessary.
2018-03-13 16:37:41 +00:00
95cb401ae0
Merge pull request #2978 from matrix-org/erikj/refactor_replication_layer
...
Remove ReplicationLayer and user Client/Server directly
2018-03-13 15:45:08 +00:00
5d8476d8ff
Merge pull request #2981 from matrix-org/erikj/factor_remote_leave
...
Factor out _remote_reject_invite in RoomMember
2018-03-13 15:44:56 +00:00
47ce527f45
Add room_id to the response of `rooms/{roomId}/join`
...
Fixes #2349
2018-03-13 14:48:12 +01:00
56e709857c
Merge pull request #2979 from matrix-org/erikj/no_handlers
...
Don't build handlers on workers unnecessarily
2018-03-13 13:46:38 +00:00
cb9f8e527c
s/replication_client/federation_client/
2018-03-13 13:26:52 +00:00
cea462e285
s/replication_server/federation_server
2018-03-13 13:22:21 +00:00
bf8e97bd3c
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/factor_remote_leave
2018-03-13 13:17:08 +00:00
ea3442c15c
Add docstring
2018-03-13 13:16:21 +00:00
16469a4f15
Merge pull request #2980 from matrix-org/erikj/rm_priv
...
Make RoomMemberHandler functions private that can be
2018-03-13 13:11:11 +00:00
c82111a55f
Merge pull request #2982 from matrix-org/erikj/fix_extra_users
...
extra_users is actually a list of UserIDs
2018-03-13 13:11:04 +00:00
da87791975
Merge pull request #2983 from matrix-org/erikj/rename_register_3pid
...
Refactor get_or_register_3pid_guest
2018-03-13 13:10:52 +00:00
99e9b4f26c
Merge pull request #2984 from matrix-org/erikj/fix_rest_regeix
...
RoomMembershipRestServlet doesn't handle /forget
2018-03-13 13:10:44 +00:00
f5160d4a3e
RoomMembershipRestServlet doesn't handle /forget
...
Due to the order we register the REST handlers `/forget` was handled by
the correct handler.
2018-03-13 12:12:55 +00:00
8b3573a8b2
Refactor get_or_register_3pid_guest
2018-03-13 12:08:58 +00:00
299fd740c7
Merge pull request #2975 from matrix-org/rav/measure_persist_events
...
Add Measure block for persist_events
2018-03-13 12:06:25 +00:00
9a2d9b4789
Merge pull request #2977 from matrix-org/erikj/replication_move_props
...
Move property setting from ReplicationLayer to base classes
2018-03-13 11:45:25 +00:00
f43b6d6d9b
Fix docstring types
2018-03-13 11:29:35 +00:00
0f942f68c1
Factor out _remote_reject_invite in RoomMember
2018-03-13 11:22:45 +00:00
d0fcc48f9d
extra_users is actually a list of UserIDs
2018-03-13 11:20:06 +00:00
31becf4ac3
Make functions private that can be
2018-03-13 11:15:16 +00:00
d023ecb810
Don't build handlers on workers unnecessarily
2018-03-13 11:08:10 +00:00
ea7b3c4b1b
Remove unused ReplicationLayer
2018-03-13 11:00:04 +00:00
265b993b8a
Split replication layer into two
2018-03-13 10:55:47 +00:00
e05bf34117
Move property setting from ReplicationLayer to FederationBase
2018-03-13 10:51:30 +00:00
c3f79c9da5
Split out edu/query registration to a separate class
2018-03-13 10:24:27 +00:00
889a2a853a
Add Measure block for persist_events
...
This seems like a useful thing to measure.
2018-03-13 10:01:42 +00:00
e48c7aac4d
Add transactional API to history purge
...
Make the purge request return quickly, and allow scripts to poll for updates.
2018-03-12 16:22:55 +00:00
1708412f56
Return an error when doing two purges on a room
...
Queuing up purges doesn't sound like a good thing.
2018-03-12 16:22:54 +00:00
b984dd0b73
Merge pull request #2961 from matrix-org/rav/run_in_background
...
Factor run_in_background out from preserve_fn
2018-03-12 16:19:13 +00:00
ba1d08bc4b
Merge pull request #2965 from matrix-org/rav/request_logging
...
Add a metric which increments when a request is received
2018-03-12 09:45:33 +00:00
58dd148c4f
Add some docstrings to help figure this out
2018-03-09 18:05:41 +00:00
88541f9009
Add a metric which increments when a request is received
...
It's useful to know when there are peaks in incoming requests - which isn't
quite the same as there being peaks in outgoing responses, due to the time
taken to handle requests.
2018-03-09 16:30:26 +00:00
dbe80a286b
refactor JsonResource
...
rephrase the OPTIONS and unrecognised request handling so that they look
similar to the common flow.
2018-03-09 16:22:16 +00:00
20f40348d4
Factor run_in_background out from preserve_fn
...
It annoys me that we create temporary function objects when there's really no
need for it. Let's factor the gubbins out of preserve_fn and start using it.
2018-03-08 11:50:11 +00:00
a56d54dcb7
Fix up log message
2018-03-07 11:55:31 +00:00
02a1296ad6
Fix typo
2018-03-07 11:55:31 +00:00
8cb44da4aa
Fix race in sync when joining room
...
The race happens when the user joins a room at the same time as doing a
sync. We fetch the current token and then get the rooms the user is in.
If the join happens after the current token, but before we get the rooms
we end up sending down a partial room entry in the sync.
This is fixed by looking at the stream ordering of the membership
returned by get_rooms_for_user, and handling the case when that stream
ordering is after the current token.
2018-03-07 11:55:31 +00:00
8ffaacbee3
Merge pull request #2949 from krombel/use_bcrypt_checkpw
...
use bcrypt.checkpw
2018-03-06 11:56:06 +00:00
b2932107bb
Merge pull request #2946 from matrix-org/rav/timestamp_to_purge
...
Implement purge_history by timestamp
2018-03-06 11:20:23 +00:00
7aed50a038
Merge pull request #2948 from matrix-org/erikj/kill_as_sync
...
Remove ability for AS users to call /events and /sync
2018-03-06 11:10:09 +00:00
b6c4b851f1
Merge pull request #2947 from matrix-org/erikj/split_directory_store
...
Split Directory store
2018-03-05 18:17:32 +00:00
ed9b5eced4
use bcrypt.checkpw
...
in bcrypt 3.1.0 checkpw got introduced (already 2 years ago)
This makes use of that with enhancements which might get introduced
by that
Signed-Off-by: Matthias Kesler <krombel@krombel.de>
2018-03-05 18:02:59 +01:00
d4ffe61d4f
Remove ability for AS users to call /events and /sync
...
This functionality has been deprecated for a while as well as being
broken for a while. Instead of fixing it lets just remove it entirely.
See: https://github.com/matrix-org/matrix-doc/issues/1144
2018-03-05 15:44:46 +00:00
69ce365b79
Fix cache invalidation on deletion
2018-03-05 15:29:03 +00:00
2e223163ff
Split Directory store
2018-03-05 15:11:30 +00:00
f8bfcd7e0d
Provide a means to pass a timestamp to purge_history
2018-03-05 14:37:23 +00:00
d032785aa7
Merge pull request #2943 from matrix-org/rav/fix_find_first_stream_ordering_after_ts
...
Test and fix find_first_stream_ordering_after_ts
2018-03-05 12:26:14 +00:00
c818fcab11
Test and fix find_first_stream_ordering_after_ts
...
It seemed to suffer from a bunch of off-by-one errors.
2018-03-05 12:04:02 +00:00
06a14876e5
Add find_first_stream_ordering_after_ts
...
Expose this as a public function which can be called outside a txn
2018-03-05 11:53:39 +00:00
42174946f8
Merge pull request #2934 from matrix-org/erikj/cache_fix
...
Fix bug with delayed cache invalidation stream
2018-03-05 11:33:17 +00:00
f394f5574d
Merge pull request #2929 from matrix-org/erikj/split_regististration_store
...
Split registration store
2018-03-05 11:33:07 +00:00
af7ed8e1ef
Return 401 for invalid access_token on logout
...
Signed-off-by: Duncan Klug <dklug@ucmerced.edu>
2018-03-02 22:01:27 -08:00
efb79820b4
Fix bug with delayed cache invalidation stream
...
We poked the notifier before updated the current token for the cache
invalidation stream. This mean that sometimes the update wouldn't be
sent until the next time a cache was invalidated.
2018-03-02 14:45:15 +00:00
fafa3e7114
Split registration store
2018-03-02 13:48:27 +00:00
d960d23830
Add missing yield during 3pid signature checks
2018-03-02 11:03:18 +00:00
1a6c7cdf54
Merge pull request #2928 from matrix-org/erikj/read_marker_caches
...
Fix typo in getting replication account data processing
2018-03-01 17:56:14 +00:00
89b7232ff8
Fix typo in getting replication account data processing
2018-03-01 17:50:30 +00:00
1773df0632
Merge pull request #2925 from matrix-org/erikj/split_sig_fed
...
Split out SignatureStore and EventFederationStore
2018-03-01 17:32:58 +00:00
65cf454fd1
Remove unused DataStore
2018-03-01 17:27:53 +00:00
9e08a93a7b
Merge pull request #2927 from matrix-org/erikj/read_marker_caches
...
Improve caching for read_marker API
2018-03-01 17:12:34 +00:00
4b44f05f19
Fewer lies are better
2018-03-01 17:08:17 +00:00
a83c514d1f
Improve caching for read_marker API
...
We add a new storage function to get a paritcular type of room account
data. This allows us to prefill the cache when updating that acount
data.
2018-03-01 17:08:17 +00:00
33bebb63f3
Add some caches to help read marker API
2018-03-01 17:08:17 +00:00
483e8104db
Merge pull request #2926 from matrix-org/erikj/member_handler_move
...
Move RoomMemberHandler out of Handlers
2018-03-01 17:01:25 +00:00
2ad4d5b5bb
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fed
2018-03-01 16:59:39 +00:00
529c026ac1
Move back to hs.is_mine
2018-03-01 16:49:12 +00:00
7c371834cc
Stub out broken function only used for cache
2018-03-01 16:44:13 +00:00
64346be26d
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_stream_store
2018-03-01 16:26:42 +00:00
22518e2833
Merge pull request #2923 from matrix-org/erikj/stream_ago_worker
...
Calculate stream_ordering_month_ago correctly on workers
2018-03-01 16:23:54 +00:00
884b26ae41
Remove unused variables
2018-03-01 16:23:48 +00:00
1b2af11650
Document abstract class and method better
2018-03-01 16:20:57 +00:00
872ff95ed4
Default stream_ordering_*_ago to None
2018-03-01 16:00:05 +00:00
22004b524e
Fix comment typo
2018-03-01 15:59:40 +00:00
4bc4236faf
Merge pull request #2922 from matrix-org/erikj/split_room_store
...
Split up RoomStore
2018-03-01 15:55:01 +00:00
2324124a72
Merge pull request #2921 from matrix-org/rav/unyielding_make_deferred_yieldable
...
Rewrite make_deferred_yieldable avoiding inlineCallbacks
2018-03-01 15:39:10 +00:00
f793bc3877
Split out stream store
2018-03-01 15:13:08 +00:00
784f036306
Move RoomMemberHandler out of Handlers
2018-03-01 14:36:50 +00:00
6411f725be
Calculate stream_ordering_month_ago correctly on workers
2018-03-01 14:20:53 +00:00
a9a2d66cdd
Split out SignatureStore and EventFederationStore
2018-03-01 14:17:53 +00:00
0c8ba5dd1c
Split up RoomStore
2018-03-01 14:01:19 +00:00
3a75de923b
Rewrite make_deferred_yieldable avoiding inlineCallbacks
...
... because (a) it's actually simpler (b) it might be marginally more
performant?
2018-03-01 12:40:05 +00:00
126b9bf96f
Log in the correct places
2018-03-01 12:05:33 +00:00
157298f986
Don't do preserve_fn for every request
2018-03-01 11:59:45 +00:00
89f90d808a
Add some logging
2018-03-01 11:59:16 +00:00
8ded8ba2c7
Make repl send_event idempotent and retry on timeouts
...
If we treated timeouts as failures on the worker we would attempt to
clean up e.g. push actions while the master might still process the
event.
2018-03-01 11:20:34 +00:00
f381d63813
Check event auth on the worker
2018-03-01 10:18:37 +00:00
6b8604239f
Correctly send ratelimit and extra_users params
2018-03-01 10:08:39 +00:00
f756f961ea
Fixup comments
2018-03-01 10:05:27 +00:00
28e973ac11
Calculate push actions on worker
2018-02-28 18:02:30 +00:00
493e25d554
Move storage functions for push calculations
...
This will allow push actions for an event to be calculated on workers.
2018-02-27 13:58:16 +00:00
3594dbc6dc
Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidation
...
Fix missing invalidations for receipt storage
2018-02-27 11:34:26 +00:00
2311189ee4
Merge pull request #2903 from matrix-org/erikj/split_roommember_store
...
Split out RoomMemberStore
2018-02-27 11:32:10 +00:00
c57607874c
Merge pull request #2901 from matrix-org/erikj/split_as_stores
...
Split AS stores
2018-02-27 10:07:07 +00:00
8956f0147a
Add comment
2018-02-27 10:06:51 +00:00
e5b4a208ce
Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions
...
Batch inserts into event_push_actions_staging
2018-02-26 14:45:40 +00:00
45b5fe9122
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/handle_unpersisted_events_push
2018-02-26 13:49:24 +00:00
d62ce972f8
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_roommember_store
2018-02-23 11:46:24 +00:00
6ae9a3d2a6
Update copyright
2018-02-23 11:44:49 +00:00
a90c60912f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_event_push_actions
2018-02-23 11:26:31 +00:00
50e8657867
Merge pull request #2902 from matrix-org/erikj/split_events_store
...
Split out get_events and co into a worker store
2018-02-23 11:23:52 +00:00
1cf9e071dd
Merge pull request #2899 from matrix-org/erikj/split_pushers
...
Split PusherStore
2018-02-23 11:23:35 +00:00
d0957753bf
Merge pull request #2898 from matrix-org/erikj/split_push_rules_store
...
Split PushRulesStore
2018-02-23 11:23:23 +00:00
199dba6c15
Merge pull request #2897 from matrix-org/erikj/split_account_data
...
Split AccountDataStore and TagStore
2018-02-23 11:23:11 +00:00
70349872c2
Update copyright
2018-02-23 11:14:35 +00:00
eba93b05bf
Split EventsWorkerStore into separate file
2018-02-23 11:01:21 +00:00
bf8a36e080
Update copyright
2018-02-23 10:52:10 +00:00
5d0f665848
Remove redundant clock
2018-02-23 10:49:58 +00:00
3bd760628b
_event_persist_queue shouldn't be in worker store
2018-02-23 10:49:18 +00:00
eb9b5eec81
Update copyright
2018-02-23 10:42:39 +00:00
c2ecfcc3a4
Update copyright
2018-02-23 10:41:34 +00:00
7e6cf89dc2
Update copyright
2018-02-23 10:39:19 +00:00
26d37f7a63
Update copyright
2018-02-23 10:33:55 +00:00
bb73f55fc6
Use absolute imports
2018-02-23 10:31:16 +00:00
faeb369f15
Fix missing invalidations for receipt storage
2018-02-21 15:19:54 +00:00
3dec9c66b3
Split out RoomMemberStore
2018-02-21 12:07:26 +00:00
46244b2759
Split AS stores
2018-02-21 11:49:34 +00:00
27b094f382
Split out get_events and co into a worker store
2018-02-21 11:41:48 +00:00
573712da6b
Update comments
2018-02-21 11:29:49 +00:00
c96d547f4d
Actually use new param
2018-02-21 11:03:42 +00:00
d15d237b0d
Split out EventPushActionWorkerStore
2018-02-21 11:01:13 +00:00
27939cbb0e
Merge pull request #2893 from matrix-org/erikj/delete_from_staging_fed
...
Delete from push_actions_staging in federation too
2018-02-21 11:00:06 +00:00
6f72765371
Split PusherStore
2018-02-21 10:54:21 +00:00
cbaad969f9
Split PushRulesStore
2018-02-21 10:43:31 +00:00
ca9b9d9703
Split AccountDataStore and TagStore
2018-02-21 10:15:04 +00:00
8fbb4d0d19
Raise exception in abstract method
2018-02-20 17:59:23 +00:00
95e4cffd85
Fix comment
2018-02-20 17:58:40 +00:00
e316bbb4c0
Use abstract base class to access stream IDs
2018-02-20 17:43:57 +00:00
f5ac4dc2d4
Split ReceiptsStore
2018-02-20 16:28:28 +00:00
24087bffa9
Ensure all push actions are deleted from staging
2018-02-20 12:34:31 +00:00
ad0ccf15ea
Refactor _set_push_actions_for_event_and_users_txn to use events_and_contexts
2018-02-20 12:34:28 +00:00
d874d4f2d7
Delete from push_actions_staging in federation too
2018-02-20 11:37:52 +00:00
6ff8c87484
Batch inserts into event_push_actions_staging
2018-02-20 11:33:07 +00:00
324c3e9399
Merge pull request #2868 from matrix-org/erikj/refactor_media_storage
...
Make store_file use store_into_file
2018-02-20 11:31:24 +00:00
3acd616979
Update pynacl dependency to 1.2.1 or higher
...
Signed-off-by: Pascal Bach <pascal.bach@nextrem.ch>
2018-02-19 10:45:22 +01:00
d1a3325f99
(Really) fix tablescan of event_push_actions on purge
...
commit 278d21b5
2018-02-16 14:02:31 +00:00
6af025d3c4
Fix typo of double is_highlight
2018-02-16 11:35:31 +00:00
012e8e142a
Comments
2018-02-16 11:35:01 +00:00
b96278d6fe
Ensure that we delete staging push actions on errors
2018-02-15 15:47:06 +00:00
4810f7effd
Remove context.push_actions
2018-02-15 15:47:06 +00:00
c714c61853
Update event_push_actions table from staging table
2018-02-15 15:47:06 +00:00
acac21248c
Store push actions in staging area
2018-02-15 15:47:04 +00:00
106906a65e
Don't serialize current state over replication
2018-02-15 13:53:18 +00:00
5fb347fc41
Merge pull request #2872 from matrix-org/erikj/event_worker_dont_log
...
Don't log errors propogated from send_event
2018-02-15 12:31:49 +00:00
fd1601c596
Fix state group storage bug in workers
...
We needed to move `_count_state_group_hops_txn` to the
StateGroupWorkerStore.
2018-02-15 11:04:32 +00:00
ef344b10e5
Don't log errors propogated from send_event
2018-02-15 11:03:49 +00:00
b8d821aa68
Merge pull request #2867 from matrix-org/rav/rework_purge
...
purge_history cleanups
2018-02-15 09:49:07 +00:00
92c52df702
Make store_file use store_into_file
2018-02-14 17:55:18 +00:00
d28ec43e15
Merge pull request #2769 from matrix-org/matthew/hit_the_gin
...
switch back from GIST to GIN indexes
2018-02-14 16:59:03 +00:00
39bf47319f
purge_history: fix sqlite syntax error
...
apparently sqlite insists on indexes being named
2018-02-14 16:42:19 +00:00
ac27f6a35e
purge_history: handle sqlite asshattery
...
apparently creating a temporary table commits the transaction. because that's a
useful thing.
2018-02-14 16:41:12 +00:00
5978dccff0
remove overzealous exception handling
2018-02-14 15:54:09 +00:00
278d21b5e4
purge_history: fix index use
...
event_push_actions doesn't have an index on event_id, so we need to specify
room_id.
2018-02-14 15:44:51 +00:00
5fcbf1e07c
Rework event purge to use a temporary table
...
... which should speed things up by reducing the amount of data being shuffled
across the connection
2018-02-14 11:02:22 +00:00
c0c9327fe0
Merge pull request #2854 from matrix-org/erikj/event_create_worker
...
Create a worker for event creation
2018-02-13 18:07:10 +00:00
d627174da2
Fix log message in purge_history
...
(we don't just remove remote events)
2018-02-13 16:51:21 +00:00
ddb6a79b68
Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin
2018-02-13 16:45:36 +00:00
0b27ae8dc3
move search reindex to schema 47
...
We're up to schema v47 on develop now, so this will have to go in there to have
an effect.
This might cause an error if somebody has already run it in the v46 guise, and
runs it again in the v47 guise, because it will cause a duplicate entry in the
bbackground_updates table. On the other hand, the entry is removed once it is
complete, and it is unlikely that anyone other than matrix.org has run it on
v46. The update itself is harmless to re-run because it deliberately copes with
the index already existing.
2018-02-13 16:44:46 +00:00
4a6d551704
GIN reindex: Fix syntax errors, improve exception handling
2018-02-13 16:44:46 +00:00
bfdf7b9237
Merge pull request #2864 from matrix-org/rav/persist_event_caching
...
Use StateResolutionHandler to resolve state in persist_events
2018-02-13 14:45:57 +00:00
630caf8a70
style nit
2018-02-13 14:29:22 +00:00
8fd1a32456
Fix typos in purge api & doc
...
* It's supposed to be purge_local_events, not ..._history
* Fix the doc to have valid json
2018-02-13 13:09:39 +00:00
4d09366656
Merge pull request #2695 from okurz/feature/allow_recent_pysaml
...
Allow use of higher versions of saml2
2018-02-13 12:24:08 +00:00
a9b712e9dc
Merge branch 'develop' into matthew/gin_work_mem
2018-02-13 12:16:01 +00:00
1026690cd2
Merge pull request #2857 from matrix-org/erikj/upload_store
...
Tell storage providers about new file so they can upload
2018-02-12 13:52:58 +00:00
10b34dbb9a
Merge pull request #2858 from matrix-org/rav/purge_updates
...
delete_local_events for purge_room_history
2018-02-09 14:11:00 +00:00
39a6b35496
purge: move room_depth update to end
...
... to avoid locking the table for too long
2018-02-09 13:07:41 +00:00
74fcbf741b
delete_local_events for purge_history
...
Add a flag which makes the purger delete local events
2018-02-09 13:07:41 +00:00
e571aef06d
purge: Move cache invalidation to more appropriate place
...
it was a bit of a non-sequitur there
2018-02-09 13:07:41 +00:00
61ffaa8137
bump purge logging to info
...
this thing takes ages and the only sign of any progress is the logs, so having
some logs is useful.
2018-02-09 13:07:41 +00:00
671540dccf
rename delete_old_state -> purge_history
...
(beacause it deletes more than state)
2018-02-09 13:07:41 +00:00
Luke Barnard