Commit Graph

6812 Commits

Author SHA1 Message Date
Erik Johnston 2f9eafdd36 Add local group server support 2017-07-17 12:03:49 +01:00
Erik Johnston bfde076022 Increase cache hit ratio for push
We don't update the cache in all code paths, which causes subsequent
calls to miss the cache
2017-07-14 16:11:26 +01:00
Erik Johnston cb3aee8219 Ensure category and role ids are non-null 2017-07-14 14:06:55 +01:00
Erik Johnston 85fda57208 Add DEFAULT_ROLE_ID 2017-07-14 14:03:54 +01:00
Erik Johnston 4b203bdba5 Correctly increment orders 2017-07-14 14:02:31 +01:00
Erik Johnston 8d26385d76 Add more metrics to push rule evaluation 2017-07-13 14:37:30 +01:00
Erik Johnston 3b0470dba5 Remove unused functions 2017-07-13 13:54:01 +01:00
Erik Johnston 8575e3160f Comments 2017-07-13 13:52:41 +01:00
Erik Johnston f60218ec41 Push: Don't acquire lock unless necessary 2017-07-13 11:23:53 +01:00
Erik Johnston a78cda4baf Remove TODO 2017-07-13 11:17:07 +01:00
Erik Johnston 7a39da8cc6 Add summary APIs to federation 2017-07-13 11:13:19 +01:00
Erik Johnston 5bbb53580a raise NotImplementedError 2017-07-13 10:25:29 +01:00
Erik Johnston 26451a09eb Comments 2017-07-12 14:47:18 +01:00
Erik Johnston 8d55877c9e Simplify checking if admin 2017-07-12 11:43:39 +01:00
Erik Johnston a62406aaa5 Add group summary APIs 2017-07-12 11:36:15 +01:00
Erik Johnston e9aec001f4 Use less DB for device list handling in sync 2017-07-12 10:30:10 +01:00
Erik Johnston 28e8c46f29 Merge pull request #2352 from matrix-org/erikj/group_server_split
Initial Group Server
2017-07-12 10:26:29 +01:00
Erik Johnston 6d586dc05c Comment 2017-07-12 09:58:37 +01:00
Erik Johnston 410b4e14a1 Move comment 2017-07-11 15:44:18 +01:00
Erik Johnston fe4e885f54 Add federation API for adding room to group 2017-07-11 14:35:07 +01:00
Erik Johnston bbb739d24a Comment 2017-07-11 14:31:36 +01:00
Erik Johnston 26752df503 Typo 2017-07-11 14:29:03 +01:00
Erik Johnston e52c391cd4 Rename column to attestation_json 2017-07-11 14:25:46 +01:00
Erik Johnston 0aac30d53b Comments 2017-07-11 14:23:50 +01:00
Erik Johnston 0184a97dbd Merge pull request #2354 from krombel/reduce_static_sync_reply
encode sync-response statically
2017-07-11 14:19:56 +01:00
Krombel 85b9f76f1d split out reducing stuff; just make encode_* static 2017-07-11 13:14:35 +02:00
Erik Johnston 6322fbbd41 Comment 2017-07-11 11:52:03 +01:00
Erik Johnston 8ba89f1050 Remove u/ requirement 2017-07-11 11:45:32 +01:00
Erik Johnston 429925a5e9 Lift out visibility parsing 2017-07-11 11:44:08 +01:00
Erik Johnston 83936293eb Comments 2017-07-11 11:42:25 +01:00
Erik Johnston e2cb760dcc Merge pull request #2357 from matrix-org/erikj/push
Don't compute push actions for backfilled events
2017-07-11 10:53:22 +01:00
Erik Johnston 925b3638ff Reduce log levels in tcp replication 2017-07-11 10:04:21 +01:00
Erik Johnston 9a6fd3ef29 Don't compute push actions for backfilled events 2017-07-11 10:02:21 +01:00
Krombel 2f82de18ee fix test 2017-07-10 17:34:58 +02:00
Erik Johnston b8ca494ee9 Initial group server implementation 2017-07-10 15:44:15 +01:00
Krombel 6e16aca8b0 encode sync-response statically; omit empty objects from sync-response 2017-07-10 16:42:17 +02:00
Erik Johnston d4d12daed9 Include registration and as stores in frontend proxy 2017-07-07 18:36:45 +01:00
Erik Johnston f467a8f66d Merge branch 'master' of github.com:matrix-org/synapse into develop 2017-07-07 18:26:28 +01:00
Erik Johnston 1fc4a962e4 Add a frontend proxy 2017-07-07 18:19:46 +01:00
Erik Johnston f502b0dea1 Perf: Don't filter events for push
We know the users are joined and we can explicitly check for if they are
ignoring the user, so lets do that.
2017-07-07 14:04:40 +01:00
Erik Johnston 76ed3476d3 Bump version and changelog 2017-07-06 18:11:22 +01:00
Erik Johnston 58dc1f2c78 Merge pull request #2342 from matrix-org/erikj/pusher_pool_instantiate
Fix bug where pusherpool didn't start and broke some rooms
2017-07-06 18:08:43 +01:00
Erik Johnston 5a7f561a9b Fix bug where pusherpool didn't start and broke some rooms
Since we didn't instansiate the PusherPool at start time it could fail
at run time, which it did for some users.

This may or may not fix things for those users, but it should happen at
start time and stop the server from starting.
2017-07-06 17:55:51 +01:00
Erik Johnston ed9a7f5436 Merge pull request #2309 from matrix-org/erikj/user_ip_repl
Fix up user_ip replication commands
2017-07-06 14:33:14 +01:00
Erik Johnston 1f64207f26 Merge branch 'master' of github.com:matrix-org/synapse into develop 2017-07-06 13:57:45 +01:00
Erik Johnston 6264cf9666 Bump version and changelog 2017-07-06 10:35:56 +01:00
Erik Johnston f386632800 Merge pull request #2334 from matrix-org/erikj/refactor_transport_server
Separate federation servlet into different lists
2017-07-05 17:09:07 +01:00
Erik Johnston 5e49a57ecc Separate federation servlet into different lists 2017-07-05 14:32:24 +01:00
Richard van der Hoff 3d31b39297 Merge pull request #2332 from matrix-org/rav/fix_pushes
Fix caching error in the push evaluator
2017-07-05 11:10:53 +01:00
Richard van der Hoff 73cfe48031 Fix caching error in the push evaluator
Initialising `result` to `{}` in the parameters meant that every call to
_flatten_dict used the *same* target dictionary.

I'm hopeful this will fix https://github.com/matrix-org/synapse/issues/2270,
but I suspect it won't. (This code seems to have been here since forever,
unlike the bug, and I don't really think it explains the observed
behaviour). Still, it makes it hard to investigate the problem.
2017-07-05 00:28:43 +01:00
Erik Johnston 05538587ef Bump version and changelog 2017-07-04 14:02:21 +01:00
Erik Johnston f92d7416d7 Merge pull request #2330 from matrix-org/erikj/cache_size_factor
Increase default cache size
2017-07-04 10:51:21 +01:00
Erik Johnston 495f075b41 Increase default cache factor size. 2017-07-04 09:58:32 +01:00
Erik Johnston b5e8d529e6 Define CACHE_SIZE_FACTOR once 2017-07-04 09:56:44 +01:00
Mark Haines 3e279411fe Improve the error handling for bad invites received over federation 2017-06-30 16:20:30 +01:00
Erik Johnston 6ff14ddd2e Make into list 2017-06-29 15:47:37 +01:00
Erik Johnston 5946aa0877 Prefill forward extrems and event to state groups 2017-06-29 15:38:48 +01:00
Erik Johnston 2c365f4723 Cache macaroon parse and validation
Turns out this can be quite expensive for requests, and is easily
cachable. We don't cache the lookup to the DB so invalidation still
works.
2017-06-29 14:50:18 +01:00
Erik Johnston c72058bcc6 Use an ExpiringCache for storing registration sessions
This is because pruning them was a significant performance drain on
matrix.org
2017-06-29 14:08:37 +01:00
Erik Johnston 27f26e48b7 Serialize user ip command as json 2017-06-27 16:25:38 +01:00
Erik Johnston 8c23221666 Fix up 2017-06-27 15:53:45 +01:00
Erik Johnston 731f3c37a0 Merge branch 'release-v0.22.0' of github.com:matrix-org/synapse into develop 2017-06-27 15:41:34 +01:00
Erik Johnston 4b444723f0 Merge pull request #2308 from matrix-org/erikj/user_ip_repl
Make workers report to master for user ip updates
2017-06-27 15:36:47 +01:00
Erik Johnston 816605a137 Merge pull request #2307 from matrix-org/erikj/user_ip_batch
Batch upsert user ips
2017-06-27 15:08:32 +01:00
Erik Johnston 78cefd78d6 Make workers report to master for user ip updates 2017-06-27 14:58:10 +01:00
Erik Johnston a0a561ae85 Fix up client ips to read from pending data 2017-06-27 14:46:12 +01:00
Erik Johnston ed3d0170d9 Batch upsert user ips 2017-06-27 13:37:04 +01:00
Erik Johnston 976128f368 Update version and changelog 2017-06-26 16:14:56 +01:00
Erik Johnston d04d672a80 Merge pull request #2290 from matrix-org/erikj/ensure_round_trip
Reject local events that don't round trip the DB
2017-06-26 15:12:02 +01:00
Erik Johnston 1bce3e6b35 Remove unused variables 2017-06-26 14:03:27 +01:00
Erik Johnston e3cbec10c1 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/ensure_round_trip 2017-06-26 14:02:44 +01:00
Erik Johnston 8abdd7b553 Fix up indices for users_who_share_rooms 2017-06-26 14:01:30 +01:00
Erik Johnston ff13c5e7af Merge pull request #2301 from xwiki-labs/push-redact-content
Add configuration parameter to allow redaction of content from push m…
2017-06-24 13:13:51 +01:00
Caleb James DeLisle 27bd0b9a91 Change the config file generator to more descriptive explanation of push.redact_content 2017-06-24 10:32:12 +02:00
Caleb James DeLisle bce144595c Fix TravisCI tests for PR #2301 - Fat finger mistake 2017-06-23 15:26:09 +02:00
Caleb James DeLisle 75eba3b07d Fix TravisCI tests for PR #2301 2017-06-23 15:15:18 +02:00
Caleb James DeLisle 1591eddaea Add configuration parameter to allow redaction of content from push messages for google/apple devices 2017-06-23 13:01:04 +02:00
Erik Johnston 7fe8ed1787 Store URL cache preview downloads seperately
This makes it easier to clear old media out at a later date
2017-06-23 11:14:11 +01:00
Erik Johnston 44c722931b Make some more params configurable 2017-06-22 14:59:52 +01:00
Erik Johnston 2d520a9826 Typo. ARGH. 2017-06-22 14:42:39 +01:00
Erik Johnston 24d894e2e2 Fix thinko in unhandled user spam 2017-06-22 14:39:05 +01:00
Matthew Hodgson ccfcef6b59 Merge branch 'master' into develop 2017-06-22 13:03:44 +01:00
Erik Johnston e0004aa28a Add desc 2017-06-22 10:03:48 +01:00
Erik Johnston b668112320 Merge pull request #2296 from matrix-org/erikj/dont_appserver_shar
Don't work out users who share room with appservice users
2017-06-21 14:50:24 +01:00
Erik Johnston dae9a00a28 Initialise exclusive_user_regex 2017-06-21 14:19:33 +01:00
Erik Johnston 8177563ebe Fix for workers 2017-06-21 13:57:49 +01:00
Krombel 4202fba82a Merge branch 'develop' into avoid_duplicate_filters 2017-06-21 14:48:21 +02:00
Krombel 812c030e87 replaced json.dumps with encode_canonical_json 2017-06-21 14:48:12 +02:00
Erik Johnston 1217c7da91 Don't work out users who share room with appservice users 2017-06-21 12:00:41 +01:00
Erik Johnston 385dcb7c60 Handle thumbnail urls 2017-06-19 17:48:28 +01:00
Erik Johnston b8b936a6ea Add API to quarantine media 2017-06-19 17:39:21 +01:00
Erik Johnston e5ae386ea4 Handle all cases of sending membership events 2017-06-19 16:07:54 +01:00
Erik Johnston 36e51aad3c Remove unused import 2017-06-19 14:42:21 +01:00
Erik Johnston b490299a3b Change to create new room and join other users 2017-06-19 14:10:13 +01:00
Erik Johnston 5db7070dd1 Forget room 2017-06-19 12:40:29 +01:00
Erik Johnston d7fe6b356c Add shutdown room API 2017-06-19 12:37:27 +01:00
Erik Johnston fcf01dd88e Reject local events that don't round trip the DB 2017-06-19 11:33:40 +01:00
Johannes Löthberg 4f66312df8 python_dependencies: Use bcrypt module instead of py-bcrypt
py-bcrypt has been unmaintained for a long while, while bcrypt is
actively maintained. And since ff8b87118d
we're compatible with the bcrypt anyway.

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2017-06-17 17:39:35 +02:00
Erik Johnston dfeca6cf40 Merge pull request #2286 from matrix-org/erikj/split_out_user_dir
Split out user directory to a separate process
2017-06-16 13:01:19 +01:00
Erik Johnston 6aa5bc8635 Initial worker impl 2017-06-16 11:47:11 +01:00
Erik Johnston d8f47d2efa Merge pull request #2280 from matrix-org/erikj/share_room_user_dir
Include users who you share a room with in user directory
2017-06-16 11:06:00 +01:00
Krombel 1ff419d343 allow Authorization header which handling got implemented in #1098
Signed-off-by: Matthias Kesler <krombel@krombel.de>
2017-06-16 11:21:14 +02:00
Erik Johnston fdf1ca30f0 Bump version and changelog 2017-06-15 12:59:06 +01:00
Erik Johnston 052c5d19d5 Merge pull request #2281 from matrix-org/erikj/phone_home_stats
Fix phone home stats
2017-06-15 12:46:23 +01:00
Erik Johnston 5ddd199870 Typo 2017-06-15 10:49:10 +01:00
Erik Johnston a9d6fa8b2b Include users who share room with requester in user directory 2017-06-15 10:17:21 +01:00
Erik Johnston 4564b05483 Implement updating users who share rooms on the fly 2017-06-15 10:17:17 +01:00
Erik Johnston 72613bc379 Implement initial population of users who share rooms table 2017-06-15 09:59:04 +01:00
Erik Johnston ebcd55d641 Add DB schema for tracking users who share rooms 2017-06-15 09:45:48 +01:00
Erik Johnston 4b461a6931 Add some more stats 2017-06-15 09:39:39 +01:00
Erik Johnston 617304b2cf Fix phone home stats 2017-06-14 19:47:15 +01:00
Erik Johnston d9fd937e39 Fix user directory insertion due to missing room_id 2017-06-13 11:50:24 +01:00
Erik Johnston 505e7e8b9d Fix up sql 2017-06-13 11:19:18 +01:00
Erik Johnston 6fd7e6db3d Fix user dir to not assume existence of user 2017-06-13 11:11:26 +01:00
Erik Johnston fdca6e36ee Merge pull request #2274 from matrix-org/erikj/cache_is_host_joined
Add cache for is_host_joined
2017-06-13 10:58:53 +01:00
Erik Johnston 90ae0cffec Merge pull request #2275 from matrix-org/erikj/tweark_user_directory_search
Tweak the ranking of PG user dir search
2017-06-13 10:58:43 +01:00
Erik Johnston de4cb50ca6 Merge pull request #2276 from matrix-org/erikj/fix_user_di
Don't assume existence of events when updating user directory
2017-06-13 10:55:41 +01:00
Erik Johnston 48d2949416 Throw exception when not retrying when downloading media 2017-06-13 10:23:14 +01:00
Erik Johnston 6ae8373d40 Don't assume existance of events when updating user directory 2017-06-13 10:19:26 +01:00
Erik Johnston b58e24cc3c Tweak the ranking of PG user dir search 2017-06-13 10:16:31 +01:00
Erik Johnston d53fe399eb Add cache for is_host_joined 2017-06-13 09:56:18 +01:00
Erik Johnston a837765e8c Merge pull request #2266 from matrix-org/erikj/host_in_room
Change is_host_joined to use current_state table
2017-06-12 09:49:51 +01:00
Erik Johnston 8060974344 Fix replication 2017-06-09 16:40:52 +01:00
Erik Johnston b0d975e216 Comments 2017-06-09 16:25:42 +01:00
Erik Johnston e54d7d536e Cache state deltas 2017-06-09 16:24:00 +01:00
Erik Johnston 1e9b4d5a95 Merge pull request #2268 from matrix-org/erikj/entity_has_changed
Fix has_any_entity_changed
2017-06-09 15:30:55 +01:00
Erik Johnston efc2b7db95 Rewrite conditional 2017-06-09 13:35:15 +01:00
Erik Johnston bfd68019c2 Merge pull request #2267 from matrix-org/erikj/missing_notifier
Fix removing of pushers when using workers
2017-06-09 13:07:29 +01:00
Erik Johnston 1664948e41 Comment 2017-06-09 13:05:05 +01:00
Erik Johnston 935e588799 Tweak SQL 2017-06-09 13:01:23 +01:00
Erik Johnston eed59dcc1e Fix has_any_entity_changed
Occaisonally has_any_entity_changed would throw the error: "Set changed
size during iteration" when taking the max of the `sorteddict`. While
its uncertain how that happens, its quite inefficient to iterate over
the entire dict anyway so we change to using the more traditional
`bisect_*` functions.
2017-06-09 11:44:01 +01:00
Erik Johnston 2cac7623a5 Add missing notifier 2017-06-09 11:24:41 +01:00
Erik Johnston 298d83b340 Fix replication 2017-06-09 11:01:28 +01:00
Erik Johnston 0185b75381 Change is_host_joined to use current_state table
This bypasses a bug where using the state groups to figure out if a host
is in a room sometimes errors if the servers isn't in the room. (For
example when the server rejected an invite to a remote room)
2017-06-09 10:52:26 +01:00
Erik Johnston 7132e5cdff Mark remote invite rejections as outliers 2017-06-09 10:08:18 +01:00
Erik Johnston ea11ee09f3 Ensure we don't use unpersisted state group as prev group 2017-06-08 11:59:57 +01:00
Erik Johnston c62c480dc6 Merge pull request #2259 from matrix-org/erikj/fix_state_woes
Fix bug where state_group tables got corrupted
2017-06-07 17:51:25 +01:00
Erik Johnston 197bd126f0 Fix bug where state_group tables got corrupted
This is due to the fact that we prefilled caches using txn.call_after,
which always gets called including on error.

We fix this by making txn.call_after only fire when a transaction
completes successfully, which is what we want most of the time anyway.
2017-06-07 17:39:36 +01:00
Erik Johnston f45f07ab86 Merge pull request #2258 from matrix-org/erikj/user_dir
Don't start user_directory handling on workers
2017-06-07 14:04:50 +01:00
Erik Johnston a053ff3979 Merge pull request #2248 from matrix-org/erikj/state_fixup
Faster cache for get_joined_hosts
2017-06-07 14:01:06 +01:00
Erik Johnston ecdd2a3658 Don't start user_directory handling on workers 2017-06-07 12:02:53 +01:00
Erik Johnston 2f34ad31ac Add some logging to user directory 2017-06-07 11:50:44 +01:00
Erik Johnston 64ed74c01e When pruning, delete from device_lists_outbound_last_success 2017-06-07 11:20:47 +01:00
Erik Johnston 1a81a1898e Keep pruning background task 2017-06-07 11:16:56 +01:00
Erik Johnston 6ba21bf2b8 Comments 2017-06-07 11:08:36 +01:00
Erik Johnston 09e4bc0501 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_fixup 2017-06-07 11:05:23 +01:00
Erik Johnston 6e2a7ee1bc Remove spurious log lines 2017-06-07 11:05:17 +01:00
Erik Johnston 65f0513a33 Split up device_lists_outbound_pokes table for faster updates. 2017-06-07 11:02:38 +01:00
Erik Johnston 6f83c4537c Increase size of IP cache 2017-06-07 10:18:44 +01:00
Erik Johnston cca94272fa Fix typo when getting app name 2017-06-06 11:50:07 +01:00
Erik Johnston 66b121b2fc Fix wrong number of arguments 2017-06-06 11:46:38 +01:00
Erik Johnston 1a01af079e Handle profile updates in user directory 2017-06-01 15:39:51 +01:00
Erik Johnston 4d039aa2ca Fix sqlite 2017-06-01 14:58:48 +01:00
Erik Johnston 21e255a8f1 Split the table in two 2017-06-01 14:50:46 +01:00
Erik Johnston d5477c7afd Tweak search query 2017-06-01 13:28:01 +01:00
Erik Johnston 02a6108235 Tweak search query 2017-06-01 13:16:40 +01:00
Erik Johnston 7233341eac Comments 2017-06-01 13:11:38 +01:00
Erik Johnston 8be6fd95a3 Check if host is still in room 2017-06-01 13:05:39 +01:00
Erik Johnston 59dbb47065 Remove spurious inlineCallbacks 2017-06-01 11:41:29 +01:00
Erik Johnston 9c7db2491b Fix removing users 2017-06-01 11:36:50 +01:00
Erik Johnston 0fe6f3c521 Bug fixes and logging
- Check if room is public when a user joins before adding to user dir
- Fix typo of field name "content.join_rules" -> "content.join_rule"
2017-06-01 11:09:49 +01:00
Erik Johnston 036362ede6 Order by if they have profile info 2017-06-01 09:41:08 +01:00
Erik Johnston a757dd4863 Use prefix matching 2017-06-01 09:40:37 +01:00
Erik Johnston f5cc22bdc6 Comment on why arbitrary comments 2017-05-31 17:30:26 +01:00
Erik Johnston 5dd1b2c525 Use unique indices 2017-05-31 17:29:12 +01:00
Erik Johnston cc7609aa9f Comment briefly on how we keep user_directory up to date 2017-05-31 17:11:18 +01:00
Erik Johnston f1378aef91 Convert to int 2017-05-31 17:03:08 +01:00
Erik Johnston b2d8d07109 Lifts things into separate function 2017-05-31 17:00:24 +01:00
Erik Johnston f9791498ae Typos 2017-05-31 16:50:57 +01:00
Erik Johnston f091061711 Fix tests 2017-05-31 16:34:40 +01:00
Erik Johnston 4abcff0177 Fix typo 2017-05-31 16:22:36 +01:00
Erik Johnston 63c58c2a3f Limit number of things we fetch out of the db 2017-05-31 16:17:58 +01:00
Erik Johnston 304880d185 Add stream change cache 2017-05-31 15:46:36 +01:00
Erik Johnston 5d79d728f5 Split out directory and search tables 2017-05-31 15:23:49 +01:00
Erik Johnston dc51af3d03 Pull max id from correct table 2017-05-31 15:13:49 +01:00
Erik Johnston 350622a107 Handle the server leaving a public room 2017-05-31 15:11:36 +01:00
Erik Johnston 63fda37e20 Add comments 2017-05-31 15:00:29 +01:00
Erik Johnston 293ef29655 Weight differently 2017-05-31 14:29:32 +01:00
Erik Johnston 535c99f157 Use POST 2017-05-31 14:15:45 +01:00
Erik Johnston 45a5df5914 Add REST API 2017-05-31 14:11:55 +01:00
Erik Johnston 3b5f22ca40 Add search 2017-05-31 14:00:01 +01:00
Erik Johnston b5db4ed5f6 Update room column when room becomes unpublic 2017-05-31 13:40:28 +01:00
Erik Johnston 168524543f Add call later 2017-05-31 11:59:36 +01:00
Erik Johnston 3e123b8497 Start later 2017-05-31 11:56:27 +01:00
Erik Johnston 42137efde7 Don't go round in circles 2017-05-31 11:55:13 +01:00
Erik Johnston eeb2f9e546 Add user_directory to database 2017-05-31 11:51:01 +01:00
Erik Johnston dd48f7204c Add comment 2017-05-30 15:01:22 +01:00
Erik Johnston 04095f7581 Add clobbered event_id 2017-05-30 14:53:01 +01:00
Erik Johnston a584a81b3e Add current_state_delta_stream table 2017-05-30 14:44:09 +01:00
Erik Johnston 619e8ecd0c Handle None state group correctly 2017-05-26 10:46:03 +01:00
Erik Johnston 23da638360 Fix typing tests 2017-05-26 10:02:04 +01:00
Erik Johnston dfbda5e025 Faster cache for get_joined_hosts 2017-05-25 17:24:44 +01:00
Erik Johnston 2b03751c3c Don't return weird prev_group 2017-05-25 14:47:39 +01:00
Erik Johnston dbc0dfd2d5 Remove unused options 2017-05-25 14:28:34 +01:00
Erik Johnston 11f139a647 Merge pull request #2247 from matrix-org/erikj/auth_event
Only store event_auth for state events
2017-05-24 16:46:34 +01:00
Erik Johnston 6e614e9e10 Add background task to clear out old event_auth 2017-05-24 15:23:34 +01:00
Erik Johnston c049472b8a Only store event_auth for state events 2017-05-24 15:23:31 +01:00
Erik Johnston 9a804b2812 Merge pull request #2243 from matrix-org/matthew/fix-url-preview-length-again
actually trim oversize og:description meta
2017-05-23 13:26:28 +01:00
Erik Johnston fbbc40f385 Merge pull request #2237 from matrix-org/erikj/sync_key_count
Add count of one time keys to sync stream
2017-05-23 11:18:13 +01:00
Erik Johnston 8cf9f0a3e7 Remove redundant invalidation 2017-05-23 09:46:59 +01:00
Erik Johnston e6618ece2d Missed an invalidation 2017-05-23 09:36:52 +01:00
Matthew Hodgson 836d5c44b6 actually trim oversize og:description meta 2017-05-22 21:14:20 +01:00
Erik Johnston 11c2a3655f Only load jinja2 templates once
Instead of every time a new email pusher is created, as loading jinja2
templates is slow.
2017-05-22 17:48:58 +01:00
Erik Johnston 539aa4d333 Merge pull request #2241 from matrix-org/erikj/fix_notifs
Correctly calculate push rules for member events
2017-05-22 16:46:58 +01:00
Erik Johnston f85a415279 Add missing storage function to slave store 2017-05-22 16:31:24 +01:00
Erik Johnston 6489455bed Comment 2017-05-22 16:22:04 +01:00
Erik Johnston d668caa79c Remove spurious log level guards 2017-05-22 16:21:06 +01:00
Erik Johnston 74bf4ee7bf Stream count_e2e_one_time_keys cache invalidation 2017-05-22 16:19:22 +01:00
Erik Johnston 33ba90c6e9 Merge pull request #2240 from matrix-org/erikj/cache_list_fix
Update list cache to handle one arg case
2017-05-22 16:10:46 +01:00
Erik Johnston bd7bb5df71 Pull out if statement from for loop 2017-05-22 15:12:19 +01:00
Erik Johnston e3417a06e2 Update list cache to handle one arg case
We update the normal cache descriptors to handle caches with a single
argument specially so that the key wasn't a 1-tuple. We need to update
the cache list to be aware of this.
2017-05-22 15:04:42 +01:00
Erik Johnston 7fb80b5eae Check if current event is a membership event 2017-05-22 15:02:12 +01:00
Erik Johnston 2d17b09a6d Add debug logging 2017-05-22 15:01:36 +01:00
Erik Johnston 24c8f38784 Comment 2017-05-22 14:59:27 +01:00
Erik Johnston 25f03cf8e9 Use tuple unpacking 2017-05-22 14:58:22 +01:00
Erik Johnston 270e1c904a Speed up calculating push rules 2017-05-19 16:51:05 +01:00
Erik Johnston b4f59c7e27 Add count of one time keys to sync stream 2017-05-19 15:47:55 +01:00
Erik Johnston 58ebb96cce Fix invalidation of get_users_with_read_receipts_in_room 2017-05-19 14:38:50 +01:00
Erik Johnston 1c1c0257f4 Move invalidation cb to its own structure 2017-05-19 11:44:11 +01:00
Erik Johnston cafe659f72 Store ActionGenerator in HomeServer 2017-05-19 10:09:56 +01:00
Erik Johnston 72ed8196b3 Don't push users who have left 2017-05-18 17:48:36 +01:00
Erik Johnston 107ac7ac96 Increase size of push rule caches 2017-05-18 17:17:53 +01:00
Erik Johnston 760625acba Make get_if_app_services_interested_in_user faster 2017-05-18 16:34:44 +01:00
Erik Johnston c57789d138 Remove size of push get_rules cache 2017-05-18 16:17:23 +01:00
Erik Johnston f33df30732 Merge branch 'master' of github.com:matrix-org/synapse into develop 2017-05-18 13:56:37 +01:00
Erik Johnston a5425b2e5b Bump changelog and version 2017-05-18 13:53:48 +01:00
Erik Johnston 6e381180ae Merge pull request #2177 from matrix-org/erikj/faster_push_rules
Make calculating push actions faster
2017-05-18 11:46:18 +01:00
Erik Johnston 056ba9b795 Add comment 2017-05-18 11:45:56 +01:00
Erik Johnston d9e3a4b5db Merge pull request #2230 from matrix-org/erikj/speed_up_get_state
Make get_state_groups_from_groups faster.
2017-05-17 17:23:04 +01:00
Erik Johnston 66d8ffabbd Faster push rule calculation via push specific cache
We add a push rule specific cache that ensures that we can reuse
calculated push rules appropriately when a user join/leaves.
2017-05-17 16:55:40 +01:00
Erik Johnston ace23463c5 Merge pull request #2216 from slipeer/app_services_interested_in_user
Fix users claimed non-exclusively by an app service don't get notific…
2017-05-17 16:28:50 +01:00
Erik Johnston bbfe4e996c Make get_state_groups_from_groups faster.
Most of the time was spent copying a dict to filter out sentinel values
that indicated that keys did not exist in the dict. The sentinel values
were added to ensure that we cached the non-existence of keys.

By updating DictionaryCache to keep track of which keys were known to
not exist itself we can remove a dictionary copy.
2017-05-17 15:12:15 +01:00
Erik Johnston a8bc7cae56 Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.21.0 2017-05-17 13:11:43 +01:00
Erik Johnston 3a431a126d Bump changelog and version 2017-05-17 11:26:57 +01:00
Erik Johnston 85e8092cca Comment 2017-05-17 10:03:09 +01:00
Erik Johnston ad53fc3cf4 Short circuit when we have delta ids 2017-05-17 09:57:34 +01:00
Erik Johnston 6fa8148ccb Merge pull request #2228 from matrix-org/erikj/speed_up_get_hosts
Speed up get_joined_hosts
2017-05-16 17:40:55 +01:00
Erik Johnston 7c69849a0d Merge pull request #2227 from matrix-org/erikj/presence_caches
Make presence use cached users/hosts in room
2017-05-16 17:40:47 +01:00
Erik Johnston 13f540ef1b Speed up get_joined_hosts 2017-05-16 16:05:22 +01:00
Erik Johnston ec5c4499f4 Make presence use cached users/hosts in room 2017-05-16 16:01:43 +01:00
Erik Johnston f2a5b6dbfd Speed up get_domain_from_id 2017-05-16 15:59:37 +01:00
Erik Johnston b8492b6c2f Merge pull request #2224 from matrix-org/erikj/prefill_state
Prefill state caches
2017-05-16 15:50:11 +01:00
Erik Johnston 331570ea6f Remove spurious merge artifacts 2017-05-16 15:33:07 +01:00
Krombel 55af207321 Merge branch 'develop' into avoid_duplicate_filters 2017-05-16 15:29:59 +02:00
Richard van der Hoff d648f65aaf Merge pull request #2218 from matrix-org/rav/event_search_index
Add an index to event_search
2017-05-16 13:26:07 +01:00
Erik Johnston 608b5a6317 Take a copy before prefilling, as it may be a frozendict 2017-05-16 12:55:29 +01:00
Krombel 64953c8ed2 avoid access-error if no filter_id matches 2017-05-15 18:36:37 +02:00
Erik Johnston f451b64c8f Merge branch 'develop' of github.com:matrix-org/synapse into erikj/prefill_state 2017-05-15 16:09:32 +01:00
Erik Johnston 2c9475b58e Merge pull request #2221 from psaavedra/sync_timeline_limit_filter_by_name
Configurable maximum number of events requested by /sync and /messages
2017-05-15 16:08:46 +01:00
Erik Johnston d12ae7fd1c Don't log exceptions for NotRetryingDestination 2017-05-15 15:42:18 +01:00
Pablo Saavedra 224137fcf9 Fixed syntax nits 2017-05-15 16:21:02 +02:00
Erik Johnston e4435b014e Update comment 2017-05-15 15:11:30 +01:00
Erik Johnston 871605f4e2 Comments 2017-05-15 15:11:30 +01:00
Erik Johnston e0d2f6d5b0 Add more granular event send metrics 2017-05-15 15:11:30 +01:00
Erik Johnston bfbc907cec Prefill state caches 2017-05-15 15:11:13 +01:00
Pablo Saavedra 627e6ea2b0 Fixed implementation errors
* Added HS as property in SyncRestServlet
* Fixed set_timeline_upper_limit function implementat¡ion
2017-05-15 14:51:43 +02:00
Pablo Saavedra 9da4316ca5 Configurable maximum number of events requested by /sync and /messages (#2220)
Set the limit on the returned events in the timeline in the get and sync
operations. The default value is -1, means no upper limit.

For example, using `filter_timeline_limit: 5000`:

POST /_matrix/client/r0/user/user:id/filter
{
room: {
    timeline: {
      limit: 1000000000000000000
    }
}
}

GET /_matrix/client/r0/user/user:id/filter/filter:id

{
room: {
    timeline: {
      limit: 5000
    }
}
}

The server cuts down the room.timeline.limit.
2017-05-13 18:17:54 +02:00
Krombel eb7cbf27bc insert whitespace to fix travis build 2017-05-12 12:09:42 +02:00
Krombel 6b95e35e96 add check to only add a new filter if the same filter does not exist previously
Signed-off-by: Matthias Kesler <krombel@krombel.de>
2017-05-11 16:05:30 +02:00
Richard van der Hoff ff3d810ea8 Add a comment to old delta 2017-05-11 12:48:50 +01:00
Richard van der Hoff 34194aaff7 Don't create event_search index on sqlite
... because the table is virtual
2017-05-11 12:46:55 +01:00
Richard van der Hoff 114f290947 Add more logging for purging
Log the number of events we will be deleting at info.
2017-05-11 12:08:47 +01:00
Richard van der Hoff baafb85ba4 Add an index to event_search
- to make the purge API quicker
2017-05-11 12:05:22 +01:00
Richard van der Hoff 29ded770b1 Merge pull request #2214 from matrix-org/rav/hurry_up_purge
When purging, don't de-delta state groups we're about to delete
2017-05-11 12:04:25 +01:00
Richard van der Hoff dc026bb16f Tidy purge code and add some comments
Try to make this clearer with more comments and some variable renames
2017-05-11 10:56:12 +01:00
Slipeer 328378f9cb Fix users claimed non-exclusively by an app service don't get notifications #2211 2017-05-11 11:42:08 +03:00
Richard van der Hoff 8e345ce465 Don't de-delta state groups we're about to delete 2017-05-10 18:44:22 +01:00
Richard van der Hoff b64d312421 add some logging to purge_history 2017-05-10 18:44:22 +01:00
Luke Barnard ccad2ed824 Modify condition on empty localpart 2017-05-10 17:34:30 +01:00
Luke Barnard 369195caa5 Modify register/available to be GET with query param
- GET is now the method for register/available
- a query parameter "username" is now used

Also, empty usernames are now handled with an error message on registration or via register/available: `User ID cannot be empty`
2017-05-10 17:23:55 +01:00
Erik Johnston a3648f84b2 Merge pull request #2208 from matrix-org/erikj/ratelimit_overrid
Add per user ratelimiting overrides
2017-05-10 15:54:48 +01:00
Richard van der Hoff 5331cd150a Merge pull request #2206 from matrix-org/rav/one_time_key_upload_change_sig
Allow clients to upload one-time-keys with new sigs
2017-05-10 14:18:40 +01:00
Luke Barnard f7278e612e Change register/available to POST (from GET) 2017-05-10 11:40:18 +01:00
Erik Johnston b990b2fce5 Add per user ratelimiting overrides 2017-05-10 11:05:43 +01:00
Richard van der Hoff aedaba018f Replace some instances of preserve_context_over_deferred 2017-05-09 19:04:56 +01:00
Richard van der Hoff de042b3b88 Do some logging when one-time-keys get claimed
might help us figure out if https://github.com/vector-im/riot-web/issues/3868
has happened.
2017-05-09 19:04:56 +01:00
Richard van der Hoff a7e9d8762d Allow clients to upload one-time-keys with new sigs
When a client retries a key upload, don't give an error if the signature has
changed (but the key is the same).

Fixes https://github.com/vector-im/riot-android/issues/1208, hopefully.
2017-05-09 19:04:56 +01:00
Erik Johnston ca238bc023 Merge branch 'release-v0.21.0' of github.com:matrix-org/synapse into develop 2017-05-08 17:35:11 +01:00
Erik Johnston 40dcf0d856 Merge pull request #2203 from matrix-org/erikj/event_cache_hit_ratio
Don't update event cache hit ratio from get_joined_users
2017-05-08 16:52:39 +01:00
Erik Johnston d3c3026496 Merge pull request #2201 from matrix-org/erikj/store_device_cache
Cache check to see if device exists
2017-05-08 16:23:04 +01:00
Erik Johnston 093f7e47cc Expand docstring a bit 2017-05-08 16:14:46 +01:00
Erik Johnston 6a12998a83 Add missing yields 2017-05-08 16:10:51 +01:00
Erik Johnston ffad4fe35b Don't update event cache hit ratio from get_joined_users
Otherwise the hit ration of plain get_events gets completely skewed by
calls to get_joined_users* functions.
2017-05-08 16:06:17 +01:00
Erik Johnston 94e6ad71f5 Invalidate cache on device deletion 2017-05-08 15:55:59 +01:00
Erik Johnston 8571f864d2 Cache one time key counts 2017-05-08 15:34:27 +01:00
Erik Johnston fc6d4974a6 Comment 2017-05-08 15:33:57 +01:00
Erik Johnston 738ccf61c0 Cache check to see if device exists 2017-05-08 15:32:18 +01:00
Erik Johnston dcabef952c Increase client_ip cache size 2017-05-08 15:09:19 +01:00
Erik Johnston 771c8a83c7 Bump version and changelog 2017-05-08 13:23:46 +01:00
Erik Johnston 6631985990 Merge pull request #2200 from matrix-org/erikj/revert_push
Revert speed up push
2017-05-08 13:20:52 +01:00
Erik Johnston e0f20e9425 Revert "Remove unused import"
This reverts commit ab37bef83b.
2017-05-08 13:07:43 +01:00
Erik Johnston fe7c1b969c Revert "We don't care about forgotten rooms"
This reverts commit ad8b316939.
2017-05-08 13:07:43 +01:00
Erik Johnston 78f306a6f7 Revert "Speed up filtering of a single event in push"
This reverts commit 421fdf7460.
2017-05-08 13:07:41 +01:00
Erik Johnston 9ac98197bb Bump version and changelog 2017-05-08 11:07:54 +01:00
Erik Johnston 27c28eaa27 Merge pull request #2190 from matrix-org/erikj/mark_remote_as_back_more
Always mark remotes as up if we receive a signed request from them
2017-05-05 14:08:12 +01:00
Erik Johnston be2672716d Merge pull request #2189 from matrix-org/erikj/handle_remote_device_list
Handle exceptions thrown in handling remote device list updates
2017-05-05 14:01:27 +01:00
Erik Johnston 653d90c1a5 Comment 2017-05-05 14:01:17 +01:00
Erik Johnston 310b1ccdc1 Use preserve_fn and add logs 2017-05-05 13:41:19 +01:00
Kegsay a59b0ad1a1 Merge pull request #2192 from matrix-org/kegan/simple-http-client-timeouts
Rewrite SimpleHttpClient.request to include timeouts
2017-05-05 11:52:43 +01:00
Erik Johnston 7b222fc56e Remove redundant reset of destination timers 2017-05-05 11:14:09 +01:00