f9340ea0d5
Merge branch 'erikj/store_engine' into erikj/search
2015-10-13 13:29:02 +01:00
ec398af41c
Expose error more nicely
2015-10-13 11:43:43 +01:00
40b6a5aad1
Split out the schema preparation and update logic into its own module
2015-10-13 11:38:48 +01:00
1a40afa756
Add sqlite schema
2015-10-13 10:36:25 +01:00
ca53ad7425
Filter events to only thsoe that the user is allowed to see
2015-10-12 15:52:55 +01:00
927004e349
Remove unused room_id parameter
2015-10-12 15:06:14 +01:00
61561b9df7
Keep FTS indexes up to date. Only search through rooms currently joined
2015-10-12 10:49:53 +01:00
c85c912562
Add basic full text search impl.
2015-10-09 15:48:31 +01:00
301141515a
Merge pull request #288 from matrix-org/markjh/unused_definitions
...
Remove some of the unused definitions from synapse
2015-09-28 14:22:44 +01:00
cf1100887b
Fix order of ON constraints in _get_rooms_for_user_where_membership_is_txn
2015-09-24 17:35:10 +01:00
1cd65a8d1e
synapse/storage/state.py: _make_group_id was unused
2015-09-23 10:37:58 +01:00
973ebb66ba
Remove unused functions from synapse/storage/signatures.py
2015-09-23 10:36:33 +01:00
e51aa4be96
synapse/storage/roommember.py:_get_members_query was unused
2015-09-23 10:35:10 +01:00
92d8d724c5
Remove unused functions from synapse/storage/events.py
2015-09-23 10:33:06 +01:00
c292dba70c
Remove unused functions from synapse/storage/event_federation.py
2015-09-23 10:31:25 +01:00
396834f1c0
synapse/storage/_base.py:_simple_max_id was unused
2015-09-23 10:30:38 +01:00
1d9036aff2
synapse/storage/_base.py:_simple_delete was unused
2015-09-23 10:30:25 +01:00
1ee3d26432
synapse/storage/_base.py:_simple_selectupdate_one was unused
2015-09-23 10:30:03 +01:00
3559a835a2
synapse/storage/event_federation.py:_get_auth_events is unused
2015-09-22 18:39:46 +01:00
7dd4f79c49
synapse/storage/_base.py:_execute_and_decode was unused
2015-09-22 18:37:07 +01:00
527d95dea0
synapse/storage/_base.py:Table was unused
2015-09-22 18:14:15 +01:00
6d59ffe1ce
Add some docstrings
2015-09-22 13:47:40 +01:00
eb011cd99b
Add docstring
2015-09-22 13:29:36 +01:00
7213588083
Implement configurable stats reporting
...
SYN-287
This requires that HS owners either opt in or out of stats reporting.
When --generate-config is passed, --report-stats must be specified
If an already-generated config is used, and doesn't have the
report_stats key, it is requested to be set.
2015-09-22 12:57:40 +01:00
ee2d722f0f
Merge pull request #276 from matrix-org/markjh/history_for_rooms_that_have_been_left
...
SPEC-216: Allow users to view the history of rooms that they have left.
2015-09-21 14:38:13 +01:00
2048388cfd
Merge pull request #281 from matrix-org/erikj/update_extremeties
...
When updating a stored event from outlier to non-outlier, remember to update the extremeties
2015-09-15 16:57:25 +01:00
8148c48f11
"Comments"
2015-09-15 16:54:48 +01:00
3b05b67c89
When updating a stored event from outlier to non-outlier, remember to update the extremeties
2015-09-15 16:34:42 +01:00
dffc9c4ae0
Drop unused index
2015-09-14 14:41:37 +01:00
09cb5c7d33
Allow users that have left a room to get the messages that happend in the room before they left
2015-09-09 17:31:09 +01:00
3c166a24c5
Remove undocumented and unimplemented 'feedback' parameter from the Client-Server API
2015-09-09 16:05:09 +01:00
89ae0166de
Allow room initialSync for users that have left the room, returning a snapshot of how the room was when they left it
2015-09-09 13:25:22 +01:00
417485eefa
Include the event_id and stream_ordering of membership events when looking up which rooms a user is in
2015-09-08 18:14:54 +01:00
b62c1395d6
Merge branch 'release-v0.10.0' of github.com:matrix-org/synapse into develop
2015-09-01 13:11:55 +01:00
cd800ad99a
Lower size of 'stateGroupCache' now that we have data from matrix.org to support doing so
2015-09-01 10:09:03 +01:00
a7122692d9
Merge branch 'release-v0.10.0' into develop
...
Conflicts:
synapse/handlers/auth.py
synapse/python_dependencies.py
synapse/rest/client/v1/login.py
2015-08-28 11:15:27 +01:00
d3c0e48859
Merge erikj/user_dedup to develop
2015-08-26 13:42:45 +01:00
a2355fae7e
Merge pull request #251 from matrix-org/removeadmin
...
Stop looking up "admin", which we never read
2015-08-25 17:23:05 +01:00
ee3fa1a99c
Merge pull request #248 from matrix-org/deviceid
...
Remove completely unused concepts from codebase
2015-08-25 17:19:06 +01:00
a9d8bd95e7
Stop looking up "admin", which we never read
2015-08-25 16:29:39 +01:00
a0b181bd17
Remove completely unused concepts from codebase
...
Removes device_id and ClientInfo
device_id is never actually written, and the matrix.org DB has no
non-null entries for it. Right now, it's just cluttering up code.
This doesn't remove the columns from the database, because that's
fiddly.
2015-08-25 16:23:06 +01:00
1c847af28a
Merge pull request #243 from matrix-org/markjh/remove_syutil
...
Replace syutil dependency with smaller, single-purpose libraries
2015-08-25 10:52:16 +01:00
037481a033
Remove autoincrement since we incrementing the ID in the storage layer
2015-08-24 17:48:57 +01:00
01fc3943f1
Fix indent
2015-08-24 17:18:58 +01:00
78323ccdb3
Remove syutil dependency in favour of smaller single-purpose libraries
2015-08-24 16:17:38 +01:00
457970c724
Don't insert events into 'event_*_extremeties' tables if they're outliers
2015-08-23 13:44:23 +01:00
1bd1a43073
Actually check if event_id isn't returned by _get_state_groups
2015-08-21 14:30:34 +01:00
745b72660a
Merge branch 'release-v0.10.0' of github.com:matrix-org/synapse into develop
2015-08-21 11:39:38 +01:00
42f12ad92f
When logging in fetch user by user_id case insensitively, *unless* there are multiple case insensitive matches, in which case require the exact user_id
2015-08-21 11:38:44 +01:00
aa3c9c7bd0
Don't allow people to register user ids which only differ by case to an existing one
2015-08-21 10:57:47 +01:00
1f7642efa9
Fix bug where we didn't correctly serialize the redacted_because key over federation
2015-08-21 09:36:07 +01:00
e8cf77fa49
Merge branch 'develop' into refresh
...
Conflicts:
synapse/rest/client/v1/login.py
2015-08-20 16:25:40 +01:00
cecbd636e9
/tokenrefresh POST endpoint
...
This allows refresh tokens to be exchanged for (access_token,
refresh_token).
It also starts issuing them on login, though no clients currently
interpret them.
2015-08-20 16:21:35 +01:00
13a6517d89
s/by_token/by_access_token/g
...
We're about to have two kinds of token, access and refresh
2015-08-20 16:01:29 +01:00
a0b8e5f2fe
Merge pull request #211 from matrix-org/email_in_use
...
Changes for unique emails
2015-08-20 10:04:04 +01:00
e624cdec64
Merge pull request #228 from matrix-org/erikj/_get_state_for_groups
...
Ensure we never return a None event from _get_state_for_groups
2015-08-18 16:30:17 +01:00
38a965b816
Merge pull request #227 from matrix-org/erikj/receipts_take2
...
Re-enable receipts API.
2015-08-18 16:30:04 +01:00
a82938416d
Remove newline because vertical whitespace makes mjark sad
2015-08-18 16:28:13 +01:00
0bfdaf1f4f
Rejig the code to make it nicer
2015-08-18 16:26:07 +01:00
128ed32e6b
Bump size of get_presence_state cache
2015-08-18 15:51:23 +01:00
f704c10f29
Rename unhelpful variable name
2015-08-18 11:54:03 +01:00
8199475ce0
Ensure we never return a None event from _get_state_for_groups
2015-08-18 11:44:10 +01:00
85d0bc3bdc
Reduce cache size from obscenely large to quite large
2015-08-18 11:00:38 +01:00
cfc503681f
Comments
2015-08-18 10:49:23 +01:00
8f4165628b
Add index receipts_linearized_room_stream
2015-08-17 14:43:54 +01:00
d3d582bc1c
Remove unused import
2015-08-17 13:38:09 +01:00
afef6f5d16
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts_take2
2015-08-17 13:23:44 +01:00
2d97e65558
Remember to invalidate caches
2015-08-17 10:46:55 +01:00
1a9510bb84
Implement a batched presence_handler.get_state and use it
2015-08-17 10:40:23 +01:00
47abebfd6d
Add batched version of store.get_presence_state
2015-08-17 09:50:50 +01:00
f9d4da7f45
Fix bug where we were leaking None into state event lists
2015-08-17 09:39:45 +01:00
30883d8409
Merge pull request #221 from matrix-org/auth
...
Simplify LoginHander and AuthHandler
2015-08-14 17:02:22 +01:00
68b255c5a1
Batch _get_linearized_receipts_for_rooms
2015-08-14 15:06:22 +01:00
129ee4e149
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts_take2
2015-08-13 17:28:43 +01:00
2bb2c02571
Remove some vertical space
2015-08-13 17:11:30 +01:00
57877b01d7
Replace list comprehension
2015-08-13 17:00:17 +01:00
5db5677969
Add metrics to the receipts cache
2015-08-13 16:58:23 +01:00
0fbed2a8fa
Comment
2015-08-12 17:22:54 +01:00
c10ac7806e
Explain why we're prefilling dict with Nones
2015-08-12 17:16:30 +01:00
101ee3fd00
Better variable name
2015-08-12 17:08:05 +01:00
df361d08f7
Split _get_state_for_group_from_cache into two
2015-08-12 17:06:21 +01:00
415c2f0549
Simplify LoginHander and AuthHandler
...
* Merge LoginHandler -> AuthHandler
* Add a bunch of documentation
* Improve some naming
* Remove unused branches
I will start merging the actual logic of the two handlers shortly
2015-08-12 15:49:37 +01:00
2df8dd9b37
Move all the caches into their own package, synapse.util.caches
2015-08-11 18:00:59 +01:00
53a817518b
Comments
2015-08-11 11:40:40 +01:00
6eaa116867
Comment
2015-08-11 11:35:24 +01:00
4762c276cb
Docs
2015-08-11 11:33:41 +01:00
1b994a97dd
Fix application of ACLs
2015-08-11 10:41:40 +01:00
10b874067b
Fix state cache
2015-08-11 09:12:41 +01:00
017b798e4f
Clean up StateStore
2015-08-10 15:01:06 +01:00
2c019eea11
Remove unused function
2015-08-10 14:44:41 +01:00
bb0a475c30
Comments
2015-08-10 14:27:38 +01:00
559c51debc
Use TypeError instead of ValueError and give a nicer error mesasge
...
when someone calls Cache.invalidate with the wrong type.
2015-08-10 14:07:17 +01:00
7ce71f2ffc
Merge branch 'erikj/cache_varargs_interface' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-10 13:47:51 +01:00
8c3a62b5c7
Merge pull request #215 from matrix-org/erikj/cache_varargs_interface
...
Change Cache to not use *args in its interface
2015-08-10 13:47:45 +01:00
86eaaa885b
Rename keyargs to args in CacheDescriptor
2015-08-10 13:44:44 +01:00
e0b6e49466
Merge branch 'erikj/cache_varargs_interface' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-10 10:39:22 +01:00
2cd6cb9f65
Rename keyargs to args in Cache
2015-08-10 10:38:47 +01:00
aa88582e00
Do bounds check
2015-08-10 10:08:15 +01:00
5119e416e8
Line length
2015-08-10 10:05:30 +01:00
8f04b6fa7a
Merge branch 'erikj/cache_varargs_interface' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-07 19:30:25 +01:00
7dec0b2bee
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-07 19:28:39 +01:00
06218ab125
Merge pull request #212 from matrix-org/erikj/cache_deferreds
...
Make CacheDescriptor cache deferreds rather than the deferreds' values
2015-08-07 19:28:05 +01:00
2352974aab
Merge branch 'erikj/cache_deferreds' of github.com:matrix-org/synapse into erikj/cache_varargs_interface
2015-08-07 19:26:54 +01:00
9c5385b53a
s/observed/observer/
2015-08-07 19:26:38 +01:00
ffab798a38
Merge branch 'erikj/cache_deferreds' of github.com:matrix-org/synapse into erikj/cache_varargs_interface
2015-08-07 19:18:47 +01:00
62126c996c
Propogate stale cache errors to calling functions
2015-08-07 19:17:58 +01:00
3213ff630c
Remove unnecessary cache
2015-08-07 19:14:05 +01:00
20addfa358
Change Cache to not use *args in its interface
2015-08-07 18:32:47 +01:00
9eb5b23d3a
Batch up various DB requests for event -> state
2015-08-07 18:16:02 +01:00
0211890134
Implement a CacheListDescriptor
2015-08-07 18:14:49 +01:00
0db40d3e93
Don't complain about extra .pyc files we find while hunting for database schemas
2015-08-07 17:22:11 +01:00
b2c7bd4b09
Cache get_recent_events_for_room
2015-08-07 14:42:34 +01:00
b3768ec10a
Remove unncessary cache
2015-08-07 13:41:05 +01:00
b8e386db59
Change Cache to not use *args in its interface
2015-08-07 11:52:21 +01:00
fe994e728f
Store absence of state in cache
2015-08-07 10:17:38 +01:00
1d08bf7c17
Merge branch 'erikj/cache_deferreds' into erikj/dictionary_cache
2015-08-06 14:03:15 +01:00
63b1eaf32c
Docs
2015-08-06 14:02:50 +01:00
b811c98574
Remove failed deferreds from cache
2015-08-06 14:01:27 +01:00
433314cc34
Re-implement DEBUG_CACHES flag
2015-08-06 14:01:05 +01:00
f596ff402e
Merge branch 'erikj/cache_deferreds' into erikj/dictionary_cache
2015-08-06 13:37:56 +01:00
2efb93af52
Merge branch 'erikj/cached_keyword_args' into erikj/cache_deferreds
2015-08-06 13:35:28 +01:00
953dbd28a7
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cached_keyword_args
2015-08-06 13:35:03 +01:00
7eea3e356f
Make @cached cache deferreds rather than the deferreds' values
2015-08-06 13:33:34 +01:00
3e1b77efc2
Merge branch 'erikj/cached_keyword_args' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-05 16:45:56 +01:00
b52b4a84ec
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/dictionary_cache
2015-08-05 15:41:20 +01:00
1e62a3d3a9
Up the cache size for 'get_joined_hosts_for_room' and 'get_users_in_room'
2015-08-05 15:40:40 +01:00
a89559d797
Use LRU cache by default
2015-08-05 15:39:47 +01:00
07507643cb
Use dictionary cache to do group -> state fetching
2015-08-05 15:11:42 +01:00
c67ba143fa
Move DictionaryCache
2015-08-04 15:58:28 +01:00
e7768e77f5
Add basic dictionary cache
2015-08-04 15:56:56 +01:00
e124128542
Bump schema version
2015-08-04 14:50:31 +01:00
c77048e12f
Add endpoint that proxies ID server request token and errors if the given email is in use on this Home Server.
2015-08-04 14:37:09 +01:00
413a4c289b
Add comment
2015-08-04 11:08:07 +01:00
4d6cb8814e
Speed up event filtering (for ACL) logic
2015-08-04 09:32:23 +01:00
39e21ea51c
Add support for using keyword arguments with cached functions
2015-07-27 13:57:29 +01:00
20c0324e9c
Dodesn't seem to make any difference: guess it does work with the object reference
2015-07-21 16:21:37 -07:00
cf7a40b08a
I think this was what was intended...
2015-07-21 16:08:00 -07:00
3b5823c74d
s/take/claim/ for end to end key APIs
2015-07-20 18:23:54 +01:00
4f973eb657
Up default cache size for _RoomStreamChangeCache
2015-07-18 19:07:33 +01:00
b6d4a4c6d8
Merge pull request #199 from matrix-org/erikj/receipts
...
Implement read receipts.
2015-07-16 18:18:36 +01:00
baa55fb69e
Merge pull request #193 from matrix-org/erikj/bulk_persist_event
...
Add bulk insert events API
2015-07-14 10:49:24 +01:00
4624d6035e
Docs
2015-07-14 10:19:07 +01:00
5989637f37
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts
2015-07-13 13:50:57 +01:00
e5991af629
Comments
2015-07-13 13:30:43 +01:00
17bb9a7eb9
Remove commented out code
2015-07-10 14:07:57 +01:00
532fcc997a
Merge pull request #196 from matrix-org/erikj/room_history
...
Add ability to restrict room history.
2015-07-10 13:47:04 +01:00
0b3389bcd2
Merge pull request #194 from matrix-org/erikj/bulk_verify_sigs
...
Implement bulk verify_signed_json API
2015-07-10 13:46:53 +01:00
0d7f0febf4
Uniquely name unique constraint
2015-07-10 13:43:03 +01:00
ed88720952
Handle error slightly better
2015-07-09 16:14:46 +01:00
bf0d59ed30
Don't bother with a timeout for one time keys on the server.
2015-07-09 14:04:03 +01:00
1af188209a
Change format of receipts to allow inclusion of data
2015-07-09 11:39:30 +01:00
8fb79eeea4
Only remove one time keys when new one time keys are added
2015-07-08 17:04:29 +01:00
304343f4d7
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts
2015-07-08 15:37:33 +01:00
af812b68dd
Add a cache to fetching of receipt streams
2015-07-08 15:35:00 +01:00
87311d1b8c
Hook up receipts to v1 initialSync
2015-07-08 11:02:04 +01:00
ca041d5526
Wire together receipts and the notifer/federation
2015-07-07 15:25:30 +01:00
716e426933
Fix various typos
2015-07-07 10:55:31 +01:00
b5770f8947
Add store for client end to end keys
2015-07-06 18:46:47 +01:00
65e69dec8b
Don't explode if we don't recognize one of the event_ids in the backfill request
2015-07-06 09:33:03 +01:00
1a60545626
Add basic impl for room history ACL on GET /messages client API
2015-07-02 16:20:10 +01:00
ac78e60de6
Add stream_id index
2015-07-02 13:18:41 +01:00
80a61330ee
Add basic storage functions for handling of receipts
2015-07-01 17:19:12 +01:00
901f56fa63
Add tables for receipts
2015-06-30 15:29:47 +01:00
b5f55a1d85
Implement bulk verify_signed_json API
2015-06-26 10:39:34 +01:00
5130d80d79
Add bulk insert events API
2015-06-25 17:29:34 +01:00
6924852592
Batch SELECTs in _get_auth_chain_ids_txn
2015-06-23 11:01:04 +01:00
2eda996a63
Add a dummy.sql into delta/20 as pip isn't packinging the pushers.py
2015-06-13 12:21:58 +01:00
1e365e88bd
Bump schema version
2015-06-11 15:50:39 +01:00
da84946de4
pep8
2015-06-04 16:43:45 +01:00
63a7b3ad1e
Add script to (re)convert the pushers table to changing the unique key. Also give the python db upgrade scripts the database engine so they can convert parameter strings, and add *args **kwargs to the upgrade function so we can add more args in future and previous scripts will ignore them.
2015-06-04 16:16:01 +01:00
0d7d9c37b6
Add cache to get_state_groups
2015-06-03 14:45:55 +01:00
d8866d7277
Caches should be bound to instances.
...
Before, caches were global and so different instances of the stores
would share caches. This caused problems in the unit tests.
2015-06-03 14:45:17 +01:00
6e17463228
Don't explode if we don't have the event
2015-06-01 16:39:43 +01:00
a7b65bdedf
Add config option to turn off freezing events. Use new encode_json api and ujson.loads
2015-05-29 12:17:33 +01:00
d9f60e8dc8
Merge pull request #163 from matrix-org/markjh/presence_list_cache
...
Add a cache for the presence list
2015-05-22 17:02:23 +01:00
0e42dfbe22
Merge pull request #164 from matrix-org/markjh/pusher_performance_2
...
Add a cache for get_push rules for user, fix cache invalidation
2015-05-22 17:01:56 +01:00
1a9a9abcc7
Add a cache for getting the presence list for a user
2015-05-22 16:11:17 +01:00
74b7de83ec
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/backfill_fixes
2015-05-22 16:10:42 +01:00
36317f3dad
Merge pull request #156 from matrix-org/erikj/join_perf
...
Make joining #matrix:matrix.org over federation quicker
2015-05-22 16:09:54 +01:00
b6adfc59f5
Invalidate the get_latest_event_ids_in_room cache when deleting from event_forward_extremities
2015-05-22 13:01:03 +01:00
a04cde613e
Add a cache for get_push rules for user, fix cache invalidation
2015-05-22 10:39:45 +01:00
4429e720ae
Merge branch 'master' of github.com:matrix-org/synapse into develop
2015-05-22 10:33:00 +01:00
2043527b9b
Don't try to use a txn when not in one, remove spurious debug logging
2015-05-21 16:53:03 +01:00
53447e9cd3
Add caches for things requested by the pushers
2015-05-21 16:41:39 +01:00
d61ce3f670
Add a cache for get_current_state with state_key
2015-05-21 16:41:39 +01:00
a910984b58
Actually return something from lambda
2015-05-21 15:58:41 +01:00
e309b1045d
Sort backfill events
2015-05-21 15:57:35 +01:00
0180bfe4aa
Remove dead code
2015-05-21 15:53:41 +01:00
1f3d1d85a9
Only get non-state
2015-05-21 15:52:29 +01:00
39a3340f73
Skip events we've already seen
2015-05-21 15:48:56 +01:00
ae3bff3491
Correctly prepopulate queue
2015-05-21 15:46:07 +01:00
dc085ddf8c
Don't prepopulate event_results
2015-05-21 15:44:05 +01:00
73d23c6ae8
Don't readd things that are already in event_results
2015-05-21 15:40:22 +01:00
6189d8e54d
PriorityQueue gives lowest first
2015-05-21 15:38:08 +01:00
115ef3ddac
Correctly capture Queue.Empty exception
2015-05-21 15:37:43 +01:00
c2633907c5
Merge branch 'erikj/join_perf' of github.com:matrix-org/synapse into erikj/backfill_fixes
2015-05-21 14:58:47 +01:00
ebfdd2eb5b
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/join_perf
2015-05-21 14:54:52 +01:00
a551c5dad7
Merge pull request #155 from matrix-org/erikj/perf
...
Bulk and batch retrieval of events.
2015-05-21 14:54:40 +01:00
27e4b45c06
s/for events/for requests for events/
2015-05-21 14:52:23 +01:00
ac5f2bf9db
s/for events/for requests for events/
2015-05-21 14:50:57 +01:00
80a167b1f0
Add comments
2015-05-21 11:19:04 +01:00
9118a92862
Split up _get_events into defer and txn versions
2015-05-20 13:27:16 +01:00
ab45e12d31
Make not return a deferred _get_event_from_row_txn
2015-05-20 13:07:19 +01:00
f407cbd2f1
PEP8
2015-05-20 13:02:01 +01:00
227f8ef031
Split out _get_event_from_row back into defer and _txn version
2015-05-20 13:00:57 +01:00
2bc60c55af
Fix _get_backfill_events to return events in the correct order
2015-05-20 12:57:00 +01:00
9084cdd70f
Ensure event_results is a set
2015-05-19 16:34:31 +01:00
aa729349dd
Fix event_backwards_extrem insertion to ignore outliers
2015-05-19 15:27:00 +01:00
99914ec9f8
Merge pull request #152 from matrix-org/notifier_performance
...
Notifier performance
2015-05-18 17:49:59 +01:00
165eb2dbe6
Comments and shuffle of functions
2015-05-18 15:18:41 +01:00
65878a2319
Remove unused metric
2015-05-18 14:06:30 +01:00
4d1b6f4ad1
Remove rejected events if we don't want rejected events
2015-05-18 14:03:46 +01:00
0b0033c40b
Merge branch 'develop' into notifier_performance
2015-05-18 13:50:01 +01:00
10f1bdb9a2
Move get_events functions to storage.events
2015-05-18 10:21:40 +01:00
c71176858b
Newline, remove debug logging
2015-05-18 10:11:14 +01:00
c3b37abdfd
PEP8
2015-05-15 16:59:58 +01:00
6c74fd62a0
Revert limiting of fetching, it didn't help perf.
2015-05-15 16:45:35 +01:00
9ff7f66a2b
init j
2015-05-15 16:36:03 +01:00
70f272f71c
Don't completely drain the list
2015-05-15 16:34:17 +01:00
8763dd80ef
Don't fetch prev_content for current_state
2015-05-15 15:33:01 +01:00
acb12cc811
Make store.get_current_state fetch events asyncly
2015-05-15 15:20:05 +01:00
d62dee7eae
Remove more debug logging
2015-05-15 15:06:37 +01:00
0f29cfabc3
Remove debug logging
2015-05-15 14:06:42 +01:00
e275a9c0d9
preserve log context
2015-05-15 11:54:51 +01:00
aa32bd38e4
Add a wait
2015-05-15 11:35:04 +01:00
372d4c6d7b
Srsly. Don't use closures. Baaaaaad
2015-05-15 11:26:00 +01:00
575ec91d82
Correctly pass through params
2015-05-15 11:15:10 +01:00
de01438a57
Sort out error handling
2015-05-15 11:00:50 +01:00
a2c4f3f150
Fix daedlock
2015-05-15 10:54:04 +01:00
1d566edb81
Remove race condition
2015-05-14 16:54:35 +01:00
ef3d8754f5
Call from right thread
2015-05-14 15:41:55 +01:00
142934084a
Count and loop
2015-05-14 15:40:21 +01:00
96c5b9f87c
Don't start up more fetch_events
2015-05-14 15:36:04 +01:00
7cd6a6f6cf
Awful idea for speeding up fetching of events
2015-05-14 15:34:02 +01:00
7f4105a5c9
Turn off preemptive transactions
2015-05-14 14:51:06 +01:00
f4d58deba1
PEP8
2015-05-14 14:45:42 +01:00
386b7330d2
Move from _base to events
2015-05-14 14:45:22 +01:00
7d6a1dae31
Jump out early
2015-05-14 14:27:58 +01:00
656223fbd3
Actually, we probably want to run this in a transaction
2015-05-14 14:26:35 +01:00
2f7f8e1c2b
Preemptively jump into a transaction if we ask for get_prev_content
2015-05-14 14:17:36 +01:00
e1e9f0c5b2
loop -> gatherResults
2015-05-14 13:58:49 +01:00
ab78a8926e
Err, we probably want a bigger limit
2015-05-14 13:47:16 +01:00
f6f902d459
Move fetching of events into their own transactions
2015-05-14 13:45:48 +01:00
cdb3757942
Refactor _get_events
2015-05-14 13:31:55 +01:00
36ea26c5c0
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/perf
2015-05-14 12:01:38 +01:00
7c549dd557
Add ID generator for push_rules_enable to #resolve SYN-378
2015-05-14 11:44:03 +01:00
899d4675dd
Merge branch 'notifier_unify' into notifier_performance
2015-05-14 11:36:44 +01:00
4f1d984e56
Add index on events
2015-05-13 17:22:26 +01:00
968b01a91a
Actually use async method
2015-05-13 17:02:46 +01:00
4071f29653
Fetch events from events_id in their own transactions
2015-05-13 16:59:41 +01:00
a988361aea
Typo
2015-05-13 15:44:15 +01:00
8888982db3
Don't insert None
2015-05-13 15:43:32 +01:00
cf706cc6ef
Don't return None
2015-05-13 15:31:25 +01:00
5971d240d4
Limit batch size
2015-05-13 15:26:49 +01:00
ca4f458787
Fetch events in bulk
2015-05-13 15:13:42 +01:00
6edff11a88
Don't fetch redaction and rejection stuff for each event, so we can use index only scan
2015-05-13 14:39:05 +01:00
63878c0379
Don't bother checking for updates if the stream token hasn't advanced for a user
2015-05-13 13:42:21 +01:00
02590c3e1d
Temp turn off checking for rejections and redactions
2015-05-13 11:31:28 +01:00
619a21812b
defer.gatherResults loop
2015-05-13 11:29:03 +01:00
fec4485e28
Batch fetching of events for state groups
2015-05-13 11:22:42 +01:00
409bcc76bd
Load events for state group seperately
2015-05-13 11:13:31 +01:00
e122685978
You need to call contextmanager
2015-05-12 16:12:37 +01:00
a6fb2aa2a5
Merge pull request #144 from matrix-org/erikj/logging_context
...
Preserving logging contexts
2015-05-12 15:23:50 +01:00
1fce36b111
Merge pull request #149 from matrix-org/erikj/backfill
...
Backfill support
2015-05-12 15:20:32 +01:00
8b28209c60
Err, delete the right stuff
2015-05-12 15:02:53 +01:00
30c72d377e
Newlines
2015-05-12 14:47:40 +01:00
e4eddf9b36
We do actually want to delete rows out of event_backward_extremities
2015-05-12 14:47:23 +01:00
c1779a79bc
Fix up _handle_prev_events to not try to insert duplicate rows
2015-05-12 14:41:50 +01:00
74850d7f75
Do state groups persistence /after/ checking if we have already persisted the event
2015-05-12 14:14:58 +01:00
da6a7bbdde
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/logging_context
2015-05-12 13:10:42 +01:00
5002056b16
SYN-377: Make sure that the StreamIdGenerator.get_next.__exit__ is called from the main thread after the transaction completes, not from database thread before the transaction completes.
2015-05-12 11:20:40 +01:00
4df11b5039
Make get_current_token accept a direction parameter, which tells whether the source whether we want a token for going 'forwards' or 'backwards'
2015-05-12 10:28:10 +01:00
84e6b4001f
Initial hack at wiring together pagination and backfill
2015-05-11 18:01:31 +01:00
17653a5dfe
Move storage.stream._StreamToken to types.RoomStreamToken
2015-05-11 18:01:01 +01:00
b036596b75
Prefer to use _simple_*.
2015-05-11 11:24:01 +01:00
cd525c0f5a
push_rules table expects an 'id' field
2015-05-11 11:24:01 +01:00
22d2f498fa
Fix push rule bug: can't insert bool into small int column
2015-05-10 10:50:51 +01:00
476899295f
Change the way we do logging contexts so that they survive divergences
2015-05-08 16:32:18 +01:00
1c7912751e
Drop the old table not the new table
2015-05-08 16:04:32 +01:00
9d36eb4eab
Rename unique constraint
2015-05-08 16:01:55 +01:00
b0f71db3ff
Remove unsigned
2015-05-08 15:59:51 +01:00
84e1cacea4
Bump schema version
2015-05-08 15:58:14 +01:00
6538d445e8
Make the timestamps in server_keys_json bigints
2015-05-08 15:55:17 +01:00
0c4ac271ca
Merge branch 'erikj/executemany' of github.com:matrix-org/synapse into erikj/SYN-371
2015-05-05 18:21:19 +01:00
0cf7e480b4
And use buffer(...) there as well
2015-05-05 18:20:01 +01:00
ed2584050f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/executemany
2015-05-05 18:15:20 +01:00
977338a7af
Use buffer(...) when inserting into bytea column
2015-05-05 18:12:53 +01:00
31049c4d72
Merge pull request #139 from matrix-org/bugs/SYN-369
...
Fix race with cache invalidation. SYN-369
2015-05-05 17:46:13 +01:00
deb0237166
Add some doc-string
2015-05-05 17:45:11 +01:00
d18f37e026
Collect the invalidate callbacks on the transaction object rather than passing around a separate list
2015-05-05 17:32:21 +01:00
9951542393
Add a comment about the zip(*[zip(sorted(...),...)])
2015-05-05 17:06:55 +01:00
041b6cba61
SYN-369: Add comments to the sequence number logic in the cache
2015-05-05 16:32:44 +01:00
63075118a5
Add debug flag in synapse/storage/_base.py for debugging the cache logic by comparing what is in the cache with what was in the database on every access
2015-05-05 16:24:04 +01:00
531d7955fd
Don't insert without deduplication. In this case we never actually use this table, so simply remove the insert entirely
2015-05-05 16:12:28 +01:00
bfa4a7f8b0
Invalidate the room_member cache if the current state events updates
2015-05-05 15:43:49 +01:00
d0fece8d3c
Missing return for when the event was already persisted
2015-05-05 15:39:09 +01:00
bdcd7693c8
Fix indentation
2015-05-05 15:14:48 +01:00
43c2e8deae
Add support for using executemany
2015-05-05 15:13:25 +01:00
1692dc019d
Don't call 'encode_parameter' no-op
2015-05-05 15:00:30 +01:00
a9aea68fd5
Invalidate the caches from the correct thread
2015-05-05 14:57:08 +01:00
261d809a47
Sequence the modifications to the cache so that selects don't race with inserts
2015-05-05 14:13:50 +01:00
d9cc5de9e5
Correctly name transaction
2015-05-05 10:24:10 +01:00
b8940cd902
Remove some unused indexes
2015-05-01 16:14:25 +01:00
eb9bd2d949
user_id now in user_threepids
2015-05-01 15:04:37 +01:00
9eeb03c0dd
Don't use self.execute: it's designed for fetching stuff
2015-05-01 14:21:25 +01:00
7693f24792
No id field on user
2015-05-01 13:55:42 +01:00
92b20713d7
More missed get_user_by_id API changes
2015-05-01 13:45:54 +01:00
4011cf1c42
Cache latest_event_ids_in_room
2015-05-01 13:06:26 +01:00
657298cebd
Don't lock user_ips table for upsert.
2015-05-01 10:46:48 +01:00
fabb7acd45
Fix bug where we reconnected to the database on every query.
2015-05-01 10:24:24 +01:00
23c639ff32
Split a storage function in two so that we don't have to do extra work.
2015-05-01 10:17:19 +01:00
8be5284e91
Remove pointless join
2015-04-30 18:49:26 +01:00
503e4d3d52
Fix broken SQL
2015-04-30 18:44:47 +01:00
00718ae7a9
Need more yield
2015-04-30 18:43:39 +01:00
0465560c1a
Add missing param
2015-04-30 18:42:44 +01:00
61d05daab1
More join conditions
2015-04-30 18:41:05 +01:00
6ead27ddda
Add more conditions on JOINs to make postgres go a little faster.
2015-04-30 18:32:03 +01:00
f383d5a801
Fix up get_current_state and get_room_name_and_aliases queries to parse events in transaction
2015-04-30 17:12:52 +01:00
69d4063651
Add get_rooms_for_user cache
2015-04-30 16:47:51 +01:00
3c4c229788
Don't use sub queries, it makes postgres sad
2015-04-30 16:16:53 +01:00
29400b45b9
SYN-367: Use upsert rather than insert_or_replace
2015-04-30 15:21:31 +01:00
2366d28780
Don't needlessly join on state_events
2015-04-30 14:02:06 +01:00
1d7702833d
Make simple query rather than long one and then throw away half the results
2015-04-30 10:16:12 +01:00
b1ca784aca
Correctly decode, for sqlite and postgres, rows from pushers table
2015-04-29 19:41:14 +01:00
4a9dc5b2f5
pushkey' are also bytes.
2015-04-29 19:27:02 +01:00
0ade2712d1
Typo
2015-04-29 19:17:25 +01:00
50f96f256f
Also remove yield from within lock in the other generator
2015-04-29 19:17:00 +01:00
d2d61a8288
Fix deadlock in id_generators. No idea why this was an actual deadlock.
2015-04-29 19:15:23 +01:00
3864b3a8e6
Actually return rows
2015-04-29 18:07:36 +01:00
0618978238
Typo, args wrong way round
2015-04-29 18:04:35 +01:00
09177f4f2e
Decode buffers in same thread
2015-04-29 18:03:42 +01:00
472be88674
We store pusher data as bytes
2015-04-29 17:43:46 +01:00
12d381bd5d
Decode the data json in the storage layer (was moved but this part was missed)
2015-04-29 17:13:51 +01:00
f8c30faf25
Oops, update the contraint too
2015-04-29 16:58:42 +01:00
61cd5d9045
Be more postgressive
2015-04-29 16:57:14 +01:00
fb95035a65
Be postgressive
2015-04-29 16:53:41 +01:00
4669def000
Oops, forgot the schema delta file
2015-04-29 16:50:16 +01:00
0337eaf321
txn.execute doesn't return cursors
2015-04-29 16:43:39 +01:00
884fb88e28
txn.execute doesn't return cursors
2015-04-29 16:35:20 +01:00
d76c058eea
Fix invalid SQL to work in postgres land
2015-04-29 16:30:25 +01:00
c8ed9bd278
pushers table requires an access_token
2015-04-29 15:33:27 +01:00
970a9b9d2b
We can't use REPLACE when upgrading databases now we have postgres
2015-04-29 13:55:44 +01:00
64991b0c8b
Merge pull request #129 from matrix-org/key_distribution
...
Key distribution v2
2015-04-29 13:34:38 +01:00
e26a3d8d9e
bump database schema version
2015-04-29 13:32:32 +01:00
a9549fdce3
Use bytea rather than BLOB
2015-04-29 13:16:09 +01:00
4ad8b45155
Merge branch 'develop' into key_distribution
...
Conflicts:
synapse/config/homeserver.py
2015-04-29 13:15:14 +01:00
cd0864121b
Make postgres database error slightly more helpful
2015-04-29 12:12:25 +01:00
4932a7e2d9
Use __all__ instead of assert to stop pyflakes from warning about unused import in __init__
2015-04-29 12:12:25 +01:00
a08bf11138
Appease PEP8
2015-04-29 11:44:48 +01:00
204132a998
Check that postgres database has correct charset set
2015-04-29 11:42:28 +01:00
327ca883ec
Merge branch 'develop' of github.com:matrix-org/synapse into postgres
2015-04-28 13:39:42 +01:00
18f8247701
Use TEXT instead of VARCHAR(n), since PostgreSQL treats them the same except for a limit
2015-04-28 12:41:33 +01:00
9182f87664
Merge pull request #126 from matrix-org/csauth
...
Client / Server Auth Refactor
2015-04-28 11:00:27 +01:00
55e1bc8920
And don't bump the schema version unnecessarily
2015-04-28 10:54:15 +01:00
b96c133034
Add server_keys.sql to the current delta rather than creating a new delta
2015-04-28 10:50:00 +01:00
8558e1ec73
Make get_max_token into inlineCallbacks so that the lock works.
2015-04-27 15:19:44 +01:00
6f8e2d517e
Merge branch 'develop' of github.com:matrix-org/synapse into postgres
2015-04-27 14:41:40 +01:00
2732be83d9
Shuffle operations so that locking upsert happens last in the txn. This ensures the lock is held for the least amount of time possible.
2015-04-27 13:22:30 +01:00
e4c4664d73
Handle the fact that postgres databases can be restarted from under us
2015-04-27 12:40:49 +01:00
8a5be236e0
pep8
2015-04-27 11:49:18 +01:00
b02e1006b9
Run database check before daemonizing, at the cost of database hygiene.
2015-04-27 11:46:00 +01:00
f8152f2708
rename db method to be more informative
2015-04-27 10:16:26 +01:00
2f475bd5d5
pep8
2015-04-24 18:15:07 +01:00
a7b51f4539
Check users in our table aren't on a different domain to the one we're configured with to try & fix SYN-266
2015-04-24 18:11:21 +01:00
eede182df7
Merge branch 'develop' into key_distribution
2015-04-24 10:35:49 +01:00
4e2f8b8722
Copyright notices
2015-04-24 10:35:29 +01:00
f30d47c876
Implement remote key lookup api
2015-04-22 14:21:08 +01:00
91c8f828e1
pep8
2015-04-17 19:56:04 +01:00
8db6832db8
Password reset, finally.
2015-04-17 19:53:47 +01:00
117f35ac4a
Add endpoint to get threepids from server
2015-04-17 17:20:18 +01:00
4eea5cf6c2
pep8
2015-04-17 16:46:45 +01:00
f96ab9d18d
make add3pid servlet work
2015-04-17 16:44:49 +01:00
e3417bbbe0
Revert needless change to storage.profile
2015-04-17 16:13:14 +01:00
cb03fafdf1
Merge branch 'develop' into csauth
2015-04-17 13:51:10 +01:00
0ec346d942
Add unique index to room_aliases, remove duplicates on upgrade. Convert some columns back to TEXT from bytea
2015-04-17 11:10:20 +01:00
e1c0970c11
PEP8
2015-04-16 11:18:45 +01:00
b8092fbc82
Go back to storing JSON in TEXT
2015-04-16 11:17:52 +01:00
bc9e69e160
Move encoding and decoding of JSON into storage layer
2015-04-16 11:01:09 +01:00
ffad75bd62
Remove mysql/maria support
2015-04-15 17:00:50 +01:00
a429515bdd
Add methods for storing and retrieving the raw key json
2015-04-15 16:58:35 +01:00
cf04cedf21
Change full_schemas/11 to work with postgres
2015-04-15 16:53:47 +01:00
5b31afcbd1
Remove debug logging
2015-04-15 16:27:04 +01:00
6e91f14d09
Add missing yield
2015-04-15 16:25:07 +01:00
ed26e4012b
pushers table requires a unique id.
2015-04-15 16:24:14 +01:00
a5c72780e6
Don't pass in removed flag
2015-04-15 15:13:22 +01:00
4af32a2817
Postgres does not allow you to continue using a cursor after a DB exception has been raised, so move _simple_insert or_ignore flag out of transaction
2015-04-15 14:51:21 +01:00
25d2914fba
Turn off persistance of sent_transactions.response_json
2015-04-15 14:09:35 +01:00
cce5d057d3
Add index on events (topological_ordering, stream_ordering) to help with ORDER BY clauses
2015-04-15 11:43:25 +01:00
a971fa9d58
Use try..finally in contextlib.contextmanager
2015-04-15 10:25:43 +01:00
ded4128965
Use True for True rather than 1
2015-04-15 10:24:24 +01:00
f9e12f79ca
Add missing yield in storage func
2015-04-15 10:24:07 +01:00
c756dfeb14
Correctly identify deadlocks
2015-04-15 10:23:42 +01:00
32e14d8181
Return a sha256 fingerprint rather than the entire tls certificate
2015-04-14 19:10:09 +01:00
127fad17dd
Add postgres database engine
2015-04-14 14:50:29 +01:00
5a95cd4442
Rename user_ips.user -> user_id
2015-04-14 13:54:09 +01:00
58d8339966
Add support for postgres instead of mysql. Change sql accourdingly. blob + varbinary -> bytea. No support for UNSIGNED or CREATE INDEX IF NOT EXISTS.
2015-04-14 13:53:20 +01:00
3c741682e5
Correctly increment the _next_id initially
2015-04-14 09:54:44 +01:00
86fc9b617c
For backwards compat, make state_groups.id have a type of int, not varchar
2015-04-13 17:03:49 +01:00
93937b2b31
Remove duplicate rows
2015-04-10 18:47:03 +01:00
c5365dee56
Use case sensitive collations
2015-04-10 18:46:33 +01:00
4103b1c470
DROP indexes before recreating them
2015-04-10 18:46:09 +01:00
4d5b098626
Use LONGBLOB and TEXT for arbitary length rows
2015-04-10 18:45:09 +01:00
7ed2ec3061
Handle the fact that in sqlite binary data might be stored as unicode or bytes
2015-04-10 13:41:54 +01:00
ce797ad373
Bump schema version
2015-04-10 11:22:30 +01:00
7e863c51e6
Use unsigned bigint
2015-04-10 11:22:04 +01:00
0f12772e32
SQLite wants INTEGER and not BIGINT for primary keys
2015-04-10 11:16:09 +01:00
d5d4281647
Update full_schemas/16 to match delta files. Add delta/16 scripts
2015-04-10 10:59:46 +01:00
cda4a6f93f
Revert non-trivial changes to upgrade scripts
2015-04-10 10:19:50 +01:00
e2722f58ee
Fix schema again
2015-04-10 10:16:29 +01:00
a1665c5094
Revert non-trivial schema changes and move them to a new schema version.
2015-04-10 10:05:44 +01:00
2ded344620
Remove unused import
2015-04-09 13:46:06 +01:00
9707acfc40
Remove spurious spaces
2015-04-09 13:45:20 +01:00
8bf285e082
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-04-09 13:13:26 +01:00
8ad0f4912e
Stream ordering and out of order insertions.
...
Handle the fact that events can be persisted out of order, and so to get
the "current max" stream token becomes non trivial - as we need to make
sure that *all* stream tokens less than the current max have also
successfully been persisted.
2015-04-09 11:41:36 +01:00
279a547a8b
Use generic db exceptions rather than sqlite3 specific ones
2015-04-08 16:53:48 +01:00
83f5125d52
Merge pull request #114 from matrix-org/improve_get_event_cache
...
Improve get event cache
2015-04-08 16:50:27 +01:00
3887350e47
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-04-08 14:11:07 +01:00
c1b34af441
Move database timer logging to seperate logger
2015-04-08 13:31:19 +01:00
9a05795619
Retry transaction, not SQL query
2015-04-08 13:11:28 +01:00
24d8134ac1
Fix maria engine to correctly recognize deadlocks
2015-04-08 13:10:54 +01:00
4fe95094d1
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-04-07 18:05:39 +01:00
49d6aa1394
Retry on deadlock
2015-04-07 15:28:37 +01:00
0bfa78b39b
PEP8
2015-04-07 12:16:05 +01:00
6bc9edd8b2
Fix prepare_sqlite3_database's convert_param_style
2015-04-07 12:13:58 +01:00
05a35d62b6
Bump database version
2015-04-07 12:10:15 +01:00
8574bf62dc
Add index to presence table
2015-04-07 12:09:36 +01:00
0af5f5efaf
Don't use multiple UNIQUE constraints; it will cause deadlocks
2015-04-07 12:08:35 +01:00
c8d3f6486d
Implement or_ignore flag on inserts
2015-04-07 12:06:22 +01:00
304111afd0
Don't use AUTOINCREMENT, use an in memory version
2015-04-07 12:05:36 +01:00
779f7b0f44
Fix unicode support
2015-04-02 10:06:22 +01:00
9236136f3a
Make work in both Maria and SQLite. Fix tests
2015-04-01 14:12:33 +01:00
5e88a09a42
Add same user_id char checks as registration.
2015-03-31 14:00:25 +01:00
cf1fa59f4b
Use a sender localpart instead of a user ID.
...
Form the user ID at runtime instead, This gives less room for error in AS
config files since they cannot specify the domain of another HS.
2015-03-31 13:48:03 +01:00
c217504949
Edit SQL schema to use string IDs not ints. Use token as ID. Update tests.
2015-03-31 12:07:56 +01:00
b59aa74556
Fix tests and missing returns on deferreds.
2015-03-31 11:35:45 +01:00
d33ae65efc
Remove more reg/unreg methods. Read config not database for cache.
2015-03-31 11:00:00 +01:00
3e420aebd8
Revert "Add another @cached wrapper, this time on get_presence_state()"
...
This reverts commit ff1fa0fbf8
2015-03-27 16:16:58 +00:00
ff1fa0fbf8
Add another @cached wrapper, this time on get_presence_state()
2015-03-27 15:57:16 +00:00
a32e876ef4
Delete pushers when changing password
2015-03-26 13:40:16 +00:00
a198894bf7
Appease pep8
2015-03-26 11:53:58 +00:00
32206dde3f
Fixes from PR comments
2015-03-26 10:11:52 +00:00
4edcbcee3b
Merge branch 'develop' into application-services-txn-reliability
...
Conflicts:
synapse/storage/__init__.py
2015-03-26 10:07:59 +00:00
953e40f9dc
Implement the main getEvent cache using Cache() instead of a custom application of LruCache; also unify its two-level structure into just one
2015-03-25 19:12:16 +00:00
c1a256cc4c
Allow multiple pushers for a single app ID & pushkey, honouring the 'append' flag in the API.
2015-03-25 19:06:22 +00:00
f173d40a32
Use FrozenEvent's reject_reason to decide whether to return it; don't include allow_rejected in the main getEvents cache key
2015-03-25 19:06:05 +00:00
1b988b051b
Store the rejected reason in (Frozen)Event structs
2015-03-25 19:06:05 +00:00
033a517feb
Indirect invalidations of _get_event_cache via a helper method to keep all uses of the cache lexically within one .py file
2015-03-25 19:06:05 +00:00
9ba6487b3f
Allow a choice of LRU behaviour for Cache() by using LruCache() or OrderedDict()
2015-03-25 19:05:34 +00:00
0e8f5095c7
Fix unicode database support
2015-03-25 17:15:20 +00:00
c7023f2155
1) Pushers are now associated with an access token
...
2) Change places where we mean unauthenticated to 401, not 403, in C/S v2: hack so it stays as 403 in v1 because web client relies on it.
2015-03-24 17:24:15 +00:00
0ba393924a
Escape non printing ascii character
2015-03-24 16:31:52 +00:00
f488293d96
Don't reinsert into event_edges
2015-03-24 16:20:26 +00:00
1aa44939fc
Fix bugs in transactions storage
2015-03-24 16:20:05 +00:00
5a447098dd
Don't use room hosts table
2015-03-24 16:19:24 +00:00
9e98f1022a
Don't order by rowid
2015-03-24 16:19:01 +00:00
9115421ace
Use _simple_upsert
2015-03-24 16:17:39 +00:00
d19e79ecc9
Make deleting other access tokens when you change your password actually work
2015-03-24 15:33:48 +00:00
6e7131f02f
Remove uses of REPLACE and ON CONFLICT IGNORE to make the SQL more portable.
2015-03-23 15:38:56 +00:00
9a7f496298
Sanitize RoomMemberStore
2015-03-23 15:29:04 +00:00
d98660a60d
Implement password changing (finally) along with a start on making client/server auth more general.
2015-03-23 14:20:28 +00:00
d5272b1d2c
Use 'update or insert' rather than on 'conflict replace'
2015-03-23 14:02:34 +00:00
278149f533
Sanitize TransactionStore
2015-03-23 13:43:21 +00:00
72d8406409
Put a cache on get_aliases_for_room
2015-03-20 19:21:13 +00:00
a63b4f7101
Remember the 'last seen' time for a given user/IP/device combination and only bother INSERTing another if it's stale
2015-03-20 18:25:49 +00:00
0f86312c4c
Pull out the cache logic from the @cached wrapper into its own class we can reuse
2015-03-20 18:25:42 +00:00
b1022ed8b5
func(*EXPR) is valid Python syntax, really...
2015-03-20 17:47:45 +00:00
f6583796fe
Merge branch 'develop' of github.com:matrix-org/synapse into mysql
2015-03-20 16:31:48 +00:00
80cd08c190
PEP8
2015-03-20 16:03:25 +00:00
9517f4da4d
Merge branch 'develop' of github.com:matrix-org/synapse into store_rearrangement
2015-03-20 16:02:47 +00:00
dc0c989ef4
Give sensible names for '_simple_...' transactions
2015-03-20 15:59:18 +00:00
ceb61daa70
Add the tiniest of tiny one-element caches to get_room_events_max_id() as it's read every time someone hits eventstream
2015-03-20 15:44:06 +00:00
fce0114005
Start removing Tables
2015-03-20 15:05:44 +00:00
7e282a53a5
Tidy up _simple_... methods
2015-03-20 15:05:10 +00:00
91cb46191d
Allow @cached-wrapped functions to have more or fewer than 1 argument; assert on the total count of them though
2015-03-20 14:59:45 +00:00
87db64b839
Rearrange storage modules
2015-03-20 14:11:38 +00:00
cb8162d3d1
Rearrange storage modules
2015-03-20 13:52:56 +00:00
d288d273e1
Generate transaction id in code
2015-03-20 10:57:44 +00:00
d4f50f3ae5
decode_result takes an iterable
2015-03-20 10:57:26 +00:00
0d0610870d
Fix up schemas some more
2015-03-20 10:55:31 +00:00
d7a0496f3e
Convert storage layer to be mysql compatible
2015-03-19 15:59:48 +00:00
58ed393235
Remove redundant key
2015-03-19 15:12:05 +00:00
fae059cc18
Fix up schemas to work with mariadb
2015-03-19 13:42:39 +00:00
bb24609158
Clean out event_forward_extremities table when the server rejoins the room
2015-03-18 11:19:47 +00:00
89036579ed
Update schema to work with mariadb
2015-03-18 11:18:49 +00:00
93978c5e2b
@cached() annotate get_user_by_token() - achieves a minor DB performance improvement
2015-03-17 17:24:51 +00:00
1489521ee5
Be polite and ensure we use @functools.wraps() when creating a function decorator
2015-03-17 17:19:22 +00:00
be170b1426
Add a metric for the scheduling latency of SQL queries
2015-03-16 17:21:59 +00:00
cd2539ab2a
Merge pull request #110 from matrix-org/fix_ban
...
Fix ban
2015-03-16 15:36:52 +00:00
f0d6f724a2
Set the service ID as soon as it is known.
2015-03-16 15:24:32 +00:00
f1d2b94e0b
Copy dict of context.current_state before changing it.
2015-03-16 15:13:05 +00:00
835e01fc70
Minor PR comment tweaks.
2015-03-16 10:16:59 +00:00
f9232c7917
Merge branch 'develop' into application-services-txn-reliability
...
Conflicts:
synapse/storage/appservice.py
2015-03-16 10:09:15 +00:00
e75fa8bbbf
Bugfix to sql_txn_timer increment - add only the per-TXN duration, not the total time ever spent since boot
2015-03-12 16:24:51 +00:00
63cb7ece62
Rename the timer metrics exported by synapse.storage to append _time, so the meaning of ':total' is clearer
2015-03-12 16:24:51 +00:00
f1fbe3e09f
Rename TimerMetric to DistributionMetric; as it could count more than just time
2015-03-12 16:24:51 +00:00
642f725fd7
Pretend the 'getEvent' cache is just another cache in the set of all the others for metric
2015-03-12 16:24:51 +00:00
b0cf867319
Use _ instead of . as a metric namespacing separator, for Prometheus
2015-03-12 16:24:51 +00:00
f9478e475b
Rename Metrics' "keys" to "labels"
2015-03-12 16:24:51 +00:00
fa319a5786
Add TimerMetrics to shadow the PerformanceCounters in synapse.storage; with the view to eventually replacing them entirely
2015-03-12 16:24:51 +00:00
25187ab674
Collect per-SQL-verb timer stats on query execution time
2015-03-12 16:24:50 +00:00
e1a7e3564f
Delete a couple of TODO markers of monitoring stats now done
2015-03-12 16:24:50 +00:00
8664599af7
Rename CacheCounterMetric to just CacheMetric; add a CallbackMetric component to give the size of the cache
2015-03-12 16:24:50 +00:00
b0cdf097f4
Sprinkle some CacheCounterMetrics around the synapse.storage layer
2015-03-12 16:24:50 +00:00
a7a28f85ae
Appease pep8
2015-03-11 17:32:43 +00:00
59a5f012cc
Also give _execute() a description
2015-03-11 17:19:17 +00:00
099e4b88d8
Add a description to storage layer's _execute_and_decode()
2015-03-11 17:08:57 +00:00
04f8478aaa
Add the master push rule for the break-my-push button. Allow server default rules to be disabled by default.
2015-03-10 17:26:25 +00:00
8916acbc13
These aren't defined for redacted events so don't crash
2015-03-10 11:21:37 +00:00
db1fbc6c6f
Fix remaining scheduler bugs. Add more informative logging.
2015-03-10 10:04:20 +00:00
7e0bba555c
Remove unused import
2015-03-09 17:48:37 +00:00
04c9751f24
Bug fixes whilst putting it all together
2015-03-09 17:45:41 +00:00
b98cd03193
Use event IDs instead of dumping event content in the txns table.
2015-03-09 17:25:20 +00:00
21fd84dcb8
Use seconds; start gluing in the AS scheduler into the AS handler.
2015-03-09 17:01:19 +00:00
0a60bbf4fa
Finish appservice txn storage impl and tests.
2015-03-09 15:53:03 +00:00
1ead1caa18
Implement create_appservice_txn with tests.
2015-03-09 13:54:20 +00:00
1c2dcf762a
Partially implement txn store methods with tests.
2015-03-09 13:10:31 +00:00
406d32f8b5
Start implementing ApplicationServiceTransactionStore
2015-03-06 17:35:14 +00:00
34ce2ca62f
Merge branch 'develop' into application-services-txn-reliability
2015-03-06 17:28:49 +00:00
4a6afa6abf
Assign the AS ID from the database; replace old placeholder txn id.
2015-03-06 17:27:55 +00:00
01c099d9ef
Add appservice txns sql schema
2015-03-06 17:16:47 +00:00
2602ddc379
Apply clarity and docstrings
2015-03-06 16:16:14 +00:00
0354659f9d
Finish synapse.appservice.scheduler implementation.
...
With tests to assert behaviour. Not hooked up yet. Stub datastore methods
not implemented yet.
2015-03-06 16:09:05 +00:00
f260cb72cd
Flesh out more stub functions.
2015-03-06 15:12:24 +00:00
141ec04d19
Add stub ApplicationServiceTransactionStore. Bootstrap Recoverers. Fill in stub Transaction functions.
2015-03-06 14:53:35 +00:00
cf66ddc1b4
Schema change as delta in v14
2015-03-06 14:11:49 +00:00
ed72fc3a50
Merge branch 'develop' into pushrules2
...
Conflicts:
synapse/storage/schema/pusher.sql
2015-03-04 15:24:21 +00:00
d56c01fff4
Note that we don't specify execution order
2015-03-04 15:10:05 +00:00
17d319a20d
s/schema_deltas/applied_schema_deltas/
2015-03-04 15:06:22 +00:00
92b3dc3219
Merge branch 'develop' into pushrules2
2015-03-04 14:56:41 +00:00
5681264faa
s/%r/%s/
2015-03-04 14:21:53 +00:00
f701197227
Add example directory structures in doc
2015-03-04 14:20:14 +00:00
2a45f3d448
Use if not results rather than len, as per feedback.
2015-03-04 14:17:59 +00:00
16dd87d848
Don't assume db conn is a Context Manager.
...
Twisted adbapi wrapped connections aren't context managers.
2015-03-04 14:03:41 +00:00
5eefd1f618
Add unique constraint on schema_version.lock schema. Use conflict clause in sql.
2015-03-04 13:52:18 +00:00
b4c38738f4
Change to use logger in db upgrade script
2015-03-04 13:43:35 +00:00
640e53935d
Use context manager with db conn to correctly commit and rollback
2015-03-04 13:43:17 +00:00
8c8354e85a
Actually add full_schemas dir
2015-03-04 13:34:38 +00:00
c3530c3fb3
More docs. Rename 'schema/current' to 'schema/full_schemas'
2015-03-04 13:34:11 +00:00
811355ccd0
Add some docs and remove unused variables
2015-03-04 13:11:01 +00:00
82b34e813d
SYN-67: Finish up implementing new database schema management
2015-03-04 12:04:19 +00:00
8d33adfbbb
SYN-67: Begin changing the way we handle schema versioning
2015-03-02 18:23:55 +00:00
6fab7bd2c1
s/user_name/user/ as per mjark's comment
2015-03-02 18:17:19 +00:00
9f03553f48
Add missing comma
2015-03-02 16:38:40 +00:00
b41dc68773
We purposefully don't have a version 14 delta script.
2015-03-02 16:36:19 +00:00
20436cdf75
Blank lines
2015-03-02 15:58:12 +00:00
8ad024ea80
Merge pull request #93 from matrix-org/application-services-exclusive
...
Application services exclusive flag support
2015-03-02 14:56:32 +00:00
0a036944bd
Merge branch 'develop' of github.com:matrix-org/synapse into batched_get_pdu
2015-03-02 13:53:30 +00:00
c3c01641d2
Run deltas and bump user_version in upgrade script
2015-03-02 13:38:57 +00:00
cb97ea3ec2
PEP8
2015-03-02 11:23:46 +00:00
377ae369c1
Wrap all of get_app_service_rooms in a txn.
2015-03-02 11:20:51 +00:00
b216b36892
JOIN state_events rather than parsing unrecognized_keys to pull out member state_keys
2015-03-02 10:41:35 +00:00
3d73383d18
Modify _simple_select_list to allow an empty WHERE clause. Use it for get_all_rooms and get_all_users.
2015-03-02 10:16:24 +00:00
ebc4830666
PR tweaks: set earlier on and use 'as json' for compat
2015-03-02 09:53:00 +00:00
16b90764ad
Convert expected format for AS regex to include exclusivity.
...
Previously you just specified the regex as a string, now it expects a JSON
object with a 'regex' key and an 'exclusive' boolean, as per spec.
2015-02-27 10:44:32 +00:00
806a6c886a
PEP8
2015-02-27 09:48:57 +00:00
1959088156
Add API for getting/setting enabled-ness of push rules.
2015-02-26 18:07:44 +00:00
f0995436e7
Check for membership invite events correctly.
2015-02-26 17:21:17 +00:00
dcec7175dc
Finish impl to get new events for AS. ASes should now be able to poll /events
2015-02-26 16:23:01 +00:00
92478e96d6
Finish impl to extract all room IDs an AS may be interested in when polling the event stream.
2015-02-26 14:35:28 +00:00
944003021b
whitespace
2015-02-26 13:43:05 +00:00
94fa334b01
Add enable/disable overlay for push rules (REST API not yet hooked up)
2015-02-25 19:17:07 +00:00
29267cf9d7
PEP8 and pyflakes
2015-02-25 17:42:28 +00:00
978ce87c86
Comment unused variables.
2015-02-25 17:37:48 +00:00
2c79c4dc7f
Fix alias query.
2015-02-25 17:37:14 +00:00
2b8ca84296
Add support for extracting matching room_ids and room_aliases for a given AS.
2015-02-25 17:15:25 +00:00
2d20466f9a
Add stub functions and work out execution flow to implement AS event stream polling.
2015-02-25 15:00:59 +00:00
9640510de2
Use OrderedDict for @cached backing store, so we can evict the oldest key unbiased
2015-02-23 18:41:58 +00:00
f53fcbce97
Use cache.pop() instead of a separate membership test + del []
2015-02-23 18:30:45 +00:00
27080698e7
Fix code style warning
2015-02-23 18:19:13 +00:00
74048bdd41
Remove unused import
2015-02-23 18:17:43 +00:00
a09e59a698
Pull the _get_event_cache.setdefault() call out of the try block, as it doesn't need to be there and is confusing
2015-02-23 16:55:57 +00:00
044d813ef7
Use the @cached decorator to implement the destination_retry_timings cache
2015-02-23 16:04:40 +00:00
357fba2c24
RoomMemberStore no longer needs a _user_rooms_cache member
2015-02-23 15:57:41 +00:00
e76d485e29
Allow @cached-wrapped functions to have a prefill method for setting entries
2015-02-23 15:41:54 +00:00
4631b737fd
Squash out the now-redundant ApplicationServicesCache object class
2015-02-23 14:38:44 +00:00
42b972bccd
Revert get_auth_chain changes
2015-02-23 14:35:23 +00:00
55022d6ca5
Remove a TODO note
2015-02-19 18:38:09 +00:00
ebc3db295b
Take named arguments to @cached() decorator, add a 'max_entries' limit
2015-02-19 18:36:02 +00:00
077d200342
Move @cached decorator out into synapse.storage._base; add minimal docs
2015-02-19 17:29:39 +00:00
0ac2a79faa
Initial stab at implementing a batched get_missing_pdus request
2015-02-19 17:24:14 +00:00
61959928bb
Pull out the 'get_rooms_for_user' cache logic into a reüsable @cached decorator
2015-02-19 14:58:07 +00:00
2c29ed3e84
Use absolute path when loading delta sql files
2015-02-17 17:22:24 +00:00
1a989c436c
Bump schema version
2015-02-17 15:45:55 +00:00
fa6c93bd26
Merge branch 'consumeErrors' of github.com:matrix-org/synapse into develop
2015-02-17 15:18:17 +00:00
72a4de2ce6
Use consumeErrors=True on all DeferredLists.
...
This is so that the DeferredLists actually consume the error instead of
propogating down the non-existent errback chain. This should reduce the
number of unhandled errors we are seeing.
2015-02-17 10:07:01 +00:00
64def4f953
Merge branch 'hotfixes-0.7.0e' into develop
2015-02-13 16:18:34 +00:00
8d5cce62ab
Update pushers by app id and pushkey, not user id and pushkey
2015-02-13 16:16:16 +00:00
0d872f5aa6
Merge pull request #50 from matrix-org/application-services
...
Application Services
2015-02-13 15:06:14 +00:00
183b3d4e47
Prepare the database whenever a connection is opened from the db_pool so that in-memory databases will work
2015-02-13 14:38:24 +00:00
6b186a57ba
Merge branch 'fix' of github.com:matrix-org/synapse into hotfixes-v0.7.0a
2015-02-12 14:51:46 +00:00
717687e1fc
Get an auth query one at a time
2015-02-12 14:39:31 +00:00
83d41f25d8
Set database schema version in delta
2015-02-12 10:05:47 +00:00
8bbdf32849
Convert get_rooms to use runInteraction so the transacion has a more helpful description
2015-02-11 18:56:13 +00:00
42bc56dad3
Merge branch 'develop' of github.com:matrix-org/synapse into use-simplejson
2015-02-11 17:01:38 +00:00
7eef84a95b
pyflakes
2015-02-11 16:52:22 +00:00
f2fdcb7c4b
Merge branch 'develop' into application-services
2015-02-11 16:43:26 +00:00
b164e0896c
Merge branch 'bugs/SYN-264' into develop
2015-02-11 16:23:30 +00:00
7f47ba7383
Added another TODO note
2015-02-11 16:18:21 +00:00
45b56609ae
Cache the result of a get_rooms_for_user query, to make user_rooms_intersect() much lighter in the read-common case
2015-02-11 16:04:08 +00:00
7be0f6594e
First step of making user_rooms_intersect() faster - implement in intersection logic in Python code terms of a DB query that is cacheable per user
2015-02-11 15:53:56 +00:00
ddb816cf60
Don't unfreeze when using FreezeEvent.get_dict, as we are using a JSONEncoder that understands FrozenDict
2015-02-11 15:44:28 +00:00
ddc25cf4e2
Invalidate the cache for an event if it is redacted
2015-02-11 15:23:28 +00:00
aff892ce79
Fix formatting
2015-02-11 15:02:35 +00:00
f5a70e0d2e
Add a cache for get_event
2015-02-11 15:01:15 +00:00
4ebbaf0d43
Blunty replace json with simplejson
2015-02-11 14:23:10 +00:00
fd40d992ad
PEP8-ify
2015-02-11 10:41:33 +00:00
9978c5c103
Merge branch 'develop' into application-services
2015-02-11 10:03:24 +00:00
b085fac735
Code-style fixes
2015-02-10 16:30:48 +00:00
fda4422bc9
Fix pyflakes
2015-02-10 14:54:07 +00:00
d7c7efb691
Add performance counters for different stages of loading events
2015-02-10 14:50:53 +00:00
0c4536da8f
Use the transaction 'desc' rather than 'name', increment the txn_ids in
...
txn names
2015-02-09 18:06:31 +00:00
347b497db0
Formatting
2015-02-09 17:57:09 +00:00
3a5ad7dbd5
Performance counters for database transaction names
2015-02-09 17:55:56 +00:00
53557fc532
Merge branch 'develop' into application-services
2015-02-09 15:20:56 +00:00
c4ee4ce93e
Fix typo
2015-02-09 15:00:37 +00:00
66fde49f07
Log database time every 10s and log as percentage
2015-02-09 14:45:15 +00:00
75656712e3
Time how long we're spending on the database thread
2015-02-09 14:22:52 +00:00
5a7dd05818
Modify auth.get_user_by_req for authing appservices directly.
...
Add logic to map the appservice token to the autogenned appservice user ID.
Add unit tests for all forms of get_user_by_req (user/appservice,
valid/bad/missing tokens)
2015-02-09 14:14:15 +00:00
ac3183caaa
Register a user account for the AS when the AS registers. Add 'sender' column to AS table.
2015-02-09 12:03:37 +00:00
0cd66885e3
Move delta/v13.sql to delta/v12.sql
2015-02-06 14:38:04 +00:00
a3c6010718
Add delta sql file.
2015-02-05 16:48:57 +00:00
fc8bcc809d
Merge branch 'develop' into application-services
2015-02-05 15:32:45 +00:00
aaf50bf6f3
Give server default rules the 'default' attribute and fix various brokenness.
2015-02-05 15:11:38 +00:00
951690e54d
Merge branch 'develop' into application-services
2015-02-05 14:28:03 +00:00
f08bd95880
Merge pull request #47 from matrix-org/signature_failures
...
Federation fixes.
2015-02-05 14:03:00 +00:00
27091f146a
Add hs_token column and generate a different token f.e application service.
2015-02-05 10:08:12 +00:00
89f2e8fbdf
Fix bug in store defer. Add more unit tests.
2015-02-04 15:21:03 +00:00
17753f0c20
Add stub ApplicationServiceApi and glue it with the handler.
2015-02-04 11:19:18 +00:00
03d415a6a2
Brief comment on why we do some things on every call to persist_event and not others
2015-02-04 10:40:59 +00:00
c0462dbf15
Rearrange persist_event so that do all the queries that need to be done before returning early if we have already persisted that event.
2015-02-04 10:16:51 +00:00
02be8da5e1
Add doc to get_event
2015-02-03 17:34:07 +00:00
dc7bb70f22
s/instance_handle/profile_tag/
2015-02-03 16:51:07 +00:00
94a5db9f4d
Add appservice package and move ApplicationService into it.
2015-02-03 14:44:16 +00:00
f2c039bfb9
Implement restricted namespace checks. Begin fleshing out the main hook for notifying application services.
2015-02-03 13:29:27 +00:00
a060b47b13
Add namespace constants. Add restrict_to option to limit namespace checks.
2015-02-03 13:17:28 +00:00
3bd2841fdb
Everyone loves SQL typos
2015-02-03 11:37:52 +00:00
197f3ea4ba
Implement regex checks for app services.
...
Expose handler.get_services_for_event which manages the checks for all
services.
2015-02-03 11:26:33 +00:00
e7ca813dd4
Try to ensure we don't persist an event we have already persisted. In persist_event check if we already have the event, if so then update instead of replacing so that we don't cause a bump of the stream_ordering.
2015-02-03 10:39:41 +00:00
9ff349a3cb
Add defers in the right places.
2015-02-02 17:42:49 +00:00
1a2de0c5fe
Implement txns for AS (un)registration.
2015-02-02 17:39:41 +00:00
941f59101b
Don't fail an entire request if one of the returned events fails a signature check. If an event does fail a signature check, look in the local database and request it from the originator.
2015-02-02 16:56:01 +00:00
a006d168c5
Actually merge into develop.
2015-02-02 16:05:34 +00:00
c059c9fea5
Merge branch 'develop' into application-services
...
Conflicts:
synapse/handlers/__init__.py
synapse/storage/__init__.py
2015-02-02 15:57:59 +00:00
7d897f5bfc
Merge pull request #43 from matrix-org/rejections
...
Rejections
2015-01-30 16:11:14 +00:00
6dc92d3427
Merge pull request #41 from matrix-org/client_v2_sync
...
Client v2 sync
2015-01-30 15:54:25 +00:00
1bd540ef79
Merge branch 'develop' of github.com:matrix-org/synapse into rejections
...
Conflicts:
synapse/storage/schema/im.sql
2015-01-30 15:16:38 +00:00
9ec9d6f2cb
Merge pull request #42 from matrix-org/replication_split
...
Replication split
2015-01-30 15:14:10 +00:00
91015ad008
Remove merge conflict
2015-01-30 14:58:54 +00:00
4f7fe63b6d
Remember to add schema file to list
2015-01-30 14:57:53 +00:00
fdd2ac495a
Merge branch 'develop' of github.com:matrix-org/synapse into rejections_storage
...
Conflicts:
synapse/storage/__init__.py
2015-01-30 14:57:33 +00:00
Erik Johnston