Commit Graph

318 Commits

Author SHA1 Message Date
Erik Johnston 2732be83d9 Shuffle operations so that locking upsert happens last in the txn. This ensures the lock is held for the least amount of time possible. 2015-04-27 13:22:30 +01:00
Erik Johnston e4c4664d73 Handle the fact that postgres databases can be restarted from under us 2015-04-27 12:40:49 +01:00
Erik Johnston b8092fbc82 Go back to storing JSON in TEXT 2015-04-16 11:17:52 +01:00
Erik Johnston 5b31afcbd1 Remove debug logging 2015-04-15 16:27:04 +01:00
Erik Johnston ed26e4012b pushers table requires a unique id. 2015-04-15 16:24:14 +01:00
Erik Johnston a5c72780e6 Don't pass in removed flag 2015-04-15 15:13:22 +01:00
Erik Johnston 4af32a2817 Postgres does not allow you to continue using a cursor after a DB exception has been raised, so move _simple_insert or_ignore flag out of transaction 2015-04-15 14:51:21 +01:00
Erik Johnston 58d8339966 Add support for postgres instead of mysql. Change sql accourdingly. blob + varbinary -> bytea. No support for UNSIGNED or CREATE INDEX IF NOT EXISTS. 2015-04-14 13:53:20 +01:00
Erik Johnston 7ed2ec3061 Handle the fact that in sqlite binary data might be stored as unicode or bytes 2015-04-10 13:41:54 +01:00
Erik Johnston 8bf285e082 Merge branch 'develop' of github.com:matrix-org/synapse into mysql 2015-04-09 13:13:26 +01:00
Erik Johnston 8ad0f4912e
Stream ordering and out of order insertions.
Handle the fact that events can be persisted out of order, and so to get
the "current max" stream token becomes non trivial - as we need to make
sure that *all* stream tokens less than the current max have also
successfully been persisted.
2015-04-09 11:41:36 +01:00
Mark Haines 83f5125d52 Merge pull request #114 from matrix-org/improve_get_event_cache
Improve get event cache
2015-04-08 16:50:27 +01:00
Erik Johnston 3887350e47 Merge branch 'develop' of github.com:matrix-org/synapse into mysql 2015-04-08 14:11:07 +01:00
Erik Johnston c1b34af441 Move database timer logging to seperate logger 2015-04-08 13:31:19 +01:00
Erik Johnston 9a05795619 Retry transaction, not SQL query 2015-04-08 13:11:28 +01:00
Erik Johnston 4fe95094d1 Merge branch 'develop' of github.com:matrix-org/synapse into mysql 2015-04-07 18:05:39 +01:00
Erik Johnston 49d6aa1394 Retry on deadlock 2015-04-07 15:28:37 +01:00
Erik Johnston 0af5f5efaf Don't use multiple UNIQUE constraints; it will cause deadlocks 2015-04-07 12:08:35 +01:00
Erik Johnston c8d3f6486d Implement or_ignore flag on inserts 2015-04-07 12:06:22 +01:00
Erik Johnston 304111afd0 Don't use AUTOINCREMENT, use an in memory version 2015-04-07 12:05:36 +01:00
Erik Johnston 779f7b0f44 Fix unicode support 2015-04-02 10:06:22 +01:00
Erik Johnston 9236136f3a Make work in both Maria and SQLite. Fix tests 2015-04-01 14:12:33 +01:00
Paul "LeoNerd" Evans a198894bf7 Appease pep8 2015-03-26 11:53:58 +00:00
Paul "LeoNerd" Evans 953e40f9dc Implement the main getEvent cache using Cache() instead of a custom application of LruCache; also unify its two-level structure into just one 2015-03-25 19:12:16 +00:00
Paul "LeoNerd" Evans f173d40a32 Use FrozenEvent's reject_reason to decide whether to return it; don't include allow_rejected in the main getEvents cache key 2015-03-25 19:06:05 +00:00
Paul "LeoNerd" Evans 1b988b051b Store the rejected reason in (Frozen)Event structs 2015-03-25 19:06:05 +00:00
Paul "LeoNerd" Evans 033a517feb Indirect invalidations of _get_event_cache via a helper method to keep all uses of the cache lexically within one .py file 2015-03-25 19:06:05 +00:00
Paul "LeoNerd" Evans 9ba6487b3f Allow a choice of LRU behaviour for Cache() by using LruCache() or OrderedDict() 2015-03-25 19:05:34 +00:00
Erik Johnston 0e8f5095c7 Fix unicode database support 2015-03-25 17:15:20 +00:00
Erik Johnston 9e98f1022a Don't order by rowid 2015-03-24 16:19:01 +00:00
Erik Johnston 6e7131f02f Remove uses of REPLACE and ON CONFLICT IGNORE to make the SQL more portable. 2015-03-23 15:38:56 +00:00
Paul "LeoNerd" Evans 0f86312c4c Pull out the cache logic from the @cached wrapper into its own class we can reuse 2015-03-20 18:25:42 +00:00
Paul "LeoNerd" Evans b1022ed8b5 func(*EXPR) is valid Python syntax, really... 2015-03-20 17:47:45 +00:00
Erik Johnston f6583796fe Merge branch 'develop' of github.com:matrix-org/synapse into mysql 2015-03-20 16:31:48 +00:00
Erik Johnston 80cd08c190 PEP8 2015-03-20 16:03:25 +00:00
Erik Johnston 9517f4da4d Merge branch 'develop' of github.com:matrix-org/synapse into store_rearrangement 2015-03-20 16:02:47 +00:00
Erik Johnston dc0c989ef4 Give sensible names for '_simple_...' transactions 2015-03-20 15:59:18 +00:00
Erik Johnston 7e282a53a5 Tidy up _simple_... methods 2015-03-20 15:05:10 +00:00
Paul "LeoNerd" Evans 91cb46191d Allow @cached-wrapped functions to have more or fewer than 1 argument; assert on the total count of them though 2015-03-20 14:59:45 +00:00
Erik Johnston 87db64b839 Rearrange storage modules 2015-03-20 14:11:38 +00:00
Erik Johnston cb8162d3d1 Rearrange storage modules 2015-03-20 13:52:56 +00:00
Erik Johnston d7a0496f3e Convert storage layer to be mysql compatible 2015-03-19 15:59:48 +00:00
Paul "LeoNerd" Evans 1489521ee5 Be polite and ensure we use @functools.wraps() when creating a function decorator 2015-03-17 17:19:22 +00:00
Paul "LeoNerd" Evans be170b1426 Add a metric for the scheduling latency of SQL queries 2015-03-16 17:21:59 +00:00
Paul "LeoNerd" Evans e75fa8bbbf Bugfix to sql_txn_timer increment - add only the per-TXN duration, not the total time ever spent since boot 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans 63cb7ece62 Rename the timer metrics exported by synapse.storage to append _time, so the meaning of ':total' is clearer 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans f1fbe3e09f Rename TimerMetric to DistributionMetric; as it could count more than just time 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans 642f725fd7 Pretend the 'getEvent' cache is just another cache in the set of all the others for metric 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans b0cf867319 Use _ instead of . as a metric namespacing separator, for Prometheus 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans f9478e475b Rename Metrics' "keys" to "labels" 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans fa319a5786 Add TimerMetrics to shadow the PerformanceCounters in synapse.storage; with the view to eventually replacing them entirely 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans 25187ab674 Collect per-SQL-verb timer stats on query execution time 2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans e1a7e3564f Delete a couple of TODO markers of monitoring stats now done 2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans 8664599af7 Rename CacheCounterMetric to just CacheMetric; add a CallbackMetric component to give the size of the cache 2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans b0cdf097f4 Sprinkle some CacheCounterMetrics around the synapse.storage layer 2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans 59a5f012cc Also give _execute() a description 2015-03-11 17:19:17 +00:00
Paul "LeoNerd" Evans 099e4b88d8 Add a description to storage layer's _execute_and_decode() 2015-03-11 17:08:57 +00:00
Kegan Dougal 3d73383d18 Modify _simple_select_list to allow an empty WHERE clause. Use it for get_all_rooms and get_all_users. 2015-03-02 10:16:24 +00:00
Paul "LeoNerd" Evans 9640510de2 Use OrderedDict for @cached backing store, so we can evict the oldest key unbiased 2015-02-23 18:41:58 +00:00
Paul "LeoNerd" Evans f53fcbce97 Use cache.pop() instead of a separate membership test + del [] 2015-02-23 18:30:45 +00:00
Mark Haines 27080698e7 Fix code style warning 2015-02-23 18:19:13 +00:00
Paul "LeoNerd" Evans a09e59a698 Pull the _get_event_cache.setdefault() call out of the try block, as it doesn't need to be there and is confusing 2015-02-23 16:55:57 +00:00
Paul "LeoNerd" Evans e76d485e29 Allow @cached-wrapped functions to have a prefill method for setting entries 2015-02-23 15:41:54 +00:00
Paul "LeoNerd" Evans 55022d6ca5 Remove a TODO note 2015-02-19 18:38:09 +00:00
Paul "LeoNerd" Evans ebc3db295b Take named arguments to @cached() decorator, add a 'max_entries' limit 2015-02-19 18:36:02 +00:00
Paul "LeoNerd" Evans 077d200342 Move @cached decorator out into synapse.storage._base; add minimal docs 2015-02-19 17:29:39 +00:00
Erik Johnston 42bc56dad3 Merge branch 'develop' of github.com:matrix-org/synapse into use-simplejson 2015-02-11 17:01:38 +00:00
Mark Haines aff892ce79 Fix formatting 2015-02-11 15:02:35 +00:00
Mark Haines f5a70e0d2e Add a cache for get_event 2015-02-11 15:01:15 +00:00
Erik Johnston 4ebbaf0d43 Blunty replace json with simplejson 2015-02-11 14:23:10 +00:00
Mark Haines b085fac735 Code-style fixes 2015-02-10 16:30:48 +00:00
Mark Haines fda4422bc9 Fix pyflakes 2015-02-10 14:54:07 +00:00
Mark Haines d7c7efb691 Add performance counters for different stages of loading events 2015-02-10 14:50:53 +00:00
Mark Haines 0c4536da8f Use the transaction 'desc' rather than 'name', increment the txn_ids in
txn names
2015-02-09 18:06:31 +00:00
Mark Haines 347b497db0 Formatting 2015-02-09 17:57:09 +00:00
Mark Haines 3a5ad7dbd5 Performance counters for database transaction names 2015-02-09 17:55:56 +00:00
Erik Johnston c4ee4ce93e Fix typo 2015-02-09 15:00:37 +00:00
Erik Johnston 66fde49f07 Log database time every 10s and log as percentage 2015-02-09 14:45:15 +00:00
Erik Johnston 75656712e3 Time how long we're spending on the database thread 2015-02-09 14:22:52 +00:00
Erik Johnston 2f4cb04f45 Be more specific in naming columns in selects. 2015-01-30 14:48:11 +00:00
Erik Johnston e0b7c521cb Merge branch 'develop' of github.com:matrix-org/synapse into rejections_storage
Conflicts:
	synapse/storage/__init__.py
	synapse/storage/schema/delta/v12.sql
2015-01-30 14:08:28 +00:00
David Baker 6d485dd1c7 unnecessary newlines 2015-01-28 14:48:42 +00:00
David Baker fb0928097a More magic commas (including the place I copied it from...) 2015-01-28 14:48:07 +00:00
David Baker 0cbb6b0f52 Google doc style 2015-01-28 14:44:41 +00:00
Erik Johnston b1b85753d7 Add support for storing rejected events in EventContext and data stores 2015-01-22 15:50:17 +00:00
David Baker c06a9063e1 Merge branch 'develop' into pushers 2015-01-13 13:15:51 +00:00
Erik Johnston 36a2a877e2 Use time.time() instead of time.clock() 2015-01-06 16:34:41 +00:00
Erik Johnston 52b2c6c9c7 Don't include None's in _get_events_txn 2015-01-06 14:56:57 +00:00
Mark Haines 5e23a19204 Merge pull request #28 from matrix-org/erikj-perf
Database performance improvements.
2015-01-06 13:33:40 +00:00
Mark Haines adb04b1e57 Update copyright notices 2015-01-06 13:21:39 +00:00
Erik Johnston af1c7c7808 PEP8 2015-01-06 13:13:17 +00:00
Erik Johnston 52d8519008 Don't do batching when getting events. 2015-01-06 13:10:27 +00:00
Erik Johnston 98933e3db6 Only fetch prev_content when a client is streaming/paginating. Use transactions for event streams. 2015-01-06 13:03:23 +00:00
Erik Johnston 3e26720e05 Temporarily turn off 'redacted_because' and 'prev_content' keys 2015-01-06 11:26:58 +00:00
Erik Johnston f4ea78e9e2 More debug logging 2015-01-06 11:24:18 +00:00
Erik Johnston d7e8ea67b3 Reformat 2015-01-06 11:18:02 +00:00
David Baker b56730bb6e Merge branch 'develop' into pushers
Conflicts:
	synapse/api/errors.py
	synapse/server.py
	synapse/storage/__init__.py
2014-12-18 15:15:22 +00:00
David Baker 9728c305a3 after a few rethinks, a working implementation of pushers. 2014-12-18 14:49:22 +00:00
Erik Johnston f3788e3c78 Test some ideas that might help performance a bit 2014-12-17 23:37:08 +00:00
Erik Johnston 5b39cfff69 Don't assume an event exists 2014-12-16 18:25:24 +00:00
Erik Johnston 882dc8dcab Persist internal_metadata 2014-12-16 13:17:09 +00:00
Erik Johnston c8dd3314d6 Fix bug where we ignored event_edge_hashes table 2014-12-15 13:55:22 +00:00
Erik Johnston 75085bb4d1 Pyflakes 2014-12-12 14:34:34 +00:00
Erik Johnston 0b04369238 Fix public room joining by making sure replaces_state never points to itself. 2014-12-11 15:56:06 +00:00
Erik Johnston 9191292b0f Fix prev_content 2014-12-11 15:16:55 +00:00
Erik Johnston 8cdebce470 Fix redactions. Fix 'age' key 2014-12-11 13:25:19 +00:00
Erik Johnston 02e4c18171 Remove dead code 2014-12-10 18:00:36 +00:00
Erik Johnston aae8a37e63 Merge branch 'develop' of github.com:matrix-org/synapse into events_refactor 2014-12-10 13:18:40 +00:00
Erik Johnston aa3f66cf7f Change the way we implement get_events to be less sucky 2014-12-09 13:35:26 +00:00
Erik Johnston ba3d1e2fc0 Remove unused import 2014-12-08 12:01:25 +00:00
Matthew Hodgson aed62a3583 track replication destination health, and perform exponential back-off when sending transactions. does *not* yet retry transactions, but drops them on the floor if waiting for a server to recover. 2014-12-07 02:26:07 +00:00
Erik Johnston 6630e1b579 Start making more things use EventContext rather than event.* 2014-12-05 16:20:48 +00:00
Erik Johnston ec2b5d8c28 Store full JSON of events in db 2014-12-01 16:22:07 +00:00
Erik Johnston a46e5ef621 SYN-163: Add an order by rowid to selects.
This should fix the bug where the edges of the graph get returned in a
different order than they were inserted in, and so no get_event no
longer returned the exact same JSON as was inserted. This meant that
signature checks failed.
2014-11-24 10:56:36 +00:00
Mark Haines db9ce032a4 Fix pep8 codestyle warnings 2014-11-20 17:26:36 +00:00
Mark Haines dfdda2c871 Use module loggers rather than the root logger. Exceptions caused by bad clients shouldn't cause ERROR level logging. Fix sql logging to use 'repr' rather than 'str' 2014-11-20 17:10:37 +00:00
Mark Haines 32090aee16 Add a few missing yields, Move deferred lists inside PreserveLoggingContext because they don't interact well with the logging contexts 2014-11-20 16:24:00 +00:00
Mark Haines cb4b6c844a Merge PDUs and Events into one object 2014-11-14 21:25:02 +00:00
Mark Haines e903c941cb Merge branch 'develop' into request_logging
Conflicts:
	setup.py
	synapse/storage/_base.py
	synapse/util/async.py
2014-11-14 11:16:50 +00:00
Erik Johnston f04b3d5042 Store all signatures on events rather than just dropping them 2014-11-12 17:02:34 +00:00
Erik Johnston a8e565eca8 Add an EventValidator. Fix bugs in auth ++ storage 2014-11-10 18:25:42 +00:00
Erik Johnston cdc1b5d629 Fix regression where we did not return redacted events. 2014-11-10 15:21:30 +00:00
Erik Johnston 6447db063a Fix backfill to work. Add auth to backfill request 2014-11-10 11:59:51 +00:00
Erik Johnston 1c06806f90 Finish redaction algorithm. 2014-11-10 10:21:32 +00:00
Erik Johnston bf6b72eb55 Start implementing auth chains 2014-11-07 10:53:38 +00:00
Erik Johnston 4317c8e583 Implement new replace_state and changed prev_state
`prev_state` is now a list of previous state ids, similiar to
prev_events. `replace_state` now points to what we think was replaced.
2014-11-06 15:10:55 +00:00
Erik Johnston cc44ecc62f Get correct prev_events 2014-11-05 13:23:35 +00:00
Erik Johnston aa76bf39ab Remove unused imports 2014-11-04 14:14:02 +00:00
Mark Haines b29517bd01 Add a request-id to each log line 2014-10-30 01:21:33 +00:00
Erik Johnston e7858b6d7e Start filling out and using new events tables 2014-10-29 16:59:24 +00:00
Erik Johnston 2d1dfb3b34 Begin implementing all the PDU storage stuff in Events land 2014-10-28 16:42:35 +00:00
Erik Johnston da1dda3e1d Add transaction level logging and timing information. Add a _simple_delete method 2014-10-28 11:18:04 +00:00
Erik Johnston 967ce43b59 Clean up LoggingTransaction 2014-10-28 10:53:11 +00:00
Erik Johnston 8e358ef35a Add timer to LoggingTransaction 2014-10-28 10:34:05 +00:00
Mark Haines 82c5820767 keep 'origin_server_ts' as 'ts' in the database to avoid needlessly updating schema 2014-10-17 17:31:48 +01:00
Mark Haines f5cf7ac25b SPEC-7: Rename 'ts' to 'origin_server_ts' 2014-10-17 17:12:25 +01:00
Mark Haines 07639c79d9 Respond with more helpful error messages for unsigned requests 2014-10-13 16:39:15 +01:00
Erik Johnston 70899d3ab2 Rename deletions to redactions 2014-09-24 15:27:59 +01:00
Erik Johnston 1e6c5b205c Fix bug where we didn't correctly pull out the event_id of the deletion 2014-09-24 13:29:20 +01:00
Erik Johnston efea61dc50 Rename 'pruned' to 'pruned_because' 2014-09-23 17:40:58 +01:00
Erik Johnston 78af6bbb98 Add m.room.deletion. If an event is deleted it will be returned to clients 'pruned', i.e. all client specified keys will be removed. 2014-09-23 15:28:32 +01:00
Paul "LeoNerd" Evans b0406b9ead Merge remote-tracking branch 'origin/develop' into test-sqlite-memory 2014-09-15 14:15:10 +01:00
Erik Johnston 5bd9369a62 Correctly handle the 'age' key in events and pdus 2014-09-15 13:26:11 +01:00
Paul "LeoNerd" Evans 1c51c8ab7d Merge remote-tracking branch 'origin/develop' into test-sqlite-memory
Conflicts:
	synapse/storage/pdu.py
2014-09-12 17:20:06 +01:00
Erik Johnston 39e3fc69e5 Make the state resolution use actual power levels rather than taking them from a Pdu key. 2014-09-12 17:11:09 +01:00
Paul "LeoNerd" Evans a840ff8f3f Now don't need the other logger.debug() call in _execute 2014-09-12 14:38:27 +01:00
Paul "LeoNerd" Evans 1c20249884 Logging of all SQL queries via the 'synapse.storage.SQL' logger 2014-09-12 14:37:55 +01:00
Paul "LeoNerd" Evans e53d77b501 Add a .runInteraction() method on SQLBaseStore itself to wrap the .db_pool 2014-09-12 14:28:07 +01:00
Erik Johnston 781ff713ba When getting a state event also include the previous content 2014-09-06 02:23:36 +01:00
Paul "LeoNerd" Evans d703e712f7 Add support to _simple_insert() to do INSERT OR REPLACE 2014-09-03 18:18:41 +01:00
Matthew Hodgson 8a7c1d6a00 fix the copyright holder from matrix.org to OpenMarket Ltd, as matrix.org hasn't been incorporated in time for launch. 2014-09-03 17:31:57 +01:00
Mark Haines 4b63b06cad Merge branch 'develop' into storage_transactions
Conflicts:
	synapse/api/auth.py
	synapse/handlers/room.py
	synapse/storage/__init__.py
2014-08-26 16:07:05 +01:00
Mark Haines 4b2ad549d5 Move the event storage into a single transaction 2014-08-26 14:36:03 +01:00
Erik Johnston 063e1b22e6 Stop internal keys from getting into SynapseEvents 2014-08-21 15:06:00 +01:00
Erik Johnston d260a42ca2 PEP8 cleanups 2014-08-15 16:17:36 +01:00
Erik Johnston 114984a236 Start chagning the events stream to work with the new DB schema 2014-08-15 13:58:28 +01:00
Erik Johnston 5002efa31b Reimplement the get public rooms api to work with new DB schema 2014-08-15 10:26:35 +01:00
Erik Johnston 2c46bb6208 Fix up typos and correct sql queries 2014-08-14 18:40:50 +01:00
Erik Johnston 661c711765 Start fixing places that use the data store. 2014-08-14 17:34:37 +01:00
Erik Johnston 2529f2bc01 Rename _execute_query 2014-08-14 16:58:51 +01:00
Erik Johnston 937c175029 Fix up RoomMemberStore to work with the new schema. 2014-08-14 16:02:10 +01:00
Erik Johnston 6d6a1c3454 Actually encode dicts as json in the DB 2014-08-14 14:30:25 +01:00
Erik Johnston 10294b6082 Merge branch 'master' of github.com:matrix-org/synapse into sql_refactor
Conflicts:
	synapse/storage/_base.py
2014-08-14 10:01:04 +01:00
Paul "LeoNerd" Evans d05aa651f8 An initial hack at storing presence state-change mtimes in database and presenting age durations to clients/federation events 2014-08-13 19:19:15 +01:00
Paul "LeoNerd" Evans baf04be5cf Set datastore's .hs field in SQLBaseStore rather than in the toplevel DataStore mixed-in result class 2014-08-13 19:19:15 +01:00
Erik Johnston 336987bb8d Initial stab at refactoring the SQL tables, including rejigging some of the storage layer. 2014-08-13 18:03:41 +01:00
Matthew Hodgson f98e6380f1 add in whitespace after copyright statements to improve legibility 2014-08-13 03:14:34 +01:00
matrix.org 4f475c7697 Reference Matrix Home Server 2014-08-12 15:10:52 +01:00