Commit Graph

180 Commits

Author SHA1 Message Date
Erik Johnston 2284eb3a53
Add database config class (#6513)
This encapsulates config for a given database and is the way to get new
connections.
2019-12-18 10:45:12 +00:00
Erik Johnston 71ee22c0ba Fix port db script 2019-12-06 16:41:48 +00:00
Erik Johnston 8b77fc6506 Fix DB scripts 2019-12-05 11:20:49 +00:00
Erik Johnston 756d4942f5 Move DB pool and helper functions into dedicated Database class 2019-12-05 10:46:37 +00:00
Erik Johnston ddbbfc9512
Merge pull request #6464 from matrix-org/erikj/make_public_sql_base
Clean up SQLBaseStore private function usage
2019-12-05 10:43:49 +00:00
Brendan Abolivier f8421a1404
Fix background updates for synapse_port_db 2019-12-04 17:57:35 +00:00
Brendan Abolivier 6cd11109db
Make synapse_port_db exit with a non-0 code if something failed 2019-12-04 17:48:23 +00:00
Erik Johnston ee86abb2d6 Remove underscore from SQLBaseStore functions 2019-12-04 16:23:43 +00:00
Andrew Morgan 57f09e01f5 Fix error when using synapse_port_db on a vanilla synapse db 2019-12-02 18:23:41 +00:00
Andrew Morgan 54fef094b3
Remove usage of deprecated logger.warn method from codebase (#6271)
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
2019-10-31 10:23:24 +00:00
Brendan Abolivier 0ef0b09d68
Merge pull request #6247 from matrix-org/babolivier/port_db_column
Add new boolean column to synapse_port_db
2019-10-24 17:07:34 +01:00
Andrew Morgan ff39cb46de Fix up changelog 2019-10-24 13:43:04 +01:00
Brendan Abolivier 2891693da6
Add new boolean column to synapse_port_db 2019-10-24 12:38:48 +01:00
Brendan Abolivier b2510dce85
Fix import paths in synapse_port_db 2019-10-23 15:54:17 +01:00
Brendan Abolivier c97ed64db3
Make synapse_port_db correctly create indexes (#6102)
Make `synapse_port_db` correctly create indexes in the PostgreSQL database, by having it run the background updates on the database before migrating the data.

To ensure we're migrating the right data, also block the port if the SQLite3 database still has pending or ongoing background updates.

Fixes #4877
2019-10-23 15:31:59 +01:00
Bart Noordervliet dc4bec885d Add missing BOOLEAN_COLUMNs to synapse_port_db (#6216)
Small fix to synapse_port_db to be able to convert from database schema v56.
2019-10-18 11:13:59 +02:00
Amber Brown be3b901ccd
Update the TLS cipher string and provide configurability for TLS on outgoing federation (#5550) 2019-06-28 18:19:09 +10:00
Amber Brown 32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Amber Brown 6312d6cc7c
Expose statistics on extrems to prometheus (#5384) 2019-06-13 22:40:52 +10:00
Brendan Abolivier 843dd714cb
Merge pull request #5325 from matrix-org/babolivier/port_db_account_validity
Add account_validity's email_sent column to the list of boolean columns in synapse_port_db
2019-06-10 13:51:46 +01:00
Richard van der Hoff fb98c05e03 add a script to generate new signing_key files 2019-06-05 15:33:02 +01:00
Brendan Abolivier 83827c4922
Add account_validity's email_sent column to the list of boolean columns in synapse_port_db
Fixes #5306
2019-06-03 17:06:47 +01:00
Neil Johnson 2d951686a7
drop tables listed in #1830 (#4992)
Tables dropped: 
 * application_services, 
 * application_services_regex, 
 * transaction_id_to_pdu, 
 * stats_reporting
 * current_state_resets
 * event_content_hashes
 * event_destinations
 * event_edge_hashes
 * event_signatures
 * feedback
 * room_hosts
 * state_forward_extremities
2019-04-08 15:37:26 +01:00
Anders 2a59e8e429 Fix issue #4596
Make synapse_port_db --curses work with Python 3.
Signed-off-by: Anders Jensen-Waud <anders@jensenwaud.com>
2019-04-03 21:59:48 +00:00
Richard van der Hoff 8e28bc5eee
Include a default configuration file in the 'docs' directory. (#4791) 2019-03-04 17:14:58 +00:00
Eric 8184ae8a09 Consider e2e_room_keys.is_verified column as boolean
This column was considered as an int, crashing the whole
migration process

Signed-off-by: Eric <eric@pedr0.net>
2019-02-20 23:18:00 +01:00
Richard van der Hoff 9c2af7b2c5 Add a script to generate a clean config file (#4315) 2018-12-22 02:04:57 +11:00
Amber Brown 264cb14402
Port hash_password to Python 3 (#4161)
* port hash_password

* changelog
2018-11-08 04:57:28 +11:00
Amber Brown 77d70a7646
Port register_new_matrix_user to Python 3 and add tests (#4085) 2018-10-26 22:05:22 +11:00
Richard van der Hoff ef771cc4c2 Fix a number of flake8 errors
Broadly three things here:

* disable W504 which seems a bit whacko
* remove a bunch of `as e` expressions from exception handlers that don't use
  them
* use `r""` for strings which include backslashes

Also, we don't use pep8 any more, so we can get rid of the duplicate config
there.
2018-10-24 10:39:03 +01:00
Amber Brown e1728dfcbe
Make scripts/ and scripts-dev/ pass pyflakes (and the rest of the codebase on py3) (#4068) 2018-10-20 11:16:55 +11:00
Bruno Windels 9a8bbc9a59 add --no-admin flag to registration script (#3836) 2018-09-28 13:36:56 +01:00
Amber Brown e1a237eaab
Admin API for creating new users (#3415) 2018-07-20 22:41:13 +10:00
Adrian Tschira e54c202b81 Replace some type checks with six type checks
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-07 01:02:32 +02:00
Richard van der Hoff 6a9777ba02 Port script: Set up state_group_id_seq
Fixes https://github.com/matrix-org/synapse/issues/3050.
2018-04-06 15:33:30 +01:00
Richard van der Hoff 01579384cc Port script: clean up a bit
Improve logging and comments. Group all the stuff to do with inspecting tables
together rather than creating the port tables in the middle.
2018-04-06 15:33:30 +01:00
Richard van der Hoff e01ba5bda3 Port script: avoid nasty errors when setting up
We really shouldn't spit out "Failed to create port table", it looks scary.
2018-04-06 15:33:30 +01:00
Richard van der Hoff 326189c25a Script to move remote media to another media store 2018-01-31 18:45:10 +00:00
Matthew Hodgson 3ab2cfec47 sanity checks 2017-11-21 12:10:20 +00:00
Matthew Hodgson 2145ee1976 don't double-invite in sync_room_to_group.pl 2017-11-19 00:48:47 +00:00
Matthew Hodgson 9d50b6f0ea quick and dirty room membership<->group membership sync script 2017-11-17 01:54:02 +00:00
Erik Johnston 9ecf621404 Less s's 2017-11-14 15:55:15 +00:00
Erik Johnston 22db751d1e Add new boolean columns to port script 2017-11-14 15:48:50 +00:00
David Baker 9d419f48e6 Make the port script drop NUL values in all tables
Postgres doesn't support NULs in strings so it makes the script
throw an exception and stop if any values contain \0. Drop them
with appropriate warning.
2017-10-31 16:58:49 +00:00
Luke Barnard e86cefcb6f Add groups table to BOOLEAN_COLUMNS in synapse_port_db 2017-10-26 17:24:54 +01:00
Erik Johnston 37d766aedd Fix port script
We changed _simple_update_one_txn to use _simple_update_txn but didn't
yank it out in the port script.

Fixes #2565
2017-10-26 10:01:03 +01:00
Richard van der Hoff 7fc1aad195 Drop search values with nul characters
https://github.com/matrix-org/synapse/issues/2187 contains a report of a port
failing due to nul characters somewhere in the search table. Let's try dropping
the offending rows.
2017-10-02 00:53:32 +01:00
Erik Johnston 60a9a49f83 Extend comment 2017-07-20 16:16:29 +01:00
Erik Johnston d7d24750be Fix port script for user directory tables 2017-07-20 10:47:01 +01:00
Matthew 3fafb7b189 add missing boolean to synapse_port_db 2017-06-16 20:51:19 +01:00
Matthew 776a070421 fix synapse_port script 2017-06-16 20:24:14 +01:00
John Kristensen be44558886 Don't assume postgres tables are in the public schema during db port
When fetching the list of tables from the postgres database during the
db port, it is assumed that the tables are in the public schema. This is
not always the case, so lets just rely on postgres to determine the
default schema to use.
2017-03-17 10:53:32 +11:00
Kevin Liu 70a00eacf9 Fix typo
This is what I get for not proofreading
2017-02-11 20:49:31 -05:00
Kevin Liu a02d609b1f Fix synapse_port_db failure (fixes #1902)
See https://matrix.to/#/!cURbafjkfsMDVwdRDQ:matrix.org/$148686272020hCgRD:potatofrom.space

Signed-off-by: Kevin Liu <kevin@potatofrom.space>
2017-02-11 20:44:16 -05:00
Erik Johnston adae348fdf Update port script with recently added tables
This also fixes a bug where the port script would explode when it
encountered the newly added boolean column
`public_room_list_stream.visibility`
2016-09-27 11:19:48 +01:00
Erik Johnston 7c7786d4e1 Allow upgrading from old port_from_sqlite3 format 2016-08-04 11:38:08 +01:00
Erik Johnston b0a14bf53e Handle the fact that some tables have negative rowid rows 2016-08-04 11:28:02 +01:00
Erik Johnston f0c06ac65c Merge pull request #909 from matrix-org/erikj/shared_secret
Add an admin option to shared secret registration (breaks backwards compat)
2016-07-06 14:08:51 +01:00
Erik Johnston 0da24cac8b Add null separator to hmac 2016-07-06 11:05:16 +01:00
Kent Shikama 896bc6cd46
Update hash_password script
Signed-off-by: Kent Shikama <kent@kentshikama.com>
2016-07-06 12:17:54 +09:00
Erik Johnston 651faee698 Add an admin option to shared secret registration 2016-07-05 17:30:22 +01:00
Erik Johnston caf33b2d9b Protect password when registering using shared secret 2016-07-05 17:18:19 +01:00
Erik Johnston 4063fe0283 Update port script 2016-04-22 10:35:53 +01:00
Erik Johnston 8aab9d87fa Don't require config to create database 2016-04-06 14:15:45 +01:00
Richard van der Hoff a877209c8b Password reset docs and script
Replace the bash/perl gen_password script with a python one, and write a note
on how to use it.
2016-03-16 09:45:37 +00:00
Daniel Wagner-Hall 763360594d Mark AS users with their AS's ID 2016-02-11 17:26:42 +00:00
Matthew Hodgson 6c28ac260c copyrights 2016-01-07 04:26:29 +00:00
Matthew Hodgson 5577a61090 throwaway 1-liner for generating password hashes 2015-12-10 19:03:06 +00:00
Erik Johnston b361440738 Spelling 2015-11-19 09:11:42 +00:00
Erik Johnston 162e2c1ce5 Fix database port script to work with new event_search table 2015-11-18 15:54:50 +00:00
Mark Haines 7d55314277 Remove unused _execute_and_decode from scripts/synapse_port_db 2015-09-23 10:42:02 +01:00
Mark Haines 73e4ad4b8b Merge branch 'master' into develop
Conflicts:
	setup.py
2015-09-07 15:06:46 +01:00
Matthew Hodgson bdf2e5865a update logger to match new ambiguous script name... 2015-09-03 09:51:42 +03:00
Erik Johnston a9ad647fb2 Make port script handle empty sent_transactions table 2015-09-02 11:11:11 +01:00
Erik Johnston c961cd7736 Clean up scripts/ 2015-08-27 13:03:17 +01:00
Erik Johnston 254aa3c986 Revert register_new_matrix_user to use v1 api 2015-05-22 11:59:48 +01:00
Erik Johnston 4fa0f53521 Support reading directly from a config 2015-05-08 13:45:58 +01:00
Erik Johnston f08c33e834 Fix port_from_sqlite_to_postgres after changes to storage layer. 2015-05-08 13:29:00 +01:00
Erik Johnston 3c92231094 Re-add scripts/register_new_matrix_user 2015-05-06 11:45:18 +01:00
Erik Johnston 119e5d7702 Seperate scripts/ into scripts/ and scripts-dev/, where scripts/* are automatically added to the package 2015-05-06 11:41:19 +01:00
Erik Johnston 204132a998 Check that postgres database has correct charset set 2015-04-29 11:42:28 +01:00
Erik Johnston f41a9a1ffc Add better help to scripts/port_from_sqlite_to_postgres.py 2015-04-28 17:42:36 +01:00
Erik Johnston a1d4813a54 Quickly fix dodgy est. time remaining 2015-04-28 12:55:29 +01:00
Erik Johnston af27b84ff7 Correctly handle total/remaining counts in the presence of sent_transasctions table 2015-04-28 12:40:04 +01:00
Erik Johnston 4a13ae7201 Correctly handle total/remaining counts in the presence of sent_transasctions table 2015-04-28 11:16:44 +01:00
Erik Johnston ce8b0b2868 Remove accidentally committed debug hardcode hack 2015-04-28 10:45:05 +01:00
Erik Johnston 1ccaea5b92 Typo in port script 2015-04-28 10:34:06 +01:00
Erik Johnston 5b8b1a43bd Split setuping up and processing of tables 2015-04-27 17:53:40 +01:00
Erik Johnston 40cbd6b6ee Shuffle progress stuff 2015-04-27 17:53:15 +01:00
Erik Johnston 4e49f52375 Don't port over all of the sent_transactions table 2015-04-27 17:36:37 +01:00
Erik Johnston dfc46c6220 PEP8 2015-04-17 12:46:29 +01:00
Erik Johnston 427bcb7608 Fix port script after storage changes. Add very simple (off by default) curses UI to see progress. 2015-04-17 11:13:05 +01:00
Erik Johnston ffad75bd62 Remove mysql/maria support 2015-04-15 17:00:50 +01:00
Erik Johnston 63677d1f47 Change port script to work with postgres 2015-04-15 10:23:24 +01:00
Erik Johnston 90bcb86957 Support running porting script multiple times 2015-04-13 13:45:04 +01:00
Erik Johnston 1bede47843 Add beginnings of migration sqlite->maria db script 2015-04-10 18:47:20 +01:00
Erik Johnston 82b34e813d SYN-67: Finish up implementing new database schema management 2015-03-04 12:04:19 +00:00
Kegan Dougal c3c01641d2 Run deltas and bump user_version in upgrade script 2015-03-02 13:38:57 +00:00
Kegan Dougal 3f6b36d96e Add upgrade script 2015-03-02 11:39:58 +00:00
Erik Johnston 963256638d Correctly handle all the places that can throw exceptions 2015-02-12 18:17:11 +00:00