Add a Postgres `REPLICA IDENTITY` to tables that do not have an implicit one. This should allow use of Postgres logical replication. (take 2, now with no added deadlocks!) (#16658)

* Add `ALTER TABLE ... REPLICA IDENTITY ...` for individual tables

We can't combine them into one file as it makes it likely to hit a deadlock

if Synapse is running, as it only takes one other transaction to access two

tables in a different order to the schema delta.

* Add notes

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* Re-introduce REPLICA IDENTITY test

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
This commit is contained in:
reivilibre 2023-12-04 14:57:28 +00:00 committed by GitHub
parent 0aa4d3b6f7
commit 51e4e35653
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
123 changed files with 218 additions and 1 deletions

1
changelog.d/16658.misc Normal file
View File

@ -0,0 +1 @@
Add a Postgres `REPLICA IDENTITY` to tables that do not have an implicit one. This should allow use of Postgres logical replication.

View File

@ -0,0 +1 @@
ALTER TABLE applied_module_schemas REPLICA IDENTITY USING INDEX applied_module_schemas_module_name_file_key;

View File

@ -0,0 +1 @@
ALTER TABLE applied_schema_deltas REPLICA IDENTITY USING INDEX applied_schema_deltas_version_file_key;

View File

@ -0,0 +1 @@
ALTER TABLE background_updates REPLICA IDENTITY USING INDEX background_updates_uniqueness;

View File

@ -0,0 +1 @@
ALTER TABLE schema_compat_version REPLICA IDENTITY USING INDEX schema_compat_version_lock_key;

View File

@ -0,0 +1 @@
ALTER TABLE schema_version REPLICA IDENTITY USING INDEX schema_version_lock_key;

View File

@ -0,0 +1,13 @@
The `10_replica_identity_xxx.sql.postgres` series of schema deltas adds replica identities for tables that do not have one implicitly as a result of having a primary key.
This is needed to use logical replication with Synapse (at least without `UPDATE` and `DELETE` statements failing!).
Where possible, we use an existing `UNIQUE` index on `NOT NULL` columns as the replica identity. Otherwise, we have to fall back to using the full row as a replica identity.
Unfortunately, by running all the `ALTER TABLE` statements in one schema delta per database, it was too likely to hit a deadlock as it would only take
one other transaction from a running Synapse worker to access the tables out of order and trigger a deadlock.
By having each statement in its own delta file, each one is run in its own transaction and only needs to take a very brief (instant) lock on the table but no other tables,
so there should be no chance of deadlock.
Like many schema deltas we already apply to Synapse, it is probably blocked by an ongoing `pg_dump`.

View File

@ -0,0 +1 @@
ALTER TABLE account_data REPLICA IDENTITY USING INDEX account_data_uniqueness;

View File

@ -0,0 +1 @@
ALTER TABLE application_services_txns REPLICA IDENTITY USING INDEX application_services_txns_as_id_txn_id_key;

View File

@ -0,0 +1 @@
ALTER TABLE appservice_room_list REPLICA IDENTITY USING INDEX appservice_room_list_idx;

View File

@ -0,0 +1 @@
ALTER TABLE appservice_stream_position REPLICA IDENTITY USING INDEX appservice_stream_position_lock_key;

View File

@ -0,0 +1 @@
ALTER TABLE batch_events REPLICA IDENTITY USING INDEX chunk_events_event_id;

View File

@ -0,0 +1 @@
ALTER TABLE blocked_rooms REPLICA IDENTITY USING INDEX blocked_rooms_idx;

View File

@ -0,0 +1 @@
ALTER TABLE cache_invalidation_stream_by_instance REPLICA IDENTITY USING INDEX cache_invalidation_stream_by_instance_id;

View File

@ -0,0 +1 @@
ALTER TABLE current_state_delta_stream REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE current_state_events REPLICA IDENTITY USING INDEX current_state_events_event_id_key;

View File

@ -0,0 +1 @@
ALTER TABLE deleted_pushers REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE device_auth_providers REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE device_federation_inbox REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE device_federation_outbox REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE device_inbox REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE device_lists_changes_converted_stream_position REPLICA IDENTITY USING INDEX device_lists_changes_converted_stream_position_lock_key;

View File

@ -0,0 +1 @@
ALTER TABLE device_lists_changes_in_room REPLICA IDENTITY USING INDEX device_lists_changes_in_stream_id;

View File

@ -0,0 +1 @@
ALTER TABLE device_lists_outbound_last_success REPLICA IDENTITY USING INDEX device_lists_outbound_last_success_unique_idx;

View File

@ -0,0 +1 @@
ALTER TABLE device_lists_outbound_pokes REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE device_lists_remote_cache REPLICA IDENTITY USING INDEX device_lists_remote_cache_unique_id;

View File

@ -0,0 +1 @@
ALTER TABLE device_lists_remote_extremeties REPLICA IDENTITY USING INDEX device_lists_remote_extremeties_unique_idx;

View File

@ -0,0 +1 @@
ALTER TABLE device_lists_remote_resync REPLICA IDENTITY USING INDEX device_lists_remote_resync_idx;

View File

@ -0,0 +1 @@
ALTER TABLE device_lists_stream REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE devices REPLICA IDENTITY USING INDEX device_uniqueness;

View File

@ -0,0 +1 @@
ALTER TABLE e2e_cross_signing_keys REPLICA IDENTITY USING INDEX e2e_cross_signing_keys_stream_idx;

View File

@ -0,0 +1 @@
ALTER TABLE e2e_cross_signing_signatures REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE e2e_device_keys_json REPLICA IDENTITY USING INDEX e2e_device_keys_json_uniqueness;

View File

@ -0,0 +1 @@
ALTER TABLE e2e_fallback_keys_json REPLICA IDENTITY USING INDEX e2e_fallback_keys_json_uniqueness;

View File

@ -0,0 +1 @@
ALTER TABLE e2e_one_time_keys_json REPLICA IDENTITY USING INDEX e2e_one_time_keys_json_uniqueness;

View File

@ -0,0 +1 @@
ALTER TABLE e2e_room_keys REPLICA IDENTITY USING INDEX e2e_room_keys_with_version_idx;

View File

@ -0,0 +1 @@
ALTER TABLE e2e_room_keys_versions REPLICA IDENTITY USING INDEX e2e_room_keys_versions_idx;

View File

@ -0,0 +1 @@
ALTER TABLE erased_users REPLICA IDENTITY USING INDEX erased_users_user;

View File

@ -0,0 +1 @@
ALTER TABLE event_auth REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE event_auth_chain_links REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE event_backward_extremities REPLICA IDENTITY USING INDEX event_backward_extremities_event_id_room_id_key;

View File

@ -0,0 +1 @@
ALTER TABLE event_edges REPLICA IDENTITY USING INDEX event_edges_event_id_prev_event_id_idx;

View File

@ -0,0 +1 @@
ALTER TABLE event_forward_extremities REPLICA IDENTITY USING INDEX event_forward_extremities_event_id_room_id_key;

View File

@ -0,0 +1 @@
ALTER TABLE event_json REPLICA IDENTITY USING INDEX event_json_event_id_key;

View File

@ -0,0 +1 @@
ALTER TABLE event_push_actions REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE event_push_actions_staging REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE event_push_summary REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE event_push_summary_last_receipt_stream_id REPLICA IDENTITY USING INDEX event_push_summary_last_receipt_stream_id_lock_key;

View File

@ -0,0 +1 @@
ALTER TABLE event_push_summary_stream_ordering REPLICA IDENTITY USING INDEX event_push_summary_stream_ordering_lock_key;

View File

@ -0,0 +1 @@
ALTER TABLE event_relations REPLICA IDENTITY USING INDEX event_relations_id;

View File

@ -0,0 +1 @@
ALTER TABLE event_search REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE event_to_state_groups REPLICA IDENTITY USING INDEX event_to_state_groups_event_id_key;

View File

@ -0,0 +1 @@
ALTER TABLE event_txn_id REPLICA IDENTITY USING INDEX event_txn_id_event_id;

View File

@ -0,0 +1 @@
ALTER TABLE event_txn_id_device_id REPLICA IDENTITY USING INDEX event_txn_id_device_id_event_id;

View File

@ -0,0 +1 @@
ALTER TABLE events REPLICA IDENTITY USING INDEX events_event_id_key;

View File

@ -0,0 +1 @@
ALTER TABLE federation_inbound_events_staging REPLICA IDENTITY USING INDEX federation_inbound_events_staging_instance_event;

View File

@ -0,0 +1 @@
ALTER TABLE federation_stream_position REPLICA IDENTITY USING INDEX federation_stream_position_instance;

View File

@ -0,0 +1 @@
ALTER TABLE ignored_users REPLICA IDENTITY USING INDEX ignored_users_uniqueness;

View File

@ -0,0 +1 @@
ALTER TABLE insertion_event_edges REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE insertion_event_extremities REPLICA IDENTITY USING INDEX insertion_event_extremities_event_id;

View File

@ -0,0 +1 @@
ALTER TABLE insertion_events REPLICA IDENTITY USING INDEX insertion_events_event_id;

View File

@ -0,0 +1 @@
ALTER TABLE local_current_membership REPLICA IDENTITY USING INDEX local_current_membership_idx;

View File

@ -0,0 +1 @@
ALTER TABLE local_media_repository REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE local_media_repository_thumbnails REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE local_media_repository_url_cache REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE monthly_active_users REPLICA IDENTITY USING INDEX monthly_active_users_users;

View File

@ -0,0 +1 @@
ALTER TABLE partial_state_events REPLICA IDENTITY USING INDEX partial_state_events_event_id_key;

View File

@ -0,0 +1 @@
ALTER TABLE partial_state_rooms_servers REPLICA IDENTITY USING INDEX partial_state_rooms_servers_room_id_server_name_key;

View File

@ -0,0 +1 @@
ALTER TABLE presence_stream REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE profiles REPLICA IDENTITY USING INDEX profiles_user_id_key;

View File

@ -0,0 +1 @@
ALTER TABLE push_rules_stream REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE ratelimit_override REPLICA IDENTITY USING INDEX ratelimit_override_idx;

View File

@ -0,0 +1 @@
ALTER TABLE receipts_graph REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE receipts_linearized REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE received_transactions REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE redactions REPLICA IDENTITY USING INDEX redactions_event_id_key;

View File

@ -0,0 +1 @@
ALTER TABLE registration_tokens REPLICA IDENTITY USING INDEX registration_tokens_token_key;

View File

@ -0,0 +1 @@
ALTER TABLE rejections REPLICA IDENTITY USING INDEX rejections_event_id_key;

View File

@ -0,0 +1 @@
ALTER TABLE remote_media_cache REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE remote_media_cache_thumbnails REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE room_account_data REPLICA IDENTITY USING INDEX room_account_data_uniqueness;

View File

@ -0,0 +1 @@
ALTER TABLE room_alias_servers REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE room_aliases REPLICA IDENTITY USING INDEX room_aliases_room_alias_key;

View File

@ -0,0 +1 @@
ALTER TABLE room_depth REPLICA IDENTITY USING INDEX room_depth_room_id_key;

View File

@ -0,0 +1 @@
ALTER TABLE room_forgetter_stream_pos REPLICA IDENTITY USING INDEX room_forgetter_stream_pos_lock_key;

View File

@ -0,0 +1 @@
ALTER TABLE room_memberships REPLICA IDENTITY USING INDEX room_memberships_event_id_key;

View File

@ -0,0 +1 @@
ALTER TABLE room_stats_earliest_token REPLICA IDENTITY USING INDEX room_stats_earliest_token_idx;

View File

@ -0,0 +1 @@
ALTER TABLE room_stats_state REPLICA IDENTITY USING INDEX room_stats_state_room;

View File

@ -0,0 +1 @@
ALTER TABLE room_tags REPLICA IDENTITY USING INDEX room_tag_uniqueness;

View File

@ -0,0 +1 @@
ALTER TABLE room_tags_revisions REPLICA IDENTITY USING INDEX room_tag_revisions_uniqueness;

View File

@ -0,0 +1 @@
ALTER TABLE server_keys_json REPLICA IDENTITY USING INDEX server_keys_json_uniqueness;

View File

@ -0,0 +1 @@
ALTER TABLE server_signature_keys REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE sessions REPLICA IDENTITY USING INDEX sessions_session_type_session_id_key;

View File

@ -0,0 +1 @@
ALTER TABLE state_events REPLICA IDENTITY USING INDEX state_events_event_id_key;

View File

@ -0,0 +1 @@
ALTER TABLE stats_incremental_position REPLICA IDENTITY USING INDEX stats_incremental_position_lock_key;

View File

@ -0,0 +1 @@
ALTER TABLE stream_ordering_to_exterm REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE stream_positions REPLICA IDENTITY USING INDEX stream_positions_idx;

View File

@ -0,0 +1 @@
ALTER TABLE threads REPLICA IDENTITY USING INDEX threads_uniqueness;

View File

@ -0,0 +1 @@
ALTER TABLE threepid_guest_access_tokens REPLICA IDENTITY FULL;

View File

@ -0,0 +1 @@
ALTER TABLE timeline_gaps REPLICA IDENTITY FULL;

Some files were not shown because too many files have changed in this diff Show More