16d78be315
make use of _simple_select_one_onecol, improved comments
2018-08-06 17:51:15 +01:00
165e067033
Revert "change monthly_active_users table to be a single column"
...
This reverts commit ec716a35b2
2018-08-02 10:59:58 +01:00
ec716a35b2
change monthly_active_users table to be a single column
2018-08-01 17:54:37 +01:00
f9f5559971
fix comment
2018-08-01 12:03:42 +01:00
6ef983ce5c
api into monthly_active_users table
2018-07-31 16:36:24 +01:00
df2235e7fa
coding style
2018-07-31 13:16:20 +01:00
51d7df1915
Create the column nullable
...
There's no real point in ever making the column non-nullable, and doing so
breaks the sytests.
2018-07-26 14:54:04 +01:00
5c1d301fd9
Stop populating events.content
...
This field is no longer read from, so we should stop populating it. Once we're
happy that this doesn't break everything, and a rollback is unlikely, we can
think about dropping the column.
2018-07-26 14:43:02 +01:00
1b4d73fa52
comment on event_edges
2018-07-26 12:53:51 +01:00
49af402019
run isort
2018-07-09 16:09:20 +10:00
244484bf3c
Revert "Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility""
...
This reverts commit 1d009013b3
2018-06-25 13:42:55 +01:00
1d009013b3
Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility"
...
This reverts commit ce0d911156b4a5d767a9
2018-06-22 16:35:10 +01:00
3ff8a619f5
UserErasureStore
...
to store which users have been erased
2018-06-12 09:52:22 +01:00
4986b084f8
remove unnecessary INSERT
2018-06-01 10:50:40 +01:00
ab0ef31dc7
create users index on creation_ts
2018-05-29 17:51:08 +01:00
9ea219c514
Send users a server notice about consent
...
When a user first syncs, we will send them a server notice asking them to
consent to the privacy policy if they have not already done so.
2018-05-22 11:54:51 +01:00
dc8930ea9e
Merge pull request #3163 from matrix-org/cohort_analytics
...
user visit data
2018-05-16 10:09:24 +00:00
8030a825c8
Merge pull request #3213 from matrix-org/rav/consent_handler
...
ConsentResource to gather policy consent from users
2018-05-16 07:19:18 +01:00
47815edcfa
ConsentResource to gather policy consent from users
...
Hopefully there are enough comments and docs in this that it makes sense on its
own.
2018-05-15 15:11:59 +01:00
977765bde2
Merge branch 'develop' of https://github.com/matrix-org/synapse into cohort_analytics
2018-05-14 09:31:42 +01:00
46df23f581
Add the schema file
2018-05-09 15:07:54 +01:00
d0857702e8
add inidexes based on usage
2018-05-01 12:12:57 +01:00
99a54bf2af
Merge pull request #3129 from matrix-org/matthew/fix_group_dups
...
remove duplicates from groups tables
2018-04-30 11:47:25 +01:00
adaf3ec87f
fix missing import
2018-04-28 22:39:15 +01:00
006e18b6bb
pep8
2018-04-28 22:32:24 +01:00
42c89c8215
make it work with sqlite
2018-04-28 22:27:30 +01:00
d82b6ea9e6
Move more xrange to six
...
plus a bonus next()
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:57:00 +02:00
617bf40924
Generate user daily stats
2018-04-25 17:37:29 +01:00
e3a373f002
remove duplicates from groups tables
...
and rename inconsistently named indexes.
Based on https://github.com/matrix-org/synapse/pull/3128 - thanks @vurpo\!
2018-04-25 14:58:43 +01:00
e089100c62
Merge pull request #3045 from matrix-org/dbkr/group_joinable
...
Add joinability for groups
2018-04-05 15:57:49 +01:00
68b0ee4e8d
Merge pull request #3041 from matrix-org/r30_stats
...
R30 stats
2018-04-05 15:37:37 +01:00
917380e89d
NON NULL -> NOT NULL
2018-04-05 14:32:12 +01:00
700e5e7198
Use DEFAULT join_policy of "invite" in db
2018-04-05 14:01:17 +01:00
eb8d8d6f57
Use join_policy API instead of joinable
...
The API is now under
/groups/$group_id/setting/m.join_policy
and expects a JSON blob of the shape
```json
{
"m.join_policy": {
"type": "invite"
}
}
```
where "invite" could alternatively be "open".
2018-04-03 16:16:40 +01:00
fcfe7f6ad3
Use simplejson throughout
...
Let's use simplejson rather than json, for consistency.
2018-03-29 22:45:52 +01:00
929b34963d
OK, smallint it is then
2018-03-28 14:53:55 +01:00
a838444a70
Grr. Copy the definition from is_admin
2018-03-28 14:50:30 +01:00
a164270833
Make column definition that works on both dbs
2018-03-28 14:23:00 +01:00
352e1ff9ed
Add schema delta file
2018-03-28 14:07:57 +01:00
788e69098c
Add user_ips last seen index
2018-03-28 12:03:13 +01:00
fa72803490
Merge branch 'master' of github.com:matrix-org/synapse into develop
2018-03-19 11:41:01 +00:00
926ba76e23
Replace ujson with simplejson
2018-03-15 23:43:31 +00:00
012e8e142a
Comments
2018-02-16 11:35:01 +00:00
acac21248c
Store push actions in staging area
2018-02-15 15:47:04 +00:00
ddb6a79b68
Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin
2018-02-13 16:45:36 +00:00
0b27ae8dc3
move search reindex to schema 47
...
We're up to schema v47 on develop now, so this will have to go in there to have
an effect.
This might cause an error if somebody has already run it in the v46 guise, and
runs it again in the v47 guise, because it will cause a duplicate entry in the
bbackground_updates table. On the other hand, the entry is removed once it is
complete, and it is unlikely that anyone other than matrix.org has run it on
v46. The update itself is harmless to re-run because it deliberately copes with
the index already existing.
2018-02-13 16:44:46 +00:00
3d33eef6fc
Store state groups separately from events ( #2784 )
...
* Split state group persist into seperate storage func
* Add per database engine code for state group id gen
* Move store_state_group to StateReadStore
This allows other workers to use it, and so resolve state.
* Hook up store_state_group
* Fix tests
* Rename _store_mult_state_groups_txn
* Rename StateGroupReadStore
* Remove redundant _have_persisted_state_group_txn
* Update comments
* Comment compute_event_context
* Set start val for state_group_id_seq
... otherwise we try to recreate old state groups
* Update comments
* Don't store state for outliers
* Update comment
* Update docstring as state groups are ints
2018-02-06 14:31:24 +00:00
2fb3a28c98
Remove lost comment
2018-01-17 14:59:44 +00:00
05f98a2224
Keep track of last access time for local media
2018-01-17 10:24:43 +00:00
a66f489678
fix GIST->GIN switch
2018-01-09 16:55:51 +00:00
f397153dfc
Merge branch 'develop' into matthew/search-all-local-users
2017-11-30 01:51:38 +00:00
f61e107f63
remove null constraint on user_dir.room_id
2017-11-30 01:43:50 +00:00
7098b65cb8
Fix error on sqlite 3.7
...
Create the url_cache index on local_media_repository as a background update, so
that we can detect whether we are on sqlite or not and create a partial or
complete index accordingly.
To avoid running the cleanup job before we have built the index, add a bailout
which will defer the cleanup if the bg updates are still running.
Fixes https://github.com/matrix-org/synapse/issues/2572 .
2017-11-21 11:14:17 +00:00
06e5bcfc83
Avoid locking for upsert on pushers tables
...
* replace the upsert into deleted_pushers with an insert
* no need to lock for upsert on pusher_throttle
2017-11-16 17:52:23 +00:00
631fa4a1b7
create new indexes before dropping old ones to keep safetynet in place
2017-11-07 10:41:55 +00:00
4ad883398f
s/users_in_pubic_room/users_in_public_rooms/g
2017-11-04 19:39:40 +00:00
d802e8ca6a
s/users_in_pubic_room/users_in_public_rooms/g
2017-11-04 19:38:13 +00:00
a100700630
fix copyright....
2017-11-04 19:35:49 +00:00
b6b075fd49
s/popualte/populate/
2017-11-04 19:35:33 +00:00
356bcafc44
Remove the last vestiges of refresh_tokens
2017-10-31 20:35:58 +00:00
1650eb5847
DB schema interface for password auth providers
...
Provide an interface by which password auth providers can register db schema
files to be run at startup
2017-10-31 14:01:53 +00:00
007cd48af6
Recreate groups table instead of adding column
...
Adding a column with non-constant default not possible in sqlite3
2017-10-26 17:55:22 +01:00
713e60b9b6
Awful hack to get default true
2017-10-26 17:38:14 +01:00
9b2feef9eb
Add is_public to groups table to allow for private groups
...
Prevent group API access to non-members for private groups
Also make all the group code paths consistent with `requester_user_id` always being the User ID of the requesting user.
2017-10-26 16:51:32 +01:00
eaaabc6c4f
replace 'except:' with 'except Exception:'
...
what could possibly go wrong
2017-10-23 15:52:32 +01:00
c2c47550f9
Fix schema delta versions
2017-10-11 13:23:15 +01:00
27955056e0
Merge branch 'develop' into erikj/groups_merged
2017-10-02 16:20:41 +01:00
4dc07e93a8
Add old indices
2017-09-28 14:10:33 +01:00
ae79764fe5
Change expires column to expires_ts
2017-09-28 12:37:53 +01:00
77f1d24de3
More brackets
2017-09-28 12:23:15 +01:00
9ccb4226ba
Delete expired url cache data
2017-09-28 12:18:06 +01:00
a8e2a3df32
Add unique index to group_rooms table
2017-09-26 15:39:21 +01:00
27ebc5c8f2
Add remote profile cache
2017-08-25 11:25:47 +01:00
8b50fe5330
Use BOOLEAN rather than TEXT type
2017-08-21 16:37:29 +01:00
05e21285aa
Store whether the user wants to publicise their membership of a group
2017-08-08 13:01:46 +01:00
c544188ee3
Add groups to sync stream
2017-07-20 16:36:42 +01:00
508460f240
Remove sync stuff
2017-07-18 09:55:46 +01:00
4540730111
Remove unused tables
2017-07-18 09:38:15 +01:00
2f9eafdd36
Add local group server support
2017-07-17 12:03:49 +01:00
26451a09eb
Comments
2017-07-12 14:47:18 +01:00
a62406aaa5
Add group summary APIs
2017-07-12 11:36:15 +01:00
6d586dc05c
Comment
2017-07-12 09:58:37 +01:00
410b4e14a1
Move comment
2017-07-11 15:44:18 +01:00
e52c391cd4
Rename column to attestation_json
2017-07-11 14:25:46 +01:00
0aac30d53b
Comments
2017-07-11 14:23:50 +01:00
83936293eb
Comments
2017-07-11 11:42:25 +01:00
b8ca494ee9
Initial group server implementation
2017-07-10 15:44:15 +01:00
8abdd7b553
Fix up indices for users_who_share_rooms
2017-06-26 14:01:30 +01:00
7fe8ed1787
Store URL cache preview downloads seperately
...
This makes it easier to clear old media out at a later date
2017-06-23 11:14:11 +01:00
b8b936a6ea
Add API to quarantine media
2017-06-19 17:39:21 +01:00
d7fe6b356c
Add shutdown room API
2017-06-19 12:37:27 +01:00
ebcd55d641
Add DB schema for tracking users who share rooms
2017-06-15 09:45:48 +01:00
65f0513a33
Split up device_lists_outbound_pokes table for faster updates.
2017-06-07 11:02:38 +01:00
21e255a8f1
Split the table in two
2017-06-01 14:50:46 +01:00
5dd1b2c525
Use unique indices
2017-05-31 17:29:12 +01:00
5d79d728f5
Split out directory and search tables
2017-05-31 15:23:49 +01:00
350622a107
Handle the server leaving a public room
2017-05-31 15:11:36 +01:00
63fda37e20
Add comments
2017-05-31 15:00:29 +01:00
eeb2f9e546
Add user_directory to database
2017-05-31 11:51:01 +01:00
dd48f7204c
Add comment
2017-05-30 15:01:22 +01:00
04095f7581
Add clobbered event_id
2017-05-30 14:53:01 +01:00
a584a81b3e
Add current_state_delta_stream table
2017-05-30 14:44:09 +01:00
6e614e9e10
Add background task to clear out old event_auth
2017-05-24 15:23:34 +01:00
ff3d810ea8
Add a comment to old delta
2017-05-11 12:48:50 +01:00
baafb85ba4
Add an index to event_search
...
- to make the purge API quicker
2017-05-11 12:05:22 +01:00
b990b2fce5
Add per user ratelimiting overrides
2017-05-10 11:05:43 +01:00
d766343668
Add index to device_lists_stream
2017-03-01 15:56:30 +00:00
e4919b9329
Add stream_id index to device_lists_outbound_pokes
...
As this is used for replication streaming
2017-02-28 11:19:06 +00:00
4aa29508af
Use TEXT rather than VARCHAR
...
While we're changing anyway
2017-02-17 10:51:49 +00:00
b4017539d4
Make the pushers lang field column longer
...
To accommodate things like zh-Hans-CN
Fixes https://github.com/vector-im/riot-ios/issues/1031
2017-02-17 10:42:57 +00:00
095b45c165
Aggregate event push actions
2017-02-14 13:39:41 +00:00
21b7375778
Add an index to make membership queries faster
2017-01-31 15:15:57 +00:00
d1e1fd6210
Add ts column to device_lists_outbound_pokes
2017-01-27 15:23:48 +00:00
fbfad76c03
Add comments
2017-01-26 16:33:21 +00:00
c974116f19
Implement device key caching over federation
2017-01-26 16:07:24 +00:00
2367c5568c
Add basic implementation of local device list changes
2017-01-25 14:27:27 +00:00
c050f493dd
Add comment
2017-01-13 15:14:41 +00:00
caddadfc5a
Change device_inbox stream index to include user
...
This makes fetching the nost recently changed users much tricker, and
brings it in line with e.g. presence_stream indices.
2017-01-10 15:04:57 +00:00
1574b839e0
Merge pull request #1676 from matrix-org/erikj/room_list
...
Add new API appservice specific public room list
2016-12-12 17:00:10 +00:00
c45d8e9ba2
Add profile data to the room_membership table for joins
2016-12-08 13:08:41 +00:00
2687af82d4
Comments
2016-12-07 09:58:33 +00:00
f32fb65552
Add new API appservice specific public room list
2016-12-06 16:12:27 +00:00
46ca345b06
Don't send old events as federation
2016-11-24 12:29:02 +00:00
2ac516850b
More efficient notif count queries
2016-11-23 15:57:04 +00:00
7c9cdb2245
Store federation stream positions in the database
2016-11-21 11:33:08 +00:00
f8ee66250a
Handle sending events and device messages over federation
2016-11-17 15:48:04 +00:00
8fd4d9129f
Replace postgres GIN with GIST
...
This is because GIN can be slow to write too, especially when the table
gets large.
2016-11-03 15:00:03 +00:00
a9111786f9
Use most recently added binding, not most recently seen user.
2016-10-27 14:32:45 +01:00
1fc1bc2a51
Fix user_threepids schema delta
...
The delta `37/user_threepids.sql` aimed to update all the email
addresses to be lower case, however duplicate emails may exist in the
table already.
This commit adds a step where the delta moves the duplicate emails to a
new `medium` `email_old`. Only the most recently used account keeps the
binding intact. We move rather than delete so that we retain some record
of which emails were associated with which account.
2016-10-27 14:14:44 +01:00
0108ed8ae6
Latest delta is now 37
2016-10-19 11:40:35 +01:00
a7f48320b1
Merge remote-tracking branch 'origin/develop' into dbkr/password_reset_case_insensitive
2016-10-19 11:28:56 +01:00
df2a616c7b
Convert emails to lowercase when storing
...
And db migration sql to convert existing addresses.
2016-10-19 11:13:55 +01:00
6942d68247
Bump schema version
2016-10-17 11:17:45 +01:00
b59994b454
Remove TODO
2016-10-17 11:17:02 +01:00
816988baaa
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_auth
2016-10-17 11:10:37 +01:00
2869a29fd7
Drop some unused indices
2016-10-17 11:08:19 +01:00
62073992c5
Make password reset email field case insensitive
2016-10-14 13:56:53 +01:00
8009d84364
Match against event_id, rather than room_id
2016-09-21 16:46:59 +01:00
dc78db8c56
Update correct table
2016-09-21 15:52:44 +01:00
4f78108d8c
Readd entries to public_room_list_stream that were deleted
2016-09-21 15:24:22 +01:00
baffe96d95
Add a room visibility stream
2016-09-14 17:29:19 +01:00
ed992ae6ba
Add a DB index to figure out past state at a stream ordering in a room
2016-09-14 16:20:27 +01:00
43253c10b8
Remove redundant event_auth index
2016-09-13 11:47:48 +01:00
0b32bb20bb
Index contains_url for file search queries
2016-09-12 16:57:05 +01:00
7fe42cf949
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/make_notif_highlight_query_fast
2016-09-12 12:37:09 +01:00
15ca0c6a4d
Make reindex happen in bg
2016-09-12 12:36:36 +01:00
31f85f9db9
Add comments to existing schema deltas that used "CREATE INDEX" directly
2016-09-12 11:00:26 +01:00
54417999b6
Revert "Add index to event_push_actions"
2016-09-12 10:39:55 +01:00
b91e2833b3
Merge remote-tracking branch 'origin/develop' into dbkr/make_notif_highlight_query_fast
2016-09-09 19:11:34 +01:00
f2acc3dcf9
Add index to event_push_actions
...
and remove room_id caluse so it uses it
Mostly from @negativemjark
2016-09-09 18:54:54 +01:00
647c724573
Use the previous MAX value if any to set the stream_id
2016-09-09 11:52:44 +01:00
6a6cbfcf1e
Track the max_stream_device_id in a separate table, since we delete from the inbox table
2016-09-09 11:48:23 +01:00
303b6f29f0
Merge pull request #1087 from matrix-org/markjh/reapply_delta
...
Reapply 34/device_outbox in 35/device_outbox_again.py since the schem…
2016-09-09 11:24:27 +01:00
0877157353
Just move the schema and add some DROPs
2016-09-09 11:04:47 +01:00
43b77c5d97
Only catch databas errors
2016-09-08 17:44:21 +01:00
7d5b142547
Add a stub run_upgrade
2016-09-08 17:39:11 +01:00
fa722a699c
Reapply 34/device_outbox in 35/device_outbox_again.py since the schema was bumped before it landed on develop
2016-09-08 17:35:16 +01:00
ebb46497ba
Add delta file
2016-09-08 16:38:54 +01:00
fa9d36e050
Merge branch 'develop' into markjh/direct_to_device_federation
2016-09-08 13:43:43 +01:00
d4a35ada28
Send device messages over federation
2016-09-06 18:16:20 +01:00
2ad72da931
Add tables for federated device messages
...
Adds tables for storing the messages that need to be sent to a
remote device and for deduplicating messages received.
2016-09-06 15:10:40 +01:00
485d999c8a
Correctly delete old state groups in purge history API
2016-09-05 14:49:08 +01:00
a99e933550
Add upgrade script that will slowly prune state_groups_state entries
2016-09-05 10:05:36 +01:00
657847e4c6
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage
2016-09-02 11:04:48 +01:00
9e25443db8
Move to storing state_groups_state as deltas
2016-09-01 14:31:26 +01:00
ef0cc648cf
Clean up old sent transactions
2016-08-31 11:12:02 +01:00
e993925279
Add store-and-forward direct-to-device messaging
2016-08-25 17:35:37 +01:00
aad8a1a825
Delete old received_transactions
2016-08-22 16:29:46 +01:00
9da84a9a1e
Make AppserviceHandler stream events from database
...
This is for two reasons:
1. Suppresses duplicates correctly, as the notifier doesn't do any
duplicate suppression.
2. Makes it easier to connect the AppserviceHandler to the replication
stream.
2016-08-18 11:54:41 +01:00
732cf72b86
Fix push_display_name_rename schema update
2016-08-17 18:12:21 +01:00
ad42322257
Add migration script
...
To port existing rule actions & enable entries to the new name
2016-08-16 16:56:30 +01:00
d9664344ec
Rename table. Add docs.
2016-08-15 11:45:57 +01:00
4d70d1f80e
Add some invalidations to a cache_stream
2016-08-15 11:15:17 +01:00
4fec5e57be
Default device_display_name to null
...
It turns out that it's more useful to return a null device display name (and
let clients decide how to handle it: eg, falling back to device_id) than using
a constant string like "unknown device".
2016-08-03 11:53:00 +01:00
26cb0efa88
SQL syntax fix
2016-07-27 12:30:22 +01:00
2e3d90d67c
Make the device id on e2e key upload optional
...
We should now be able to get our device_id from the access_token, so the
device_id on the upload request is optional. Where it is supplied, we should
check that it matches.
For active access_tokens without an associated device_id, we ought to register
the device in the devices table.
Also update the table on upgrade so that all of the existing e2e keys are
associated with real devices.
2016-07-26 23:38:12 +01:00
436bffd15f
Implement deleting devices
2016-07-26 07:35:48 +01:00
ec5717caf5
Create index on user_ips in the background
...
user_ips is kinda big, so really we want to add the index in the background
once we're running. Replace the schema delta with one which will do that.
I've done this in a way that's reasonably easy to reuse as there a few other
indexes I need, and I don't suppose they will be the last.
2016-07-22 13:16:39 +01:00
bc8f265f0a
GET /devices endpoint
...
implement a GET /devices endpoint which lists all of the user's devices.
It also returns the last IP where we saw that device, so there is some dancing
to fish that out of the user_ips table.
2016-07-20 16:42:32 +01:00
1e2a7f18a1
Merge pull request #922 from matrix-org/erikj/file_api2
...
Feature: Add filter to /messages. Add 'contains_url' to filter.
2016-07-20 10:40:48 +01:00
022b9176fe
schema fix
...
device_id should be text, not bigint.
2016-07-19 11:44:05 +01:00
f863a52cea
Add device_id support to /login
...
Add a 'devices' table to the storage, as well as a 'device_id' column to
refresh_tokens.
Allow the client to pass a device_id, and initial_device_display_name, to
/login. If login is successful, then register the device in the devices table
if it wasn't known already. If no device_id was supplied, make one up.
Associate the device_id with the access token and refresh token, so that we can
get at it again later. Ensure that the device_id is copied from the refresh
token to the access_token when the token is refreshed.
2016-07-18 16:39:44 +01:00
b64aa6d687
Add sender and contains_url field to events table
2016-07-14 15:35:43 +01:00
314b146b2e
Track approximate last access time for remote media
2016-06-29 11:41:20 +01:00
c8c5bf950a
Fix synapse/storage/schema/delta/30/as_users.py
2016-05-31 17:10:40 +01:00
c9aff0736c
Remove topics table
2016-05-13 14:40:38 +01:00
9295fa30a8
Annotate the removed indicies with why they were removed.
2016-05-13 14:16:57 +01:00
5e50058473
Remove unused indices
...
This includes removing both unused indices and indices that are subsets
of other indices.
2016-05-13 13:28:07 +01:00
3b1930e8ec
unbreak schema
2016-05-10 16:42:37 +01:00
997db04648
Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs
2016-05-10 14:40:19 +02:00
9c272da05f
Add an openidish mechanism for proving to third parties that you own a given user_id
2016-05-05 13:42:44 +01:00
fcd1eb642d
Add primary key
2016-05-04 16:51:51 +01:00
8e6a163f27
Add timestamp and auto incrementing ID
2016-05-04 15:19:12 +01:00
984d4a2c0f
Add /report endpoint
2016-05-04 11:28:10 +01:00
acded821c4
Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs
2016-04-29 10:05:20 +01:00
ae571810f2
Order NULLs first
2016-04-21 18:14:18 +01:00
8fae3d7b1e
Use special UPDATE syntax
2016-04-21 18:01:49 +01:00
51bb339ab2
Create index concurrently
2016-04-21 17:16:11 +01:00
b743c1237e
Add missing run_upgrade
2016-04-21 17:12:04 +01:00
c877f0f034
Optimise event_search in postgres
2016-04-21 16:56:14 +01:00
Neil Johnson