Commit Graph

11137 Commits

Author SHA1 Message Date
Richard van der Hoff 5570d1c93f
Merge pull request #6334 from matrix-org/rav/url_preview_limit_title_2
Fix exception when OpenGraph tag values are ints
2019-11-05 17:28:11 +00:00
Richard van der Hoff 81d49cbb07 Fix exception when OpenGraph tag values are ints 2019-11-05 17:22:58 +00:00
Richard van der Hoff 55a7da247a
Merge branch 'develop' into rav/url_preview_limit_title 2019-11-05 17:08:07 +00:00
Richard van der Hoff e78167c94b
Apply suggestions from code review
Co-Authored-By: Brendan Abolivier <babolivier@matrix.org>
Co-Authored-By: Erik Johnston <erik@matrix.org>
2019-11-05 16:46:39 +00:00
Richard van der Hoff e9bfe719ba Strip overlong OpenGraph data from url preview
... to stop people causing DoSes with malicious web pages
2019-11-05 15:51:18 +00:00
Erik Johnston c16e192e2f Fix caching devices for remote servers in worker.
When the `/keys/query` API is hit on client_reader worker Synapse may
decide that it needs to resync some remote deivces. Usually this happens
on master, and then gets cached. However, that fails on workers and so
it falls back to fetching devices from remotes directly, which may in
turn fail if the remote is down.
2019-11-05 15:49:43 +00:00
Brendan Abolivier cb2cbe4d26
Only filter if a filter was provided 2019-11-05 15:28:36 +00:00
Brendan Abolivier f141af4c79
Update copyright 2019-11-05 14:52:38 +00:00
Brendan Abolivier 8822b33111
Update copyrights 2019-11-05 14:46:35 +00:00
Richard van der Hoff 4086002827
Improve documentation for EventContext fields (#6319) 2019-11-05 13:23:25 +00:00
Brendan Abolivier 1dffa78701
Filter events_before and events_after in /context requests
While the current version of the spec doesn't say much about how this endpoint uses filters (see https://github.com/matrix-org/matrix-doc/issues/2338), the current implementation is that some fields of an EventFilter apply (the ones that are used when running the SQL query) and others don't (the ones that are used by the filter itself) because we don't call event_filter.filter(...). This seems counter-intuitive and probably not what we want so this commit fixes it.
2019-11-05 12:21:59 +00:00
Erik Johnston ffe595381d Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history 2019-11-05 10:27:41 +00:00
Andrew Morgan 506a63de67 Merge branch 'develop' of github.com:matrix-org/synapse into anoa/room_upgrade_groups 2019-11-04 18:22:41 +00:00
Andrew Morgan c2203bea57 Re-add docstring, with caveats detailed 2019-11-04 18:17:11 +00:00
Brendan Abolivier e252ffadbc Merge branch 'develop' into babolivier/msc2326_bg_update 2019-11-04 18:09:50 +00:00
Andrew Morgan 0287d033ee Transfer upgraded rooms on groups 2019-11-04 18:08:50 +00:00
Amber Brown 4e1c7b79fa
Remove the psutil dependency (#6318)
* remove psutil and replace with resource
2019-11-05 05:05:48 +11:00
Brendan Abolivier 09957ce0e4
Implement per-room message retention policies 2019-11-04 17:09:22 +00:00
Erik Johnston 7134ca7daa Change to not require a state_groups.room_id index.
This does mean that we won't clean up orphaned state groups (i.e. state
groups that were persisted but the associated event wasn't).
2019-11-04 13:36:57 +00:00
Erik Johnston 6a0092d371 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history 2019-11-04 13:29:35 +00:00
Richard van der Hoff cc6243b4c0
document the REPLICATE command a bit better (#6305)
since I found myself wonder how it works
2019-11-04 12:40:18 +00:00
Brendan Abolivier 3b29a73f9f
Print out the actual number of affected rows 2019-11-04 09:56:11 +00:00
Brendan Abolivier 824bba2f78
Correctly order results 2019-11-04 09:56:11 +00:00
Brendan Abolivier 49008e674f
TODO 2019-11-04 09:56:11 +00:00
Brendan Abolivier 1586f2c7e7
Fix exit condition 2019-11-04 09:56:11 +00:00
Brendan Abolivier 1c1268245d
Lint 2019-11-04 09:56:11 +00:00
Brendan Abolivier 911b03ca31
Don't try to process events we already have a label for 2019-11-04 09:56:10 +00:00
Brendan Abolivier 07cb38e965
Use a sensible default value for labels 2019-11-04 09:56:10 +00:00
Brendan Abolivier a46574281d
Use the right format for rows 2019-11-04 09:56:10 +00:00
Brendan Abolivier c9a1b80a74
MSC2326: Add background update to take previous events into account 2019-11-04 09:56:04 +00:00
Brendan Abolivier f496d25877
Merge pull request #6301 from matrix-org/babolivier/msc2326
Implement MSC2326 (label based filtering)
2019-11-01 17:04:45 +00:00
Brendan Abolivier 988d8d6507
Incorporate review 2019-11-01 16:22:44 +00:00
Richard van der Hoff c6516adbe0
Factor out an _AsyncEventContextImpl (#6298)
The intention here is to make it clearer which fields we can expect to be
populated when: notably, that the _event_type etc aren't used for the
synchronous impl of EventContext.
2019-11-01 16:19:09 +00:00
Brendan Abolivier 5598445655
Update synapse/storage/data_stores/main/schema/delta/56/event_labels.sql
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-11-01 16:18:34 +00:00
Hubert Chathi fa7e52caf1
Merge pull request #6313 from matrix-org/uhoreg/cross_signing_fix_sqlite_schema
fix hidden field in devices table for older sqlite
2019-11-01 10:52:46 -04:00
Richard van der Hoff 1cb84c6486
Support for routing outbound HTTP requests via a proxy (#6239)
The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy.

The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`.

The proxy will then be used for
 * push
 * url previews
 * phone-home stats
 * recaptcha validation
 * CAS auth validation

It will *not* be used for:
 * Application Services
 * Identity servers
 * Outbound federation
 * In worker configurations, connections from workers to masters

Fixes #4198.
2019-11-01 14:07:44 +00:00
Andrew Morgan fe1f2b4520
Remove last usages of deprecated logging.warn method (#6314) 2019-11-01 12:03:44 +00:00
Brendan Abolivier a2c63c619a
Add more data to the event_labels table and fix the indexes 2019-11-01 11:47:28 +00:00
Erik Johnston 669b6cbda3 Fix up comment 2019-11-01 11:32:20 +00:00
Brendan Abolivier e3689ac6f7
Add unstable feature flag 2019-11-01 10:41:23 +00:00
Brendan Abolivier 57cdb046e4
Lint 2019-11-01 10:39:14 +00:00
Brendan Abolivier c6dbca2422
Incorporate review 2019-11-01 10:30:51 +00:00
Andrew Morgan ace947e8da
Depublish a room from the public rooms list when it is upgraded (#6232) 2019-11-01 10:28:09 +00:00
Hubert Chathi c61db13183 fix hidden field in devices table for older sqlite 2019-10-31 22:52:55 -04:00
Hubert Chathi c3fc176c60
Update synapse/storage/data_stores/main/devices.py
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-10-31 22:49:48 -04:00
Hubert Chathi 6f4bc6d01d Merge branch 'develop' into cross-signing_federation 2019-10-31 22:38:21 -04:00
Hubert Chathi 3b4216f961
Merge pull request #6254 from matrix-org/uhoreg/cross_signing_fix_workers_notify
make notification of signatures work with workers
2019-10-31 22:35:03 -04:00
Will Hunt 42e707c663 rstrip slashes from url on appservice (#6306) 2019-10-31 17:32:25 +00:00
Hubert Chathi 9c94b48bf1 Merge branch 'develop' into uhoreg/cross_signing_fix_workers_notify 2019-10-31 12:32:07 -04:00
Hubert Chathi f7e4a582ef clean up code a bit 2019-10-31 12:01:00 -04:00
Erik Johnston fb1a6914cf Update log line to lie a little less 2019-10-31 15:45:48 +00:00
Amber Brown 020add5099
Update black to 19.10b0 (#6304)
* update version of black and also fix the mypy config being overridden
2019-11-01 02:43:24 +11:00
Erik Johnston 61be1a2926 Add state_groups.room_id index 2019-10-31 15:39:26 +00:00
Erik Johnston f91f2a1f92 Docstrings 2019-10-31 15:26:00 +00:00
Erik Johnston 8f5bbdb987 Fix purge room API 2019-10-31 15:22:08 +00:00
Erik Johnston cd581338cf Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history 2019-10-31 15:19:26 +00:00
Erik Johnston dfe0cd71b6
Merge pull request #6294 from matrix-org/erikj/add_state_storage
Add StateGroupStorage interface
2019-10-31 16:17:53 +01:00
Travis Ralston 3a74c03ffb
Expose some homeserver functionality to spam checkers (#6259)
* Offer the homeserver instance to the spam checker

* Newsfile

* Linting

* Expose a Spam Checker API instead of passing the homeserver object

* Alter changelog

* s/hs/api
2019-10-31 09:16:14 -06:00
Erik Johnston 64f2b8c3d8
Apply suggestions from code review
Fix docstring

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-10-31 15:44:31 +01:00
Erik Johnston c6bcd38841 Fix /purge_room API.
It fails trying to clean the `topic` table which was recently removed.
2019-10-31 11:17:23 +00:00
Andrew Morgan 54fef094b3
Remove usage of deprecated logger.warn method from codebase (#6271)
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
2019-10-31 10:23:24 +00:00
Hubert Chathi 998f7fe7d4 make user signatures a separate stream 2019-10-30 17:22:52 -04:00
Hubert Chathi 670972c0e1 Merge branch 'develop' into uhoreg/cross_signing_fix_workers_notify 2019-10-30 16:46:31 -04:00
Hubert Chathi bb6cec27a5 rename get_devices_by_remote to get_device_updates_by_remote 2019-10-30 14:57:34 -04:00
Richard van der Hoff 0467f33584
fix delete_existing for _persist_events (#6300)
this is part of _retry_on_integrity_error, so should only be on _persist_events_and_state_updates
2019-10-30 18:05:00 +00:00
Brendan Abolivier dcc069a2e2
Lint 2019-10-30 18:01:56 +00:00
Brendan Abolivier fe51d6cacf
Add more integration testing 2019-10-30 17:28:41 +00:00
Brendan Abolivier e7943f660a
Add unit tests 2019-10-30 16:15:04 +00:00
Brendan Abolivier 233b14ebe1
Add index on label 2019-10-30 15:58:05 +00:00
Brendan Abolivier acd16ad86a
Implement filtering 2019-10-30 15:56:33 +00:00
Erik Johnston 7c8c97e635 Split purge API into events vs state 2019-10-30 15:23:37 +00:00
Erik Johnston 69f0054ce6 Port to use state storage 2019-10-30 14:46:54 +00:00
Erik Johnston 5db03535d5 Add StateGroupStorage interface 2019-10-30 14:46:49 +00:00
Brendan Abolivier fa0dcbc8fa
Store labels for new events 2019-10-30 14:27:15 +00:00
Hubert Chathi 7d7eac61be Merge branch 'develop' into cross-signing_federation 2019-10-30 10:17:10 -04:00
Hubert Chathi bc32f102cd black 2019-10-30 10:07:36 -04:00
Hubert Chathi d78b1e339d apply changes as a result of PR review 2019-10-30 10:01:53 -04:00
Erik Johnston ec6de1cc7d Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_persistence_store 2019-10-30 13:37:04 +00:00
Erik Johnston a8d16f6c00 Review comments 2019-10-30 13:36:12 +00:00
Erik Johnston e5c3a99091
Merge pull request #6291 from matrix-org/erikj/fix_cache_descriptor
Make ObservableDeferred.observe() always return deferred.
2019-10-30 14:06:34 +01:00
Erik Johnston 6e677403b7 Clarify docstring 2019-10-30 11:52:04 +00:00
Erik Johnston 326b3dace7 Make ObservableDeferred.observe() always return deferred.
This makes it easier to use in an async/await world.

Also fixes a bug where cache descriptors would occaisonally return a raw
value rather than a deferred.
2019-10-30 11:35:46 +00:00
Andrew Morgan a2276d4d3c
Fix log line that was printing undefined value (#6278) 2019-10-30 11:28:48 +00:00
Andrew Morgan 7955abeaac
Fix small typo in comment (#6269) 2019-10-30 11:16:19 +00:00
Andrew Morgan 46c12918ad
Fix typo in domain name in account_threepid_delegates config option (#6273) 2019-10-30 11:07:42 +00:00
Andrew Morgan b39ca49db1
Handle FileNotFound error in checking git repository version (#6284) 2019-10-30 11:00:15 +00:00
Erik Johnston 770d1ef673
Merge pull request #6280 from matrix-org/erikj/receipts_async_await
Port receipt and read markers to async/wait
2019-10-30 11:44:18 +01:00
Erik Johnston ba4cc5541c
Merge pull request #6274 from matrix-org/erikj/replication_async
Port replication http server endpoints to async/await
2019-10-30 11:44:08 +01:00
Erik Johnston 72bc6294ed
Merge pull request #6275 from matrix-org/erikj/port_rest_events
Port room rest handlers to async/await
2019-10-30 11:44:02 +01:00
Brendan Abolivier 47f767269c
Add database table for keeping track of labels on events 2019-10-29 16:56:22 +00:00
Erik Johnston a287f1e804 Don't return coroutines 2019-10-29 16:36:46 +00:00
Erik Johnston 38474707b9 Merge branch 'erikj/federation_server_async_await' of github.com:matrix-org/synapse into erikj/receipts_async_await 2019-10-29 15:53:17 +00:00
Erik Johnston 74c1e16106 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/federation_server_async_await 2019-10-29 15:52:39 +00:00
Erik Johnston 307e313ef4 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/port_rest_events 2019-10-29 15:51:12 +00:00
Erik Johnston d6e40e7cbd Merge branch 'develop' of github.com:matrix-org/synapse into erikj/replication_async 2019-10-29 15:42:58 +00:00
Erik Johnston 2c35ffead2 Port receipt and read markers to async/wait 2019-10-29 15:08:22 +00:00
Erik Johnston 09a135b039 Make concurrently_execute work with async/await 2019-10-29 15:02:23 +00:00
Richard van der Hoff 65cb307e19 Merge branch 'master' into develop 2019-10-29 14:40:57 +00:00
Richard van der Hoff 9ffcf0f7ba 1.5.0 2019-10-29 14:28:54 +00:00
Erik Johnston 3f33879be4 Port federation_server to async/await 2019-10-29 14:13:08 +00:00
Erik Johnston 9be41bc121 Port room rest handlers to async/await 2019-10-29 13:09:29 +00:00
Erik Johnston e577a4b2ad Port replication http server endpoints to async/await 2019-10-29 13:00:51 +00:00
Erik Johnston 561133c3c5
Merge pull request #6263 from matrix-org/erikj/caches_return_deferreds
Quick fix to ensure cache descriptors always return deferreds
2019-10-29 12:53:21 +01:00
Erik Johnston e6c7e239ef Update docstring 2019-10-29 11:48:30 +00:00
Erik Johnston e419c44ba4 Merge branch 'release-v1.5.0' of github.com:matrix-org/synapse into develop 2019-10-29 11:41:27 +00:00
Erik Johnston 342d871d76 Make room directory search case insensitive 2019-10-29 10:24:02 +00:00
Brendan Abolivier 14504ad573
Add CI for synapse_port_db (#6140)
This adds:

* a test sqlite database
* a configuration file for the sqlite database
* a configuration file for a postgresql database (using the credentials in `.buildkite/docker-compose.pyXX.pgXX.yaml`)

as well as a new script named `.buildkite/scripts/test_synapse_port_db.sh` that:

1. installs Synapse
2. updates the test sqlite database to the latest schema and runs background updates on it
3. creates an empty postgresql database
4. run the `synapse_port_db` script to migrate the test sqlite database to the empty postgresql database (with coverage)

Step `2` is done via a new script located at `scripts-dev/update_database`.

The test sqlite database is extracted from a SyTest run, so that it can be considered as an actual homeserver's database with actual data in it.
2019-10-28 17:45:32 +00:00
Erik Johnston d0d8a22c13 Quick fix to ensure cache descriptors always return deferreds 2019-10-28 13:33:04 +00:00
Richard van der Hoff bcfc647e4d Synapse 1.5.0rc2 (2019-10-28)
=============================
 
 Bugfixes
 --------
 
 - Update list of boolean columns in `synapse_port_db`. ([\#6247](https://github.com/matrix-org/synapse/issues/6247))
 - Fix /keys/query API on workers. ([\#6256](https://github.com/matrix-org/synapse/issues/6256))
 - Improve signature checking on some federation APIs. ([\#6262](https://github.com/matrix-org/synapse/issues/6262))
 
 Internal Changes
 ----------------
 
 - Move schema delta files to the correct data store. ([\#6248](https://github.com/matrix-org/synapse/issues/6248))
 - Small performance improvement by removing repeated config lookups in room stats calculation. ([\#6255](https://github.com/matrix-org/synapse/issues/6255))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEv27Axt/F4vrTL/8QOSor00I9eP8FAl225VkACgkQOSor00I9
 eP/7wwf6AkQE1dGYDPpmS29XYocfD3EbjT8FojaHTz1gbkIt5rEF88QHzwRo4P/x
 1S1kQZpEKxKrwmLMDmSbw9STK/76LKGk9k8VycVHfTdy8SICCiL50pVPpSTJ4nXV
 905E7nY0zau3wunWJD4gEjqll967W0AtNKwCHO0D6vNP4pcQNhTkSjhnmypXCFGo
 bvV9r0wq8mtq/FS1GX0Y2CEwbbV1MoPrvlXW37t/gBPk3P1QB/vPjakxO8LgeuZf
 JzAamP5jOYZJpPCpZxwDm6ZNEQ7T4jORdA623wp1gh1nTG88cObCLXJe2jCnpjnJ
 QVb1YlJiwcehXkclIoywgIzj13iIsw==
 =7ctT
 -----END PGP SIGNATURE-----

Merge tag 'v1.5.0rc2' into develop

Synapse 1.5.0rc2 (2019-10-28)
=============================

Bugfixes
--------

- Update list of boolean columns in `synapse_port_db`. ([\#6247](https://github.com/matrix-org/synapse/issues/6247))
- Fix /keys/query API on workers. ([\#6256](https://github.com/matrix-org/synapse/issues/6256))
- Improve signature checking on some federation APIs. ([\#6262](https://github.com/matrix-org/synapse/issues/6262))

Internal Changes
----------------

- Move schema delta files to the correct data store. ([\#6248](https://github.com/matrix-org/synapse/issues/6248))
- Small performance improvement by removing repeated config lookups in room stats calculation. ([\#6255](https://github.com/matrix-org/synapse/issues/6255))
2019-10-28 12:59:13 +00:00
Richard van der Hoff c482d45822 1.5.0rc2 2019-10-28 12:48:18 +00:00
Richard van der Hoff 172f264ed3
Improve signature checking on some federation APIs (#6262)
Make sure that we check that events sent over /send_join, /send_leave, and
/invite, are correctly signed and come from the expected servers.
2019-10-28 12:43:23 +00:00
Richard van der Hoff 9aee28927b Convert EventContext to attrs (#6218)
* make EventContext use an attr
2019-10-28 14:29:55 +02:00
Hubert Chathi da78f61778
Merge pull request #6253 from matrix-org/uhoreg/e2e_backup_delete_keys
delete keys when deleting backup versions
2019-10-25 11:28:11 -04:00
Erik Johnston 64c2cfda8a Merge branch 'release-v1.5.0' of github.com:matrix-org/synapse into develop 2019-10-25 11:34:49 +01:00
Erik Johnston 87259b3a3a
Merge pull request #6255 from matrix-org/erikj/stats_config
Remove repeated calls to config.stats_enabled.
2019-10-25 11:34:38 +01:00
Erik Johnston a71b8c87ec Merge branch 'release-v1.5.0' of github.com:matrix-org/synapse into develop 2019-10-25 11:32:24 +01:00
Erik Johnston a411f2b177 Fix /keys/query API on workers.
The necessary getters were added only to the master store and not the
worker stores.
2019-10-25 11:08:03 +01:00
Erik Johnston f666d6f5d7 Remove repeated calls to config.stats_enabled.
Turns out that fetching variables from the config object is expensive,
so doing it once at startup avoids unnecessary work.
2019-10-25 10:28:36 +01:00
Erik Johnston 44ab048cfe
Merge pull request #6251 from matrix-org/michaelkaye/debug_guard_logging
Reduce debug logging overhead
2019-10-25 10:05:44 +01:00
Hubert Chathi c40d7244f8 Merge branch 'develop' into cross-signing_federation 2019-10-24 22:31:25 -04:00
Hubert Chathi 8ac766c44a make notification of signatures work with workers 2019-10-24 22:14:58 -04:00
Hubert Chathi ff05c9b760 don't error if federation query doesn't have cross-signing keys 2019-10-24 21:46:11 -04:00
Hubert Chathi 848cd388d9 delete keys when deleting backups 2019-10-24 21:21:51 -04:00
Michael Kaye e4d98188da Address codestyle concerns 2019-10-24 18:43:13 +01:00
Michael Kaye 8f4a808d9d Delay printf until logging is required.
Using % will cause the string to be generated even if debugging
is off.
2019-10-24 18:31:53 +01:00
Michael Kaye 9eebc1e73b use %r to __repr__ objects
This avoids calculating __repr__ unless we are going to log.
2019-10-24 18:18:56 +01:00
Michael Kaye f85b9842f0 Don't encode object as UTF-8 string if not needed.
I believe that string formatting ~10-15 sized events will
take a proportion of CPU time.
2019-10-24 18:08:45 +01:00
Michael Kaye 39266a9c9f Make user/room stats log line less verbose. 2019-10-24 17:55:53 +01:00
Erik Johnston 3aa2a90556 Move schema delta files to the correct data store.
They were put in the global schema delta directory due to a bad merge.
2019-10-24 16:45:03 +01:00
Andrew Morgan 8831b04a53 1.5.0rc1 2019-10-24 12:47:58 +01:00
Neil Johnson 2794b79052 Option to suppress resource exceeded alerting (#6173)
The expected use case is to suppress MAU limiting on small instances
2019-10-24 11:48:46 +01:00
Andrew Morgan 92e88a71d3
Cleanup extra quotes from IDEs (#6236) 2019-10-23 16:49:05 +01:00
Erik Johnston 3ca4c7c516 Use new EventPersistenceStore 2019-10-23 16:15:03 +01:00
Erik Johnston 73cf63784b Add DataStores and Storage classes. 2019-10-23 16:15:03 +01:00
Hubert Chathi dc2cd6f79d move get_e2e_cross_signing_key to EndToEndKeyWorkerStore so it works with workers 2019-10-23 09:13:47 -04:00
Erik Johnston 22a9847670 Move persist_events out from main data store.
This is in preparation for splitting out of state_groups_state from the
main store into it own one, as persisting events depends on calculating
state.
2019-10-23 13:29:44 +01:00
Andrew Morgan 409c62b27b
Add config linting script that checks for bool casing (#6203)
Add a linting script that enforces all boolean values in the default config be lowercase.

This has annoyed me for a while so I decided to fix it.
2019-10-23 13:22:54 +01:00
Hubert Chathi 480eac30eb black 2019-10-22 22:37:16 -04:00
Hubert Chathi 404e8c8532 vendor-prefix the EDU name until MSC1756 is merged into the spec 2019-10-22 22:33:23 -04:00
Hubert Chathi 1fabf82d50 update to work with newer code, and fix formatting 2019-10-22 21:44:58 -04:00
Hubert Chathi 41ad35b523 add missing param 2019-10-22 19:06:29 -04:00
Hubert Chathi cfdb84422d make black happy 2019-10-22 19:06:06 -04:00
Hubert Chathi a1aaf3eea6 don't crash if the user doesn't have cross-signing keys 2019-10-22 19:04:37 -04:00
Hubert Chathi 8d3542a64e implement federation parts of cross-signing 2019-10-22 19:04:35 -04:00
Erik Johnston c17efdc01c Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_stores 2019-10-22 18:44:29 +01:00
Erik Johnston 23d62eded2 Clean up prepare_database.py a bit and add comments 2019-10-22 18:43:31 +01:00
Erik Johnston 6cc497f99b Delete background_update table creation in main data_store 2019-10-22 18:02:50 +01:00
Erik Johnston 1bbc5444a8 Move README into synapse/storage/__init__.py 2019-10-22 17:59:31 +01:00
Hubert Chathi 2761731634
Merge pull request #5726 from matrix-org/uhoreg/e2e_cross-signing2-part2
Cross-signing [3/4] -- uploading signatures edition
2019-10-22 17:05:06 +01:00
Erik Johnston acf47c7698 Add a basic README to synapse.storage 2019-10-22 11:55:46 +01:00
Erik Johnston bb6264be0b Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_stores 2019-10-22 10:41:18 +01:00
Erik Johnston ffd24545bb Fix schema management to work with multiple data stores. 2019-10-21 16:08:40 +01:00
Erik Johnston c66a06ac6b Move storage classes into a main "data store".
This is in preparation for having multiple data stores that offer
different functionality, e.g. splitting out state or event storage.
2019-10-21 16:05:06 +01:00
Neil Johnson 82c8799ec7 Set room version default to 5 2019-10-19 09:06:15 +01:00
Richard van der Hoff 93eaeec75a
Remove Auth.check method (#6217)
This method was somewhat redundant, and confusing.
2019-10-18 19:43:36 +02:00
Hubert Chathi 36adfaedab Merge branch 'develop' into cross-signing_sig_upload 2019-10-18 18:34:42 +01:00
Hubert Chathi 125eb45e19 fix doc strings 2019-10-18 16:56:16 +01:00
Hubert Chathi 0a9d22808c Merge branch 'uhoreg/e2e_cross-signing_merged' into develop 2019-10-18 15:57:40 +01:00
Richard van der Hoff 2b22faded7 Merge remote-tracking branch 'origin/develop' into rav/event_auth/1 2019-10-18 12:35:13 +01:00
Richard van der Hoff 560c122267
Fix logging config for the docker image (#6197)
Turns out that loggers that are instantiated before the config is loaded get
turned off.

Also bring the logging config that is generated by --generate-config into line.

Fixes #6194.
2019-10-18 13:34:33 +02:00
Hubert Chathi cfc28325a6 Merge branch 'develop' into uhoreg/e2e_cross-signing_merged 2019-10-18 11:39:40 +01:00
Brendan Abolivier 1ee97cbd01
Merge branch 'master' into develop 2019-10-18 11:00:43 +01:00
Hubert Chathi f0f6a2b360 use the right function for when we're already in runInteraction 2019-10-18 10:56:54 +01:00
Erik Johnston d98029ea89
Merge pull request #6196 from matrix-org/erikj/await
Move rest/admin to use async/await.
2019-10-18 11:53:02 +02:00
Brendan Abolivier 41b9faed16 1.4.1 2019-10-18 10:15:12 +01:00
Richard van der Hoff 80003dfcd5 Merge remote-tracking branch 'origin/develop' into rav/event_auth/1 2019-10-18 10:11:40 +01:00
Erik Johnston 5859a5c569 Fix presence timeouts when synchrotron restarts. (#6212)
* Fix presence timeouts when synchrotron restarts.

Handling timeouts would fail if there was an external process that had
timed out, e.g. a synchrotron restarting. This was due to a couple of
variable name typoes.

Fixes #3715.
2019-10-18 06:42:26 +01:00
Richard van der Hoff 1ba359a11f rip out some unreachable code
The only possible rejection reason is AUTH_ERROR, so all of this is unreachable.
2019-10-17 21:45:59 +01:00
Richard van der Hoff 70d9759cdf Synapse 1.4.1rc1 (2019-10-17)
=============================
 
 Bugfixes
 --------
 
 - Fix bug where redacted events were sometimes incorrectly censored in the database, breaking APIs that attempted to fetch such events. ([\#6185](https://github.com/matrix-org/synapse/issues/6185), [5b0e9948](5b0e9948ea))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEv27Axt/F4vrTL/8QOSor00I9eP8FAl2oj9sACgkQOSor00I9
 eP+ZQQf9GQbjQhq92G1Mt1dI3HxXQnIvelGWaLC9rB++eGcafBkP0EJgCbQZTKda
 WntghJib0oWVHVtPbwYKtwF3z365WbME3awO6RpRoMvrFL5PDqJ8Ma5LUgJ6/GuO
 yDBfABbZk7PVS5WibftXXqibK8MkSrnId57VN3tc/SUHj3DGe4vk+A7plwjNckBj
 2SLUA8DPw5I3Y4Z6peb6VweEA8M59+i3/0cZFAJ+4Ofy2Jad3z8kVXmJ1gTjBOdG
 1bjBfnS2SrZLZwD79zJzaUR0zxfIvj62mL/fjrtR66oRmKF2T+9UnM7+9gH9JHWZ
 ct3Mk4Nnr4VaBBWOWOcD2btvByckug==
 =9lJk
 -----END PGP SIGNATURE-----

Merge tag 'v1.4.1rc1' into develop

Synapse 1.4.1rc1 (2019-10-17)
=============================

Bugfixes
--------

- Fix bug where redacted events were sometimes incorrectly censored in the database, breaking APIs that attempted to fetch such events. ([\#6185](https://github.com/matrix-org/synapse/issues/6185), [5b0e9948](5b0e9948ea))
2019-10-17 17:03:02 +01:00
Richard van der Hoff 71cd3fed66 1.4.1rc1 2019-10-17 16:40:56 +01:00
Erik Johnston 5b0e9948ea Do the update as a background index 2019-10-17 16:31:42 +01:00
Erik Johnston a2bb50c2eb Merge pull request #6185 from matrix-org/erikj/fix_censored_evnets
Fix inserting bytes as text in `censor_redactions`
2019-10-17 16:29:07 +01:00
Hubert Chathi 06fc66c81e
Merge pull request #6193 from matrix-org/uhoreg/interpret_device_key_in_storage
make storage layer in charge of interpreting the device key data
2019-10-11 15:19:06 -04:00
Hubert Chathi 132b251e29 expand on comment 2019-10-11 14:24:52 -04:00
Erik Johnston 2e97a4c197 Port synapse/rest/client/_base.py to async/await 2019-10-11 15:29:26 +01:00
Erik Johnston 3c2d6c708c Add maybe_awaitable and fix __init__ bugs 2019-10-11 15:26:09 +01:00
Hubert Chathi 691dd67fcd
Merge pull request #6189 from matrix-org/uhoreg/e2e_backup_optional_version
make version optional in body of e2e backup version update
2019-10-11 10:11:59 -04:00
Erik Johnston 6f5c6c8f60
Merge pull request #6195 from matrix-org/erikj/opentracing_preview_url
Trace non-JSON APIs, /media, /key etc
2019-10-11 13:23:52 +01:00
Valérian Rousset be9b55e0d2 cas: support setting display name (#6114)
Now, the CAS server can return an attribute stating what's the desired displayname, instead of using the username directly.
2019-10-11 12:33:12 +01:00
Erik Johnston dfbb62c28d Port synaps/rest/client/users.py to async/await 2019-10-11 12:21:40 +01:00
Erik Johnston f95325e22a Port synaps/rest/client/server_notice_servlet.py to async/await 2019-10-11 12:21:40 +01:00
Erik Johnston 281f887090 Port synaps/rest/client/media.py to async/await 2019-10-11 12:21:40 +01:00
Erik Johnston fca3a541e7 Port rest/admin/__init__.py to async/await 2019-10-11 12:17:52 +01:00
Erik Johnston f3ceaf4323 Trace non-JSON APIs, /media, /key etc 2019-10-11 11:58:52 +01:00
Neil Johnson a0d0ba7862
Fix MAU reaping where reserved users are specified. (#6168) 2019-10-11 09:38:26 +01:00
Hubert Chathi 7a0dce9259 make sure we actually return something 2019-10-10 20:31:30 -04:00
Hubert Chathi 4908fb3b30 make storage layer in charge of interpreting the device key data 2019-10-10 19:15:30 -04:00
Erik Johnston 83d86106a8
Merge pull request #6156 from matrix-org/erikj/postgres_any
Use Postgres ANY for selecting many values.
2019-10-10 16:41:36 +01:00
Erik Johnston b54b1e759a Fix SQLite take 2 2019-10-10 16:19:40 +01:00
Erik Johnston afb6d9d53b Fix SQLite 2019-10-10 15:55:41 +01:00
Erik Johnston 3bc687508f Remove add_in_list_sql_clause 2019-10-10 15:35:46 +01:00
Erik Johnston ca3e01e50d Fix store_url_cache using bytes 2019-10-10 14:52:29 +01:00
Erik Johnston 9d06fb9cb1 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/postgres_any 2019-10-10 14:38:21 +01:00
Erik Johnston 8bc529c04d Merge branch 'develop' of github.com:matrix-org/synapse into erikj/disable_sql_bytes 2019-10-10 14:30:51 +01:00
Erik Johnston 933034e2fe
Merge pull request #6127 from matrix-org/erikj/patch_inner
Add more log context checks when patching inlineCallbacks
2019-10-10 13:47:50 +01:00
Erik Johnston fe1c1e6c28
Fixup comments
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-10-10 13:17:19 +01:00
Erik Johnston b161786c14 Replace IN usage with helper funcs 2019-10-10 13:15:49 +01:00
Erik Johnston b4fbf71187 Add helper funcs to use postgres ANY
This means that we can write queries with `col = ANY(?)`, which helps
postgres.
2019-10-10 13:15:24 +01:00
Erik Johnston 91f43dca39 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/disable_sql_bytes 2019-10-10 13:10:57 +01:00
werner291 b5b03b7079 Add domain validation when creating room with list of invitees (#6121) 2019-10-10 13:05:48 +01:00