Commit Graph

6337 Commits

Author SHA1 Message Date
Erik Johnston f6124311fd Add m.room.member type to query 2017-02-01 11:59:17 +00:00
Erik Johnston 88a4d54883 Merge pull request #1867 from matrix-org/erikj/member_index
Add an index to make membership queries faster
2017-02-01 11:44:27 +00:00
Erik Johnston 368c88c487 Add a small cache get_all_new_events 2017-02-01 10:50:44 +00:00
Erik Johnston 5deaf9e30b Up get_latest_event_ids_in_room cache 2017-02-01 10:39:41 +00:00
Erik Johnston acb501c46d Comment 2017-02-01 10:32:49 +00:00
Erik Johnston 97479d0c54 Implement /keys/changes 2017-02-01 10:30:03 +00:00
Erik Johnston 06567ec513 Merge pull request #1866 from matrix-org/erikj/device_list_fixes
Better handle 404 response for federation /send/
2017-02-01 09:44:14 +00:00
Erik Johnston 458b6f4733 Only invalidate membership caches based on the cache stream
Before we completely invalidated get_users_in_room whenever we updated
any current_state_events table. This was way too aggressive.
2017-01-31 16:09:03 +00:00
Erik Johnston fe08db2713 Remove explicit < 400 check as apparently this is confusing 2017-01-31 15:21:32 +00:00
Erik Johnston 21b7375778 Add an index to make membership queries faster 2017-01-31 15:15:57 +00:00
Erik Johnston 4c0ec15bdc Comment 2017-01-31 13:53:46 +00:00
Erik Johnston 85c590105f Comment 2017-01-31 13:46:38 +00:00
Erik Johnston ae7a132f38 Better handle 404 response for federation /send/ 2017-01-31 13:40:09 +00:00
Erik Johnston ac001dabdc Merge pull request #1864 from matrix-org/erikj/device_list_fixes
Fix clearing out old device list outbound pokes
2017-01-31 13:35:35 +00:00
Erik Johnston bfb3d255b1 Merge pull request #1862 from matrix-org/erikj/presence_update
Use DB cache of joined users for presence
2017-01-31 13:23:24 +00:00
Erik Johnston ab55794b6f Fix deletion of old sent devices correctly 2017-01-31 13:22:41 +00:00
Erik Johnston d3169e8d28 Only fetch with row ts and count > 1 2017-01-31 11:20:03 +00:00
Erik Johnston 05b9f48ee5 Fix clearing out old device list outbound pokes 2017-01-31 10:08:55 +00:00
Erik Johnston 4b3403ca9b Stream cache invalidations for room membership storage functions 2017-01-30 17:28:22 +00:00
Erik Johnston 1c13c9f6b6 Don't have such a large cache 2017-01-30 17:12:14 +00:00
Erik Johnston c7a26b7c32 Fix unit tests 2017-01-30 17:11:24 +00:00
Erik Johnston fd1c18c088 Use DB cache of joined users for presence 2017-01-30 17:00:24 +00:00
Erik Johnston c2c9a78db9 Noop device key changes if they're the same 2017-01-30 16:55:04 +00:00
Erik Johnston e75a779d9e Fix query 2017-01-30 16:38:20 +00:00
Erik Johnston 828db669ec Use get_users_in_room and declare it iterable 2017-01-30 16:37:22 +00:00
Erik Johnston 9636b2407d Merge pull request #1857 from matrix-org/erikj/device_list_stream
Implement device lists updates over federation
2017-01-30 14:35:21 +00:00
Erik Johnston 3670025e64 Rename func 2017-01-30 14:11:31 +00:00
Erik Johnston 4ac363a168 Remove debug logging 2017-01-30 14:10:12 +00:00
Erik Johnston d360c97ae1 Clear out old destination pokes. 2017-01-30 10:14:37 +00:00
Erik Johnston 76100203ab Always use the latest stream_id, sent or unsent 2017-01-30 10:14:25 +00:00
Erik Johnston d1e1fd6210 Add ts column to device_lists_outbound_pokes 2017-01-27 15:23:48 +00:00
Erik Johnston 252b503fc8 Hook device list updates to replication 2017-01-27 14:31:35 +00:00
Erik Johnston 84a35f32c7 Comment 2017-01-27 10:35:12 +00:00
Erik Johnston c517a19c2d Comment 2017-01-27 10:33:26 +00:00
Erik Johnston 738a2867c8 SQL param ordering 2017-01-27 10:31:29 +00:00
Erik Johnston 755adff0e4 User if rather than for 2017-01-27 10:31:06 +00:00
Erik Johnston 888c59c955 Better name 2017-01-27 10:29:47 +00:00
Erik Johnston f25a4a4692 Remove unused param 2017-01-27 10:27:39 +00:00
Erik Johnston 31aca5589c Fix on sqlite: use left rather than outer join 2017-01-26 16:55:50 +00:00
Erik Johnston 76d40f4904 Handle users leaving rooms 2017-01-26 16:39:33 +00:00
Erik Johnston fbfad76c03 Add comments 2017-01-26 16:33:21 +00:00
Erik Johnston c974116f19 Implement device key caching over federation 2017-01-26 16:07:24 +00:00
Erik Johnston 51e9fe36e4 Fix up sending of m.device_list_update edus 2017-01-25 16:55:21 +00:00
Erik Johnston 2367c5568c Add basic implementation of local device list changes 2017-01-25 14:27:27 +00:00
Paul "LeoNerd" Evans 10e48d8310 Don't clobber a displayname or avatar_url if provided by an m.room.member event 2017-01-24 18:06:07 +00:00
Erik Johnston ba8e144554 Merge branch 'erikj/current_state_fix' into develop 2017-01-23 16:15:10 +00:00
Erik Johnston f5b46482f4 Merge pull request #1840 from matrix-org/erikj/current_state_fix
Insert delta of current_state_events to be more efficient
2017-01-23 16:14:34 +00:00
Erik Johnston fdf2a31a51 Typo 2017-01-23 16:14:14 +00:00
Erik Johnston 41dab8a222 Fix bug where current_state_events renamed to current_state_ids 2017-01-23 15:22:48 +00:00
Erik Johnston c77b24c092 Refactor to calculate state delta outside transaction 2017-01-23 14:51:33 +00:00
Erik Johnston 5d2134d485 Comments 2017-01-20 17:13:24 +00:00
Erik Johnston a55fa2047f Insert delta of current_state_events to be more efficient 2017-01-20 17:10:18 +00:00
Erik Johnston 3d9d48fffb Merge pull request #1836 from matrix-org/erikj/current_state_fix
Derive current_state_events from state groups
2017-01-20 15:14:05 +00:00
Erik Johnston d0897dead5 Spelling 2017-01-20 15:05:11 +00:00
Erik Johnston 567aa35b67 Update all call sites after rename 2017-01-20 14:40:31 +00:00
Erik Johnston f2f40e64a9 Comments 2017-01-20 14:38:13 +00:00
Erik Johnston 4c6a31cd6e Calculate the forward extremeties once 2017-01-20 14:28:53 +00:00
Erik Johnston 86063d4321 Merge pull request #1835 from matrix-org/erikj/fix_workers
Make worker listener config backwards compat
2017-01-20 11:55:56 +00:00
Erik Johnston 09eb08f910 Derive current_state_events from state groups 2017-01-20 11:52:51 +00:00
Erik Johnston 97efe99ae9 Make worker listener config backwards compat 2017-01-20 11:45:29 +00:00
Marvin Steadfast 86e6165687 Added default config for turn username and password 2017-01-19 14:35:55 +01:00
Marvin Steadfast 1e38be3a7a Added username and password for turn server
It makes it possible to use a turn server that needs a username and
password instead of a token.
2017-01-19 14:08:20 +01:00
Erik Johnston 841c228533 Merge pull request #1828 from matrix-org/erikj/iterable_cache_size
Update LruCache size estimate on clear
2017-01-18 14:57:54 +00:00
Erik Johnston c430111d0e Update LruCache size estimate on clear 2017-01-18 14:55:23 +00:00
David Baker 97d3918377 Merge pull request #1811 from aperezdc/unhardcode-riot-urls
Allow configuring the Riot URL used in notification emails
2017-01-18 14:38:49 +00:00
David Baker 8c5009b628 Lowercase all email addresses before querying db
Since we store all emails in the DB in lowercase
(https://github.com/matrix-org/synapse/pull/1170)
2017-01-18 13:25:56 +00:00
Erik Johnston ae7b4da4cc Merge pull request #1823 from matrix-org/erikj/load_events_logs
Remove loading events logs
2017-01-18 11:07:58 +00:00
Erik Johnston fc7cae8aa3 Merge pull request #1824 from matrix-org/erikj/retry_host_log
Lower the not retrying host log line to debug
2017-01-18 11:07:51 +00:00
Erik Johnston f9058ca785 Merge pull request #1822 from matrix-org/erikj/statE_logging
Change resolve_state_groups call site logging to DEBUG
2017-01-18 11:02:03 +00:00
Erik Johnston f648313f98 Merge pull request #1821 from matrix-org/erikj/cache_metrics_string_intern
Measure metrics of string_cache
2017-01-18 10:57:39 +00:00
Erik Johnston 15f012032c Merge pull request #1818 from matrix-org/erikj/state_auth_splitout_split
Optimise state resolution
2017-01-18 10:53:00 +00:00
Erik Johnston 4ec1cf49e2 Lower loading events log to DEBUG 2017-01-17 17:28:32 +00:00
Erik Johnston f878f64f43 Lower the not retrying host log line to debug 2017-01-17 17:20:39 +00:00
Erik Johnston 5f027d1fc5 Change resolve_state_groups call site logging to DEBUG 2017-01-17 17:07:15 +00:00
Erik Johnston 380dba1020 Measure metrics of string_cache 2017-01-17 17:04:46 +00:00
Erik Johnston ed4d176152 PEP8 2017-01-17 15:27:28 +00:00
Mark Haines c6064a7ba6 Only construct sets when necessary 2017-01-17 15:23:07 +00:00
Erik Johnston a8594fd19f Use better names 2017-01-17 14:59:03 +00:00
Erik Johnston 7fae460402 Merge pull request #1820 from matrix-org/erikj/push_tools
Get state at event rather than for room in push
2017-01-17 14:58:26 +00:00
Erik Johnston 37b4c7d8a9 Fix typo in return type 2017-01-17 14:43:32 +00:00
Erik Johnston e5d2df9c34 Use better variable name 2017-01-17 14:32:53 +00:00
Erik Johnston 04006bb7f0 Get state at event rather than for room in push 2017-01-17 14:31:21 +00:00
Erik Johnston ce59a2faad Correctly handle case of rejected events in state res 2017-01-17 14:18:53 +00:00
Erik Johnston 633f97151c Check event is in state_map 2017-01-17 13:33:54 +00:00
Erik Johnston e6153e1bd1 Fix couple of federation state bugs 2017-01-17 13:22:34 +00:00
Erik Johnston 5d6bad1b3c Optimise state resolution 2017-01-17 13:22:19 +00:00
Erik Johnston e8ecbb6f20 Merge pull request #1812 from matrix-org/erikj/state_auth_splitout_split
Split out static state methods from StateHandler
2017-01-17 11:55:18 +00:00
Erik Johnston d6c75cb7c2 Rename and comment tree_to_leaves_iterator 2017-01-17 11:47:03 +00:00
Erik Johnston 1ccd5676e3 Remove needless call to evict() 2017-01-17 11:42:26 +00:00
Erik Johnston d906206049 Increase state_group_cache_size 2017-01-17 11:31:08 +00:00
Erik Johnston f85b6ca494 Speed up cache size calculation
Instead of calculating the size of the cache repeatedly, which can take
a long time now that it can use a callback, instead cache the size and
update that on insertion and deletion.

This requires changing the cache descriptors to have two caches, one for
pending deferreds and the other for the actual values. There's no reason
to evict from the pending deferreds as they won't take up any more
memory.
2017-01-17 11:18:13 +00:00
Erik Johnston 6d00213e80 Use OrderedDict in ExpiringCache 2017-01-16 15:33:22 +00:00
Erik Johnston 897f8752da Up cache max entries for state 2017-01-16 15:08:17 +00:00
Erik Johnston beda469bc6 Put staticmethods at module level 2017-01-16 15:05:24 +00:00
Erik Johnston 46aebbbcbf Add support for 'iterable' to ExpiringCache 2017-01-16 14:57:23 +00:00
Erik Johnston 01521299c7 Increase cache size limit 2017-01-16 11:56:51 +00:00
Erik Johnston 2fae34bd2c Optionally measure size of cache by sum of length of values 2017-01-13 17:46:17 +00:00
Erik Johnston 95a22ae194 Merge pull request #1810 from matrix-org/erikj/state_auth_splitout_split
Split out static auth methods from Auth object
2017-01-13 16:32:27 +00:00
Erik Johnston ec0a523ac3 Split out static state methods from StateHandler 2017-01-13 15:25:06 +00:00
Erik Johnston e178feca3f Remove unused function 2017-01-13 15:16:45 +00:00
Erik Johnston f0325a9ccc Merge pull request #1793 from matrix-org/erikj/change_device_inbox_index
Change device_inbox stream index to include user
2017-01-13 15:14:51 +00:00
Erik Johnston c050f493dd Add comment 2017-01-13 15:14:41 +00:00
Adrian Perez de Castro a3e4a198e3
Allow configuring the Riot URL used in notification emails
The URLs used for notification emails were hardcoded to use either matrix.to
or vector.im; but for self-hosted setups where Riot is also self-hosted it
may be desirable to allow configuring an alternative Riot URL.

Fixes #1809.

Signed-off-by: Adrian Perez de Castro <aperez@igalia.com>
2017-01-13 17:12:04 +02:00
Erik Johnston 8b2fa38256 Split event auth code into seperate module 2017-01-13 15:07:32 +00:00
Erik Johnston 641ccdbb14 Merge pull request #1795 from matrix-org/erikj/port_defaults
Restore default bind address
2017-01-13 13:02:59 +00:00
Erik Johnston ebf94aff8d Fix spurious Unhandled Error log lines 2017-01-12 17:19:47 +00:00
Erik Johnston bf5c9706d9 Remove full_twisted_stacktraces option
The debug 'full_twisted_stacktraces' flag caused synapse to rewrite
twisted deferreds to always fire the callback on the next reactor tick.
This was to force the deferred to always store the stacktraces on
exceptions, and thus be more likely to have a full stacktrace when it
reaches the final error handlers and gets printed to the logs.

Dynamically rewriting things is generally bad, and in particular this
change violates assumptions of various bits of Twisted. This wouldn't
necessarily be so bad, but it turns out this option has been turned on
on some production servers.

Turning the option can cause e.g. #1778.

For now, lets just entirely nuke this option.
2017-01-12 10:32:52 +00:00
Erik Johnston 7b62d0bc70 Add missing None check 2017-01-11 10:57:03 +00:00
Erik Johnston 7e6c2937c3 Split out static auth methods from Auth object 2017-01-10 18:16:54 +00:00
Erik Johnston b1dfd20292 Pop bind_address 2017-01-10 17:23:18 +00:00
Erik Johnston edd6cdfc9a Restore default bind address 2017-01-10 17:21:41 +00:00
Erik Johnston 8a0fddfd73 Remove spurious for..else.. 2017-01-10 16:30:53 +00:00
Erik Johnston d524bc9110 Merge pull request #1792 from matrix-org/erikj/limit_cache_prefill_device
Limit number of entries to prefill from cache
2017-01-10 15:42:00 +00:00
Erik Johnston d2b00d0866 Merge pull request #1790 from matrix-org/erikj/linearizer
Add paranoia exception catch in Linearizer
2017-01-10 15:38:30 +00:00
Erik Johnston ab655dca33 Explicitly close the cursor 2017-01-10 15:15:25 +00:00
Erik Johnston 5a32e9273e Don't disable autocommit 2017-01-10 15:11:27 +00:00
Erik Johnston caddadfc5a Change device_inbox stream index to include user
This makes fetching the nost recently changed users much tricker, and
brings it in line with e.g. presence_stream indices.
2017-01-10 15:04:57 +00:00
Erik Johnston dd52d4de4c Limit number of entries to prefill from cache
Some tables, like device_inbox, take a long time to query at startup for
the stream change cache prefills. This is likely because they are slower
growing streams and so are more fragmented on disk. For now, lets pull
fewer entries out to make startup quicker.

In future, we should add a better index to make it even faster.
2017-01-10 14:34:50 +00:00
Mark Haines 32019c9897 Log which files we saved attachments to in the media_repository 2017-01-10 14:19:50 +00:00
Erik Johnston 657488113e Merge pull request #1789 from matrix-org/erikj/decouple_presence
Don't block messages sending on bumping presence
2017-01-10 14:06:05 +00:00
Erik Johnston 3b4de17d2b Comment 2017-01-10 14:05:53 +00:00
Erik Johnston 7d0981b312 Merge pull request #1787 from matrix-org/erikj/linearize_member
Linearize updates to membership via PUT /state/
2017-01-10 14:04:54 +00:00
Erik Johnston f477370c0c Add paranoia exception catch in Linearizer 2017-01-10 14:04:13 +00:00
Erik Johnston 586f474a44 Don't block messages sending on bumping presence 2017-01-10 12:46:00 +00:00
Erik Johnston 6823fe5241 Linearize updates to membership via PUT /state/ 2017-01-09 18:25:13 +00:00
Erik Johnston f7085ac84f Name linearizer's for better logs 2017-01-09 17:17:10 +00:00
Erik Johnston 9898bbd9dc Merge branch 'master' of github.com:matrix-org/synapse into develop 2017-01-09 14:51:17 +00:00
Erik Johnston 9a8ae6f1bf Bump version and changelog 2017-01-09 14:47:56 +00:00
Matthew Hodgson 2f4b2f4783 gah, fix mangled merge of 0.18.7 into develop 2017-01-07 04:00:42 +00:00
Matthew 6d363cea9d Merge branch 'release-v0.18.7' into develop 2017-01-07 03:46:16 +00:00
Matthew f0e4bac64e bump changelog & version 2017-01-07 03:45:38 +00:00
Matthew 4304e7e593 do the discard check in the right place to avoid grabbing dependent events 2017-01-07 03:44:18 +00:00
Matthew Hodgson 8c48971b51 Merge branch 'release-v0.18.7' into develop 2017-01-07 02:23:37 +00:00
Matthew e10c527930 Discard PDUs from invalid origins due to #1753 in 0.18.[56] 2017-01-07 02:13:14 +00:00
Matthew Hodgson 2f5be2d8dc oops, this should have been rc1 2017-01-07 01:11:56 +00:00
Matthew Hodgson 4086026524 move logging to right place 2017-01-07 00:41:46 +00:00
Matthew 19e2fb4386 bump version 2017-01-06 23:38:22 +00:00
Matthew b2850e62db fix typo breaking the fix to #1753 2017-01-06 23:23:37 +00:00
Mark Haines 06c00bd19b Merge branch 'release-v0.18.6' into develop 2017-01-06 14:46:27 +00:00
Mark Haines b42a972b71 Bump version and changelog 2017-01-06 14:44:28 +00:00
Patrik Oldsberg 1ef6084b75 handlers/room_member: fix guest access check when joining rooms
Signed-off-by: Patrik Oldsberg <patrik.oldsberg@ericsson.com>
2017-01-06 14:36:56 +00:00
Matthew Hodgson bd85434cb3 Merge branch 'release-v0.18.6' into develop 2017-01-05 13:58:19 +00:00
Mark Haines c18f7fc410 Fix flake8 and update changelog 2017-01-05 13:50:22 +00:00
Matthew Hodgson dafd50d178 Merge pull request #1767 from matrix-org/matthew/resolve_state_group_logging
log call paths for resolve_state_group
2017-01-05 13:47:42 +00:00
Matthew Hodgson 883ff92a7f Fix case 2017-01-05 13:45:02 +00:00
Matthew Hodgson d79d165761 add logging for all the places we call resolve_state_groups. my kingdom for a backtrace that actually works. 2017-01-05 13:40:39 +00:00
Matthew Hodgson 8cfc0165e9 fix annoying typos 2017-01-05 13:39:43 +00:00
Mark Haines 62451800e7 Bump version and changelog to v0.18.6-rc3 2017-01-05 13:36:10 +00:00
Matthew Hodgson b31ed22738 Merge branch 'release-v0.18.6' into develop 2017-01-05 13:03:02 +00:00
Mark Haines dd3df11c55 More logging for the linearizer and for get_events 2017-01-05 12:32:47 +00:00
Matthew Hodgson 468749c9fc fix comment 2017-01-05 12:00:11 +00:00
Matthew Hodgson eedf400d05 limit total timeout for get_missing_events to 10s 2017-01-05 11:58:15 +00:00
Matthew Hodgson 8e82611f37 fix comment 2017-01-05 11:44:44 +00:00
Mark Haines f784980d2b Only send events that originate on this server.
Or events that are sent via the federation "send_join" API.

This should match the behaviour from before v0.18.5 and #1635 landed.
2017-01-05 11:26:30 +00:00
Mark Haines e02bdaf08b Get the destinations from the state from before the event
Rather than the state after then event.
2017-01-04 15:17:15 +00:00
Mark Haines b6b67715ed Send ALL membership events to the server that was affected.
Send all membership changes to the server that was affected.
This ensures that if the last member of a room on a server
was kicked or banned they get told about it.
2017-01-04 13:56:20 +00:00
Matthew Hodgson 555d702e34 limit total timeout for get_missing_events to 10s 2016-12-31 15:21:37 +00:00
Matthew Hodgson 899a3a1268 Merge branch 'release-v0.18.6' into develop 2016-12-31 02:38:26 +00:00
Mark Haines f3de4f8cb7 Bump version and changelog 2016-12-30 20:21:04 +00:00
Mark Haines 62ce3034f3 s/aquire/acquire/g 2016-12-30 20:04:44 +00:00
Mark Haines 0aff09f6c9 Add more useful logging when we block fetching events 2016-12-30 20:00:44 +00:00
Mark Haines cc50b1ae53 Remove fallback from get_missing_events.
get_missing_events used to fallback to fetching the missing events
individually requesting from every server in the room, one by one.e

This could be unacceptably slow, possibly causing #1732
2016-12-30 18:13:15 +00:00
Mark Haines f576c34594 Merge remote-tracking branch 'origin/release-v0.18.6' into develop 2016-12-30 15:13:49 +00:00
Mark Haines 822cb39dfa Use the new twisted logging framework.
Hopefully adding an observer to the new framework will avoid a memory
leak https://twistedmatrix.com/trac/ticket/8164
2016-12-30 11:09:24 +00:00
Mark Haines 342fb8dae9 Merge branch 'release-v0.18.6' into develop 2016-12-29 17:33:46 +00:00
Mark Haines f023be9293 Bump changelog and version 2016-12-29 16:18:04 +00:00
Mark Haines 97ffc5690b Manually abort the underlying TLS connection.
The abort() method calls loseConnection() which tries to shutdown the
TLS connection cleanly. We now call abortConnection() directly which
should promptly close both the TLS connection and the underlying TCP
connection.

I also added some TODO markers to consider cancelling the old previous
timeout rather than checking time.time(). But given how urgently we want
to get this code released I'd rather leave the existing code with the
duplicate timeouts and the time.time() check.
2016-12-29 15:51:04 +00:00
Erik Johnston b4bc6fef5b Respect long_retries param and default to off 2016-12-29 00:58:34 +00:00
Erik Johnston 68030fd37b Spelling and comments 2016-12-29 00:10:49 +00:00
Erik Johnston b7336ff32d Clean up 2016-12-29 00:09:33 +00:00
Erik Johnston 5b6672c66d Wrap connections in an N minute timeout to ensure they get reaped correctly 2016-12-29 00:06:53 +00:00
David Baker 84cf00c645 Fix another comment typo 2016-12-21 09:51:43 +00:00
David Baker 0c88ab1844 Add /account/3pid/delete endpoint
Also fix a typo in a comment
2016-12-20 18:27:30 +00:00
Johannes Löthberg 702c020e58 Fix check for bind_address
The empty string is a valid setting for the bind_address option, so
explicitly check for None here instead.

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-20 01:37:50 +01:00
Matthew Hodgson 09f15918be Merge pull request #1711 from matrix-org/matthew/utf8-password-change
fix ability to change password to a non-ascii one
2016-12-20 00:02:13 +00:00
Matthew Hodgson da2c8f3c94 Merge pull request #1709 from kyrias/bind_addresses
Add support for specifying multiple bind addresses
2016-12-19 23:49:34 +00:00
Matthew Hodgson a58e4e0d48 Merge pull request #1696 from kyrias/ipv6
IPv6 support
2016-12-19 23:49:07 +00:00
Matthew Hodgson f2a5aebf98 fix ability to change password to a non-ascii one
https://github.com/vector-im/riot-web/issues/2658
2016-12-18 22:25:21 +00:00
Johannes Löthberg a9c1b419a9 Bump twisted dependency
At least 16.0.0 is needed for wrapClientTLS support.

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-18 23:16:43 +01:00
Johannes Löthberg f5cd5ebd7b Add IPv6 comment to default config
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-18 23:14:32 +01:00
Johannes Löthberg c95e9fff99 Make default homeserver config use bind_addresses
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-18 21:51:56 +01:00
Johannes Löthberg 7dfd70fc83 Add support for specifying multiple bind addresses
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-18 21:51:56 +01:00
Erik Johnston b2f8642d3d Cache network room list queries. 2016-12-16 16:11:43 +00:00
Erik Johnston b9b6d17ab1 Bump version and changelog 2016-12-16 10:18:02 +00:00
Richard van der Hoff edc6a1e4f9 Add some logging for syncing to_device events
Attempt to track down the loss of to_device events
(https://github.com/vector-im/riot-web/issues/2711 etc).
2016-12-15 18:16:10 +00:00
Erik Johnston 35129ac998 Merge pull request #1698 from matrix-org/erikj/room_list
Fix caching on public room list
2016-12-15 15:40:28 +00:00
Erik Johnston aa1336c00a Merge pull request #1700 from matrix-org/erikj/backfill_filter
Fix /backfill returning events it shouldn't
2016-12-15 14:21:30 +00:00
Marcin Bachry 24c16fc349 Fix crash in url preview when html tag has no text
Signed-off-by: Marcin Bachry <hegel666@gmail.com>
2016-12-14 22:38:18 +01:00
Erik Johnston b8255eba26 Comment 2016-12-14 13:49:54 +00:00
Erik Johnston b2999a7055 Fix /backfill returning events it shouldn't 2016-12-14 13:41:45 +00:00
Erik Johnston c3208e45c9 Fixup membership query 2016-12-14 10:46:58 +00:00
Erik Johnston 9d95351cad Merge branch 'develop' of github.com:matrix-org/synapse into erikj/room_list 2016-12-13 17:57:08 +00:00
Erik Johnston 1de53a7a1a Fix caching on public room list 2016-12-13 17:33:24 +00:00
Erik Johnston b3d398343e Bump changelog and version 2016-12-13 11:07:27 +00:00
Johannes Löthberg 0648e76979 Remove spurious newline
Apparently I just removed the spaces instead...

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-12 18:41:30 +01:00
Erik Johnston 8588d0eb3d Merge pull request #1697 from matrix-org/erikj/fix_bg_member
Fix background update that prematurely stopped
2016-12-12 17:20:17 +00:00
Erik Johnston 1574b839e0 Merge pull request #1676 from matrix-org/erikj/room_list
Add new API appservice specific public room list
2016-12-12 17:00:10 +00:00
Erik Johnston 7ec2bf9b77 Fix background update that prematurely stopped 2016-12-12 16:54:58 +00:00
Erik Johnston 2bf5a47b3e Rename network_id to instance_id on client side 2016-12-12 16:05:45 +00:00
Johannes Löthberg d3bd94805f Fixup for #1689 and #1690
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-12 16:32:47 +01:00
Erik Johnston 09cbcb78d3 Add cache to get_public_room_ids_at_stream_id 2016-12-12 14:41:51 +00:00
Erik Johnston 631376e2ac Notify replication. Use correct network_id 2016-12-12 14:28:15 +00:00
Richard van der Hoff abed247182 Remove unspecced GET endpoints for e2e keys
GET /keys/claim is a terrible idea, since it isn't idempotent; also it throws
500 errors if you call it without all the right params.

GET /keys/query is arguable, but it's unspecced, so let's get rid of it too to
stop people relying on unspecced APIs.
2016-12-12 12:31:40 +00:00
Erik Johnston d45c984653 Docstring 2016-12-12 11:00:27 +00:00
Erik Johnston d53a80af25 Merge pull request #1620 from matrix-org/erikj/concurrent_room_access
Limit the number of events that can be created on a given room concurrently
2016-12-12 10:30:23 +00:00
Richard van der Hoff 85cd30b1fd Merge pull request #1686 from matrix-org/rav/fix_federation_key_fails
E2E key query: handle federation fails
2016-12-12 09:33:39 +00:00
Richard van der Hoff deca951241 Remove unused import 2016-12-12 09:24:35 +00:00
Glyph 9f07f4c559 IPv6 support for endpoint.py
Similar to https://github.com/matrix-org/synapse/pull/1689, but for endpoint.py
2016-12-11 11:10:32 +01:00
Glyph 6e18805ac2 IPv6 support for client.py
This is an (untested) general sketch of how to use wrapClientTLS to implement TLS over IPv6, as well as faster connections over IPv4.
2016-12-11 11:10:32 +01:00
Richard van der Hoff efa4ccfaee E2E key query: handle federation fails
Don't fail the whole request if we can't connect to a particular server.
2016-12-09 18:31:01 +00:00
Erik Johnston fbaf868f62 Correctly handle timeout errors 2016-12-09 16:30:29 +00:00
Erik Johnston be14c24cea Fix rare notifier bug where listeners dont timeout
There was a race condition that caused the notifier to 'miss' the
timeout notification, since there were no other checks for the timeout
this caused listeners to get stuck in a loop until something happened.
2016-12-09 15:43:18 +00:00
Erik Johnston 1697f6a323 Merge pull request #1680 from matrix-org/erikj/joined_rooms
Add new room membership APIs
2016-12-09 11:35:48 +00:00
Erik Johnston 52d12ca782 Add /room/<room_id>/joined_members API
This returns the currently joined members in the room with their display
names and avatar urls. This is more efficient than /members for large
rooms where you don't need the full events.
2016-12-08 13:32:07 +00:00
Erik Johnston c45d8e9ba2 Add profile data to the room_membership table for joins 2016-12-08 13:08:41 +00:00
Richard van der Hoff b08f76bd23 Fix ignored read-receipts
Don't ignore read-receipts which arrive in the same EDU as a read-receipt for
an old event.
2016-12-08 12:13:01 +00:00
Richard van der Hoff bd07a35c29 Fix result of insert_receipt
This should fix the absence of notifications when new receipts arrive.
2016-12-08 12:11:34 +00:00
Erik Johnston de796f27e6 Add joined_rooms servlet 2016-12-08 11:39:03 +00:00
Erik Johnston 2687af82d4 Comments 2016-12-07 09:58:33 +00:00
Erik Johnston 3727d66a0e Don't include appservice id 2016-12-06 17:04:26 +00:00
Erik Johnston f32fb65552 Add new API appservice specific public room list 2016-12-06 16:12:27 +00:00
Richard van der Hoff 1529c19675 Prevent user tokens being used as guest tokens (#1675)
Make sure that a user cannot pretend to be a guest by adding 'guest = True'
caveats.
2016-12-06 15:31:37 +00:00
Erik Johnston 9609c91e7d Merge pull request #653 from matrix-org/erikj/preset_guest_join
Enable guest access for private rooms by default
2016-12-05 17:47:14 +00:00
Erik Johnston 338df4f409 Merge pull request #1649 from matrix-org/dbkr/log_ui_auth_args
Log the args that we have on UI auth completion
2016-12-05 16:40:58 +00:00
pik c46e7a9c9b Bugfix: Console logging handler missing default filter 2016-12-03 20:14:58 -03:00
Johannes Löthberg 32c8b5507c preview_url_resource: Ellipsis must be in unicode string
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-01 13:12:13 +01:00
Richard van der Hoff 6841d8ff55 Fix doc-string
Remove refresh_token reference
2016-12-01 11:42:17 +00:00
Richard van der Hoff aa09d6b8f0 Rip out more refresh_token code
We might as well treat all refresh_tokens as invalid. Just return a 403 from
/tokenrefresh, so that we don't have a load of dead, untestable code hanging
around.

Still TODO: removing the table from the schema.
2016-11-30 17:40:18 +00:00
Richard van der Hoff dc4b23e1a1 Merge branch 'develop' into rav/no_more_refresh_tokens 2016-11-30 17:10:04 +00:00
Richard van der Hoff 8379a741cc Merge pull request #1660 from matrix-org/rav/better_content_type_validation
More intelligent Content-Type parsing
2016-11-30 16:54:03 +00:00
Richard van der Hoff 321fe5c44c Merge pull request #1656 from matrix-org/rav/remove_time_caveat
Stop putting a time caveat on access tokens
2016-11-30 16:53:20 +00:00
Richard van der Hoff b5b3a7e867 More intelligent Content-Type parsing
Content-Type is allowed to contain options (`; charset=utf-8`, for
instance). We should allow that.
2016-11-30 15:07:32 +00:00
Richard van der Hoff 4febfe47f0 Comments
Update comments in verify_macaroon
2016-11-30 07:36:32 +00:00
Richard van der Hoff 77eca2487c Merge pull request #1653 from matrix-org/rav/guest_e2e
Implement E2E for guests
2016-11-29 17:41:35 +00:00
Richard van der Hoff 1c4f05db41 Stop putting a time caveat on access tokens
The 'time' caveat on the access tokens was something of a lie, since we weren't
enforcing it; more pertinently its presence stops us ever adding useful time
caveats.

Let's move in the right direction by not lying in our caveats.
2016-11-29 16:49:41 +00:00
Richard van der Hoff 5c4edc83b5 Stop generating refresh tokens
Since we're not doing refresh tokens any more, we should start killing off the
dead code paths. /tokenrefresh itself is a bit of a thornier subject, since
there might be apps out there using it, but we can at least not generate
refresh tokens on new logins.
2016-11-28 10:13:01 +00:00
Richard van der Hoff b6146537d2 Merge pull request #1655 from matrix-org/rav/remove_redundant_macaroon_checks
Remove redundant list of known caveat prefixes
2016-11-25 16:57:19 +00:00
Richard van der Hoff f62b69e32a Allow guest access to endpoints for E2E
Expose /devices, /keys, and /sendToDevice to guest users, so that they can use
E2E.
2016-11-25 15:26:34 +00:00
Richard van der Hoff 7f02e4d008 Give guest users a device_id
We need to create devices for guests so that they can use e2e, but we don't
have anywhere to store it, so just use a fixed one.
2016-11-25 15:25:30 +00:00
Erik Johnston 11bfe438a2 Use correct var 2016-11-24 15:26:53 +00:00
Erik Johnston aaecffba3a Correctly handle 500's and 429 on federation 2016-11-24 15:04:49 +00:00
Richard van der Hoff e1d7c96814 Remove redundant list of known caveat prefixes
Also add some comments.
2016-11-24 12:38:17 +00:00
Erik Johnston 7e03f9a484 Bump version and changelog 2016-11-24 12:29:58 +00:00
Erik Johnston 46ca345b06 Don't send old events as federation 2016-11-24 12:29:02 +00:00
Erik Johnston f36ea03741 Bump changelog and version 2016-11-24 11:08:01 +00:00
David Baker c9d4e7b716 Clarify that creds doesn not contain passwords. 2016-11-24 10:54:59 +00:00
David Baker f681aab895 Log the args that we have on UI auth completion
This will be super helpful for debugging if we have more
registration woes.
2016-11-24 10:11:45 +00:00
Erik Johnston 1985860c6e Comment 2016-11-23 15:59:59 +00:00
Erik Johnston 2ac516850b More efficient notif count queries 2016-11-23 15:57:04 +00:00
Erik Johnston 302fbd218d Merge pull request #1635 from matrix-org/erikj/split_out_fed_txn
Split out federation transaction sending to a worker
2016-11-23 15:39:12 +00:00
Erik Johnston b2d6e63b79 Merge pull request #1641 from matrix-org/erikj/as_pushers
Ignore AS users when fetching push rules
2016-11-23 15:21:52 +00:00
Erik Johnston feec718265 Shuffle receipt handler around so that worker apps don't need to load it 2016-11-23 15:14:24 +00:00
Erik Johnston 26072df6af Ensure only main or federation_sender process can send federation traffic 2016-11-23 14:09:47 +00:00
Erik Johnston b69f76c106 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_fed_txn 2016-11-23 11:31:53 +00:00
Erik Johnston 4d9b5c60f9 Comment 2016-11-23 11:11:41 +00:00
Erik Johnston 0163466d72 Ignore AS users when fetching push rules
By ignoring AS users early on when fetching push rules for a room we can
avoid needlessly hitting the DB and filling up the caches.
2016-11-23 11:01:01 +00:00
Erik Johnston 4c79a63fd7 Explicit federation ack 2016-11-23 10:40:44 +00:00
Erik Johnston 54fed21c04 Fix tests and flake8 2016-11-22 18:18:31 +00:00
Erik Johnston 90565d015e Invalidate retry cache in both directions 2016-11-22 17:45:44 +00:00
Kegan Dougal 83bcdcee61 Return early on /sync code paths if a '*' filter is used
This is currently very conservative in that it only does this if there is no
`since` token. This limits the risk to clients likely to be doing one-off
syncs (like bridges), but does mean that normal human clients won't benefit
from the time savings here. If the savings are large enough, I would consider
generalising this to just check the filter.
2016-11-22 16:38:35 +00:00
Kegsay d4a459f7cb Merge pull request #1638 from matrix-org/kegan/sync-event-fields
Implement "event_fields" in filters
2016-11-22 14:02:38 +00:00
Kegan Dougal c3d963ac24 Review comments 2016-11-22 13:42:11 +00:00
Kegan Dougal 6d4e6d4cba Also check for dict since sometimes they aren't frozen 2016-11-22 10:39:41 +00:00
Erik Johnston baf9e74a73 Merge branch 'master' of github.com:matrix-org/synapse into develop 2016-11-22 10:31:48 +00:00
Kegan Dougal cea4e4e7b2 Glue only_event_fields into the sync rest servlet 2016-11-22 10:14:05 +00:00
Kegan Dougal 0a8b0eeca1 More tests 2016-11-22 09:59:27 +00:00
Erik Johnston 51e89709aa Comments 2016-11-21 17:59:39 +00:00
Kegan Dougal 70a2157b64 Start adding some tests 2016-11-21 17:52:45 +00:00
Kegan Dougal f97511a1f3 Move event_fields filtering to serialize_event
Also make it an inclusive not exclusive filter, as the spec demands.
2016-11-21 17:42:16 +00:00
Erik Johnston 88d85ebae1 Add some metrics 2016-11-21 17:36:05 +00:00
Erik Johnston 50934ce460 Comments 2016-11-21 16:55:23 +00:00
Kegan Dougal e90fcd9edd Add filter_event_fields and filter_field to FilterCollection 2016-11-21 15:18:18 +00:00
Erik Johnston 9687e039e7 Remove explicit calls to send_pdu 2016-11-21 14:48:51 +00:00
Kegan Dougal a2a6c1c22f Fail with a coherent error message if `/sync?filter=` is invalid 2016-11-21 13:15:25 +00:00
Erik Johnston 524d61bf7e Fix tests 2016-11-21 11:53:02 +00:00
Erik Johnston 7c9cdb2245 Store federation stream positions in the database 2016-11-21 11:33:08 +00:00
Mark Haines a289150943 Fix flake8 2016-11-18 17:15:02 +00:00
David Baker 544722bad2 Work around client replacing reg params
Works around https://github.com/vector-im/vector-android/issues/715
and equivalent for iOS
2016-11-18 17:07:35 +00:00
Erik Johnston f8ee66250a Handle sending events and device messages over federation 2016-11-17 15:48:04 +00:00
Erik Johnston ed787cf09e Hook up the send queue and create a federation sender worker 2016-11-16 17:34:44 +00:00
Erik Johnston 1587b5a033 Add initial cut of federation send queue 2016-11-16 14:47:52 +00:00
Erik Johnston 59ef517e6b Use new federation_sender DI 2016-11-16 14:47:52 +00:00
Erik Johnston 847d5db1d1 Add transaction queue and transport layer to DI 2016-11-16 14:47:52 +00:00
Erik Johnston daec6fc355 Move logic into transaction_queue 2016-11-16 14:47:52 +00:00
Erik Johnston 0e830d3770 Rename transaction queue functions to send_* 2016-11-16 14:47:52 +00:00
Erik Johnston dc6cede78e Merge pull request #1628 from matrix-org/erikj/ldap_split_out
Use external ldap auth pacakge
2016-11-15 16:53:34 +00:00
Erik Johnston c7546b3cdb Merge pull request #1617 from matrix-org/erikj/intern_state_dict
Correctly intern keys in state cache
2016-11-15 16:45:55 +00:00
Erik Johnston d56c39cf24 Use external ldap auth pacakge 2016-11-15 13:03:19 +00:00
Erik Johnston f9d156d270 New Flake8 fixes 2016-11-15 11:22:29 +00:00
Erik Johnston 9d58ccc547 Bump changelog and version 2016-11-14 15:05:04 +00:00
Kegsay 9355a5c42b Merge pull request #1624 from matrix-org/kegan/idempotent-requests
Store Promise<Response> instead of Response for HTTP API transactions
2016-11-14 12:45:30 +00:00
Kegan Dougal 3991b4cbdb Clean transactions based on time. Add HttpTransactionCache tests. 2016-11-14 11:19:24 +00:00
Kegan Dougal af4a1bac50 Move .observe() up to the cache to make things neater 2016-11-14 09:52:41 +00:00
Erik Johnston 0964005d84 Merge pull request #1625 from DanielDent/patch-1
Add support for durations in minutes
2016-11-12 11:20:46 +00:00
Daniel Dent 1c93cd9f9f Add support for durations in minutes 2016-11-12 00:10:23 -08:00
Kegan Dougal 8ecaff51a1 Review comments 2016-11-11 17:47:03 +00:00
Kegan Dougal f6c48802f5 More flake8 2016-11-11 15:08:24 +00:00
Kegan Dougal a88bc67f88 Flake8 and fix whoopsie 2016-11-11 15:02:29 +00:00
Kegan Dougal 42c43cfafd Use ObservableDeferreds instead of Deferreds as they behave as intended 2016-11-11 14:54:10 +00:00
Kegan Dougal c7daf3136c Use observable deferreds because they are sane 2016-11-11 14:13:32 +00:00
Erik Johnston 64038b806c Comments 2016-11-11 10:42:08 +00:00
Erik Johnston 2bd4513a4d Limit the number of events that can be created on a given room concurretnly 2016-11-10 16:44:35 +00:00
Erik Johnston d073cb7ead Add Limiter: limit concurrent access to resource 2016-11-10 16:29:51 +00:00
Kegan Dougal 8a8ad46f48 Flake8 2016-11-10 15:22:11 +00:00
Kegan Dougal 2771447c29 Store Promise<Response> instead of Response for HTTP API transactions
This fixes a race whereby:
 - User hits an endpoint.
 - No cached transaction so executes main code.
 - User hits same endpoint.
 - No cache transaction so executes main code.
 - Main code finishes executing and caches response and returns.
 - Main code finishes executing and caches response and returns.

 This race is common in the wild when Synapse is struggling under load.
 This commit fixes the race by:
  - User hits an endpoint.
  - Caches the promise to execute the main code and executes main code.
  - User hits same endpoint.
  - Yields on the same promise as the first request.
  - Main code finishes executing and returns, unblocking both requests.
2016-11-10 14:49:26 +00:00
Erik Johnston ac507e7ab8 Don't assume providers raise ConfigError's 2016-11-08 17:23:28 +00:00
Erik Johnston e6651e8046 Merge branch 'master' of github.com:matrix-org/synapse into develop 2016-11-08 14:43:49 +00:00
Erik Johnston 3c09818d91 Bump version and changelog 2016-11-08 14:39:55 +00:00
Erik Johnston 27d3f2e7ab Explicitly set authentication mode in ldap3
This only makes a difference for versions of ldap3 before 1.0, but a)
its best to be explicit and b) there are distributions that package
ancient versions for ldap3 (e.g. debian).
2016-11-08 14:35:25 +00:00
Erik Johnston 17e0a58020 Merge pull request #1615 from matrix-org/erikj/limit_prev_events
Limit the number of prev_events of new events
2016-11-08 12:06:15 +00:00
Erik Johnston 587d8ac60f Correctly intern keys in state cache 2016-11-08 11:53:25 +00:00
Erik Johnston a4632783fb Sample correctly 2016-11-08 11:20:26 +00:00
Erik Johnston 24772ba56e Respect use_frozen_dicts option in workers 2016-11-08 11:07:18 +00:00
Erik Johnston eeda4e618c Limit the number of prev_events of new events 2016-11-08 11:02:29 +00:00
Erik Johnston d24197bead Merge pull request #1198 from euank/more-ip-blacklist
default config: blacklist more internal ips
2016-11-07 09:41:34 +00:00
Euan Kemp c6bbad109b default config: blacklist more internal ips 2016-11-06 17:02:25 -08:00
Erik Johnston 63772443e6 Comment 2016-11-04 10:53:42 +00:00
Erik Johnston a3f6576084 Remove unused but buggy function 2016-11-04 10:48:20 +00:00
Paul "LeoNerd" Evans 2938a00825 Rename the python-specific metrics now the docs claim that we have done 2016-11-03 17:03:52 +00:00
Paul "LeoNerd" Evans 5219f7e060 Since we don't export per-filetype fd counts any more, delete all the code related to that too 2016-11-03 16:41:32 +00:00
Paul "LeoNerd" Evans 93ebeb2aa8 Remove now-unused 'resource' import 2016-11-03 16:37:09 +00:00
Paul "LeoNerd" Evans c1b077cd19 Now we have new-style metrics don't bother exporting legacy-named process ones 2016-11-03 16:34:16 +00:00
Erik Johnston 64c6566980 Remove spurious comment 2016-11-03 15:04:32 +00:00
Erik Johnston 8fd4d9129f Replace postgres GIN with GIST
This is because GIN can be slow to write too, especially when the table
gets large.
2016-11-03 15:00:03 +00:00
David Baker 9084720993 Don't error on non-ascii passwords 2016-11-03 10:42:14 +00:00
Mark Haines b1c27975d0 Set CORs headers on responses from the media repo 2016-11-02 11:29:25 +00:00
Erik Johnston 2746e805fe Merge pull request #1188 from matrix-org/erikj/sent_transactions
Remove sent_transactions table.
2016-11-01 14:52:55 +00:00
Erik Johnston 3c91c5b216 Bump version and changelog 2016-11-01 13:06:36 +00:00
Erik Johnston 760469c812 Continue to clean up received_transactions 2016-11-01 11:42:08 +00:00
Erik Johnston f09d2b692f Removed unused stuff 2016-10-31 17:10:56 +00:00
Erik Johnston 4c3eb14d68 Increase batching of sent transaction inserts
This should further reduce the number of individual inserts,
transactions and updates that are required for keeping sent_transactions
up to date.
2016-10-31 16:07:45 +00:00
Mark Haines 82e71a259c Bump changelog and version 2016-10-28 11:16:05 +01:00
Mark Haines 490b97d3e7 Merge branch 'develop' into release-v0.18.2 2016-10-28 11:12:31 +01:00
Paul "LeoNerd" Evans 1cc22da600 Set up the process collector during metrics __init__; that way all split-process workers have it 2016-10-27 18:09:34 +01:00
Paul "LeoNerd" Evans aac13b1f9a Pass the Metrics group into the process collector instead of having it find its own one; this avoids it needing to import from synapse.metrics 2016-10-27 18:08:15 +01:00
Paul "LeoNerd" Evans ccc1a3d54d Allow creation of a 'subspace' within a Metrics object, returning another one 2016-10-27 18:07:34 +01:00
Erik Johnston 665e53524e Bump changelog and version 2016-10-27 14:52:47 +01:00
Erik Johnston a9111786f9 Use most recently added binding, not most recently seen user. 2016-10-27 14:32:45 +01:00
Erik Johnston 1fc1bc2a51 Fix user_threepids schema delta
The delta `37/user_threepids.sql` aimed to update all the email
addresses to be lower case, however duplicate emails may exist in the
table already.

This commit adds a step where the delta moves the duplicate emails to a
new `medium` `email_old`. Only the most recently used account keeps the
binding intact. We move rather than delete so that we retain some record
of which emails were associated with which account.
2016-10-27 14:14:44 +01:00
Erik Johnston ab731d8f8e Bump changelog and version 2016-10-27 11:07:02 +01:00
Mark Haines 177f104432 Merge pull request #1098 from matrix-org/markjh/bearer_token
Allow clients to supply access_tokens as headers
2016-10-25 17:33:15 +01:00
Erik Johnston 22fbf86e4f Merge branch 'release-v0.18.2' of github.com:matrix-org/synapse into develop 2016-10-25 15:09:46 +01:00
Erik Johnston 855645c719 Bump version and changelog 2016-10-25 11:16:16 +01:00
Erik Johnston 2ef617bc06 Fix infinite typing bug
There's a bug somewhere that causes typing notifications to not be timed
out properly. By adding a paranoia timer and using correct inequalities
notifications should stop being stuck, even if it the root cause hasn't
been fixed.
2016-10-24 15:51:22 +01:00
Erik Johnston b6800a8ecd Actually use the new function 2016-10-24 13:39:49 +01:00
Erik Johnston d04e2ff3a4 Fix incredubly slow back pagination query
If a client didn't specify a from token when paginating backwards
synapse would attempt to query the (global) maximum topological token.
This a) doesn't make much sense since they're room specific and b) there
are no indices that lets postgres do this efficiently.
2016-10-24 13:35:51 +01:00
Paul Evans a842fed418 Merge pull request #1177 from matrix-org/paul/standard-metric-names
Standardise prometheus metrics
2016-10-21 13:06:19 +01:00
Luke Barnard e01a1bc92d Merge pull request #1175 from matrix-org/luke/feature-configurable-as-rate-limiting
Allow Configurable Rate Limiting Per AS
2016-10-20 16:21:10 +01:00
Luke Barnard 6fdd31915b Style 2016-10-20 13:53:15 +01:00
Luke Barnard 07caa749bf Closing brace on following line 2016-10-20 12:07:16 +01:00
Luke Barnard f09db236b1 as_user->app_service, less redundant comments, better positioned comments 2016-10-20 12:04:54 +01:00
Luke Barnard 8bfd01f619 flake8 2016-10-20 11:52:46 +01:00
Luke Barnard 1b17d1a106 Use real AS object by passing it through the requester
This means synapse does not have to check if the AS is interested, but instead it effectively re-uses what it already knew about the requesting user
2016-10-20 11:43:05 +01:00
Paul "LeoNerd" Evans b01aaadd48 Split callback metric lambda functions down onto their own lines to keep line lengths under 90 2016-10-19 18:26:13 +01:00
Paul "LeoNerd" Evans 1071c7d963 Adjust code for <100 char line limit 2016-10-19 18:23:25 +01:00
Paul "LeoNerd" Evans 6453d03edd Cut the raw /proc/self/stat line up into named fields at collection time 2016-10-19 18:21:40 +01:00
Paul "LeoNerd" Evans 3ae48a1f99 Move the process metrics collector code into its own file 2016-10-19 18:10:24 +01:00
Paul "LeoNerd" Evans 4cedd53224 A slightly neater way to manage metric collector functions 2016-10-19 17:54:09 +01:00
Paul "LeoNerd" Evans 5663137e03 appease pep8 2016-10-19 16:09:42 +01:00
Paul "LeoNerd" Evans b202531be6 Also guard /proc/self/fds-related code with a suitable psuedoconstant 2016-10-19 15:37:41 +01:00
Paul "LeoNerd" Evans 1b179455fc Guard registration of process-wide metrics by existence of the requisite /proc entries 2016-10-19 15:34:38 +01:00
Paul "LeoNerd" Evans 981f852d54 Add standard process_start_time_seconds metric 2016-10-19 15:05:22 +01:00
Paul "LeoNerd" Evans def63649df Add standard process_max_fds metric 2016-10-19 15:05:21 +01:00
Paul "LeoNerd" Evans 06f1ad1625 Add standard process_open_fds metric 2016-10-19 15:05:21 +01:00
Paul "LeoNerd" Evans 95fc70216d Add standard process_*_memory_bytes metrics 2016-10-19 15:05:21 +01:00
Paul "LeoNerd" Evans 9b0316c75a Use /proc/self/stat to generate the new process_cpu_*_seconds_total metrics 2016-10-19 15:05:21 +01:00
Paul "LeoNerd" Evans 03c2720940 Export CPU usage metrics also under prometheus-standard metric name 2016-10-19 15:05:21 +01:00
Paul "LeoNerd" Evans b21b9dbc37 Callback metric values might not just be integers - allow floats 2016-10-19 15:05:15 +01:00
Erik Johnston 78c083f159 Merge pull request #1164 from pik/error-codes
Clarify Error codes for GET /filter/
2016-10-19 14:26:17 +01:00
Erik Johnston 3aa8925091 Merge pull request #1176 from matrix-org/erikj/eager_ratelimit_check
Check whether to ratelimit sooner to avoid work
2016-10-19 14:25:52 +01:00
Erik Johnston f2f74ffce6 Comment 2016-10-19 14:21:28 +01:00
David Baker 0108ed8ae6 Latest delta is now 37 2016-10-19 11:40:35 +01:00
David Baker a7f48320b1 Merge remote-tracking branch 'origin/develop' into dbkr/password_reset_case_insensitive 2016-10-19 11:28:56 +01:00
David Baker df2a616c7b Convert emails to lowercase when storing
And db migration sql to convert existing addresses.
2016-10-19 11:13:55 +01:00
Erik Johnston 550308c7a1 Check whether to ratelimit sooner to avoid work 2016-10-19 10:45:24 +01:00
pik e8b1d2a452 Refactor test_filter to use real DataStore
* add tests for filter api errors
2016-10-18 12:17:38 -05:00
Luke Barnard 5b54d51d1e Allow Configurable Rate Limiting Per AS
This adds a flag loaded from the registration file of an AS that will determine whether or not its users are rate limited (by ratelimit in _base.py). Needed for IRC bridge reasons - see https://github.com/matrix-org/matrix-appservice-irc/issues/240.
2016-10-18 17:04:09 +01:00
Erik Johnston 8ca05b5755 Fix push notifications for a single unread message 2016-10-18 10:57:33 +01:00
Erik Johnston f0ca088280 Reduce redundant database work in email pusher
Update the last stream ordering if the
`get_unread_push_actions_for_user_in_range_for_email` returns no new
push actions. This reduces the range that it needs to check next
iteration.
2016-10-18 10:52:47 +01:00
Erik Johnston b95dbdcba4 Bump version and changelog 2016-10-17 15:59:12 +01:00
Erik Johnston 6942d68247 Bump schema version 2016-10-17 11:17:45 +01:00
Erik Johnston b59994b454 Remove TODO 2016-10-17 11:17:02 +01:00
Erik Johnston 816988baaa Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_auth 2016-10-17 11:10:37 +01:00
Erik Johnston 2869a29fd7 Drop some unused indices 2016-10-17 11:08:19 +01:00
Erik Johnston a68ade6ed3 Merge pull request #1162 from larroy/master
Use sys.executable instead of hardcoded python. fixes #1161
2016-10-14 21:42:55 +01:00
David Baker 29c5922021 Revert part of 6207399
older sqlite doesn't support indexes on expressions, lets just
store things lowercase in the db
2016-10-14 16:20:24 +01:00
Alexander Maznev d9350b0db8 Error codes for filters
* add tests

Signed-off-by: Alexander Maznev <alexander.maznev@gmail.com>
2016-10-14 10:18:28 -05:00
David Baker bcb1245a2d Merge remote-tracking branch 'origin/develop' into dbkr/password_reset_case_insensitive 2016-10-14 15:10:38 +01:00
David Baker 62073992c5 Make password reset email field case insensitive 2016-10-14 13:56:53 +01:00
Erik Johnston 6f7540ada4 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fix_email_notifs 2016-10-14 10:22:43 +01:00
Erik Johnston 1d107d8484 Fix email push notifs being dropped
A lot of email push notifications were failing to be sent due to an
exception being thrown along one of the (many) paths. This was due to a
change where we moved from pulling out the full state for each room, but
rather pulled out the event ids for the state and separately loaded the
full events when needed.
2016-10-13 13:40:38 +01:00
Richard van der Hoff f7aed3d7a2 Merge pull request #1168 from matrix-org/rav/ui_auth_on_device_delete
User-interactive auth on delete device
2016-10-13 09:38:41 +01:00
Richard van der Hoff 9009143fb9 Handle delete device requests with no body
We should probably return a 401 rather than a 400 for existing clients that
don't know they have to do the UIA dance to delete a device.
2016-10-12 18:47:28 +01:00
Richard van der Hoff fbd3866bc6 User-interactive auth on delete device 2016-10-12 16:16:31 +01:00
Mark Haines 9e18e0b1cb Merge pull request #1167 from matrix-org/markjh/fingerprints
Add config option for adding additional TLS fingerprints
2016-10-12 15:27:44 +01:00
Mark Haines c61ddeedac Explain how long the servers can cache the TLS fingerprints for 2016-10-12 14:48:24 +01:00
Mark Haines 0af6213019 Improve comment formatting 2016-10-12 14:45:13 +01:00
Erik Johnston 35e2cc8b52 Merge pull request #1155 from matrix-org/erikj/pluggable_pwd_auth
Implement pluggable password auth
2016-10-12 11:41:20 +01:00
Mark Haines 6e9f3ab415 Add config option for adding additional TLS fingerprints 2016-10-11 19:14:46 +01:00
Erik Johnston 3061dac53e Merge branch 'develop' of github.com:matrix-org/synapse into erikj/replication_noop 2016-10-11 14:08:29 +01:00
Erik Johnston 668f91d707 Fix check of wrong variable 2016-10-11 13:57:22 +01:00
Richard van der Hoff 0061e8744f Merge pull request #1166 from matrix-org/rav/grandfather_broken_riot_signup
Work around email-spamming Riot bug
2016-10-11 11:58:58 +01:00
Richard van der Hoff fa74fcf512 Work around email-spamming Riot bug
5d9546f9 introduced a change to synapse behaviour, in that failures in the
interactive-auth process would return the flows and params data as well as an
error code (as specced in https://github.com/matrix-org/matrix-doc/pull/397).

That change exposed a bug in Riot which would make it request a new validation
token (and send a new email) each time it got a 401 with a `flows` parameter
(see https://github.com/vector-im/vector-web/issues/2447 and the fix at
https://github.com/matrix-org/matrix-react-sdk/pull/510).

To preserve compatibility with broken versions of Riot, grandfather in the old
behaviour for the email validation stage.
2016-10-11 11:34:40 +01:00
Erik Johnston a2f2516199 Merge pull request #1157 from Rugvip/nolimit
Remove rate limiting from app service senders and fix get_or_create_user requester
2016-10-11 11:20:54 +01:00
Erik Johnston a940618c94 Merge pull request #1150 from Rugvip/state_key
api/auth: fix for not being allowed to set your own state_key
2016-10-11 11:19:55 +01:00
Pedro Larroy c57f871184 Use sys.executable instead of hardcoded python. fixes #1161 2016-10-08 23:55:20 +02:00
Richard van der Hoff 8681aff4f1 Merge pull request #1160 from matrix-org/rav/401_on_password_fail
Interactive Auth: Return 401 from for incorrect password
2016-10-07 10:57:43 +01:00
Richard van der Hoff 5d9546f9f4 Interactive Auth: Return 401 from for incorrect password
This requires a bit of fettling, because I want to return a helpful error
message too but we don't want to distinguish between unknown user and invalid
password. To avoid hardcoding the error message into 15 places in the code,
I've had to refactor a few methods to return None instead of throwing.

Fixes https://matrix.org/jira/browse/SYN-744
2016-10-07 00:00:00 +01:00
Patrik Oldsberg 7b5546d077 rest/client/v1/register: use the correct requester in createUser
Signed-off-by: Patrik Oldsberg <patrik.oldsberg@ericsson.com>
2016-10-06 22:12:32 +02:00
Richard van der Hoff 5d34e32d42 Merge pull request #1159 from matrix-org/rav/uia_fallback_postmessage
window.postmessage for Interactive Auth fallback
2016-10-06 19:56:43 +01:00
Richard van der Hoff f382117852 window.postmessage for Interactive Auth fallback
If you're a webapp running the fallback in an iframe, you can't set set a
window.onAuthDone function. Let's post a message back to window.opener instead.
2016-10-06 18:16:59 +01:00
Patrik Oldsberg 3de7c8a4d0 handlers/profile: added admin override for set_displayname and set_avatar_url
Signed-off-by: Patrik Oldsberg <patrik.oldsberg@ericsson.com>
2016-10-06 15:24:59 +02:00
Patrik Oldsberg 2ff2d36b80 handers: do not ratelimit app service senders
Signed-off-by: Patrik Oldsberg <patrik.oldsberg@ericsson.com>
2016-10-06 15:24:59 +02:00
Patrik Oldsberg 9bfc617791 storage/appservice: make appservice methods only relying on the cache synchronous 2016-10-06 15:24:59 +02:00
Erik Johnston 4285be791d Bump changelog and version 2016-10-05 14:40:38 +01:00
Erik Johnston 850b103b36 Implement pluggable password auth
Allows delegating the password auth to an external module. This also
moves the LDAP auth to using this system, allowing it to be removed from
the synapse tree entirely in the future.
2016-10-03 10:36:40 +01:00
Patrik Oldsberg 24a70e19c7 api/auth: fix for not being allowed to set your own state_key
Signed-off-by: Patrik Oldsberg <patrik.oldsberg@ericsson.com>
2016-09-30 13:08:25 +02:00
Erik Johnston 04aa2f2863 Bump version and changelog 2016-09-30 10:34:57 +01:00
Martin Weinelt 3027ea22b0 Restructure ldap authentication
- properly parse return values of ldap bind() calls
- externalize authentication methods
- change control flow to be more error-resilient
- unbind ldap connections in many places
- improve log messages and loglevels
2016-09-29 15:30:08 +01:00
Erik Johnston 5875a65253 Merge pull request #1145 from matrix-org/erikj/fix_reindex
Fix background reindex of origin_server_ts
2016-09-29 13:53:48 +01:00
Erik Johnston 9040c9ffa1 Fix background reindex of origin_server_ts
The storage function `_get_events_txn` was removed everywhere except
from this background reindex. The function was removed due to it being
(almost) completely unused while also being large and complex.
Therefore, instead of resurrecting `_get_events_txn` we manually
reimplement the bits that are needed directly.
2016-09-27 11:23:49 +01:00
Erik Johnston 4974147aa3 Remove duplication 2016-09-27 09:27:54 +01:00
Erik Johnston 13122e5e24 Remove unused variable 2016-09-27 09:21:51 +01:00
Erik Johnston cf3e1cc200 Fix perf of fetching state in SQLite 2016-09-26 17:16:24 +01:00
Erik Johnston a38d46249e Merge pull request #1140 from matrix-org/erikj/typing_fed_timeout
Time out typing over federation
2016-09-26 11:24:14 +01:00
Erik Johnston 748d8fdc7b Reduce DB hits for replication
Some streams will occaisonally advance their positions without actually
having any new rows to send over federation. Currently this means that
the token will not advance on the workers, leading to them repeatedly
sending a slightly out of date token. This in turns requires the master
to hit the DB to check if there are any new rows, rather than hitting
the no op logic where we check if the given token matches the current
token.

This commit changes the API to always return an entry if the position
for a stream has changed, allowing workers to advance their tokens
correctly.
2016-09-23 16:49:21 +01:00
Erik Johnston 655891d179 Move FEDERATION_PING_INTERVAL timer. Update log line 2016-09-23 15:43:34 +01:00
Erik Johnston 4225a97f4e Merge branch 'master' of github.com:matrix-org/synapse into develop 2016-09-23 15:36:59 +01:00
Erik Johnston 22578545a0 Time out typing over federation 2016-09-23 14:00:52 +01:00
Erik Johnston 667fcd54e8 Merge pull request #1136 from matrix-org/erikj/fix_signed_3pid
Allow invites via 3pid to bypass sender sig check
2016-09-22 13:41:49 +01:00
Erik Johnston f96020550f Update comments 2016-09-22 12:54:22 +01:00
Erik Johnston 81964aeb90 Merge pull request #1132 from matrix-org/erikj/initial_sync_split
Support /initialSync in synchrotron worker
2016-09-22 12:45:02 +01:00
Erik Johnston 2e9ee30969 Add comments 2016-09-22 11:59:46 +01:00
Erik Johnston a61e4522b5 Shuffle things around to make unit tests work 2016-09-22 11:08:12 +01:00
Erik Johnston 1168cbd54d Allow invites via 3pid to bypass sender sig check
When a server sends a third party invite another server may be the one
that the inviting user registers with. In this case it is that remote
server that will issue an actual invitation, and wants to do it "in the
name of" the original invitee. However, the new proper invite will not
be signed by the original server, and thus other servers would reject
the invite if it was seen as coming from the original user.

To fix this, a special case has been added to the auth rules whereby
another server can send an invite "in the name of" another server's
user, so long as that user had previously issued a third party invite
that is now being accepted.
2016-09-22 10:56:53 +01:00
Erik Johnston bbc0d9617f Merge pull request #1134 from matrix-org/erikj/fix_stream_public_deletion
Fix _delete_old_forward_extrem_cache query
2016-09-21 17:04:04 +01:00
Erik Johnston 8009d84364 Match against event_id, rather than room_id 2016-09-21 16:46:59 +01:00
Erik Johnston dc692556d6 Remove spurious AS clause 2016-09-21 16:28:47 +01:00
Erik Johnston dc78db8c56 Update correct table 2016-09-21 15:52:44 +01:00
Erik Johnston 4f78108d8c Readd entries to public_room_list_stream that were deleted 2016-09-21 15:24:22 +01:00
Erik Johnston 0b78d8adf2 Fix _delete_old_forward_extrem_cache query 2016-09-21 15:20:56 +01:00
Erik Johnston 90c070c850 Add total_room_count_estimate to /publicRooms 2016-09-21 13:30:05 +01:00
Erik Johnston 87528f0756 Support /initialSync in synchrotron worker 2016-09-21 11:46:28 +01:00
Erik Johnston 2b8ff4659f Bump version and changelog 2016-09-19 17:16:56 +01:00
Erik Johnston ddfcdd4778 Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.18.0 2016-09-19 17:15:24 +01:00
Erik Johnston 6f0c5e5d9b Merge pull request #1131 from matrix-org/matthew/e2e-notifs
Notify on e2e events
2016-09-19 10:50:26 +01:00
Erik Johnston 49cf205dc7 _id field must uniquely identify different conditions 2016-09-19 10:34:01 +01:00
Erik Johnston 39af634dd2 Merge pull request #1130 from matrix-org/erikj/fix_pubroom_pag
Handle fact that _generate_room_entry may not return a room entry
2016-09-19 10:13:59 +01:00
Matthew Hodgson 3f6ec271ba proposal for notifying on e2e events 2016-09-17 22:05:06 +01:00
Erik Johnston 4d49e0bdfd PEP8 2016-09-17 18:09:22 +01:00
Erik Johnston 81570abfb2 Handle fact that _generate_room_entry may not return a room entry 2016-09-17 18:01:54 +01:00
Erik Johnston ddc89df89d Enable guest access to POST /publicRooms 2016-09-17 15:55:24 +01:00
Erik Johnston eb24aecf8c Merge pull request #1129 from matrix-org/erikj/fix_pubroom_pag
Fix and clean up publicRooms pagination
2016-09-17 15:30:34 +01:00
Erik Johnston e1ba98d724 Merge pull request #1127 from matrix-org/dbkr/publicroom_search_case_insensitive
Make public room search case insensitive
2016-09-17 15:01:17 +01:00
Erik Johnston a298331de4 Spelling 2016-09-17 14:59:40 +01:00
Erik Johnston 71edaae981 Fix and clean up publicRooms pagination 2016-09-17 14:46:19 +01:00
Matthew Hodgson 883df2e983 fix logger for client_reader worker 2016-09-17 14:12:04 +01:00
David Baker 5336acd46f Make public room search case insensitive 2016-09-16 19:02:42 +01:00
Erik Johnston 19fe990476 Update changelog and bump version 2016-09-16 17:30:59 +01:00
Erik Johnston 995f2f032f Fix public room pagination for client_reader app 2016-09-16 14:48:21 +01:00
Erik Johnston a68807d426 Comment 2016-09-16 11:36:20 +01:00
Erik Johnston 2e67cabd7f Make POST /publicRooms require auth 2016-09-16 11:32:51 +01:00
Erik Johnston b7b62bf9ea Comment 2016-09-16 11:00:29 +01:00
Erik Johnston d84319ae10 Add remote reoom cache 2016-09-16 10:31:59 +01:00
Erik Johnston 23b6701a28 Support filtering remote room lists 2016-09-16 10:24:15 +01:00
Erik Johnston e58a9d781c Filter remote rooms lists locally 2016-09-16 10:19:32 +01:00
Erik Johnston 74d4cdee25 Don't cache searched in /publicRooms 2016-09-16 09:05:11 +01:00
Erik Johnston 418bcd4309 Add new storage function to slave store 2016-09-16 08:37:39 +01:00
Erik Johnston 098db4aa52 Add very basic filter API to /publicRooms 2016-09-15 17:50:16 +01:00
Erik Johnston c33b25fd8d Change the way we calculate new_limit in /publicRooms and add POST API 2016-09-15 17:35:20 +01:00
Erik Johnston de4f798f01 Handling expiring stream extrems correctly. 2016-09-15 17:34:59 +01:00
Erik Johnston ea6dc356b0 Merge pull request #1125 from matrix-org/erikj/public_room_cache
Change get_pos_of_last_change to return upper bound
2016-09-15 15:48:53 +01:00
Erik Johnston 955f34d23e Change get_pos_of_last_change to return upper bound 2016-09-15 15:12:07 +01:00
Erik Johnston 241d7d2d62 Merge pull request #1124 from matrix-org/erikj/enable_state_caching_workers
Enable state caches on workers
2016-09-15 15:01:47 +01:00
Erik Johnston 4be85281f9 Enable state caches on workers 2016-09-15 14:31:22 +01:00
Erik Johnston cb3edec6af Use stream_change cache to make get_forward_extremeties_for_room cache more effective 2016-09-15 14:28:13 +01:00
Erik Johnston 55e6fc917c Add cache to get_forward_extremeties_for_room 2016-09-15 14:04:28 +01:00
Erik Johnston 68c1ed4d1a Remove default public rooms limit 2016-09-15 13:56:20 +01:00
Erik Johnston b82fa849c8 Merge pull request #1120 from matrix-org/erikj/push_invite_cache
Ensure we don't mutate the cache of push rules
2016-09-15 13:27:18 +01:00
Erik Johnston e457034e99 Merge pull request #1121 from matrix-org/erikj/public_room_paginate
Add pagination support to publicRooms
2016-09-15 13:27:09 +01:00
Erik Johnston 1d98cf26be By default limit /publicRooms to 100 entries 2016-09-15 13:18:35 +01:00
Erik Johnston 211786ecd6 Stream public room changes down replication 2016-09-15 11:47:23 +01:00
Erik Johnston 4fb65a1091 Base public room list off of public_rooms stream 2016-09-15 11:27:04 +01:00
Erik Johnston 5810cffd33 Pass since/from parameters over federation 2016-09-15 10:36:19 +01:00
Erik Johnston f3eead0660 Allow paginating both forwards and backwards 2016-09-15 10:15:37 +01:00
Erik Johnston 4131381123 Remove support for aggregate room lists 2016-09-15 09:28:15 +01:00
Erik Johnston 6a5ded5988 Ensure we don't mutate the cache of push rules 2016-09-15 09:16:13 +01:00
Erik Johnston 4f181f361d Accept optional token to public room list 2016-09-15 09:08:57 +01:00
Erik Johnston c566f0ee17 Calculate the public room list from a stream_ordering 2016-09-14 17:42:47 +01:00
Erik Johnston 772c6067a3 Refactor public rooms to not pull out the full state for each room 2016-09-14 17:29:25 +01:00
Erik Johnston baffe96d95 Add a room visibility stream 2016-09-14 17:29:19 +01:00
Erik Johnston 264a48aedf Merge pull request #1117 from matrix-org/erikj/fix_state
Ensure we don't mutate state cache entries
2016-09-14 16:50:37 +01:00
Erik Johnston ed992ae6ba Add a DB index to figure out past state at a stream ordering in a room 2016-09-14 16:20:27 +01:00
Erik Johnston e0b6db29ed Split out public room list into a worker process 2016-09-14 14:42:51 +01:00
Erik Johnston a70a43bc51 Move RoomListHandler into a separate file 2016-09-14 14:07:37 +01:00
Erik Johnston f2b2cd8eb4 Amalgamate two identical consecutive if statements 2016-09-14 11:16:22 +01:00
Erik Johnston 00f51493f5 Fix reindex 2016-09-14 10:18:30 +01:00
Erik Johnston d5ae1f1291 Ensure we don't mutate state cache entries 2016-09-14 10:03:48 +01:00
Matthew Hodgson 1b01488d27 Merge pull request #1111 from matrix-org/matthew/device-ids
make device IDs more useful for human disambiguation
2016-09-14 01:18:02 +01:00
Paul Evans 0f73f0e70e Merge pull request #1116 from matrix-org/paul/tiny-fixes
Fix typo "persiting"
2016-09-13 13:27:42 +01:00
Paul "LeoNerd" Evans ca35e54d6b Fix typo "persiting" 2016-09-13 13:26:33 +01:00
Mark Haines ad816b0add Limit how often we ask for keys from dead servers 2016-09-13 11:53:50 +01:00
Erik Johnston 43253c10b8 Remove redundant event_auth index 2016-09-13 11:47:48 +01:00
Mark Haines 18ab019a4a Move the E2E key handling into the e2e handler 2016-09-13 11:35:35 +01:00
Mark Haines 76b09c29b0 Merge pull request #1110 from matrix-org/markjh/e2e_timeout
Add a timeout parameter for end2end key queries.
2016-09-13 10:50:45 +01:00
Erik Johnston ba6bc2faa0 Merge pull request #1109 from matrix-org/erikj/partial_indices
Add WHERE clause support to index creation
2016-09-13 09:06:16 +01:00
Matthew Hodgson edbcb4152b make device IDs more useful for human disambiguation 2016-09-13 00:02:39 +01:00
Mark Haines 949c2c5435 Add a timeout parameter for end2end key queries.
Add a timeout parameter for controlling how long synapse will wait
for responses from remote servers. For servers that fail include how
they failed to make it easier to debug.

Fetch keys from different servers in parallel rather than in series.

Set the default timeout to 10s.
2016-09-12 18:17:09 +01:00
Erik Johnston b17af156c7 Remove where clause 2016-09-12 17:05:54 +01:00
Erik Johnston 1c9da43a95 Merge pull request #1108 from matrix-org/erikj/create_dm
Add is_direct param to /createRoom
2016-09-12 16:57:16 +01:00
Erik Johnston 0b32bb20bb Index contains_url for file search queries 2016-09-12 16:57:05 +01:00
Erik Johnston c94de0ab60 Add WHERE clause support to index creation 2016-09-12 16:55:01 +01:00
Erik Johnston 502c901e11 Merge pull request #1107 from matrix-org/erikj/backfill_none
Fix backfill when cannot find an event.
2016-09-12 16:48:01 +01:00
Erik Johnston 48a5a7552d Add is_direct param to /createRoom 2016-09-12 16:34:20 +01:00
Erik Johnston 706b5d76ed Fix backfill when cannot find an event.
`get_pdu` can succeed but return None.
2016-09-12 14:59:51 +01:00
Erik Johnston 7c679b1118 Merge pull request #1106 from matrix-org/erikj/state_reindex_concurrent
Create new index concurrently
2016-09-12 14:38:40 +01:00
Erik Johnston d080b3425c Merge pull request #1105 from matrix-org/erikj/make_notif_highlight_query_fast
Optimise /notifications query
2016-09-12 14:34:12 +01:00
Erik Johnston 03a98aff3c Create new index concurrently 2016-09-12 14:27:01 +01:00
Erik Johnston fa20c9ce94 Change the index to be stream_ordering, highlight 2016-09-12 14:04:08 +01:00
Erik Johnston 5ef5435529 Remove unused import 2016-09-12 13:32:58 +01:00
Mark Haines aa7b890cfe Merge pull request #1104 from matrix-org/markjh/direct_to_device_federation_sync
Fix direct to device messages recieved over federation to notify sync
2016-09-12 13:25:23 +01:00
Erik Johnston 7cd6edb947 Use register_background_index_update 2016-09-12 12:54:48 +01:00
Erik Johnston 0294c14ec4 Add back in query change 2016-09-12 12:43:56 +01:00
Erik Johnston 7fe42cf949 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/make_notif_highlight_query_fast 2016-09-12 12:37:09 +01:00
Erik Johnston 15ca0c6a4d Make reindex happen in bg 2016-09-12 12:36:36 +01:00
Mark Haines a232e06100 Fix direct to device messages recieved over federation to notify sync 2016-09-12 12:30:46 +01:00
Mark Haines 4a32d25d4c Merge branch 'develop' into markjh/bearer_token 2016-09-12 11:14:56 +01:00
Mark Haines 31f85f9db9 Add comments to existing schema deltas that used "CREATE INDEX" directly 2016-09-12 11:00:26 +01:00
Mark Haines ec609f8094 Fix unit tests 2016-09-12 10:46:02 +01:00
Erik Johnston 54417999b6 Revert "Add index to event_push_actions" 2016-09-12 10:39:55 +01:00
Erik Johnston 45dc260060 Merge pull request #1101 from matrix-org/erikj/state_types_idx
Change state fetch query for postgres to be faster
2016-09-12 10:20:38 +01:00
Erik Johnston d1c217c823 Merge pull request #1097 from matrix-org/erikj/replication_typing_rest
Correctly handle typing stream id resetting
2016-09-12 10:10:15 +01:00
Erik Johnston 897d57bc58 Change state fetch query for postgres to be faster
It turns out that postgres doesn't like doing a list of OR's and is
about 1000x slower, so we just issue a query for each specific type
seperately.
2016-09-12 10:05:07 +01:00
Erik Johnston 555460ae1b Merge pull request #1095 from matrix-org/erikj/batch_edus
Clobber EDUs in send queue
2016-09-12 08:04:15 +01:00
Richard van der Hoff 4162f820ff Merge pull request #1100 from VShell/fix-cas
Conform better to the CAS protocol specification
2016-09-09 21:57:42 +01:00
Shell Turner 29205e9596 Conform better to the CAS protocol specification
Redirect to CAS's /login endpoint properly, and
don't require an <attributes> element.

Signed-off-by: Shell Turner <cam.turn@gmail.com>
2016-09-09 21:20:14 +01:00
David Baker b91e2833b3 Merge remote-tracking branch 'origin/develop' into dbkr/make_notif_highlight_query_fast 2016-09-09 19:11:34 +01:00
David Baker f2acc3dcf9 Add index to event_push_actions
and remove room_id caluse so it uses it

Mostly from @negativemjark
2016-09-09 18:54:54 +01:00
Mark Haines 3ddec016ff Merge branch 'develop' into markjh/bearer_token 2016-09-09 18:51:22 +01:00
Mark Haines 8e01263587 Allow clients to supply access_tokens as headers
Clients can continue to supply access tokens as query parameters
or can supply the token as a header:

   Authorization: Bearer <access_token_goes_here>

This matches the ouath2 format of
https://tools.ietf.org/html/rfc6750#section-2.1
2016-09-09 18:17:42 +01:00
Erik Johnston 3265def8c7 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/batch_edus 2016-09-09 18:06:01 +01:00
Erik Johnston af4701b311 Fix incorrect attribute name 2016-09-09 17:36:56 +01:00
Erik Johnston 44330a21e9 Comment 2016-09-09 17:22:07 +01:00
Erik Johnston 464ffd1b5e Comment 2016-09-09 17:17:23 +01:00
Erik Johnston 327425764e Add edu.type as part of key. Remove debug logging 2016-09-09 17:13:30 +01:00
Mark Haines dbff7e9436 Merge pull request #1096 from matrix-org/markjh/get_access_token
Add helper function for getting access_tokens from requests
2016-09-09 17:09:27 +01:00
Erik Johnston a4339de9de Correctly handle typing stream id resetting 2016-09-09 16:44:26 +01:00
Mark Haines 8aee5aa068 Add helper function for getting access_tokens from requests
Rather than reimplementing the token parsing in the various places.
This will make it easier to change the token parsing to allow access_tokens
in HTTP headers.
2016-09-09 16:33:15 +01:00
Erik Johnston 52b2318777 Clobber EDUs in send queue 2016-09-09 15:59:08 +01:00
Paul Evans 56f38d1776 Merge pull request #1091 from matrix-org/paul/third-party-lookup
Improvements to 3PE lookup API
2016-09-09 15:43:11 +01:00
Paul Evans 8cb252d00c Merge pull request #1094 from matrix-org/paul/get-state-whole-event
Allow clients to ask for the whole of a single state event
2016-09-09 15:26:17 +01:00
Paul "LeoNerd" Evans 776594f99d Log if rejecting 3PE query metadata result due to type check 2016-09-09 15:11:31 +01:00
Paul "LeoNerd" Evans ed44c475d8 Reject malformed 3PE query metadata results earlier in AS API handling code 2016-09-09 15:07:04 +01:00
Erik Johnston ab80d5e0a9 Drop replication log levels 2016-09-09 14:56:50 +01:00
Paul "LeoNerd" Evans f25d74f69c Minor fixes from PR comments 2016-09-09 14:54:16 +01:00
Paul "LeoNerd" Evans d271383e63 Filter returned events for client-facing format 2016-09-09 14:40:15 +01:00
Paul "LeoNerd" Evans 0fc0a3bdff Allow clients to specify the format a room state event is returned in 2016-09-09 14:34:29 +01:00
Erik Johnston 6c4d582144 Deduplicate presence in _update_states 2016-09-09 14:28:22 +01:00
Erik Johnston 685da5a3b0 Merge pull request #1092 from matrix-org/erikj/transaction_queue_check
Check if destination is ready for retry earlier
2016-09-09 13:49:56 +01:00
Erik Johnston a6c6750166 Check if destination is ready for retry earlier 2016-09-09 13:46:05 +01:00
Paul "LeoNerd" Evans bdbcfc2a80 appease pep8 2016-09-09 13:31:39 +01:00
Paul "LeoNerd" Evans 6eb0c8a2e4 Python isn't JavaScript; have to quote dict keys 2016-09-09 13:31:17 +01:00
Mark Haines 6b54fa81de Merge pull request #1089 from matrix-org/markjh/direct_to_device_stream
Track the max device stream_id in a separate table,
2016-09-09 13:31:07 +01:00
Paul "LeoNerd" Evans 25eb769b26 Efficiency fix for lookups of a single protocol 2016-09-09 13:25:02 +01:00
Paul "LeoNerd" Evans 3328428d05 Allow lookup of a single 3PE protocol query metadata 2016-09-09 13:19:04 +01:00
Erik Johnston 4598682b43 Fix tightloop on sending transaction 2016-09-09 13:12:53 +01:00
Paul "LeoNerd" Evans 033d43e419 Don't corrupt shared cache on subsequent protocol requests 2016-09-09 13:10:36 +01:00
Mark Haines 647c724573 Use the previous MAX value if any to set the stream_id 2016-09-09 11:52:44 +01:00
Erik Johnston a15ba15e64 Merge pull request #1088 from matrix-org/erikj/transaction_queue_check
Correctly guard against multiple concurrent transactions
2016-09-09 11:49:06 +01:00
Mark Haines 6a6cbfcf1e Track the max_stream_device_id in a separate table, since we delete from the inbox table 2016-09-09 11:48:23 +01:00
Erik Johnston d2688d7f03 Correctly guard against multiple concurrent transactions 2016-09-09 11:44:36 +01:00
Mark Haines 303b6f29f0 Merge pull request #1087 from matrix-org/markjh/reapply_delta
Reapply 34/device_outbox in 35/device_outbox_again.py since the schem…
2016-09-09 11:24:27 +01:00
Erik Johnston 1fe7ca1362 Merge branch 'release-v0.17.3' of github.com:matrix-org/synapse into develop 2016-09-09 11:15:40 +01:00
Erik Johnston 66efcbbff1 Bump changelog and version 2016-09-09 11:07:39 +01:00
Mark Haines 0877157353 Just move the schema and add some DROPs 2016-09-09 11:04:47 +01:00
Erik Johnston 2ffec928e2 Reduce batch size to be under SQL limit 2016-09-09 11:03:31 +01:00
Erik Johnston b390756150 Update last_device_stream_id_by_dest if there is nothing to send 2016-09-09 11:00:15 +01:00
Matthew Hodgson b8f84f99ff Merge pull request #1081 from matrix-org/dbkr/notifications_only_highlight
Implement `only=highlight` on `/notifications`
2016-09-09 00:09:51 +01:00
Mark Haines 43b77c5d97 Only catch databas errors 2016-09-08 17:44:21 +01:00
Paul "LeoNerd" Evans 2f267ee160 Collect up all the "instances" lists of individual AS protocol results into one combined answer to the client 2016-09-08 17:43:53 +01:00
Mark Haines 7d5b142547 Add a stub run_upgrade 2016-09-08 17:39:11 +01:00
David Baker c3276aef25 Merge pull request #1080 from matrix-org/dbkr/fix_notifications_api_with_from
Fix /notifications API when used with `from` param
2016-09-08 17:35:35 +01:00
Mark Haines fa722a699c Reapply 34/device_outbox in 35/device_outbox_again.py since the schema was bumped before it landed on develop 2016-09-08 17:35:16 +01:00
Erik Johnston 5c688739d6 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/reindex_state_groups 2016-09-08 16:52:09 +01:00
Erik Johnston ebb46497ba Add delta file 2016-09-08 16:38:54 +01:00
Mark Haines 91ec972277 Merge pull request #1084 from matrix-org/markjh/direct_to_device_wildcard
Support wildcard device_ids for direct to device messages
2016-09-08 16:28:08 +01:00
Erik Johnston 5beda10bbd Reindex state_groups_state after pruning 2016-09-08 16:18:01 +01:00
Erik Johnston 257025ac89 Merge pull request #1082 from matrix-org/erikj/remote_public_rooms
Add server param to /publicRooms
2016-09-08 16:04:22 +01:00
Erik Johnston 3f9889bfd6 Use parse_string 2016-09-08 15:51:10 +01:00
Erik Johnston caa22334b3 Merge branch 'master' of github.com:matrix-org/synapse into develop 2016-09-08 15:33:00 +01:00
Erik Johnston b152ee71fe Bump version and changelog 2016-09-08 15:23:44 +01:00
Mark Haines a1c8f268e5 Support wildcard device_ids for direct to device messages 2016-09-08 15:13:05 +01:00
Erik Johnston 8b93af662d Check the user_id for presence/typing matches origin 2016-09-08 15:07:38 +01:00
Mark Haines fa9d36e050 Merge branch 'develop' into markjh/direct_to_device_federation 2016-09-08 13:43:43 +01:00
David Baker 4ef222ab61 Implement `only=highlight` on `/notifications` 2016-09-08 13:43:35 +01:00
Erik Johnston 61cd9af09b Log delta files we're applying 2016-09-08 13:40:46 +01:00
Erik Johnston 791658b576 Add server param to /publicRooms 2016-09-08 11:53:05 +01:00
David Baker c5b49eb7ca Fix /notifications API when used with `from` param 2016-09-08 09:40:10 +01:00
Erik Johnston b568ca309c Temporarily disable sequential scans for state fetching 2016-09-08 09:38:54 +01:00
Mark Haines 3c320c006c Merge pull request #1077 from matrix-org/markjh/device_logging
Log the types and values when failing to store devices
2016-09-07 18:24:24 +01:00