Commit Graph

111 Commits

Author SHA1 Message Date
Richard van der Hoff 52e87fbfbe
Run the AS senders as background processes (#4189)
This should fix some "Starting db connection from sentinel context" warnings,
and will mean we get metrics for these processes.
2018-12-04 10:53:49 +01:00
Amber Brown 2d2828dcbc
Port http/ to Python 3 (#3771) 2018-09-06 00:10:47 +10:00
Amber Brown 49af402019 run isort 2018-07-09 16:09:20 +10:00
Matthew Hodgson f4f1cda928 add ip_range_whitelist parameter to limit where ASes can connect from 2018-06-28 20:32:00 +01:00
Will Hunt 13d211edc1
Merge pull request #3344 from Half-Shot/hs/as-metrics
Add metrics to track appservice transactions
2018-06-07 11:37:12 +01:00
Michael Telatynski 592c162516
also redact __str__ of ApplicationService used for logging 2018-06-06 10:35:29 +01:00
Will Hunt d6e3c2c79b Let's try labels instead of label, that might work 2018-06-05 17:30:45 +01:00
Will Hunt 604cff1a06 Add metrics to track appservice transactions 2018-06-05 13:21:30 +01:00
Richard van der Hoff fc149b4eeb Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background 2018-04-27 14:31:23 +01:00
Richard van der Hoff 2a13af23bc Use run_in_background in preference to preserve_fn
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
2018-04-27 12:55:51 +01:00
Richard van der Hoff 9255a6cb17 Improve exception handling for background processes
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.

This is unsatisfactory for a number of reasons:
 - logging on garbage collection is best-effort and may happen some time after
   the error, if at all
 - it can be hard to figure out where the error actually happened.
 - it is logged as a scary CRITICAL error which (a) I always forget to grep for
   and (b) it's not really CRITICAL if a background process we don't care about
   fails.

So this is an attempt to add exception handling to everything we fire off into
the background.
2018-04-27 11:07:40 +01:00
Richard van der Hoff b78395b7fe Refactor ResponseCache usage
Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a
(get, set) pair, and then use it throughout the codebase.

This will be largely non-functional, but does include the following functional
changes:

* federation_server.on_context_state_request: drops use of _server_linearizer
  which looked redundant and could cause incorrect cache misses by yielding
  between the get and the set.
* RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks
* the wrap function includes some logging. I'm hoping this won't be too noisy
  on production.
2018-04-12 13:02:15 +01:00
Richard van der Hoff b3384232a0 Add metrics for ResponseCache 2018-04-10 23:14:47 +01:00
Adrian Tschira e54c202b81 Replace some type checks with six type checks
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-07 01:02:32 +02:00
Luke Barnard 7fb0e98b03 Extract group_id from the dict for multiple use 2017-11-16 18:18:30 +00:00
Luke Barnard d8391f0541 Remove unused GROUP_ID_REGEX 2017-11-16 18:05:57 +00:00
Luke Barnard 4e8374856d Document get_groups_for_user 2017-11-16 18:03:46 +00:00
Luke Barnard 270f9cd23a Flake8 2017-11-16 18:03:31 +00:00
Luke Barnard 9d83d52027 Use a generator instead of a list 2017-11-16 17:57:34 +00:00
Luke Barnard b1edf26051 Check group_id belongs to this domain 2017-11-16 17:54:27 +00:00
Luke Barnard 97bd18af4e Add automagical AS Publicised Group(s)
via registration file "users" namespace:

```YAML
...
namespaces:
  users:
    - exclusive: true
      regex: '.*luke.*'
      group_id: '+all_the_lukes:hsdomain'
...
```

This is part of giving App Services their own groups for matching users. With this, ghost users will be given the appeareance that they are in a group and that they have publicised the fact, but _only_ from the perspective of the `get_publicised_groups_for_user` API.
2017-11-16 16:44:55 +00:00
Erik Johnston 2a7e9faeec Do logcontexts outside ResponseCache 2017-10-25 15:21:08 +01:00
Richard van der Hoff eaaabc6c4f replace 'except:' with 'except Exception:'
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Erik Johnston 760625acba Make get_if_app_services_interested_in_user faster 2017-05-18 16:34:44 +01:00
Erik Johnston 69efd77749 Add comment 2017-03-29 09:50:05 +01:00
Erik Johnston 51b156d48a Cache whether an AS is interested based on members 2017-03-28 13:27:21 +01:00
Erik Johnston 30f5ffdca2 Remove param and cast at call site 2017-03-28 13:27:21 +01:00
Erik Johnston 650f0e69f2 Compile the regex's used in ASes 2017-03-28 13:27:21 +01:00
Erik Johnston 2bf5a47b3e Rename network_id to instance_id on client side 2016-12-12 16:05:45 +00:00
Erik Johnston 3727d66a0e Don't include appservice id 2016-12-06 17:04:26 +00:00
Erik Johnston f32fb65552 Add new API appservice specific public room list 2016-12-06 16:12:27 +00:00
Luke Barnard 5b54d51d1e Allow Configurable Rate Limiting Per AS
This adds a flag loaded from the registration file of an AS that will determine whether or not its users are rate limited (by ratelimit in _base.py). Needed for IRC bridge reasons - see https://github.com/matrix-org/matrix-appservice-irc/issues/240.
2016-10-18 17:04:09 +01:00
Paul "LeoNerd" Evans 776594f99d Log if rejecting 3PE query metadata result due to type check 2016-09-09 15:11:31 +01:00
Paul "LeoNerd" Evans ed44c475d8 Reject malformed 3PE query metadata results earlier in AS API handling code 2016-09-09 15:07:04 +01:00
Kegan Dougal 572acde483 Use None instead of the empty string
Change how we validate the 'url' field as a result.
2016-08-30 17:16:00 +01:00
Kegan Dougal 16b652f0a3 Flake8 2016-08-30 16:30:12 +01:00
Kegan Dougal e82247f990 Allow application services to have an optional 'url'
If 'url' is not specified, they will not be pushed for events or queries. This
is useful for bots who simply wish to reserve large chunks of user/alias
namespace, and don't care about being pushed for events.
2016-08-30 16:21:16 +01:00
Paul "LeoNerd" Evans 9459137f1e Just sprintf the 'kind' argument into uri directly 2016-08-25 18:35:38 +01:00
Paul "LeoNerd" Evans 1294d4a329 Move ThirdPartyEntityKind into api.constants so the expectation becomes that the value is significant 2016-08-25 18:34:47 +01:00
Paul "LeoNerd" Evans e7af8be5ae Root the 3PE lookup API within /_matrix/app/unstable instead of at toplevel 2016-08-25 18:06:29 +01:00
Paul Evans d9dcb2ba3a Merge pull request #1041 from matrix-org/paul/third-party-lookup
Extend 3PE lookup APIs for metadata query
2016-08-25 17:06:53 +01:00
Paul "LeoNerd" Evans adf53f04ce appease pep8 2016-08-25 16:00:31 +01:00
Paul "LeoNerd" Evans db7283cc6b Implement a ResponseCache around 3PE lookup metadata lookups 2016-08-25 15:56:27 +01:00
Paul "LeoNerd" Evans d0b8d49f71 Kill PROTOCOL_META since I'm not using it any more 2016-08-25 15:45:28 +01:00
Paul "LeoNerd" Evans 5474824975 Actually query over AS API for 3PE lookup metadata 2016-08-25 15:29:36 +01:00
Paul "LeoNerd" Evans 9899824b85 Initial hack at the 3PN protocols metadata lookup API 2016-08-24 12:33:01 +01:00
Erik Johnston 9219139351 Preserve some logcontexts 2016-08-24 11:58:40 +01:00
Paul "LeoNerd" Evans 63c19e1df9 Move 3PU/3PL lookup APIs into /thirdparty containing entity 2016-08-24 11:55:57 +01:00
Paul "LeoNerd" Evans 65201631a4 Move validation logic for AS 3PE query response into ApplicationServiceApi class, to keep the handler logic neater 2016-08-18 17:33:56 +01:00
Paul "LeoNerd" Evans b515f844ee Avoid so much copypasta between 3PU and 3PL query by unifying around a ThirdPartyEntityKind enumeration 2016-08-18 17:19:55 +01:00