Commit Graph

42 Commits

Author SHA1 Message Date
Amber Brown 32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Erik Johnston 197fae1639 Use event streams to calculate presence
Primarily this fixes a bug in the handling of remote users joining a
room where the server sent out the presence for all local users in the
room to all servers in the room.

We also change to using the state delta stream, rather than the
distributor, as it will make it easier to split processing out of the
master process (as well as being more flexible).

Finally, when sending presence states to newly joined servers we filter
out old presence states to reduce the number sent. Initially we filter
out states that are offline and have a last active more than a week ago,
though this can be changed down the line.

Fixes #3962
2019-03-27 13:41:36 +00:00
Richard van der Hoff 02e23b36bc Rename and move the classes 2019-03-13 20:02:56 +00:00
Richard van der Hoff fdcad8eabd Move client receipt processing to federation sender worker.
This is mostly a prerequisite for #4730, but also fits with the general theme
of "move everything off the master that we possibly can".
2019-03-13 17:21:19 +00:00
Richard van der Hoff 856c83f5f8
Avoid rebuilding Edu objects in worker mode (#4770)
In worker mode, on the federation sender, when we receive an edu for sending
over the replication socket, it is parsed into an Edu object. There is no point
extracting the contents of it so that we can then immediately build another Edu.
2019-03-04 12:57:44 +00:00
Krombel 79a8a347a6 fix #3445
itervalues(d) calls d.itervalues() [PY2] and d.values() [PY3]
but SortedDict only implements d.values()
2018-08-29 16:28:25 +02:00
Travis Ralston e908b86832 Remove pdu_failures from transactions
The field is never read from, and all the opportunities given to populate it are not utilized. It should be very safe to remove this.
2018-07-30 16:28:47 -06:00
Amber Brown 49af402019 run isort 2018-07-09 16:09:20 +10:00
Ivan Shapovalov c88d50aa8f federation/send_queue.py: fix usage of sortedcontainers.SortedDict 2018-06-06 02:45:18 +03:00
Amber Brown f7869f8f8b
Port to sortedcontainers (with tests!) (#3332) 2018-06-06 00:13:57 +10:00
Ivan Shapovalov 7d9d75e4e8 federation/send_queue.py: fix usage of LaterGauge
Fixes a startup crash due to commit df9f72d9e5
"replacing portions".
2018-06-03 14:16:17 +03:00
Amber Brown c936a52a9e
Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307) 2018-05-31 19:03:47 +10:00
Amber Brown 53cc2cde1f cleanup 2018-05-22 17:32:57 -05:00
Amber Brown df9f72d9e5 replacing portions 2018-05-21 19:47:37 -05:00
Adrian Tschira 36c59ce669 Use six.itervalues in some places
There's more where that came from

Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 20:39:43 +02:00
Richard van der Hoff d3347ad485 Revert "Use sortedcontainers instead of blist"
This reverts commit 9fbe70a7dc.

It turns out that sortedcontainers.SortedDict is not an exact match for
blist.sorteddict; in particular, `popitem()` removes things from the opposite
end of the dict.

This is trivial to fix, but I want to add some unit tests, and potentially some
more thought about it, before we do so.
2018-04-13 11:16:43 +01:00
Vincent Breitmoser 9fbe70a7dc Use sortedcontainers instead of blist
This commit drop-in replaces blist with SortedContainers. They are
written in pure python so work with pypy, but perform as good as
native implementations, at least in a couple benchmarks:

http://www.grantjenks.com/docs/sortedcontainers/performance.html
2018-04-10 11:29:51 +02:00
Erik Johnston 247c736b9b Merge pull request #2115 from matrix-org/erikj/dedupe_federation_repl
Reduce federation replication traffic
2017-04-12 11:07:13 +01:00
Erik Johnston 26ae5178a4 Add some comments 2017-04-12 10:36:29 +01:00
Erik Johnston a8c8e4efd4 Comment 2017-04-11 15:35:49 +01:00
Erik Johnston 2be8a281d2 Comments 2017-04-11 15:28:24 +01:00
Erik Johnston 84fbb80c8f Use generators 2017-04-10 16:55:56 +01:00
Erik Johnston 40453b3f84 Dedupe KeyedEdu and Devices federation repl traffic 2017-04-10 16:49:51 +01:00
Erik Johnston 29574fd5b3 Reduce federation presence replication traffic
This is mainly done by moving the calculation of where to send presence
updates from the presence handler to the transaction queue, so we only
need to send the presence event (and not the destinations) across the
replication connection. Before we were duplicating by sending the full
state across once per destination.
2017-04-10 16:48:30 +01:00
Erik Johnston 0018491af2 Rename variable 2017-04-10 12:44:43 +01:00
Erik Johnston 8c5f03cec7 Revert to sending the same data type as before 2017-04-10 10:07:18 +01:00
Erik Johnston f8434db549 Change name 2017-04-10 10:03:07 +01:00
Erik Johnston ab904caf33 Comments 2017-04-10 10:02:17 +01:00
Erik Johnston a828a64b75 Comment 2017-04-07 11:54:03 +01:00
Erik Johnston d4d176e5d0 Add logging 2017-04-07 11:51:28 +01:00
Erik Johnston 449d1297ca Fix up federation SendQueue and document types 2017-04-07 11:48:33 +01:00
Erik Johnston f10ce8944b Don't double json encode federation replication data 2017-04-05 11:10:28 +01:00
Erik Johnston 11880103b1 Make federation send queue take the current position 2017-03-30 12:54:36 +01:00
Erik Johnston 35b4aa04be Notify on new federation traffic 2017-03-27 14:07:47 +01:00
Erik Johnston 4c79a63fd7 Explicit federation ack 2016-11-23 10:40:44 +00:00
Erik Johnston 51e89709aa Comments 2016-11-21 17:59:39 +00:00
Erik Johnston 88d85ebae1 Add some metrics 2016-11-21 17:36:05 +00:00
Erik Johnston 50934ce460 Comments 2016-11-21 16:55:23 +00:00
Erik Johnston 9687e039e7 Remove explicit calls to send_pdu 2016-11-21 14:48:51 +00:00
Erik Johnston f8ee66250a Handle sending events and device messages over federation 2016-11-17 15:48:04 +00:00
Erik Johnston ed787cf09e Hook up the send queue and create a federation sender worker 2016-11-16 17:34:44 +00:00
Erik Johnston 1587b5a033 Add initial cut of federation send queue 2016-11-16 14:47:52 +00:00