0a00b7ff14
Update black, and run auto formatting over the codebase ( #9381 )
...
- Update black version to the latest
- Run black auto formatting over the codebase
- Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md
2021-02-16 22:32:34 +00:00
dd8da8c5f6
Precompute joined hosts and store in Redis ( #9198 )
2021-01-26 13:57:31 +00:00
921a3f8a59
Fix not sending events over federation when using sharded event persisters ( #8536 )
...
* Fix outbound federaion with multiple event persisters.
We incorrectly notified federation senders that the minimum persisted
stream position had advanced when we got an `RDATA` from an event
persister.
Notifying of federation senders already correctly happens in the
notifier, so we just delete the offending line.
* Change some interfaces to use RoomStreamToken.
By enforcing use of `RoomStreamTokens` we make it less likely that
people pass in random ints that they got from somewhere random.
2020-10-14 13:27:51 +01:00
f31f8e6319
Remove stream ordering from Metadata dict ( #8452 )
...
There's no need for it to be in the dict as well as the events table. Instead,
we store it in a separate attribute in the EventInternalMetadata object, and
populate that on load.
This means that we can rely on it being correctly populated for any event which
has been persited to the database.
2020-10-05 14:43:14 +01:00
3bd3707cb9
Fix malformed log line in new federation "catch up" logic ( #8442 )
2020-10-02 11:05:29 +01:00
c1ef579b63
Add prometheus metrics to track federation delays ( #8430 )
...
Add a pair of federation metrics to track the delays in sending PDUs to/from
particular servers.
2020-10-01 11:09:12 +01:00
36efbcaf51
Catch-up after Federation Outage (bonus): Catch-up on Synapse Startup ( #8322 )
...
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix _set_destination_retry_timings
This came about because the code assumed that retry_interval
could not be NULL — which has been challenged by catch-up.
2020-09-18 14:59:13 +01:00
576bc37d31
Catch-up after Federation Outage (split, 4): catch-up loop ( #8272 )
2020-09-15 09:07:19 +01:00
17fa4c7ca7
Catch up after Federation Outage (split, 2): Track last successful stream ordering after transmission ( #8247 )
...
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2020-09-04 15:06:51 +01:00
58f61f10f7
Catch-up after Federation Outage (split, 1) ( #8230 )
...
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2020-09-04 12:22:23 +01:00
c619253db8
Stop sub-classing object ( #8249 )
2020-09-04 06:54:56 -04:00
4535e849d7
Remove obsolete order field in `send_new_transaction` ( #8245 )
...
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2020-09-03 19:23:07 +01:00
5758dcf30c
Add type hints for state. ( #8140 )
2020-08-24 14:25:27 -04:00
eebf52be06
Be stricter about JSON that is accepted by Synapse ( #8106 )
2020-08-19 07:26:03 -04:00
ad6190c925
Convert stream database to async/await. ( #8074 )
2020-08-17 07:24:46 -04:00
ff0e894656
Drop federation transmission queues during a significant remote outage. ( #7864 )
...
* Empty federation transmission queues when we are backing off.
Fixes #7828 .
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Address feedback
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Reword newsfile
2020-08-13 12:35:04 +01:00
9d1e4942ab
Fix typing for notifier ( #8064 )
2020-08-12 14:03:08 +01:00
3aa36b782c
Merge branch 'master' into develop
2020-07-30 15:18:36 +01:00
c978f6c451
Convert federation client to async/await. ( #7975 )
2020-07-30 08:01:33 -04:00
2c1b9d6763
Update worker docs with recent enhancements ( #7969 )
2020-07-29 23:22:13 +01:00
b975fa2e99
Convert state resolution to async/await ( #7942 )
2020-07-24 10:59:51 -04:00
fefe9943ef
Convert presence handler helpers to async/await. ( #7939 )
2020-07-23 16:47:36 -04:00
649a7ead5c
Add ability to run multiple pusher instances ( #7855 )
...
This reuses the same scheme as federation sender sharding
2020-07-16 14:06:28 +01:00
12528dc42f
Remove obsolete comment.
...
It was correct at the time of our friend Jorik writing it (checking
git blame), but the world has moved now and it is no longer a
generator.
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2020-07-16 11:12:48 +01:00
f299441cc6
Add ability to shard the federation sender ( #7798 )
2020-07-10 18:26:36 +01:00
38e1fac886
Fix some spelling mistakes / typos. ( #7811 )
2020-07-09 09:52:58 -04:00
1e03513f9a
Fix new metric where we used ms instead of seconds ( #7771 )
...
Introduced in #7755 , not yet released.
2020-07-01 15:23:58 +01:00
a99658074d
Add some metrics for inbound and outbound federation processing times ( #7755 )
2020-06-30 16:58:06 +01:00
bd6dc17221
Replace iteritems/itervalues/iterkeys with native versions. ( #7692 )
2020-06-15 07:03:36 -04:00
075375bbc9
add a comment
2020-05-21 13:25:41 +01:00
d5aa7d93ed
Fix catchup-on-reconnect for the Federation Stream ( #7374 )
...
looks like we managed to break this during the refactorathon.
2020-05-05 14:15:57 +01:00
4cff617df1
Move catchup of replication streams to worker. ( #7024 )
...
This changes the replication protocol so that the server does not send down `RDATA` for rows that happened before the client connected. Instead, the server will send a `POSITION` and clients then query the database (or master out of band) to get up to date.
2020-03-25 14:54:01 +00:00
b08b0a22d5
Add typing to synapse.federation.sender ( #6871 )
2020-02-07 13:56:38 +00:00
a8a50f5b57
Wake up transaction queue when remote server comes back online ( #6706 )
...
This will be used to retry outbound transactions to a remote server if
we think it might have come back up.
2020-01-17 10:27:19 +00:00
d386f2f339
Add StateMap type alias ( #6715 )
2020-01-16 13:31:22 +00:00
3916e1b97a
Clean up newline quote marks around the codebase ( #6362 )
2019-11-21 12:00:14 +00:00
6f4bc6d01d
Merge branch 'develop' into cross-signing_federation
2019-10-31 22:38:21 -04:00
020add5099
Update black to 19.10b0 ( #6304 )
...
* update version of black and also fix the mypy config being overridden
2019-11-01 02:43:24 +11:00
54fef094b3
Remove usage of deprecated logger.warn method from codebase ( #6271 )
...
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
2019-10-31 10:23:24 +00:00
bb6cec27a5
rename get_devices_by_remote to get_device_updates_by_remote
2019-10-30 14:57:34 -04:00
c40d7244f8
Merge branch 'develop' into cross-signing_federation
2019-10-24 22:31:25 -04:00
8d3542a64e
implement federation parts of cross-signing
2019-10-22 19:04:35 -04:00
c66a06ac6b
Move storage classes into a main "data store".
...
This is in preparation for having multiple data stores that offer
different functionality, e.g. splitting out state or event storage.
2019-10-21 16:05:06 +01:00
66537e10ce
add some metrics on the federation sender ( #6160 )
2019-10-03 17:47:20 +01:00
ef20aa52eb
use access methods (duh..)
...
Co-Authored-By: Erik Johnston <erik@matrix.org>
2019-09-05 15:07:17 +01:00
1d65292e94
Link the send loop with the edus contexts
...
The contexts were being filtered too early so the send loop wasn't
being linked to them unless the destination
was whitelisted.
2019-09-05 14:42:37 +01:00
8767b63a82
Propagate opentracing contexts through EDUs ( #5852 )
...
Propagate opentracing contexts through EDUs
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-08-22 18:21:10 +01:00
4806651744
Replace returnValue with return ( #5736 )
2019-07-23 23:00:55 +10:00
a6a776f3d8
remove dead transaction persist code ( #5622 )
...
this hasn't done anything for years
2019-07-05 12:59:42 +01:00
463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ ( #5606 )
2019-07-04 00:07:04 +10:00
32e7c9e7f2
Run Black. ( #5482 )
2019-06-20 19:32:02 +10:00
b42f90470f
Add experimental option to reduce extremities.
...
Adds new config option `cleanup_extremities_with_dummy_events` which
periodically sends dummy events to rooms with more than 10 extremities.
THIS IS REALLY EXPERIMENTAL.
2019-06-18 15:02:18 +01:00
5c15039e06
Clean up code for sending federation EDUs. ( #5381 )
...
This code confused the hell out of me today. Split _get_new_device_messages
into its two (unrelated) parts.
2019-06-13 13:52:08 +01:00
2d1d7b7e6f
Prevent multiple device list updates from breaking a batch send ( #5156 )
...
fixes #5153
2019-06-06 23:54:00 +01:00
130f932cbc
Run `black` on per_destination_queue
...
... mostly to fix pep8 fails
2019-05-09 16:27:02 +01:00
11ea16777f
Limit the number of EDUs in transactions to 100 as expected by receiver ( #5138 )
...
Fixes #3951 .
2019-05-09 11:01:41 +01:00
197fae1639
Use event streams to calculate presence
...
Primarily this fixes a bug in the handling of remote users joining a
room where the server sent out the presence for all local users in the
room to all servers in the room.
We also change to using the state delta stream, rather than the
distributor, as it will make it easier to split processing out of the
master process (as well as being more flexible).
Finally, when sending presence states to newly joined servers we filter
out old presence states to reduce the number sent. Initially we filter
out states that are offline and have a last active more than a week ago,
though this can be changed down the line.
Fixes #3962
2019-03-27 13:41:36 +00:00
a902d13180
Batch up outgoing read-receipts to reduce federation traffic. ( #4890 )
...
Rate-limit outgoing read-receipts as per #4730 .
2019-03-20 16:02:25 +00:00
02e23b36bc
Rename and move the classes
2019-03-13 20:02:56 +00:00
Eric Eastwood