hf_text-generation-inference/router/src
OlivierDehaene 757223b352
feat: add SchedulerV3 (#1996)
- Refactor code to allow supporting multiple versions of the
generate.proto at the same time
- Add v3/generate.proto (ISO to generate.proto for now but allow for
future changes without impacting v2 backends)
- Add Schedule trait to abstract queuing and batching mechanisms that
will be different in the future
- Add SchedulerV2/V3 impl
2024-06-04 15:56:56 +02:00
..
infer feat: add SchedulerV3 (#1996) 2024-06-04 15:56:56 +02:00
config.rs router: send the input as chunks to the backend 2024-06-03 17:02:41 +02:00
lib.rs feat: add SchedulerV3 (#1996) 2024-06-04 15:56:56 +02:00
main.rs feat: add SchedulerV3 (#1996) 2024-06-04 15:56:56 +02:00
server.rs feat: add SchedulerV3 (#1996) 2024-06-04 15:56:56 +02:00
validation.rs feat: add SchedulerV3 (#1996) 2024-06-04 15:56:56 +02:00