31d76e238d
fix(batching): Avoid theoretical hang in batcher loop ( #5 )
...
- Avoid theoretical hang in batcher loop
- Avoid a couple of clones in the router generate method
- Keep attention mask tensors as integers
- Remove num_heads attribute
Co-authored-by: OlivierDehaene <Olivier.dehaene@gmail.com>
2022-12-05 10:10:59 +01:00
d6d5b12e03
fix(router): Handle tokenizer errors
2022-11-14 17:15:19 +01:00
91f5f86280
fix(router): Fix HTTP status codes
2022-11-14 14:34:15 +01:00
427d7cc444
feat(server): Support AutoModelForSeq2SeqLM
2022-11-04 18:03:04 +01:00
c5665f5c8b
feat(server): Support generic AutoModelForCausalLM
2022-11-04 14:22:47 +01:00
b3b7ea0d74
feat: Use json formatter by default in docker image
2022-11-02 17:29:56 +01:00
3cf6368c77
feat(server): Support all AutoModelForCausalLM on a best effort basis
2022-10-28 19:24:00 +02:00
09674e6df9
feat(server): Support bitsandbytes
2022-10-27 14:25:29 +02:00
beb552127a
feat(client): Simplify sharded logic
2022-10-22 23:40:05 +02:00
c837893370
feat(router): Add max_waiting_tokens
2022-10-21 16:40:05 +02:00
895a341d06
fix(validation): Fix error messages
2022-10-21 10:59:15 +02:00
f16f2f5ae1
v0.1.0
2022-10-20 19:14:44 +02:00
92c1ecd008
feat: Add arguments to CLI
2022-10-17 18:27:33 +02:00
5e5d8766a2
feat: Improve error handling
2022-10-17 14:59:00 +02:00
bcb53903b8
feat: Add AML deployment
2022-10-15 20:21:50 +02:00
bf99afe916
feat: Docker image
2022-10-14 15:56:21 +02:00
39df4d9975
Use axum
2022-10-11 18:14:39 +02:00
e86ecbac63
ValidationError was not correctly handled
2022-10-11 16:53:40 +02:00
4c693e6524
Refactored gRPC interface
...
Added validation logic
2022-10-11 16:50:54 +02:00
fa9a088467
Add load testing
2022-10-11 10:36:51 +02:00
295831a481
Init
2022-10-08 12:30:12 +02:00
Nick Hill