Mirrors
/
hf_text-generation-inference
mirror of https://github.com/huggingface/text-generation-inference.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Large Language Model Text Generation Inference
10 Commits 178 Branches 49 Tags 605 MiB
Python 70.9%
Rust 22.2%
Cuda 4.7%
C++ 1%
Nix 0.4%
Other 0.7%
Go to file
Olivier Dehaene 5e5d8766a2 feat: Improve error handling 2022-10-17 14:59:00 +02:00
aml feat: Improve error handling 2022-10-17 14:59:00 +02:00
k6 Add load testing 2022-10-11 10:36:51 +02:00
proto Refactored gRPC interface 2022-10-11 16:50:54 +02:00
router feat: Improve error handling 2022-10-17 14:59:00 +02:00
server feat: Improve error handling 2022-10-17 14:59:00 +02:00
.dockerignore feat: Add AML deployment 2022-10-15 20:21:50 +02:00
.gitignore Add load testing 2022-10-11 10:36:51 +02:00
Dockerfile feat: Improve error handling 2022-10-17 14:59:00 +02:00
README.md feat: Improve error handling 2022-10-17 14:59:00 +02:00
run.sh feat: Improve error handling 2022-10-17 14:59:00 +02:00

README.md

Text Generation Inference

A Rust and gRPC server for text generation inference.

Load Tests

See k6/load_test.js We send the default examples with a 1 second delay between each request.

Stages:

  • Ramp up to 50 concurrent requests per second in 1min
  • Ramp up from 50 to 100 concurrent requests per second in 2min
  • Ramp down to 0 concurrent requests per second in 1min
avg min med max p(90) p(95) RPS
Original code 8.9s 1s 9.12s 16.69s 13.7s 14.26s 5.9
ISO with original code 8.88s 959.53ms 8.89s 17.08s 13.34s 14.12s 5.94
New batching logic 5.44s 1.27s 5.28s 13.12s 7.78s 8.92s 9.08

Install

cd server
pip install .

cd router
cargo build --release

Run

python server/bloom_inference/main.py bigscience/bloom --num-gpus 8 --shard-directory /dev/shm/models

./router/target/release/router

TODO:

  • Add batching args to router CLI
  • Add docstrings + comments everywhere as the codebase is fairly complicated
  • Add tests
  • Add shutdown logic in router and server
  • Improve multi-processing logic in server
  • Improve past key layer indexing?