Mirrors
/
hf_text-generation-inference
mirror of https://github.com/huggingface/text-generation-inference.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Large Language Model Text Generation Inference
21 Commits 178 Branches 49 Tags 605 MiB
Python 70.9%
Rust 22.2%
Cuda 4.7%
C++ 1%
Nix 0.4%
Other 0.7%
Go to file
OlivierDehaene 755fc0e403 fix(models): Revert buggy support for AutoModel 2022-11-03 16:07:54 +01:00
aml feat(server): Support all AutoModelForCausalLM on a best effort basis 2022-10-28 19:24:00 +02:00
assets v0.1.0 2022-10-20 19:14:44 +02:00
k6 Add load testing 2022-10-11 10:36:51 +02:00
launcher feat: Use json formatter by default in docker image 2022-11-02 17:29:56 +01:00
proto v0.1.0 2022-10-20 19:14:44 +02:00
router feat: Use json formatter by default in docker image 2022-11-02 17:29:56 +01:00
server fix(models): Revert buggy support for AutoModel 2022-11-03 16:07:54 +01:00
.dockerignore v0.1.0 2022-10-20 19:14:44 +02:00
.gitignore v0.1.0 2022-10-20 19:14:44 +02:00
Cargo.lock feat: Use json formatter by default in docker image 2022-11-02 17:29:56 +01:00
Cargo.toml feat(server): Use safetensors 2022-10-22 20:00:15 +02:00
Dockerfile feat: Use json formatter by default in docker image 2022-11-02 17:29:56 +01:00
LICENSE Create LICENSE (#2) 2022-10-22 10:44:52 +02:00
Makefile feat(server): Support all AutoModelForCausalLM on a best effort basis 2022-10-28 19:24:00 +02:00
README.md fix(models): Revert buggy support for AutoModel 2022-11-03 16:07:54 +01:00
rust-toolchain.toml v0.1.0 2022-10-20 19:14:44 +02:00

README.md

Text Generation Inference

architecture

A Rust and gRPC server for text generation inference.

Features

Supported models

  • BLOOM
  • BLOOM-560m

Load Tests for BLOOM

See k6/load_test.js

avg min med max p(90) p(95) RPS
Original code 8.9s 1s 9.12s 16.69s 13.7s 14.26s 5.9
New batching logic 5.44s 959.53ms 5.28s 13.12s 7.78s 8.92s 9.08

Install

make install

Run

BLOOM 560-m

make run-bloom-560m

BLOOM

First you need to download the weights:

make download-bloom

make run-bloom # Requires 8xA100 80GB

You can also quantize the weights with bitsandbytes to reduce the VRAM requirement:

make run-bloom-quantize # Requires 8xA100 40GB

Test

curl 127.0.0.1:3000/generate \
    -v \
    -X POST \
    -d '{"inputs":"Testing API","parameters":{"max_new_tokens":9}}' \
    -H 'Content-Type: application/json'

Develop

make server-dev
make router-dev

TODO:

  • Support AutoModelForSeq2SeqLM
  • Add tests for the server/model logic
  • Backport custom CUDA kernels to Transformers
  • Install safetensors with pip