hf_text-generation-inference/server
..
custom_kernels
exllama_kernels
exllamav2_kernels
marlin
tests
text_generation_server
.gitignore
Makefile
Makefile-awq
Makefile-eetq
Makefile-flash-att
Makefile-flash-att-v2
Makefile-selective-scan
Makefile-vllm
README.md
poetry.lock
pyproject.toml
requirements_cuda.txt
requirements_intel.txt
requirements_rocm.txt

README.md

Text Generation Inference Python gRPC Server

A Python gRPC server for Text Generation Inference

Install

make install

Run

make run-dev