History

Michael Feil ff703cb867 Adding ctranslate2 quantization and inference: moving the contribution (#1 ) * rebaseing the commit on preemo fork. * reformatting and changes. * update dockerfile * update changes for dockerfile * adapt path * rebaseing the commit on preemo fork. * reformatting and changes. * update dockerfile * update changes for dockerfile * adapt path --------- Co-authored-by: michaelfeil <me@michaelfeil.eu>		2023-10-02 11:12:49 -07:00
..
custom_kernels	feat(server): Rework model loading (#344 )	2023-06-08 14:51:52 +02:00
exllama_kernels	feat: add cuda memory fraction (#659 )	2023-07-24 11:43:58 +02:00
tests	Adding ctranslate2 quantization and inference: moving the contribution (#1 )	2023-10-02 11:12:49 -07:00
text_generation_server	Adding ctranslate2 quantization and inference: moving the contribution (#1 )	2023-10-02 11:12:49 -07:00
.gitignore	feat(clients): Python client (#103 )	2023-03-07 18:52:22 +01:00
Makefile	Adding ctranslate2 quantization and inference: moving the contribution (#1 )	2023-10-02 11:12:49 -07:00
Makefile-flash-att	feat(server): use latest flash attention commit (#543 )	2023-07-04 20:23:55 +02:00
Makefile-flash-att-v2	feat(server): flash attention v2 (#624 )	2023-07-18 16:21:18 +02:00
Makefile-vllm	feat(server): update vllm version (#723 )	2023-07-28 15:36:38 +02:00
README.md	feat(router): refactor API and add openAPI schemas (#53 )	2023-02-03 12:43:37 +01:00
poetry.lock	fix(server): fix missing datasets in quantize	2023-07-27 14:50:45 +02:00
pyproject.toml	Adding ctranslate2 quantization and inference: moving the contribution (#1 )	2023-10-02 11:12:49 -07:00
requirements.txt	fix(server): fix missing datasets in quantize	2023-07-27 14:50:45 +02:00

README.md

Text Generation Inference Python gRPC Server

A Python gRPC server for Text Generation Inference

Install

make install

Run

make run-dev