83634dc122
* use xpu-smi to dump used memory xpu use "ZE_AFFINITY_MASK" to control card, usage is like CUDA_VISIBLE_DEVICES Signed-off-by: Wang, Yi A <yi.a.wang@intel.com> * Update server/text_generation_server/utils/import_utils.py Co-authored-by: Daniël de Kok <me@github.danieldk.eu> --------- Signed-off-by: Wang, Yi A <yi.a.wang@intel.com> Co-authored-by: Daniël de Kok <me@github.danieldk.eu> |
||
|---|---|---|
| .. | ||
| custom_kernels | ||
| exllama_kernels | ||
| exllamav2_kernels | ||
| marlin | ||
| tests | ||
| text_generation_server | ||
| .gitignore | ||
| Makefile | ||
| Makefile-awq | ||
| Makefile-eetq | ||
| Makefile-flash-att | ||
| Makefile-flash-att-v2 | ||
| Makefile-selective-scan | ||
| Makefile-vllm | ||
| README.md | ||
| poetry.lock | ||
| pyproject.toml | ||
| requirements_cuda.txt | ||
| requirements_intel.txt | ||
| requirements_rocm.txt | ||
README.md
Text Generation Inference Python gRPC Server
A Python gRPC server for Text Generation Inference
Install
make install
Run
make run-dev