hf_text-generation-inference/docs/source/index.md

Text Generation Inference

Text-Generation-Inference is, an open-source, purpose-built solution for deploying and serving Large Language Models (LLMs). TGI enables high-performance text generation using Tensor Parallelism and dynamic batching for the most popular open-source LLMs, including StarCoder, BLOOM, GPT-NeoX, Llama, and T5. Text Generation Inference implements optimization for all supported model architectures, including:

• Serve the most popular Large Language Models with a simple launcher
• Tensor Parallelism for faster inference on multiple GPUs
• Token streaming using Server-Sent Events (SSE)
• Continuous batching of incoming requests for increased total throughput
• Optimized transformers code for inference using flash-attention and Paged Attention on the most popular architectures
• Quantization with bitsandbytes and GPT-Q
• Safetensors weight loading
• Watermarking with A Watermark for Large Language Models
• Logits warper (temperature scaling, top-p, top-k, repetition penalty, more details see transformers.LogitsProcessor)
• Stop sequences
• Log probabilities
• Production ready (distributed tracing with Open Telemetry, Prometheus metrics)