diff --git a/README.md b/README.md index 7589a3a6..60fe83cd 100644 --- a/README.md +++ b/README.md @@ -52,6 +52,8 @@ Text Generation Inference (TGI) is a toolkit for deploying and serving Large Lan - Logits warper (temperature scaling, top-p, top-k, repetition penalty, more details see [transformers.LogitsProcessor](https://huggingface.co/docs/transformers/internal/generation_utils#transformers.LogitsProcessor)) - Stop sequences - Log probabilities +- [Speculation](https://huggingface.co/docs/text-generation-inference/conceptual/speculation) ~2x latency +- [Guidance/JSON](https://huggingface.co/docs/text-generation-inference/conceptual/guidance). Specify output format to speed up inference and make sure the output is valid according to some specs.. - Custom Prompt Generation: Easily generate text by providing custom prompts to guide the model's output - Fine-tuning Support: Utilize fine-tuned models for specific tasks to achieve higher accuracy and performance diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml index d57a594d..ad4f29f6 100644 --- a/docs/source/_toctree.yml +++ b/docs/source/_toctree.yml @@ -37,4 +37,8 @@ title: Safetensors - local: conceptual/flash_attention title: Flash Attention + - local: conceptual/speculation + title: Speculation (Medusa, ngram) + - local: conceptual/guidance + title: Guidance, JSON, tools (using outlines) title: Conceptual Guides diff --git a/docs/source/conceptual/guidance.md b/docs/source/conceptual/guidance.md new file mode 100644 index 00000000..8fb46466 --- /dev/null +++ b/docs/source/conceptual/guidance.md @@ -0,0 +1 @@ +## Guidance diff --git a/docs/source/conceptual/speculation b/docs/source/conceptual/speculation new file mode 100644 index 00000000..f08b41c5 --- /dev/null +++ b/docs/source/conceptual/speculation @@ -0,0 +1 @@ +## Speculation diff --git a/docs/source/conceptual/speculation.md b/docs/source/conceptual/speculation.md new file mode 100644 index 00000000..f08b41c5 --- /dev/null +++ b/docs/source/conceptual/speculation.md @@ -0,0 +1 @@ +## Speculation