Daniël de Kok 4594e6faba Add support for Marlin-quantized models ... This change adds support for Marlin-quantized models. Marlin is an FP16xINT4 matmul kernel, which provides good speedups decoding batches of 16-32 tokens. It supports quantized models with symmetric quantization, groupsize -1 or 128, and 4-bit. Tested with: - Llama 2 - Llama 3 - Phi 3		2024-06-06 13:16:52 +02:00
..
attention	Fixing rocm. (#2021)	2024-06-05 14:41:34 +02:00
awq	Refactor layers. (#1866)	2024-05-13 12:44:30 +02:00
gptq	Fix GPTQWeight import (#2020)	2024-06-05 14:49:15 +02:00
__init__.py	MLPSpeculator. (#1865)	2024-05-14 12:33:18 +02:00
bnb.py	[Bug Fix] Update torch import reference in bnb quantization (#1902)	2024-05-15 21:08:32 +02:00
conv.py	Refactor layers. (#1866)	2024-05-13 12:44:30 +02:00
eetq.py	Refactor layers. (#1866)	2024-05-13 12:44:30 +02:00
exl2.py	Add support for exl2 quantization	2024-05-30 11:28:05 +02:00
fp8.py	Refactor layers. (#1866)	2024-05-13 12:44:30 +02:00
layernorm.py	MI300 compatibility (#1764)	2024-05-17 15:30:47 +02:00
linear.py	Add support for Marlin-quantized models	2024-06-06 13:16:52 +02:00
marlin.py	Add support for Marlin-quantized models	2024-06-06 13:16:52 +02:00
medusa.py	fix: use path inside of speculator config (#1935)	2024-05-22 20:46:29 +02:00
mlp.py	MLPSpeculator. (#1865)	2024-05-14 12:33:18 +02:00
rotary.py	reenable xpu for tgi (#1939)	2024-05-23 14:11:08 +02:00
speculative.py	MLPSpeculator. (#1865)	2024-05-14 12:33:18 +02:00
tensor_parallel.py	Add support for Marlin-quantized models	2024-06-06 13:16:52 +02:00