2ce8019480
GPTQ-Marlin is currently the best-performing kernel for GPTQ models. So let's use it by default if the kernels are installed, the GPU supports it, and the kernels support the configuration. For models generated by `text-generation-server quantize`, use `sym=False`. This subcommand symmetric quantization since the beginning and incorrectly reporting the model to be symmetric will use GPTQ-Marlin (which does not support asymmetric quantization). |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| custom_autotune.py | ||
| exllama.py | ||
| exllamav2.py | ||
| quant_linear.py | ||
| quantize.py | ||