2ce8019480
GPTQ-Marlin is currently the best-performing kernel for GPTQ models. So let's use it by default if the kernels are installed, the GPU supports it, and the kernels support the configuration. For models generated by `text-generation-server quantize`, use `sym=False`. This subcommand symmetric quantization since the beginning and incorrectly reporting the model to be symmetric will use GPTQ-Marlin (which does not support asymmetric quantization). |
||
---|---|---|
.. | ||
images | ||
models | ||
conftest.py | ||
poetry.lock | ||
pyproject.toml | ||
pytest.ini | ||
requirements.txt |