hf_text-generation-inference/integration-tests
Daniël de Kok ba291dad9f
Improve the handling of quantized weights (#2250)
* Improve the handling of quantized weights

Handling of quantized weights was split between two mechanisms:

- For quantized checkpoints, we used the new weight loader
  infrastructure.
- For quantization while loading (EETQ, FP8, bitsandbytes) we
  instead relied on conditional in `get_linear`.

Weight loaders support context managers to selectively load
particular layers with different weight loaders, which is useful
for models like Idefics2 AWQ, which uses a quantized text model,
but unquantized vision and connector models. However, the context
manager would be overrided by `get_linear`, which string-checks
`quantizer`. Also, the context manager would not work with
EETQ, FP8, and bitsandbytes.

This change migrates all quantizers to the weight loader infrastructure.
This has several benefits:

- We can use context managers with all quantizers.
- All the implementation details move down to the quantizer layers,
  `get_linear` does not need to know how to handle quantizer linear
  layers.
- All quantizer weights are strongly typed, we don't pass around
  raw tensors.
- We don't have to pass around the `quantizer` string everywhere.

* Exclude non-MLP layers when using FP8 quantization with Llama
2024-07-19 09:37:39 +02:00
..
images Pali gemma modeling (#1895) 2024-05-16 06:58:47 +02:00
models Improve the handling of quantized weights (#2250) 2024-07-19 09:37:39 +02:00
conftest.py feat: simple mistral lora integration tests (#2180) 2024-07-15 09:16:15 -04:00
poetry.lock fix: improve tool type, bump pydantic and outlines (#1650) 2024-03-21 12:45:56 -04:00
pyproject.toml v2.0.1 2024-04-18 17:20:36 +02:00
pytest.ini chore: add pre-commit (#1569) 2024-02-16 11:58:58 +01:00
requirements.txt fix: improve tool type, bump pydantic and outlines (#1650) 2024-03-21 12:45:56 -04:00