hf_text-generation-inference

Commit Graph

Author	SHA1	Message	Date
Daniël de Kok	7f54b7336a	Test Marlin MoE with `desc_act=true` (#2622 ) Update the Mixtral GPTQ test to use a model with `desc_act=true` and `group_size!=-1` to ensure that we are checking activation sorting/non-full K (with tensor parallelism). The `desc_act=false` case is already checked by the Mixtral AWQ test.	2024-10-21 12:50:35 +02:00
Daniël de Kok	90a1d04a2f	Add support for GPTQ-quantized MoE models using MoE Marlin (#2557 ) This change add support for MoE models that use GPTQ quantization. Currently only models with the following properties are supported: - No `desc_act` with tensor parallelism, unless `group_size=-1`. - No asymmetric quantization. - No AWQ.	2024-09-30 11:14:32 +02:00

Author

SHA1

Message

Date

Daniël de Kok

7f54b7336a

Test Marlin MoE with `desc_act=true` (#2622 )

Update the Mixtral GPTQ test to use a model with `desc_act=true` and
`group_size!=-1` to ensure that we are checking activation
sorting/non-full K (with tensor parallelism). The `desc_act=false` case
is already checked by the Mixtral AWQ test.

2024-10-21 12:50:35 +02:00

Daniël de Kok

90a1d04a2f

Add support for GPTQ-quantized MoE models using MoE Marlin (#2557 )

This change add support for MoE models that use GPTQ quantization.
Currently only models with the following properties are supported:

- No `desc_act` with tensor parallelism, unless `group_size=-1`.
- No asymmetric quantization.
- No AWQ.

2024-09-30 11:14:32 +02:00

2 Commits