hf_text-generation-inference

Commit Graph

Author	SHA1	Message	Date
Daniël de Kok	fc9c3153e5	Add pytest release marker (#2114 ) * Add pytest release marker Annotate a test with `@pytest.mark.release` and it only gets run with `pytest integration-tests --release`. * Mark many models as `release` to speed up CI	2024-06-25 16:53:20 +02:00
Daniël de Kok	093a27c528	Add support for GPTQ Marlin (#2052 ) Add support for GPTQ Marlin kernels GPTQ Marlin extends the Marlin kernels to support common GPTQ configurations: - bits: 4 or 8 - groupsize: -1, 32, 64, or 128 - desc_act: true/false Using the GPTQ Marlin kernels requires repacking the parameters in the Marlin quantizer format. The kernels were contributed by Neural Magic to VLLM. We vendor them here for convenience.	2024-06-14 09:45:42 +02:00

Author

SHA1

Message

Date

Daniël de Kok

fc9c3153e5

Add pytest release marker (#2114 )

* Add pytest release marker

Annotate a test with `@pytest.mark.release` and it only gets run
with `pytest integration-tests --release`.

* Mark many models as `release` to speed up CI

2024-06-25 16:53:20 +02:00

Daniël de Kok

093a27c528

Add support for GPTQ Marlin (#2052 )

Add support for GPTQ Marlin kernels

GPTQ Marlin extends the Marlin kernels to support common GPTQ
configurations:

- bits: 4 or 8
- groupsize: -1, 32, 64, or 128
- desc_act: true/false

Using the GPTQ Marlin kernels requires repacking the parameters in the
Marlin quantizer format.

The kernels were contributed by Neural Magic to VLLM. We vendor them
here for convenience.

2024-06-14 09:45:42 +02:00

2 Commits