4 Commits

Author	SHA1	Message	Date
Felix Marty	284894303a	remove require_backend decorators on handles, for some reasons fails in github actions	2024-06-24 18:51:32 +02:00
fxmarty	3de8f3647b	fix decorators	2024-06-24 18:51:08 +02:00
fxmarty	406885638b	skip exl2 tests on rocm	2024-06-24 18:49:45 +02:00
Daniël de Kok	36dd16017c	Add support for exl2 quantization ... Mostly straightforward, changes to existing code: * Wrap quantizer parameters in a small wrapper to avoid passing around untyped tuples and needing to repack them as a dict. * Move scratch space computation to warmup, because we need the maximum input sequence length to avoid allocating huge scratch buffers that OOM.	2024-05-30 11:28:05 +02:00