Daniël de Kok 2358c2bb54 Add basic FP8 KV cache support (#2603) ... * Add basic FP8 KV cache support This change adds rudimentary FP8 KV cache support. The support is enabled by passing `--kv-cache-dtype fp8_e5m2` to the launcher. Doing so uses this type for the KV cache. However support is still limited: * Only the `fp8_e5m2` type is supported. * The KV cache layout is the same as `float16`/`bfloat16` (HND). * The FP8 KV cache is only supported for FlashInfer. * Loading of scales is not yet supported. * Fix Cargo.toml		2024-10-04 17:51:48 +02:00
..
custom_modeling	Add basic FP8 KV cache support (#2603)	2024-10-04 17:51:48 +02:00
__init__.py	Add basic FP8 KV cache support (#2603)	2024-10-04 17:51:48 +02:00
bloom.py	Refactor dead code - Removing all `flash_xxx.py` files. (#2166)	2024-07-05 10:29:56 +02:00
causal_lm.py	Fixing exl2 and other quanize tests again. (#2419)	2024-08-15 11:12:51 +02:00
flash_causal_lm.py	Add basic FP8 KV cache support (#2603)	2024-10-04 17:51:48 +02:00
galactica.py	feat: add ruff and resolve issue (#2262)	2024-07-26 10:29:09 -04:00
globals.py	Lots of improvements (Still 2 allocators) (#2449)	2024-08-29 16:29:01 +02:00
idefics_causal_lm.py	Mllama flash version (#2585)	2024-10-02 11:22:13 +02:00
mamba.py	Fixing exl2 and other quanize tests again. (#2419)	2024-08-15 11:12:51 +02:00
mllama_causal_lm.py	Mllama flash version (#2585)	2024-10-02 11:22:13 +02:00
model.py	feat: add ruff and resolve issue (#2262)	2024-07-26 10:29:09 -04:00
pali_gemma.py	feat: add ruff and resolve issue (#2262)	2024-07-26 10:29:09 -04:00
seq2seq_lm.py	Fixing exl2 and other quanize tests again. (#2419)	2024-08-15 11:12:51 +02:00
types.py	feat: add ruff and resolve issue (#2262)	2024-07-26 10:29:09 -04:00
vlm_causal_lm.py	Prefix test - Different kind of load test to trigger prefix test bugs. (#2490)	2024-09-11 18:10:40 +02:00