Daniël de Kok 2358c2bb54 Add basic FP8 KV cache support (#2603) ... * Add basic FP8 KV cache support This change adds rudimentary FP8 KV cache support. The support is enabled by passing `--kv-cache-dtype fp8_e5m2` to the launcher. Doing so uses this type for the KV cache. However support is still limited: * Only the `fp8_e5m2` type is supported. * The KV cache layout is the same as `float16`/`bfloat16` (HND). * The FP8 KV cache is only supported for FlashInfer. * Loading of scales is not yet supported. * Fix Cargo.toml		2024-10-04 17:51:48 +02:00
..
src	Add basic FP8 KV cache support (#2603)	2024-10-04 17:51:48 +02:00
Cargo.toml	CI (2592): Allow LoRA adapter revision in server launcher (#2602)	2024-10-02 10:51:04 -04:00
build.rs	chore(github): add templates (#264)	2023-05-02 15:43:19 +02:00