Daniël de Kok 2358c2bb54 Add basic FP8 KV cache support (#2603) ... * Add basic FP8 KV cache support This change adds rudimentary FP8 KV cache support. The support is enabled by passing `--kv-cache-dtype fp8_e5m2` to the launcher. Doing so uses this type for the KV cache. However support is still limited: * Only the `fp8_e5m2` type is supported. * The KV cache layout is the same as `float16`/`bfloat16` (HND). * The FP8 KV cache is only supported for FlashInfer. * Loading of scales is not yet supported. * Fix Cargo.toml		2024-10-04 17:51:48 +02:00
..
env_runtime.rs	add intel xpu support for TGI (#1475)	2024-04-26 15:48:58 +02:00
gpu.rs	Remove compute capability lazy cell (#2580)	2024-09-30 08:48:47 +02:00
main.rs	Add basic FP8 KV cache support (#2603)	2024-10-04 17:51:48 +02:00