hf_text-generation-inference/nix
Daniël de Kok 0f346a3296
Switch from fbgemm-gpu w8a8 scaled matmul to vLLM/marlin-kernels (#2688)
* Switch from fbgemm-gpu w8a8 scaled matmul to vLLM/marlin-kernels

Performance and accuracy of these kernels are on par (tested with Llama
70B and 405B). Removes a dependency and resolves some stability issues
we have been seeing.

* Update test snapshots
2024-10-25 16:40:47 +02:00
..
client.nix Stream options. (#2533) 2024-09-19 20:50:37 +02:00
crate-overrides.nix nix: support Python tokenizer conversion in the router (#2515) 2024-09-12 10:44:01 +02:00
docker.nix nix: experimental support for building a Docker container (#2470) 2024-10-01 18:02:06 +02:00
impure-shell.nix feat: natively support Granite models (#2682) 2024-10-23 10:04:05 +00:00
overlay.nix nix: example of local package overrides during development (#2607) 2024-10-04 16:52:42 +02:00
server.nix Switch from fbgemm-gpu w8a8 scaled matmul to vLLM/marlin-kernels (#2688) 2024-10-25 16:40:47 +02:00