OlivierDehaene
|
a6a0c97ed9
|
feat: prefill chunking (#2600)
* wip
* rollback
* refactor to use prefix/postfix namming + fix all_input_ids_tensor
* maybe patching vlms?
* fix filter and concat
* wip, no filter, no concat
* current
* add prepare_for_prefill
* working
* load tested
* re-create slots
* re-create slots
* fix slot_filtering_indices
* feedback loop
* remove log
* fix benchmarker
* fix vlm and seq2seq
* rename to cache and input lengths
* fix prefill logprobs
* fix launcher
* fix logprobs?
* idk at this point
* max input length
* omfg
* remove debugging lines
* fix tests
* fix mllama
* fix cargo tests
* remove support chunking for paged
* Fixing non blocked attentions
* Fixing dtype + AMD, Ipex targets.
* lint fix.
* rename
* Fix prefix_caching variable, remove defaults in server (confusing a lot
of the times).
* Add simple resolution when user specifies ATTENTION=paged.
* Put back non default simple tests.
* Fix env name
---------
Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
|
2024-10-16 12:49:33 +02:00 |