hf_text-generation-inference/backends/v3/src
OlivierDehaene a6a0c97ed9
feat: prefill chunking (#2600)
* wip

* rollback

* refactor to use prefix/postfix namming + fix all_input_ids_tensor

* maybe patching vlms?

* fix filter and concat

* wip, no filter, no concat

* current

* add prepare_for_prefill

* working

* load tested

* re-create slots

* re-create slots

* fix slot_filtering_indices

* feedback loop

* remove log

* fix benchmarker

* fix vlm and seq2seq

* rename to cache and input lengths

* fix prefill logprobs

* fix launcher

* fix logprobs?

* idk at this point

* max input length

* omfg

* remove debugging lines

* fix tests

* fix mllama

* fix cargo tests

* remove support chunking for paged

* Fixing non blocked attentions

* Fixing dtype + AMD, Ipex targets.

* lint fix.

* rename

* Fix prefix_caching variable, remove defaults in server (confusing a lot
of the times).

* Add simple resolution when user specifies ATTENTION=paged.

* Put back non default simple tests.

* Fix env name

---------

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
2024-10-16 12:49:33 +02:00
..
client feat: prefill chunking (#2600) 2024-10-16 12:49:33 +02:00
backend.rs feat: prefill chunking (#2600) 2024-10-16 12:49:33 +02:00
block_allocator.rs Lots of improvements (Still 2 allocators) (#2449) 2024-08-29 16:29:01 +02:00
lib.rs feat: prefill chunking (#2600) 2024-10-16 12:49:33 +02:00
main.rs feat: prefill chunking (#2600) 2024-10-16 12:49:33 +02:00
queue.rs feat: prefill chunking (#2600) 2024-10-16 12:49:33 +02:00
radix.rs Adding a test for FD. (#2516) 2024-09-16 17:00:54 +02:00