Mirrors
/
hf_text-generation-inference
mirror of https://github.com/huggingface/text-generation-inference.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,134 Commits 184 Branches 51 Tags 606 MiB
Commit Graph

70 Commits

Author SHA1 Message Date
Morgan Funtowicz 85c03e33a9 chore(trtllm): fmt 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 62f33d7ecd chore(trtllm): move dockerfile to right place 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 6687c06a21 feat(looper): minor optimizations to avoid growing too much the containers 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 027756c52d chore(cmake): download timestamp should be before URL 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 629153b44b feat(looper): check engine and executorWorker paths exist before creating the backend 2024-10-21 10:00:27 +02:00
Morgan Funtowicz f20ec28891 chore(cmake): use correct policy for download_timestamp 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 819c953771 misc(cuda): require 12.6 2024-10-21 10:00:27 +02:00
Morgan Funtowicz dd94ccc989 (fix): ore fixes for Dockerfile 2024-10-21 10:00:27 +02:00
Morgan Funtowicz f9f10a6636 (misc): improve trtllm download script robustness 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 0c3ba932cc (misc): disable logging in release mode 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 437c2aa142 (misc): update dependency in trtllm dockerfile 2024-10-21 10:00:27 +02:00
Morgan Funtowicz cb69c9a967 (misc): update dependency in trtllm dockerfile 2024-10-21 10:00:27 +02:00
Morgan Funtowicz c8a99af6c9 (fix): do not recreate the stateful hashmap at every it 2024-10-21 10:00:27 +02:00
Morgan Funtowicz eb13d8d1f3 (misc): increase verbosity of spdlog 2024-10-21 10:00:27 +02:00
Morgan Funtowicz ce0cd1fce8 (misc): build with trtllm 0.13.0 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 188e4dc64f (misc: build for sm_{75,80,86,89,90} by default 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 544c9d9dba (fix): HOPPER_SM_MAJOR is 9 not 8 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 213acc6e34 (misc) move to latest trtllm 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 507ff66692 (misc) rerun-if-changed all the cmake modules 2024-10-21 10:00:27 +02:00
Morgan Funtowicz b242f45c04 (misc) delete backend.rs 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 984ae9798f (post) impl postprocessing 2024-10-21 10:00:27 +02:00
Morgan Funtowicz fa63db0d07 (scheduler) rework submit/pull logic 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 42ccf4e77c (misc) no need to move for uint32_t items 2024-10-21 10:00:27 +02:00
Morgan Funtowicz b41875c139 (misc) simplify [make_]move_iterator by using c++20 type inference 2024-10-21 10:00:27 +02:00
Morgan Funtowicz b1846fb4e6 (backend) refactor & cleanup 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 483f172938 (ffi) do not use reference capture in lambda as we are not capturing anything 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 3d0e90b631 (ffi) missing namespace for tle::Response 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 8e648ce425 (ffi) fix usage of wrong vector constructor making a capacity fill call 2024-10-21 10:00:27 +02:00
Morgan Funtowicz dddc9a44bd (build) fetchcontent use archives instead of git 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 089c5fe668 (server) forward auth_token to server::run 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 291eaa99fb use blocking_recv in looper to consume awaiting_requests at max before pulling in a single step 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 7bebc629af (misc) missing Result types for Rust 2024-10-21 10:00:27 +02:00
Morgan Funtowicz c2e21d8725 (backend) implement the post_processor background thread 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 0dca168bcb (misc) change scope identifiers 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 933ab67aa1 (ffi) encode the provided user prompt within each request thread 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 0b0c30fe8b (ffi) remove narrowing type warning 2024-10-21 10:00:27 +02:00
Morgan Funtowicz fb759bdd2a (looper) new looper initial implementation 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 5f7c0b67c3 (ffi) add template specialization to catch and convert to Rust Result<T, tensorrt_llm::common::TllmException> 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 33c962ef41 (ffi) add missing headers imports 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 2883c042ed (ffi) cleanup again 2024-10-21 10:00:27 +02:00
Morgan Funtowicz f4a74be384 (backend) expose PullNewTokens 2024-10-21 10:00:27 +02:00
Morgan Funtowicz b8a40a0af3 (backend) cleanup a bit 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 38b5263c61 (ffi) add max_new_tokens parameters 2024-10-21 10:00:27 +02:00
Morgan Funtowicz f6f689f509 (build) setup ccache if available 2024-10-21 10:00:27 +02:00
Morgan Funtowicz 2a339f99dd (trt) 2024-10-21 10:00:25 +02:00
Morgan Funtowicz 0cd7538a48 (ffi) use const for GetSamplingConfig 2024-10-21 09:57:26 +02:00
Morgan Funtowicz cea64e234f (chore) fmt ... why? 2024-10-21 09:57:26 +02:00
Morgan Funtowicz a3f7d76f7b (launcher) default new server::run parameters to false for now 2024-10-21 09:57:24 +02:00
Morgan Funtowicz 25b20cba2a (backend) use parking_lot crate for RwLock fairness 
# Conflicts:
#	backends/trtllm/src/backend.rs
 2024-10-21 09:57:16 +02:00
OlivierDehaene a6a0c97ed9
feat: prefill chunking (#2600) 
* wip

* rollback

* refactor to use prefix/postfix namming + fix all_input_ids_tensor

* maybe patching vlms?

* fix filter and concat

* wip, no filter, no concat

* current

* add prepare_for_prefill

* working

* load tested

* re-create slots

* re-create slots

* fix slot_filtering_indices

* feedback loop

* remove log

* fix benchmarker

* fix vlm and seq2seq

* rename to cache and input lengths

* fix prefill logprobs

* fix launcher

* fix logprobs?

* idk at this point

* max input length

* omfg

* remove debugging lines

* fix tests

* fix mllama

* fix cargo tests

* remove support chunking for paged

* Fixing non blocked attentions

* Fixing dtype + AMD, Ipex targets.

* lint fix.

* rename

* Fix prefix_caching variable, remove defaults in server (confusing a lot
of the times).

* Add simple resolution when user specifies ATTENTION=paged.

* Put back non default simple tests.

* Fix env name

---------

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
 2024-10-16 12:49:33 +02:00