f9f10a6636
(misc): improve trtllm download script robustness
2024-10-21 10:00:27 +02:00
0c3ba932cc
(misc): disable logging in release mode
2024-10-21 10:00:27 +02:00
437c2aa142
(misc): update dependency in trtllm dockerfile
2024-10-21 10:00:27 +02:00
cb69c9a967
(misc): update dependency in trtllm dockerfile
2024-10-21 10:00:27 +02:00
c8a99af6c9
(fix): do not recreate the stateful hashmap at every it
2024-10-21 10:00:27 +02:00
eb13d8d1f3
(misc): increase verbosity of spdlog
2024-10-21 10:00:27 +02:00
ce0cd1fce8
(misc): build with trtllm 0.13.0
2024-10-21 10:00:27 +02:00
188e4dc64f
(misc: build for sm_{75,80,86,89,90} by default
2024-10-21 10:00:27 +02:00
544c9d9dba
(fix): HOPPER_SM_MAJOR is 9 not 8
2024-10-21 10:00:27 +02:00
213acc6e34
(misc) move to latest trtllm
2024-10-21 10:00:27 +02:00
507ff66692
(misc) rerun-if-changed all the cmake modules
2024-10-21 10:00:27 +02:00
b242f45c04
(misc) delete backend.rs
2024-10-21 10:00:27 +02:00
984ae9798f
(post) impl postprocessing
2024-10-21 10:00:27 +02:00
fa63db0d07
(scheduler) rework submit/pull logic
2024-10-21 10:00:27 +02:00
42ccf4e77c
(misc) no need to move for uint32_t items
2024-10-21 10:00:27 +02:00
b41875c139
(misc) simplify [make_]move_iterator by using c++20 type inference
2024-10-21 10:00:27 +02:00
b1846fb4e6
(backend) refactor & cleanup
2024-10-21 10:00:27 +02:00
483f172938
(ffi) do not use reference capture in lambda as we are not capturing anything
2024-10-21 10:00:27 +02:00
3d0e90b631
(ffi) missing namespace for tle::Response
2024-10-21 10:00:27 +02:00
8e648ce425
(ffi) fix usage of wrong vector constructor making a capacity fill call
2024-10-21 10:00:27 +02:00
dddc9a44bd
(build) fetchcontent use archives instead of git
2024-10-21 10:00:27 +02:00
089c5fe668
(server) forward auth_token to server::run
2024-10-21 10:00:27 +02:00
291eaa99fb
use blocking_recv in looper to consume awaiting_requests at max before pulling in a single step
2024-10-21 10:00:27 +02:00
7bebc629af
(misc) missing Result types for Rust
2024-10-21 10:00:27 +02:00
c2e21d8725
(backend) implement the post_processor background thread
2024-10-21 10:00:27 +02:00
0dca168bcb
(misc) change scope identifiers
2024-10-21 10:00:27 +02:00
933ab67aa1
(ffi) encode the provided user prompt within each request thread
2024-10-21 10:00:27 +02:00
0b0c30fe8b
(ffi) remove narrowing type warning
2024-10-21 10:00:27 +02:00
fb759bdd2a
(looper) new looper initial implementation
2024-10-21 10:00:27 +02:00
5f7c0b67c3
(ffi) add template specialization to catch and convert to Rust Result<T, tensorrt_llm::common::TllmException>
2024-10-21 10:00:27 +02:00
33c962ef41
(ffi) add missing headers imports
2024-10-21 10:00:27 +02:00
2883c042ed
(ffi) cleanup again
2024-10-21 10:00:27 +02:00
f4a74be384
(backend) expose PullNewTokens
2024-10-21 10:00:27 +02:00
b8a40a0af3
(backend) cleanup a bit
2024-10-21 10:00:27 +02:00
38b5263c61
(ffi) add max_new_tokens parameters
2024-10-21 10:00:27 +02:00
f6f689f509
(build) setup ccache if available
2024-10-21 10:00:27 +02:00
2a339f99dd
(trt)
2024-10-21 10:00:25 +02:00
0cd7538a48
(ffi) use const for GetSamplingConfig
2024-10-21 09:57:26 +02:00
cea64e234f
(chore) fmt ... why?
2024-10-21 09:57:26 +02:00
a3f7d76f7b
(launcher) default new server::run parameters to false for now
2024-10-21 09:57:24 +02:00
25b20cba2a
(backend) use parking_lot crate for RwLock fairness
...
# Conflicts:
# backends/trtllm/src/backend.rs
2024-10-21 09:57:16 +02:00
a6a0c97ed9
feat: prefill chunking ( #2600 )
...
* wip
* rollback
* refactor to use prefix/postfix namming + fix all_input_ids_tensor
* maybe patching vlms?
* fix filter and concat
* wip, no filter, no concat
* current
* add prepare_for_prefill
* working
* load tested
* re-create slots
* re-create slots
* fix slot_filtering_indices
* feedback loop
* remove log
* fix benchmarker
* fix vlm and seq2seq
* rename to cache and input lengths
* fix prefill logprobs
* fix launcher
* fix logprobs?
* idk at this point
* max input length
* omfg
* remove debugging lines
* fix tests
* fix mllama
* fix cargo tests
* remove support chunking for paged
* Fixing non blocked attentions
* Fixing dtype + AMD, Ipex targets.
* lint fix.
* rename
* Fix prefix_caching variable, remove defaults in server (confusing a lot
of the times).
* Add simple resolution when user specifies ATTENTION=paged.
* Put back non default simple tests.
* Fix env name
---------
Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
2024-10-16 12:49:33 +02:00
0204946d26
Max token capacity metric ( #2595 )
...
* adding max_token_capacity_metric
* added tgi to name of metric
* Adding max capacity metric.
* Add description for the metrics
---------
Co-authored-by: Edwinhr716 <Edandres249@gmail.com>
2024-10-02 16:32:36 +02:00
0ff6ff60ad
Hotfixing main ( #2556 )
2024-09-24 11:51:14 +02:00
10e6f29295
chore: Add old V2 backend ( #2551 )
...
* wip
* added v2
2024-09-24 08:38:17 +02:00
38fcafcf96
Adding a test for FD. ( #2516 )
...
* Adding a test for FD.
* Fixing flashdecoding (empty batch doesn't work).
* Fixing the invalid popping.
* Fixing radix with block_size > 1
* Last reference.
* Use an actual hash.
* Update hash for slice.len() == 1
* Update the locks.
* Increasing docker timeout.
2024-09-16 17:00:54 +02:00
dae3bf1d87
Fix tokenization yi ( #2507 )
...
* Fixing odd tokenization self modifications on the Rust side (load and
resave in Python).
* Fixing the builds ?
* Fix the gh action?
* Fixing the location ?
* Validation is odd.
* Try a faster runner
* Upgrade python version.
* Remove sccache
* No sccache.
* Getting libpython maybe ?
* List stuff.
* Monkey it up.
* have no idea at this point
* Tmp.
* Shot in the dark.
* Tmate the hell out of this.
* Desperation.
* WTF.
* -y.
* Apparently 3.10 is not available anymore.
* Updating the dockerfile to make libpython discoverable at runtime too.
* Put back rust tests.
* Why do we want mkl on AMD ?
* Forcing 3.11 ?
2024-09-11 22:41:56 +02:00
a4e3e8c608
Prefix test - Different kind of load test to trigger prefix test bugs. ( #2490 )
...
* Adding prefix test.
* [WIP] tmp dump of integration load tests.
* Remove other tensor creation.
* Fixed the radix tree.
Used a slice everywhere in radix.rs to keep the cheap Arc cloning
instead of recomputing the input_ids.
* Fix parsing
* Is it really flashinfer version ?
* Remove some comments.
* Revert the max prefix hit.
* Adding numpy to diff.
* Upgraded flashinfer.
* Upgrading some stuff.
* Are we done yet ?
* Minor fixup
* Remove 1 log and put back the other.
* Add comment for why slot 0 is OK.
* Mounting on the job.
* Get me a debug branch
* Debugging CIs is fun.
* Attempt #28
* wip
* Tmate.
* Praying.
* Updating VLM causal model with updated context.
* Important line got squashed.
* Tmate again.
* Fingers crossed.
* We want only 1 run of integration tests.....
---------
Co-authored-by: Guillaume LEGENDRE <glegendre01@gmail.com>
2024-09-11 18:10:40 +02:00
c1fe28d694
Fixing more correctly the invalid drop of the batch. ( #2498 )
2024-09-06 17:35:49 +02:00
379472c4c2
radix trie: add assertions ( #2491 )
...
These should all be cheap assertions.
Also:
* Fixup some comments.
* Delete a `remove` that was done unnecessarily twice.
2024-09-06 11:55:23 +02:00
Morgan Funtowicz