85c03e33a9
chore(trtllm): fmt
2024-10-21 10:00:27 +02:00
62f33d7ecd
chore(trtllm): move dockerfile to right place
2024-10-21 10:00:27 +02:00
6687c06a21
feat(looper): minor optimizations to avoid growing too much the containers
2024-10-21 10:00:27 +02:00
027756c52d
chore(cmake): download timestamp should be before URL
2024-10-21 10:00:27 +02:00
629153b44b
feat(looper): check engine and executorWorker paths exist before creating the backend
2024-10-21 10:00:27 +02:00
f20ec28891
chore(cmake): use correct policy for download_timestamp
2024-10-21 10:00:27 +02:00
819c953771
misc(cuda): require 12.6
2024-10-21 10:00:27 +02:00
dd94ccc989
(fix): ore fixes for Dockerfile
2024-10-21 10:00:27 +02:00
f9f10a6636
(misc): improve trtllm download script robustness
2024-10-21 10:00:27 +02:00
0c3ba932cc
(misc): disable logging in release mode
2024-10-21 10:00:27 +02:00
437c2aa142
(misc): update dependency in trtllm dockerfile
2024-10-21 10:00:27 +02:00
cb69c9a967
(misc): update dependency in trtllm dockerfile
2024-10-21 10:00:27 +02:00
c8a99af6c9
(fix): do not recreate the stateful hashmap at every it
2024-10-21 10:00:27 +02:00
eb13d8d1f3
(misc): increase verbosity of spdlog
2024-10-21 10:00:27 +02:00
ce0cd1fce8
(misc): build with trtllm 0.13.0
2024-10-21 10:00:27 +02:00
188e4dc64f
(misc: build for sm_{75,80,86,89,90} by default
2024-10-21 10:00:27 +02:00
544c9d9dba
(fix): HOPPER_SM_MAJOR is 9 not 8
2024-10-21 10:00:27 +02:00
213acc6e34
(misc) move to latest trtllm
2024-10-21 10:00:27 +02:00
507ff66692
(misc) rerun-if-changed all the cmake modules
2024-10-21 10:00:27 +02:00
b242f45c04
(misc) delete backend.rs
2024-10-21 10:00:27 +02:00
984ae9798f
(post) impl postprocessing
2024-10-21 10:00:27 +02:00
fa63db0d07
(scheduler) rework submit/pull logic
2024-10-21 10:00:27 +02:00
42ccf4e77c
(misc) no need to move for uint32_t items
2024-10-21 10:00:27 +02:00
b41875c139
(misc) simplify [make_]move_iterator by using c++20 type inference
2024-10-21 10:00:27 +02:00
b1846fb4e6
(backend) refactor & cleanup
2024-10-21 10:00:27 +02:00
483f172938
(ffi) do not use reference capture in lambda as we are not capturing anything
2024-10-21 10:00:27 +02:00
3d0e90b631
(ffi) missing namespace for tle::Response
2024-10-21 10:00:27 +02:00
8e648ce425
(ffi) fix usage of wrong vector constructor making a capacity fill call
2024-10-21 10:00:27 +02:00
dddc9a44bd
(build) fetchcontent use archives instead of git
2024-10-21 10:00:27 +02:00
089c5fe668
(server) forward auth_token to server::run
2024-10-21 10:00:27 +02:00
291eaa99fb
use blocking_recv in looper to consume awaiting_requests at max before pulling in a single step
2024-10-21 10:00:27 +02:00
7bebc629af
(misc) missing Result types for Rust
2024-10-21 10:00:27 +02:00
c2e21d8725
(backend) implement the post_processor background thread
2024-10-21 10:00:27 +02:00
0dca168bcb
(misc) change scope identifiers
2024-10-21 10:00:27 +02:00
933ab67aa1
(ffi) encode the provided user prompt within each request thread
2024-10-21 10:00:27 +02:00
0b0c30fe8b
(ffi) remove narrowing type warning
2024-10-21 10:00:27 +02:00
fb759bdd2a
(looper) new looper initial implementation
2024-10-21 10:00:27 +02:00
5f7c0b67c3
(ffi) add template specialization to catch and convert to Rust Result<T, tensorrt_llm::common::TllmException>
2024-10-21 10:00:27 +02:00
33c962ef41
(ffi) add missing headers imports
2024-10-21 10:00:27 +02:00
2883c042ed
(ffi) cleanup again
2024-10-21 10:00:27 +02:00
f4a74be384
(backend) expose PullNewTokens
2024-10-21 10:00:27 +02:00
b8a40a0af3
(backend) cleanup a bit
2024-10-21 10:00:27 +02:00
38b5263c61
(ffi) add max_new_tokens parameters
2024-10-21 10:00:27 +02:00
f6f689f509
(build) setup ccache if available
2024-10-21 10:00:27 +02:00
2a339f99dd
(trt)
2024-10-21 10:00:25 +02:00
0cd7538a48
(ffi) use const for GetSamplingConfig
2024-10-21 09:57:26 +02:00
cea64e234f
(chore) fmt ... why?
2024-10-21 09:57:26 +02:00
a3f7d76f7b
(launcher) default new server::run parameters to false for now
2024-10-21 09:57:24 +02:00
25b20cba2a
(backend) use parking_lot crate for RwLock fairness
...
# Conflicts:
# backends/trtllm/src/backend.rs
2024-10-21 09:57:16 +02:00
3f385991b0
More fixes trtllm ( #2342 )
...
* (backend) use parking_lot crate for RwLock fairness
* (docker) let's put rust in the TRTLLM folder when building
* (docker) build ompi with SLURM support
* (launcher) default new server::run parameters to false for now
* (chore) fmt ... why?
2024-08-14 12:02:05 +02:00
Morgan Funtowicz