Morgan Funtowicz
|
31747163e7
|
chore(trtllm): remove unused method
|
2024-10-21 14:10:23 +02:00 |
|
Morgan Funtowicz
|
fb00f985ae
|
chore(trtllm): post-rebase commit
|
2024-10-21 12:31:24 +02:00 |
|
Morgan Funtowicz
|
85c03e33a9
|
chore(trtllm): fmt
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
62f33d7ecd
|
chore(trtllm): move dockerfile to right place
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
6687c06a21
|
feat(looper): minor optimizations to avoid growing too much the containers
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
027756c52d
|
chore(cmake): download timestamp should be before URL
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
629153b44b
|
feat(looper): check engine and executorWorker paths exist before creating the backend
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
f20ec28891
|
chore(cmake): use correct policy for download_timestamp
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
819c953771
|
misc(cuda): require 12.6
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
dd94ccc989
|
(fix): ore fixes for Dockerfile
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
f9f10a6636
|
(misc): improve trtllm download script robustness
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
0c3ba932cc
|
(misc): disable logging in release mode
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
437c2aa142
|
(misc): update dependency in trtllm dockerfile
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
cb69c9a967
|
(misc): update dependency in trtllm dockerfile
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
c8a99af6c9
|
(fix): do not recreate the stateful hashmap at every it
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
eb13d8d1f3
|
(misc): increase verbosity of spdlog
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
ce0cd1fce8
|
(misc): build with trtllm 0.13.0
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
188e4dc64f
|
(misc: build for sm_{75,80,86,89,90} by default
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
544c9d9dba
|
(fix): HOPPER_SM_MAJOR is 9 not 8
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
213acc6e34
|
(misc) move to latest trtllm
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
507ff66692
|
(misc) rerun-if-changed all the cmake modules
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
b242f45c04
|
(misc) delete backend.rs
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
984ae9798f
|
(post) impl postprocessing
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
fa63db0d07
|
(scheduler) rework submit/pull logic
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
42ccf4e77c
|
(misc) no need to move for uint32_t items
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
b41875c139
|
(misc) simplify [make_]move_iterator by using c++20 type inference
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
b1846fb4e6
|
(backend) refactor & cleanup
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
483f172938
|
(ffi) do not use reference capture in lambda as we are not capturing anything
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
3d0e90b631
|
(ffi) missing namespace for tle::Response
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
8e648ce425
|
(ffi) fix usage of wrong vector constructor making a capacity fill call
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
dddc9a44bd
|
(build) fetchcontent use archives instead of git
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
089c5fe668
|
(server) forward auth_token to server::run
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
291eaa99fb
|
use blocking_recv in looper to consume awaiting_requests at max before pulling in a single step
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
7bebc629af
|
(misc) missing Result types for Rust
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
c2e21d8725
|
(backend) implement the post_processor background thread
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
0dca168bcb
|
(misc) change scope identifiers
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
933ab67aa1
|
(ffi) encode the provided user prompt within each request thread
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
0b0c30fe8b
|
(ffi) remove narrowing type warning
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
fb759bdd2a
|
(looper) new looper initial implementation
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
5f7c0b67c3
|
(ffi) add template specialization to catch and convert to Rust Result<T, tensorrt_llm::common::TllmException>
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
33c962ef41
|
(ffi) add missing headers imports
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
2883c042ed
|
(ffi) cleanup again
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
f4a74be384
|
(backend) expose PullNewTokens
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
b8a40a0af3
|
(backend) cleanup a bit
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
38b5263c61
|
(ffi) add max_new_tokens parameters
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
f6f689f509
|
(build) setup ccache if available
|
2024-10-21 10:00:27 +02:00 |
|
Morgan Funtowicz
|
2a339f99dd
|
(trt)
|
2024-10-21 10:00:25 +02:00 |
|
Morgan Funtowicz
|
0cd7538a48
|
(ffi) use const for GetSamplingConfig
|
2024-10-21 09:57:26 +02:00 |
|
Morgan Funtowicz
|
cea64e234f
|
(chore) fmt ... why?
|
2024-10-21 09:57:26 +02:00 |
|
Morgan Funtowicz
|
a3f7d76f7b
|
(launcher) default new server::run parameters to false for now
|
2024-10-21 09:57:24 +02:00 |