 Cyberes
|
33b4b8404b
|
clean up streaming
|
2023-10-03 14:10:50 -06:00 |
|
Cyberes
|
581a0fec99
|
fix exception
|
2023-10-03 13:47:18 -06:00 |
|
Cyberes
|
94141b8ecf
|
fix processing not being decremented on streaming, fix confusion over queue, adjust stop sequences
|
2023-10-02 20:53:08 -06:00 |
|
Cyberes
|
f7e9687527
|
finish openai endpoints
|
2023-10-01 16:04:53 -06:00 |
|
Cyberes
|
2a3ff7e21e
|
update openai endpoints
|
2023-10-01 14:15:01 -06:00 |
|
Cyberes
|
93d19fb95b
|
fix exception
|
2023-10-01 10:25:32 -06:00 |
|
Cyberes
|
d203973e80
|
fix routes
|
2023-10-01 01:13:13 -06:00 |
|
Cyberes
|
25ec56a5ef
|
get streaming working, remove /v2/
|
2023-10-01 00:20:00 -06:00 |
|
Cyberes
|
e0f86d053a
|
reorganize to api v2
|
2023-09-30 19:42:41 -06:00 |
|
Cyberes
|
624ca74ce5
|
mvp
|
2023-09-29 00:09:44 -06:00 |
|
Cyberes
|
e7b57cad7b
|
set up cluster config and basic background workers
|
2023-09-28 18:40:24 -06:00 |
|
Cyberes
|
e1d3fca6d3
|
try to cancel inference if disconnected from client
|
2023-09-28 09:55:31 -06:00 |
|
Cyberes
|
e42f2b6819
|
fix negative queue on stats
|
2023-09-28 08:47:39 -06:00 |
|
Cyberes
|
59f2aac8ad
|
rewrite redis usage
|
2023-09-28 03:44:30 -06:00 |
|
Cyberes
|
a4a1d6cce6
|
fix double logging
|
2023-09-28 01:34:15 -06:00 |
|
Cyberes
|
e5fbc9545d
|
add ratelimiting to websocket streaming endpoint, fix queue not decrementing IP requests, add console printer
|
2023-09-27 21:15:54 -06:00 |
|
Cyberes
|
35e9847b27
|
set inference workers to daemon, add finally to inference worker, hide estimated avg tps
|
2023-09-27 18:36:51 -06:00 |
|
Cyberes
|
105b66d5e2
|
unify error message handling
|
2023-09-27 14:48:47 -06:00 |
|
Cyberes
|
aba2e5b9c0
|
don't use db pooling, add LLM-ST-Errors header to disable formatted errors
|
2023-09-26 23:59:22 -06:00 |
|
Cyberes
|
048e5a8060
|
fix API key handling
|
2023-09-26 22:49:53 -06:00 |
|
Cyberes
|
d9bbcc42e6
|
more work on openai endpoint
|
2023-09-26 22:09:11 -06:00 |
|
Cyberes
|
11e84db59c
|
update database, tokenizer handle null prompt, convert top_p to vllm on openai, actually validate prompt on streaming,
|
2023-09-25 22:32:48 -06:00 |
|
Cyberes
|
8240a1ebbb
|
fix background log not doing anything
|
2023-09-25 18:18:29 -06:00 |
|
Cyberes
|
8184e24bff
|
fix sending error messages when streaming
|
2023-09-25 17:37:58 -06:00 |
|
Cyberes
|
30282479a0
|
fix flask exception
|
2023-09-25 17:22:28 -06:00 |
|
Cyberes
|
3eaabc8c35
|
fix copied code
|
2023-09-25 12:38:02 -06:00 |
|
Cyberes
|
1646a00987
|
implement streaming on openai, improve streaming, run DB logging in background thread
|
2023-09-25 12:30:40 -06:00 |
|
Cyberes
|
6459a1c91b
|
allow setting simultaneous IP limit per-token, fix token use tracker, fix tokens on streaming
|
2023-09-25 00:55:20 -06:00 |
|
Cyberes
|
8d6b2ce49c
|
minor changes, add admin token auth system, add route to get backend info
|
2023-09-24 15:54:35 -06:00 |
|
Cyberes
|
2678102153
|
handle error while streaming
|
2023-09-24 13:27:27 -06:00 |
|
Cyberes
|
0015e653b2
|
adjust a few final things
|
2023-09-23 22:30:59 -06:00 |
|
Cyberes
|
7ee2311183
|
whats going on
|
2023-09-23 21:10:14 -06:00 |
|
Cyberes
|
41e622d19c
|
fix two exceptions
|
2023-09-23 20:55:49 -06:00 |
|
Cyberes
|
f67ac8175b
|
fix wrong approach for streaming
|
2023-09-23 18:44:07 -06:00 |
|
Cyberes
|
8a4de7df44
|
oops
|
2023-09-23 18:01:12 -06:00 |
|
Cyberes
|
76a1428ba0
|
implement streaming for vllm
|
2023-09-23 17:57:23 -06:00 |
|
Cyberes
|
03e3ec5490
|
port to mysql, use vllm tokenizer endpoint
|
2023-09-20 20:30:31 -06:00 |
|
Cyberes
|
79b1e01b61
|
option to disable streaming, improve timeout on requests to backend, fix error handling. reduce duplicate code, misc other cleanup
|
2023-09-14 14:05:50 -06:00 |
|
Cyberes
|
9740df07c7
|
add openai-compatible backend
|
2023-09-12 16:40:09 -06:00 |
|
Cyberes
|
8c04238e04
|
disable stream for now
|
2023-08-30 19:58:59 -06:00 |
|
Cyberes
|
2816c01902
|
refactor generation route
|
2023-08-30 18:53:26 -06:00 |
|
Cyberes
|
bf648f605f
|
implement streaming for hf-textgen
|
2023-08-29 17:56:12 -06:00 |