Cyberes 79b1e01b61 | ||
---|---|---|
.. | ||
README.md | ||
build-vllm.sh | ||
vllm-gptq-setup-no-cuda.py | ||
vllm.service | ||
vllm_api_server.py |
README.md
Nginx
- Make sure your proxies all have a long timeout:
proxy_read_timeout 300;
proxy_connect_timeout 300;
proxy_send_timeout 300;
The LLM middleware has a request timeout of 120 so this longer timeout is to avoid any issues.