cyberes
/
local-llm-server
Archived
1
1
Fork 0
Code Issues Pull Requests Packages Projects Releases 1 Wiki Activity

docker: update readme

This commit is contained in:
Cyberes 2023-10-15 14:57:14 -06:00
parent 2d2d776fd7
commit 521db377b4
1 changed files with 2 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
other/vllm/Docker/README.md
View File

@ -30,9 +30,7 @@
You **must** have a GPU attached to your system when building the container (required for building VLLM). You **must** have a GPU attached to your system when building the container (required for building VLLM).
1. Install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) and CUDA 11.8. 1. Install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) and CUDA 11.8.
2. `sudo docker build .` 2. `sudo docker build .` If you want to build the latest VLLM, add `--no-cache`
If you want to build the latest VLLM, add `--no-cache`
Don't forget about `--progress=plain`
To run the container on your local machine: To run the container on your local machine:
@ -42,7 +40,7 @@ sudo docker run -it --shm-size 14g --gpus all -v /home/user/testing123/notebooks
You will need to create a directory to mount inside the container (for example: `/home/user/testing123/`). Within this should be the folder `models` that holds the model to load, `rathole-client.toml`, and `cmd.txt`. You will need to create a directory to mount inside the container (for example: `/home/user/testing123/`). Within this should be the folder `models` that holds the model to load, `rathole-client.toml`, and `cmd.txt`.
If you need to debug something you can start a shell inside the container: If you need to debug something, you can start a shell inside the container:
```bash ```bash
sudo docker run -it --shm-size 14g --gpus all -v /home/user/testing123/notebooks:/notebooks -v /home/user/testing123/storage:/storage -p 8888:8888 --entrypoint bash cyberes/vllm-paperspace:latest sudo docker run -it --shm-size 14g --gpus all -v /home/user/testing123/notebooks:/notebooks -v /home/user/testing123/storage:/storage -p 8888:8888 --entrypoint bash cyberes/vllm-paperspace:latest