81f1e2bc37
|
||
|---|---|---|
| .husky | ||
| data | ||
| docker | ||
| docs | ||
| scripts | ||
| src | ||
| .env.example | ||
| .gitattributes | ||
| .gitignore | ||
| .prettierrc | ||
| README.md | ||
| http-client.env.json | ||
| package-lock.json | ||
| package.json | ||
| render.yaml | ||
| tsconfig.json | ||
README.md
OAI Reverse Proxy
Reverse proxy server for various LLM APIs.
Table of Contents
What is this?
This project allows you to run a reverse proxy server for various LLM APIs.
Features
- Support for multiple APIs
- Translation from OpenAI-formatted prompts to any other API, including streaming responses
- Multiple API keys with rotation and rate limit handling
- Basic user management
- Simple role-based permissions
- Per-model token quotas
- Temporary user accounts
- Prompt and completion logging
- Abuse detection and prevention
Usage Instructions
If you'd like to run your own instance of this server, you'll need to deploy it somewhere and configure it with your API keys. A few easy options are provided below, though you can also deploy it to any other service you'd like if you know what you're doing and the service supports Node.js.
Self-hosting (locally or without Docker)
Follow the "Local Development" instructions below to set up prerequisites and start the server. Then you can use a service like ngrok or trycloudflare.com to securely expose your server to the internet, or you can use a more traditional reverse proxy/WAF like Cloudflare or Nginx.
Ensure you set the TRUSTED_PROXIES environment variable according to your deployment. Refer to .env.example and config.ts for more information.
Self hosting (with Docker)
If you have a Docker-capable VPS or server, use the Huggingface Dockerfile (./docker/huggingface/Dockerfile) to build an image and run it on your server.
Ensure you set the TRUSTED_PROXIES environment variable according to your deployment. Refer to .env.example and config.ts for more information.
Alternatives
Fiz and Sekrit are working on some alternative ways to deploy this conveniently. While I'm not directly involved in writing code or scripts for that project, I'm providing some advice and will include links to their work here when it's ready.
Huggingface (not advised)
See here for instructions on how to deploy to a Huggingface Space.
Render (not advised)
See here for instructions on how to deploy to Render.com.
Local Development
To run the proxy locally for development or testing, install Node.js >= 18.0.0 and follow the steps below.
- Clone the repo
- Install dependencies with
npm install - Create a
.envfile in the root of the project and add your API keys. See the .env.example file for an example. - Start the server in development mode with
npm run start:dev.
You can also use npm run start:dev:tsc to enable project-wide type checking at the cost of slower startup times. npm run type-check can be used to run type checking without starting the server.