Mirrors
/
oai-reverse-proxy
mirror of https://gitgud.io/khanon/oai-reverse-proxy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Reverse proxy server for various LLM APIs. Features translation between API formats, user management, anti-abuse, API key rotation, DALL-E support, and optional prompt/response logging.
548 Commits 14 Branches 0 Tags 2.2 MiB
TypeScript 93.6%
EJS 4.8%
CSS 1.1%
JavaScript 0.4%
Dockerfile 0.1%
Go to file
khanon 5eeb2875b4 adds haiku 3.5 2024-11-04 18:20:19 +00:00
.husky Add temporary user tokens (khanon/oai-reverse-proxy!42) 2023-09-09 22:21:38 +00:00
data OpenAI DALL-E Image Generation (khanon/oai-reverse-proxy!52) 2023-11-14 05:41:19 +00:00
docker updates render dockerfile to correctly copy patches dir into build context 2024-09-16 23:39:43 -05:00
docs updates render dockerfile to correctly copy patches dir into build context 2024-09-16 23:39:43 -05:00
patches Add configurable network interface or SOCKS/HTTP proxy for outgoing requests (khanon/oai-reverse-proxy!80) 2024-09-16 15:17:57 +00:00
public various improvements and fixes to PoW challenge UI and token refresh 2024-09-22 11:11:30 -05:00
scripts Implement priority queue by tokens (khanon/oai-reverse-proxy!79) 2024-09-09 16:48:46 +00:00
src adds haiku 3.5 2024-11-04 18:20:19 +00:00
.env.example Add configurable network interface or SOCKS/HTTP proxy for outgoing requests (khanon/oai-reverse-proxy!80) 2024-09-16 15:17:57 +00:00
.gitattributes initial commit 2023-04-08 01:54:44 -05:00
.gitignore adds dall-e full history page and metadata downloader 2024-03-10 14:53:11 -05:00
.prettierrc Temporary usertokens via proof-of-work challenge (khanon/oai-reverse-proxy!68) 2024-05-19 16:31:56 +00:00
README.md adds Sonnet 3.5v2 AWS model ID and adjusts AWS model assignment to raise error on no match 2024-10-23 13:39:34 -05:00
http-client.env.json Azure OpenAI suport (khanon/oai-reverse-proxy!48) 2023-12-04 04:21:18 +00:00
package-lock.json handles sillytavern using both camel and snake-cased parameters for gemini api 2024-10-06 11:03:48 -05:00
package.json fixes google ai language filter and updates readme 2024-10-18 23:15:02 -05:00
render.yaml Add docs and support for Render.com deployments (khanon/oai-reverse-proxy!9) 2023-05-15 21:47:30 +00:00
tsconfig.json refactors api transformers and adds oai->anthropic chat api translation 2024-03-08 20:59:19 -06:00

README.md

OAI Reverse Proxy

Reverse proxy server for various LLM APIs.

Table of Contents

What is this?

This project allows you to run a reverse proxy server for various LLM APIs.

Features

  • Support for multiple APIs
  • Translation from OpenAI-formatted prompts to any other API, including streaming responses
  • Multiple API keys with rotation and rate limit handling
  • Basic user management
    • Simple role-based permissions
    • Per-model token quotas
    • Temporary user accounts
  • Event audit logging
  • Optional full logging of prompts and completions
  • Abuse detection and prevention
    • IP address and user token model invocation rate limits
    • IP blacklists
    • Proof-of-work challenge for access by anonymous users

Usage Instructions

If you'd like to run your own instance of this server, you'll need to deploy it somewhere and configure it with your API keys. A few easy options are provided below, though you can also deploy it to any other service you'd like if you know what you're doing and the service supports Node.js.

Personal Use (single-user)

If you just want to run the proxy server to use yourself without hosting it for others:

  1. Install Node.js >= 18.0.0
  2. Clone this repository
  3. Create a .env file in the root of the project and add your API keys. See the .env.example file for an example.
  4. Install dependencies with npm install
  5. Run npm run build
  6. Run npm start

Updating

You must re-run npm install and npm run build whenever you pull new changes from the repository.

Local Development

Use npm run start:dev to run the proxy in development mode with watch mode enabled. Use npm run type-check to run the type checker across the project.

Self-hosting

See here for instructions on how to self-host the application on your own VPS or local machine and expose it to the internet for others to use.

Ensure you set the TRUSTED_PROXIES environment variable according to your deployment. Refer to .env.example and config.ts for more information.

Building

To build the project, run npm run build. This will compile the TypeScript code to JavaScript and output it to the build directory. You should run this whenever you pull new changes from the repository.

Note that if you are trying to build the server on a very memory-constrained (<= 1GB) VPS, you may need to run the build with NODE_OPTIONS=--max_old_space_size=2048 npm run build to avoid running out of memory during the build process, assuming you have swap enabled. The application itself should run fine on a 512MB VPS for most reasonable traffic levels.

Forking

If you are forking the repository on GitGud, you may wish to disable GitLab CI/CD or you will be spammed with emails about failed builds due not having any CI runners. You can do this by going to Settings > General > Visibility, project features, permissions and then disabling the "CI/CD" feature.