BrighTO-Router: Open-Source Rust-Based LLM Gateway for Self-Hosted AI Traffic

A developer has released BrighTO-Router, a free open-source AI traffic gateway written in Rust, designed to give teams a single self-hosted API endpoint for managing large language model requests. The tool supports routing for OpenAI-compatible endpoints, Anthropic Messages, local models, reranking, and speech transcription, along with team API keys and token budgets. Built with Rust for predictable memory usage and fast async networking, the entire solution deploys via a single Docker image to minimize operational complexity. The project includes a PostgreSQL-backed usage ledger and supports round-robin and weighted load balancing across model groups. Benchmarks testing up to one million token payloads have been published, measuring router overhead rather than model inference speed to ensure reproducible results.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in