Smart Router: every prompt gets a lane.
An OpenAI-compatible LLM gateway. Hand-written rules score each prompt in under a millisecond and switch it to the cheapest model tier that fits, with fallback to the next tier and a cost log against GPT-4o.
- provider
- demo
- classifier
- rules, <1 ms
- baseline
- gpt-4o
The 60-second version
Python · FastAPI- Problem
- Apps send every prompt to one expensive model, even “what is 2+2?”. Most prompts don’t need it.
- What I built
- A drop-in OpenAI-compatible gateway that scores each prompt with rules in under 1 ms, picks the cheapest tier that fits, falls back if a provider fails, and logs the cost.
- Proof
- 10 of 24 held-out prompts routed to the right tier (23 of 24 on the prompts the rules were tuned on), 150 tests, CI on every push. The misses are shown, not hidden.
Try it
3see the cost dashboard
FastAPIasync httpxPydanticRedis cachePostgres + AlembicSSE streamingpytestDockerGitHub Actions
Routing report card
raw json →Running the classifier over the labelled prompts…
Switchboard
Loading the tiers…
Complexity scorescore-
Rules firedpoints
- ··No prompt scored yet.
Cost estimate
Priced per request once it is sent: this tier against GPT-4o list price for the same tokens.
Response
Press “Route this prompt” to send it through the gateway.
Traffic on this instance
open dashboard →No requests yet.
How it works request path
The gateway speaks the OpenAI chat format, so an app only changes its base URL. No LLM is used to classify: the score comes from length, keywords, code, multi-step and math patterns.
Drop-in API
POST /v1/chat/completions, plus X-Router-Tier and X-Router-Model headers.ExplainableEvery decision traces back to the rules in the log above;
/v1/route/explain returns them.ResilientA failing provider escalates to the next tier, for normal and streamed requests.
TestedUnit tests with fake providers cover routing, fallback, streaming, cache, auth and limits.
Call it like OpenAI python · curl
Point any OpenAI client at the gateway. On this public demo the answers are simulated and requests are rate limited. Full schema in the API docs.