smart-router / control room

Smart Router: every prompt gets a lane.

An OpenAI-compatible LLM gateway. Hand-written rules score each prompt in under a millisecond and switch it to the cheapest model tier that fits, with fallback to the next tier and a cost log against GPT-4o.

provider
demo
classifier
rules, <1 ms
baseline
gpt-4o

The 60-second version

Python · FastAPI
Problem
Apps send every prompt to one expensive model, even “what is 2+2?”. Most prompts don’t need it.
What I built
A drop-in OpenAI-compatible gateway that scores each prompt with rules in under 1 ms, picks the cheapest tier that fits, falls back if a provider fails, and logs the cost.
Proof
10 of 24 held-out prompts routed to the right tier (23 of 24 on the prompts the rules were tuned on), 150 tests, CI on every push. The misses are shown, not hidden.
Try it 3see the cost dashboard
FastAPIasync httpxPydanticRedis cachePostgres + AlembicSSE streamingpytestDockerGitHub Actions

Routing report card

raw json →

Running the classifier over the labelled prompts…

Console

Demo mode

Simulated answers. No language model is called and no API key is used. The classification, routing, fallback, cost estimate and analytics are the project's real code.

Ctrl/⌘ + Enter to send
Load an example

Switchboard

Loading the tiers…

Complexity scorescore-
Rules firedpoints
  1. ··No prompt scored yet.
Cost estimate

Priced per request once it is sent: this tier against GPT-4o list price for the same tokens.

Response
Press “Route this prompt” to send it through the gateway.

Traffic on this instance

open dashboard →

No requests yet.

How it works request path

The gateway speaks the OpenAI chat format, so an app only changes its base URL. No LLM is used to classify: the score comes from length, keywords, code, multi-step and math patterns.

App sends a chat request; cache check; rule-based classifier picks a tier; the tier's provider answers, falling back to the next tier on failure; cost is logged. cache hit (Redis, SHA-256 of messages): answered with no provider call your app OpenAI SDK gateway auth · limits · cache rule classifier score 0-15, < 1 ms simple · ≤ 1 medium · 2-3 complex · ≥ 4 provider down: try the next tier up provider Groq · OpenRouter · … cost log tokens · cost vs GPT-4o
Drop-in APIPOST /v1/chat/completions, plus X-Router-Tier and X-Router-Model headers.
ExplainableEvery decision traces back to the rules in the log above; /v1/route/explain returns them.
ResilientA failing provider escalates to the next tier, for normal and streamed requests.
TestedUnit tests with fake providers cover routing, fallback, streaming, cache, auth and limits.

Call it like OpenAI python · curl

Point any OpenAI client at the gateway. On this public demo the answers are simulated and requests are rate limited. Full schema in the API docs.