The bill that broke me last year was the second month of a feature I was proud of. The product was working. Users were happy. The model was Claude Opus on every request, including the ones where a junior model would have done the job in a third of the time for a tenth of the cost. I knew the bill was going to be high. I did not know it was going to be that high. I spent a weekend rewiring the feature to pick a model per request based on the actual difficulty of the work, and the bill the next month was 70 percent lower with no measurable drop in quality.
That weekend is the experience that taught me the LLM router pattern is not a nice-to-have. It is the difference between a feature you can run profitably and a feature that quietly destroys your margins. By 2026, every serious AI product I know of routes per request, falls over to a backup provider when the primary one blinks, and treats the choice of model as a runtime decision rather than a hard-coded constant. The teams that have not figured this out are paying multiples of what they should be, or shipping features that fall over the first time their provider has a hiccup.
This is what the pattern looks like, why it works, and how to build it without ending up with a worse abstraction than the one you started with.
Why One Model Is The Wrong Default
The instinct to pick one model and ship is reasonable. It is simpler. It is one set of prompts to tune, one provider to monitor, one bill to pay. The reason it stops working is that real apps have a wide spread of request difficulty, and one model cannot be optimal for all of them.
A summarization that takes a paragraph in and a sentence out does not need a frontier model. A code refactor across a multi-file diff probably does. A classification with three possible answers and a clear schema can be handled by a model that costs a tenth of what you are paying right now. A long-context analysis with a thousand-page PDF needs the model that actually has a thousand-page context window without going into denial.
When you pick one model, you are setting the price floor for your easy requests at the cost of your hard requests, and the price ceiling for your hard requests at the capability of whatever model you picked. Easy requests overpay. Hard requests sometimes underdeliver. The economics get worse as your traffic mix shifts.
The other half of the problem is reliability. Every model provider has had at least one bad day in the last year. Some of them have had bad weeks. If your app cannot route around an outage, your uptime is bounded by the worst day of your single provider, and that is not a number you want to put in a customer SLA.
What An LLM Router Actually Is
The router is the layer between your app and the providers. It looks at the request, decides which model should handle it, calls that model, and falls back to another model if the call fails or returns something unusable. The decision can be based on the prompt, the user, the cost budget, the latency budget, or any combination.
There are three layers of routing that matter, and they compose.
The first is provider routing. The same model is available from multiple providers (OpenAI’s GPT-4-class models also surface through Azure, Anthropic’s Claude through AWS Bedrock and Google Vertex, open weights through a dozen inference providers). Routing across providers gives you redundancy without changing the model. When the primary provider rate-limits or returns 500s, the secondary handles the request and the user does not notice.
The second is model routing within a provider. You have access to a small, fast, cheap model and a large, slow, expensive one from the same provider. The router picks the smaller model when the work is simple and the larger model when it is not. The classification can be a heuristic, a small classifier model, or just the prompt template you are using.
The third is strategy routing across model families. Different model families have different shapes of strength. Anthropic models tend to follow long instructions better. OpenAI’s reasoning models are strong at structured tasks with verifiable outputs. Open-weight models are unbeatable on cost for tasks that fit them. Google models lead on long-context retrieval. The router can pick the right family for the task, not just the right size.
In production, all three layers run together. A single request might be routed to the cheap variant of the right family on the secondary provider during an outage, transparently, with a fallback to the primary variant on the primary provider if the cheap one returns something the verifier rejects.
The Tools That Make This Manageable In 2026
You can build a router from scratch with HTTP calls and a switch statement. People do. It works for a while. The reason it stops working is that the surface area of “which provider, which model, which version, which key, which region” expands faster than your switch statement can keep up with, and every change is a deploy.
The tooling that has converged in 2026 to handle this for you is worth knowing.
Vercel AI Gateway is the option that has the most traction in the JS and TS world. It exposes a single endpoint that fronts dozens of providers and models, with a unified API. You pass "provider/model" as a string and the gateway handles auth, retries, and observability. Fallbacks are configured per request or per project. Spend dashboards are built in. If you are using the AI SDK, the gateway is the default and you should not be using direct provider calls without a reason. The combination removes most of the integration tax of multi-provider routing. I covered the integration shape in the AI SDK v6 guide.
OpenRouter has been around longer and has a similar shape, with broader coverage of niche providers and open-weight inference endpoints. It is a strong choice if your routing needs are heavy on the long tail of providers, especially for cost-optimized open-weight serving.
Portkey leans more enterprise-y, with stronger access control, audit logging, and a UI for building routes that nontechnical stakeholders can review. If your organization has procurement processes around AI usage, Portkey’s positioning is built for that conversation.
LiteLLM is the open-source one. You self-host it and call it like an OpenAI-compatible endpoint. It is the right answer when you cannot or will not put your prompts through a third-party gateway, but you still want a normalized API across providers. The tradeoff is that you are now operating a piece of infrastructure that is on the critical path of every model call your app makes.
The decision between them is mostly about which compromises you are willing to take. Hosted gateways like Vercel AI Gateway and OpenRouter remove the operational burden but ask you to send your prompts and responses through their infrastructure. Self-hosted options like LiteLLM keep everything in your boundary at the cost of running another service. There is no clean win on both axes.
For most teams, a hosted gateway is the right call. The traffic is already going to a model provider’s infrastructure. Adding a routing hop that has zero data retention and a real SLA is not a meaningful change to the trust model, and the operational cost of self-hosting is not worth the perceived control.
How To Decide Which Model Handles A Request
Picking a model per request is the part most teams get wrong. The two failure modes are picking too coarsely (one prompt template, one model) or picking too finely (a meta-LLM-call to decide which LLM to call, which doubles your latency and your bill).
The pattern that works is to classify requests into a small number of buckets and assign a primary plus a fallback to each bucket.
Buckets I have ended up with on multiple projects look something like:
Cheap classification. Three to five output classes, short input, clear schema. Routes to a small fast model with the largest model in the same family as the fallback. The verifier is “did the output match the schema.” If not, retry on the fallback.
Structured extraction. Pull a defined schema out of unstructured text. Routes to a mid-size model with strong JSON-mode support. The verifier is schema validation. The fallback is a larger model in the same family or a different family known to be strong at structured output.
Open-ended generation. Write a paragraph, summarize a document, draft an email. Routes to a mid-size model with conversational strength. The fallback is a different family in the same size class.
Long-context analysis. A PDF, a transcript, a long document. Routes to a model with a context window that fits the document and pricing that is not punitive at that length. The fallback is the next best long-context model from a different provider.
High-stakes reasoning. Code generation, analytical answers, anything where being wrong is expensive. Routes to a frontier model. The fallback is the next-best frontier model from a different family.
Embedding and retrieval. Vector search and reranking. Routes to a fast embedding model from one provider with a fallback to another. Reranking goes to a small dedicated reranker rather than a general model.
The classifier into these buckets does not need to be an LLM. It can be the route the request came in on, the kind of request the user made in the UI, or a small heuristic on the prompt length. Reserve LLM-based classification for the cases where the heuristic genuinely cannot tell. Most cases can.
The reason this works is that you have already done the hard thinking at design time, and the runtime decision is cheap. The router is a function that maps (bucket, attempt, conditions) to (provider, model). It does not need to be smart. It needs to be predictable.
Fallbacks That Hold Up
Fallbacks are the part everyone agrees they need and almost no one tests. The first time a fallback fires in production is the wrong time to find out it does not work.
A few rules I have ended up with after watching fallbacks fail.
The fallback model has to be capable enough to actually substitute for the primary. Falling back from a frontier model to a tiny one because it is “available” produces outputs that are worse than just retrying the primary. Pick a fallback in the same capability tier or close to it. The cost saved by a worse fallback is rarely worth the quality drop.
The fallback should run on a different provider, not just a different region. Provider-wide outages are real. Region failovers within one provider do not protect you from those. The point of a fallback is to keep working when your primary is broken at the API level, not just at the rack level.
The fallback should produce a verifiable result. If the primary returns something invalid (broken JSON, hallucinated tool call, refusal), the verifier should reject it and the router should fall back. If the verifier is just “the call succeeded,” you will fall back to the same kind of bad output and call it a day. Real verifiers check the shape of the output against the schema, the signals in the content, and the expected ranges of the result.
The fallback should not loop. If both the primary and the fallback fail, the user gets a clean error, not an infinite retry. Cap the chain at two or three attempts and surface the failure to your error path. Quietly retrying forever is how outages turn into bills.
The fallback should be tested. Synthetic outage drills should run on a schedule. Force the primary off in staging, run real traffic patterns, watch the fallback handle them. The drills find the boring bugs, like a missing API key for the secondary, before the real outage does.
Cost Control Without Quality Regression
Routing for cost is the part most teams notice first, because the bill is the most visible signal. The trap is that aggressive cost optimization tends to ship quality regressions that take a while to show up in metrics.
The pattern I have ended up trusting is to optimize cost in the buckets where the work is well-defined and verifiable, and leave the open-ended buckets on the better model.
Classification, structured extraction, and embedding are well-defined work with clear ground truth. You can run an offline eval. You can swap in a smaller model. You can verify the eval did not regress. You can ship. The cost savings are real and the quality risk is small. The same eval discipline I wrote about for AI evals as a solo developer applies here. The eval is the part that lets you optimize cost without flying blind.
Open-ended generation is harder. The output is text. The eval is fuzzy. Swapping a smaller model in usually does not show up in a metric for weeks, and when it does, it shows up as user feedback that the product feels worse, which is hard to roll back without losing trust. For these buckets, I tend to leave the better model in place and find the cost savings elsewhere.
The other tactic that has worked is escalation. Try the cheap model first. If the verifier is happy, ship. If the verifier rejects the output, escalate to the better model. The cost shape is “cheap most of the time, expensive when it has to be,” and the quality shape is “as good as the better model at the worst case.” The latency cost is a single retry on the rejection path. The dollar cost is dominated by the cheap path because most requests succeed there. For workloads with a tight verifier, escalation is hard to beat.
The thing not to do is route by cost without an eval to back it up. “We saved 40 percent” with no quality measurement is a vibes-based win that quietly becomes a vibes-based loss over the next quarter.
Observability For The Router Itself
Every layer that sits between your app and your dependencies is a layer that needs observability of its own. The router is no exception.
The minimum I have found I need:
Per-request log of which bucket was chosen, which provider and model handled it, which fallback fired, how long it took, how many tokens it used, and whether the verifier accepted the result. Without all of this, debugging a slow or expensive request is a guessing game.
Aggregates over those per-request logs by bucket, by provider, by model. The signal you want is “the structured-extraction bucket on provider X is failing 8 percent of requests now versus 1 percent last week.” The signal you do not want is to find this out from a customer.
A spend dashboard that totals by bucket and by user. Cost spikes at the bucket level point at routing changes. Spikes at the user level point at abuse or at one customer’s prompt template doing something pathological. Both happen and both need to be visible.
Provider health that is independent of the router’s own success rate. Hitting a provider’s status page programmatically is fine for a coarse signal. Synthetic probes that hit each provider with a known-good request once a minute are better. They tell you when a provider is degraded before your real traffic notices.
The same patterns I covered in agent observability and debugging apply, with the addition of routing-specific signals. The router is part of the agent’s runtime. Treat it as a component you can debug, not a black box you hope is working.
Patterns That Almost Always Backfire
A few patterns look smart in the design doc and turn into pain in production.
LLM-based router decisions in the hot path. Calling a model to decide which model to call doubles the latency of every request and adds a category of bug where the router itself can fail. Reserve LLM-based routing for the cases where it is genuinely needed and bake the rest into deterministic logic. If you find yourself routing through a model, ask whether a heuristic on the request shape would do.
Per-user model choices exposed in the UI. Letting users pick the model is a feature for power users and a footgun for everyone else. Most users will not pick the best model for their task, will report inconsistent behavior across their sessions, and will turn into support tickets. If you do expose the choice, make the default smart and the override hidden in advanced settings.
Routing tables in code. A switch statement that maps prompt types to providers gets stale fast and turns every model swap into a deploy. Move the routing table to config, with a version, and reload it without redeploying the app. The cycle time of “we want to try a new model on bucket X” should be minutes, not days.
Caching across users without thinking. Routing and caching are tempting to combine. The risk is that caching identical prompts across users leaks data when the prompt contains anything user-specific. Per-user caches are safer. Per-request caches are safest. Shared caches need a hard policy on what is allowed to be cached. The same caution as prompt caching in production.
Fallback chains longer than three. Five-deep fallbacks are not redundancy. They are slow failure. Cap the chain. Fail loud when it exhausts.
What To Build First
If you are starting from a one-model app and want to land the router pattern without the migration becoming a project of its own, the steps that have worked for me are roughly:
Move all model calls behind a single function in your app. One call site, one function, no scattered SDK calls. This is the precondition for everything else.
Move that function to a hosted gateway. Set the model as a parameter. The behavior should be identical to before. This is the layer where you get fallbacks for free without changing the app’s logic.
Add buckets for the easiest wins. Classification, structured extraction, embedding. Set primary and fallback per bucket. Write an eval for each. Verify before shipping.
Add escalation for the buckets where it makes sense. Cheap model first, fallback to better model on verifier rejection. Watch the cost and the verifier rejection rate together. Tune.
Build the observability. Per-request logs, aggregates, spend by bucket. Wire alerts to the metrics that matter, especially provider health and verifier rejection rate.
Iterate from there. New buckets when new request shapes appear. New providers when the existing ones underperform. New model versions when they become available. The router is not a thing you build once. It is a thing you keep tuning as your traffic and your providers evolve.
Where This Is Going
The interesting frontier is automatic routing. Right now, the bucketing is hand-designed. The next step is routers that learn from observed performance which model handles which kind of prompt best, and route accordingly without being told. Early versions of this exist in some gateways. They are not yet good enough that I would trust them blind, but the trajectory is real.
The other thing happening is the consolidation of provider APIs. Every provider has had to add OpenAI-compatible endpoints because that is the API the ecosystem standardized on. The pain of integrating with a new provider is dropping. The router pattern was a workaround for an ecosystem with too many shapes. As the shapes converge, the router becomes thinner and the gateway becomes more about policy and observability than translation.
The thing that is not changing is that one model is not enough for a real product. The frontier is going to keep producing better models. Older models are going to keep getting cheaper. The right answer is going to keep being “a mix, picked per request, with fallbacks,” and the teams that build for that pattern from day one are the ones whose costs and reliability stay sane as the ecosystem keeps moving.
The bill that scared me into this pattern was the cost of not having it. The next one I do not have to worry about. Routing the easy work to a cheap model and the hard work to a frontier model is not clever architecture. It is just paying attention to what the work needs. That, more than anything else I have built into AI products, has held up across model generations and provider shake-ups. The pattern outlasts the parts.
If you are still calling one model from one provider for every request your app makes, the router is the first piece of infrastructure to add. It pays for itself in the first month, and it keeps paying.