How to switch LLM providers without rewriting agent code

Your agent runs on GPT-4o. Then pricing moves, a Claude model scores higher on your evals, or an EU customer asks where inference runs. These triggers arrive often. OpenAI cut o3 prices by 80% in June 2025. Anthropic launched Claude Opus 4.5. Each change affects which provider is cheapest for a given task. That ranking won't hold for long.

Direct provider SDKs turn that change into an engineering project. You rewrite API calls, swap authentication headers, change response parsing, and retest every tool chain that reads model output. Nearly 39% of respondents find switching AI infrastructure or service providers very or extremely difficult.

That difficulty is self-inflicted when agent code names the provider directly. Blaxel's Model Gateway is the implementation used in this guide.

An LLM gateway, also called an LLM proxy, removes the coupling. It's an abstraction layer between your agents and the model providers. Agents send requests to one endpoint with a model identifier. The gateway resolves that identifier to a provider, attaches credentials, and presents one request contract to the agent.

TL;DR

• Provider lock-in costs engineering time: Without an abstraction layer, switching providers requires changes to API calls, authentication, and response parsing.
• An LLM gateway decouples agents from providers: Your agent calls one endpoint. The gateway routes to the configured provider and handles provider-specific formats.
• Provider changes stay out of agent code: Update the provider or model behind the endpoint. Agent code and deployments stay untouched.
• Telemetry and cost tracking come free: Every inference call passes through the gateway. Token counts, latency, and cost land in one place.
• Policy-based routing adds operational control: Route by region, model tier, or token budget without touching agent code.

Why agent code shouldn't know which LLM it's calling

A direct provider client looks harmless in a small prototype. The problems surface as a team adds agents across providers or receives a compliance request.

Provider-specific SDKs create hidden coupling

The coupling runs deeper than the base URL. OpenAI expects Authorization: Bearer <key>. Anthropic requires x-api-key plus an anthropic-version header. Gemini uses x-goog-api-key.

Request bodies differ next. Anthropic puts the system prompt in a top-level system field. It allows only user and assistant roles in messages. Code that sends role: "system" inside messages[] fails there.

Responses differ too.

• OpenAI returns text output in choices[].message.content. Anthropic uses content[].text, while Gemini uses candidates[].content.parts[].text.
• OpenAI puts tool calls in tool_calls[]. Anthropic uses tool_use content blocks. Gemini uses functionCall parts.
• OpenAI signals termination with finish_reason: "stop". Anthropic returns stop_reason: "end_turn", while Gemini returns finishReason: STOP.
• OpenAI ends streaming with a data: [DONE] sentinel. Anthropic emits named events from message_start to message_stop. Gemini streams complete response fragments.

Direct integrations track providers and API revisions. Each SDK adds its own dependency tree, error contract, and retry semantics. Multiple providers mean multiple versions of everything to maintain.

Model routing belongs in the gateway, not application code

Load balancing left application code years ago, and model routing follows the same logic. Availability and latency influence which provider serves a request, as do cost and compliance policy. None of those inputs are visible to the agent, so the agent shouldn't decide.

Put routing in a gateway, not in clients. Centralized routing avoids repeated code. It also provides one place to test the logic. This approach follows Microsoft's guidance. Uber encountered "repeated work and inconsistencies in the approach" before it centralized LLM serving.

Centralized routing means centralized control. A policy can route European traffic to EU-hosted endpoints. It applies to every agent the moment you save it. No redeploy or code review is needed across a dozen repositories.

Failover also belongs in the gateway. Gateways with automatic failover can reroute requests without exposing the first failure to the agent. Provider failover requires separate planning.

Provider outages are routine. OpenAI's status history shows an API error incident lasting nearly seven hours on August 26, 2025. Standard API plans from OpenAI, Anthropic, and Google carry no contractual uptime commitment. Your reliability strategy depends on the gateway.

How to implement provider-agnostic model access

A switchable endpoint replaces direct SDK calls and keeps provider changes out of agent code. Blaxel's Model Gateway is the implementation used below. Azure API Management's AI gateway, Google's Apigee, and Envoy AI Gateway implement the same pattern. They use different configuration surfaces.

1. Replace direct provider SDKs with a unified model endpoint

Delete the provider client constructors from agent code. Replace them with one base URL and a model identifier. The gateway resolves the identifier to a provider and attaches the right credentials.

Blaxel's Model Gateway implements this pattern. Adding external model endpoints gives you single endpoints on one base URL. You no longer call each provider separately. You create a model endpoint in the console and pick the provider.

Supported providers include OpenAI, Anthropic, Mistral AI, Cohere, xAI, and DeepSeek. Integrations add Azure AI Foundry, AWS Bedrock, Gemini, Vertex AI, and HuggingFace. The provider API key is stored as an encrypted secret in Workspace Settings. Non-sensitive parameters like organization ID or region go in config fields.

Agent code references the endpoint name:

from blaxel.langgraph import bl_model
model = await bl_model("my-model")

The SDK wraps the provider SDK internally and handles authentication for you. That line stays identical whether my-model points at OpenAI, Anthropic, or Mistral. The endpoint configuration determines the backend.

Frameworks without a Blaxel adapter can call the gateway as an OpenAI-compatible endpoint. Send requests to POST https://run.blaxel.ai/{YOUR-WORKSPACE}/models/{YOUR-MODEL}/v1/chat/completions. Include the x-Blaxel-authorization and x-Blaxel-workspace headers.

Start by grepping your repositories for provider client constructors such as OpenAI( and Anthropic(. Every hit is a call site the swap will touch.

2. Configure routing rules and model selection

• Capability tier: The gateway evaluates every request against its configured routing policies. Send classification and extraction to a smaller model. Reserve frontier models for multi-step reasoning.
• Geographic compliance: route EU user data to EU-hosted endpoints.
• Token budget: cap spend per model tier or per team.

Route routine, high-frequency tasks to efficient small models. Gate frontier inference for complex reasoning. Agentic workflows can sharply amplify token use per task. A wrong tier therefore costs far more in an agent.

Blaxel's Model Gateway expresses these as deployment policies attached to a model API. A location policy restricts routing to specific regions. A token usage policy sets maxToken limits on input, output, or total tokens. Limits can apply per month, day, hour, or minute. Requests over the limit get an HTTP 429. Policies apply to every agent calling that endpoint, with no code changes.

Plan provider resilience by configuring a second endpoint on a different provider. Switch by configuration when the primary degrades. Automatic provider failover is planned but not currently available in Blaxel Model Gateway.

3. Centralize telemetry and cost tracking

The gateway sees every inference call and is the observation point for usage data. Token counts and cost per request land in one stream together with latency. You do not need to instrument each agent. Log every call with team, application, model, and token counts. That metadata makes AI cost allocation possible.

Only 45.4% of enterprises had real-time dashboards for token and cost tracking in IDC's July 2026 survey. Without one, the first cost signal is the invoice.

Blaxel's Model Gateway records request count and rate. It records end-to-end latency at average, p50, p90, and p99. It also records input and output tokens, plus their total.

Alert on token anomalies at this layer. A publicly reported Claude Code loop on July 21, 2025 consumed 1,673,680,266 tokens in about five hours. A maxToken policy with hourly granularity rejects that traffic long before the bill arrives. Configure one ceiling per agent endpoint. Then set an alert at half the ceiling.

4. Test provider switching with zero downtime

Validate the architecture by switching a live endpoint. Change my-model from one provider to another in the endpoint config. The next inference call uses the new provider; no agent redeployment is required.

Run a shadow comparison before cutting over production traffic. Create a second endpoint on the target provider. Replay a sample of recent requests against both. Compare output quality and latency side by side, then review token cost. Commit to the switch only when the target meets your eval thresholds.

Document OpenAI Chat Completions' response_format with a json_schema among the provider-specific behaviors your agents depend on. Anthropic uses forced tool use through tool_choice. Gemini uses generationConfig.responseSchema. These incompatible controls make structured output the usual break.

Chat Completions provides prompt_tokens. Anthropic and the Responses API use input_tokens, while Gemini provides promptTokenCount.

The OpenAI Agents SDK documents a buffering option because some providers stream tool-call deltas incompatibly. LangChain disables stream_usage by default when it detects a non-OpenAI base URL. One endpoint covers the common paths. Provider differences still surface in complex tool-use workflows, so test those first.

Operational benefits beyond provider switching

Provider switching is the headline benefit. Teams that route every call through one gateway gain additional operational benefits. Those benefits grow more valuable with each agent added.

Unified observability across all inference calls

One telemetry stream covers every model call from every agent. You get latency percentiles and error rates by provider, plus tokens by model and agent. When agents call providers directly, each SDK logs independently. Cross-provider comparison then requires a merge job nobody owns.

The gateway also insulates you from unstable telemetry standards. OpenTelemetry's GenAI semantic conventions use gen_ai.provider.name for the provider and gen_ai.request.model for the model. Input-token usage is recorded in gen_ai.usage.input_tokens. Every attribute carries Development stability. The dedicated repository has no published release. gen_ai.system was already deprecated in favor of gen_ai.provider.name. Updating attribute names in one gateway beats updating them in every agent.

Request-level traces also catch failures that uptime monitoring misses. Anthropic's September 2025 postmortem found 16% of requests to Sonnet 4 hit incorrect routing at the worst point. No availability outage was declared.

Gateway metrics can expose latency and error anomalies before users report them. Detecting semantic degradation requires shadow traffic or an eval harness. Cost attribution follows from the same data once every call carries a client identity. You can attribute costs to each agent and roll them up by project or team.

Compliance and data governance at the gateway

Data residency is enforced by endpoint selection. Endpoint selection is the gateway's job. Azure AI Foundry DataZoneStandard deployments process prompts inside the EU data zone. Global deployments may process them in any region worldwide.

Google exposes an EU jurisdictional endpoint at aiplatform.eu.rep.googleapis.com. For AWS Bedrock workloads, EU geographic inference profiles are available. Claude requires a different path because Anthropic documents no first-party EU endpoint. EU-resident Claude traffic therefore goes through Vertex AI or Bedrock. A location policy encodes those choices once. Agent code never sees them.

Audit trails also come from the gateway. The EU AI Act requires deployers of high-risk systems to keep logs for at least six months. A gateway log records which agent called which model, when, and with what parameters. Compliance reviewers get one source instead of a log hunt across repositories.

Zero Data Retention (ZDR) is a provider-side arrangement. Keep regulated agents inside it.

• OpenAI: requires prior approval.
• AWS Bedrock: applies zero retention by default for most models.

Restrict the regulated endpoint to providers where your ZDR agreement is active.

Cost control without limiting developer velocity

Per-model and per-team token ceilings stop runaway spend without slowing anyone down. Developers keep calling the same endpoint. The gateway enforces the limit. For high-volume or autonomous agent workflows, treat budget controls and maximum iteration limits as core cost controls.

Nearly 93% of organizations exceeded their AI budgets in May 2026. Token usage for one task can vary by up to 30 times. Budgets built on average token counts miss by design.

Average monthly agent inference spend reached six figures in IDC's same survey. Most enterprises were over their budget. Once every call carries a team identity, the same stream bills each team for its actual usage.

Project next quarter's inference cost from observed tokens per agent per day. Do not use vendor list prices. Set the first ceilings near the high end of observed daily usage. Tighten them as data accumulates.

[CTA banner: get-started]

Conclusion: decouple your agents from your LLM provider

An LLM gateway between your agents and model providers decouples agent logic from model selection. The agent keeps calling one endpoint name while the gateway controls its provider-specific model configuration and deployment region.

Teams without this gateway pay for its absence repeatedly. Price cuts and deprecation notices become migration projects, as does every compliance request. Anthropic retired Claude Sonnet 4 and Opus 4 on June 15, 2026. The engineering cost multiplies by the number of agents in the fleet.

Blaxel, the perpetual sandbox platform, runs its Model Gateway alongside sandboxes for agent code execution. Blaxel supports coding and codegen agents, along with workloads that need centralized routing, telemetry, or token controls. It gives you unified model endpoints across OpenAI, Anthropic, Mistral AI, and other providers. You also get location and token policies. Built-in telemetry and maxToken controls are included; the Models overview documents capabilities and providers. Start building at app.blaxel.ai or talk to the team at blaxel.ai/contact.

FAQ

How much latency does an LLM gateway add?

A gateway adds a routing hop, but usually far less latency than inference itself. Use a latency analysis dashboard to separate gateway processing from target processing. Claude 3.5 Haiku reached 0.6 seconds p50 time to first token using Bedrock's latency-optimized inference. Run the gateway next to the agent's execution environment to keep the hop short.

Which agent frameworks can use an LLM gateway?

Any framework with a configurable base URL can use a gateway. LangChain's ChatOpenAI accepts base_url. CrewAI's LLM takes base_url with custom_openai=True. The Vercel AI SDK offers createOpenAICompatible. Blaxel also ships bl_model and blModel adapters for LangGraph, CrewAI, LlamaIndex, Google ADK, and the Vercel AI SDK. Any OpenAI-compatible client can target the gateway URL directly.

What happens during a provider outage?

The gateway preserves a stable client interface during an outage, but provider diversity still requires operational planning. Prepare a secondary endpoint on a different provider and account for Blaxel Model Gateway's current lack of automatic provider failover. A gateway does not erase provider-specific tool-use behavior or guarantee compatibility across providers, so validate critical workflows before an outage.