The AI agent infrastructure stack: what engineering leaders need to get right

Your agent worked in staging. It parsed documents, called tools, generated code, and returned clean results. Then you deployed it. Real users exposed every infrastructure gap you didn't know existed.

Cold start latency stacked across sequential tool calls. State vanished between sessions, forcing expensive re-initialization. Security gaps from running untrusted code on shared infrastructure blocked your first enterprise deal. Memory consumption spiked during tool call bursts, killing processes that worked fine under test load.

Compliance teams flagged shared-kernel container isolation as insufficient for multi-tenant deployments. These failures don't surface during development. Production traffic with hundreds of concurrent users hitting unpredictable execution paths will find every weak point.

This article maps the AI agent infrastructure stack as it exists in production today. It covers execution, orchestration, and observability across the full stack. Each section addresses specific architectural decisions and identifies where poor choices create the most downstream pain.

Why AI agent infrastructure diverges from traditional ML infrastructure

Your existing ML infrastructure handles two workload types well. Batch training jobs run for hours on persistent GPU clusters. Inference endpoints process stateless requests in predictable sub-second cycles. Agent workloads break both models.

Agents are non-deterministic, multi-step, and code-generating. A single interaction might query a database, call APIs, generate code, and execute it in a sandbox. The agent evaluates the output, then repeats with different tools.

The AgentCgroup study found that OS-level execution accounts for 56–74% of end-to-end task latency. Tool calls, container initialization, and agent setup dominate. LLM reasoning accounts for only 26–44%. Teams optimizing GPU throughput are addressing the smaller portion of the problem.

Memory patterns diverge even more sharply. Traditional inference has stable per-request memory footprints. Agent workloads exhibit peak-to-average memory ratios above 15x during tool call bursts, according to the same AgentCgroup research. Average-based resource allocation leads to out-of-memory kills during spikes.

Traditional autoscaling reacts to metrics averaged over minutes. Tool calls create sub-second bursts that complete before autoscaling detects them. For engineering leaders, this means standard capacity planning doesn't work for agent workloads.

The five layers of the AI agent infrastructure stack

Each layer represents an architectural decision with real tradeoffs. Getting one layer right doesn't compensate for getting another wrong. Latency compounds across layers. Security gaps at the execution layer undermine compliance work elsewhere. Poor observability makes debugging any other layer harder.

Reasoning and model routing

Foundation models sit at the base of every agent system. The first decision is single-provider or multi-model routing.

Single-provider setups are operationally simpler. One API key, one billing relationship, one set of rate limits. The risk became concrete during Claude outages in early March 2026 that left single-provider teams completely down for hours.

Multi-model routing adds complexity but protects against outages. A model gateway handles authentication, budget verification, rate limiting, caching, and routing. Track input and output tokens separately per request. Attribution should cover specific users, teams, and endpoints. Token cost control at this layer prevents runaway spending from reasoning loops.

The tradeoff is a business continuity decision, not a technical preference.

Orchestration and workflow control

Orchestration governs how agents plan, execute, and recover across multi-step tasks. Production patterns fall into three categories:

• Graph-based frameworks: LangGraph and similar tools provide native cycle support and state checkpointing for complex workflows.
• Role-based frameworks: CrewAI and others offer fast abstraction-driven builds for multi-agent collaboration.
• Custom systems: Maximum control at the cost of building everything from scratch.

Regardless of framework, production orchestration requires explicit state checkpointing, circuit breakers for failure isolation, and end-to-end telemetry. Teams also face long runtimes requiring checkpoint management, complex routing between agents, and data governance at the orchestration layer.

Tool integration and protocol standards

Before Model Context Protocol (MCP), integrating agents with external tools required custom code for every combination. MCP standardizes this through a stateful JSON-RPC session protocol defined in its specification. The three-tier architecture separates hosts, clients, and servers. Servers expose tools, resources, and prompts.

Adoption has been rapid. MCP SDK downloads reached millions monthly by late 2025. Salesforce, SAP, Google, Microsoft, OpenAI, and Google DeepMind have committed to integration. For teams building new tool layers, this adoption level means proprietary integration approaches carry migration risk.

The operational challenges in production are real, though. The specification doesn't define rate limiting, connection pooling, or disaster recovery. MCP's stateful session design complicates horizontal scaling. Sessions can't be routed to arbitrary server instances. Authentication relies on OAuth 2.1, and the specification prohibits simple token passthrough. Each tool server needs its own auth implementation.

Secure execution environments

Agents that generate and execute code need stronger isolation than standard containers. Containers share the host operating system kernel. Kernel exploits in one container can affect all co-located workloads.

MicroVMs operate through a hypervisor. Each workload gets an independent, dedicated kernel with hardware-partitioned resources. The Firecracker USENIX paper quantifies the difference. Firecracker exposes only five emulated devices to guests. The implementation is roughly 50,000 lines of Rust. Breaking microVM isolation requires escaping both the guest kernel and the hypervisor.

State persistence is the second architectural choice. Ephemeral sandboxes that reset after each invocation force agents to re-clone repositories and rebuild context. Persistent sandboxes maintain state between sessions, eliminating re-initialization cost.

Perpetual sandbox platforms like Blaxel combine microVM isolation with sandboxes that remain in standby indefinitely. Sandboxes resume in under 25ms with complete state restored: filesystem plus running processes. They return to standby within 15 seconds when connections close, with zero compute charges during standby.

Observability and evaluation

Traditional application performance monitoring (APM) tools were built for deterministic, linear request-response flows. Agent workloads violate every assumption behind them. The same input produces different outputs across runs. Multi-step reasoning loops can't be represented as linear traces.

Traditional APM also misses the economic dimension. Token usage is both a cost signal and a behavioral signal. Excessive token consumption often indicates runaway reasoning loops. By the time CPU and memory metrics alert, unexpected costs have already accumulated.

Agent-specific observability needs hierarchical tracing that captures reasoning steps, tool invocations with semantic purpose, decision points, and per-call token metrics. Evaluation frameworks for agent reliability remain in early development as of early 2026. Human eval remains essential for validating agent outputs where automated metrics fail to capture reasoning quality.

Where infrastructure decisions create the most downstream pain

The layers above interact. Poor choices in one layer compound problems across others.

Cold start latency compounds across tool calls

Agents making sequential tool calls accumulate latency at each step. Standard serverless platforms show cold start latency of 200–400ms at p50 and roughly 1.2 seconds at p99.

An agent making five sequential tool calls compounds that to roughly 1.5 seconds at the median. Tail latency hits 6 seconds. With ten sequential calls, tail latency can reach 12 seconds. For coding assistants and PR review agents, this kind of delay breaks the user experience entirely.

Always-on compute eliminates cold starts but means paying for idle time. Scale-to-zero cuts idle costs but introduces multi-second delays. A third option exists: sandboxes that return to standby after inactivity, then resume in under 25ms. This eliminates cold start accumulation without the always-on cost.

State loss forces expensive re-initialization

Agents that lose state between invocations repeat expensive operations. Context rebuilt from scratch creates quadratic token growth because each subsequent turn reprocesses all prior context.

Research from Stevens Institute found that unconstrained agents incur $5–$8 per task when context must be repeatedly reloaded. Adding a dedicated memory layer reduced initialization latency from 30 seconds to 300 milliseconds. That cost difference multiplies fast in production.

Multi-agent systems make this worse. An orchestrator spawning 50 workers that each re-clone the same repository pays for that operation 50 times per cycle. Cost and latency scale linearly with agent count.

Perpetual sandbox platforms address this by maintaining a complete filesystem and memory state in standby mode. Agents resume exactly where they stopped. For longer-term persistence beyond individual sessions, persistent volumes provide storage across sandbox sessions for months.

Security isolation gaps block enterprise adoption

Enterprise buyers require tenant isolation that containers can't guarantee. American Institute of Certified Public Accountants (AICPA) SOC 2 Trust Services Criteria CC6 governs logical access controls for multi-tenant platforms. Container architectures require extensive documentation of compensating controls around shared-kernel risk.

MicroVM architectures provide cleaner audit evidence because hypervisor-level boundary enforcement directly demonstrates protection.

Agents processing electronic protected health information face additional requirements. Health Insurance Portability and Accountability Act (HIPAA) technical safeguards under 45 CFR § 164.312 apply.

No HHS guidance explicitly addresses container isolation for multi-tenant AI agent environments. MicroVMs reduce compliance ambiguity because hardware-enforced boundaries require fewer interpretive arguments during audits.

Building versus buying AI agent infrastructure

Building production-grade sandbox infrastructure with Firecracker requires significant investment. Experienced practitioners estimate the work takes years of skilled engineering effort to reach minimum security standards. SOC 2 Type II or ISO 27001 certification adds another 12–18 months. Ongoing operations require a dedicated platform engineering team.

The build path makes sense for teams with deep KVM expertise, highly differentiated security requirements, or prohibitive scale economics. For most Series A through Series C startups, that investment delays product development by years.

The vendor dependency risk is real. Evaluate financial stability, incident transparency, and support responsiveness before committing. The honest calculation: Does allocating a dedicated infrastructure team serve your company better than a managed platform?

How the AI agent infrastructure stack is evolving

Several shifts are reshaping agent infrastructure with production evidence.

MCP has achieved near-universal vendor adoption with official integrations from every major cloud provider. Building proprietary tool integration layers today creates migration risk as the ecosystem standardizes.

Execution environments are converging on microVM isolation. The same Firecracker research shows microVMs boot in 125–200ms with roughly 5MB memory overhead. The performance gap that once made microVMs impractical has closed. The choice is now a threat model decision, not a performance tradeoff.

Agent-to-agent communication patterns are emerging but lack standardization. Teams build asynchronous JSON-based messaging and pipeline orchestration with specialized roles. No dominant protocol has emerged yet. Build orchestration layers that can accommodate a future standard.

Evaluate your AI agent infrastructure stack now

The infrastructure decisions you make today constrain what your agents can do for years. Wrong execution layer choices mean cold start latency that breaks the user experience. Security mistakes stall enterprise deals during procurement. Poor state persistence compounds cost and latency with every interaction.

Start by auditing your current cold start and resume behavior under a realistic load with sequential tool calls.

Perpetual sandbox platforms like Blaxel address these constraints across the full stack. Blaxel Sandboxes provide microVM isolation with sub-25ms resume from perpetual standby – even after months – and zero compute cost while idle. Agents Hosting deploys agent logic on the same infrastructure, eliminating network latency between agent and execution environment – no matter the agentic framework. Sandboxes can scale up to 50,000+ concurrent machines.

MCP Servers Hosting manages tool integration with 100+ pre-built tools. The Model Gateway unifies LLM access with token cost controls. Batch Jobs handle parallel processing for fan-out workloads.

Pricing

• Free: Up to $200 in free credits plus usage costs
• Pre-configured sandbox tiers and usage-based pricing: See Blaxel's pricing page for the most up-to-date information
• Available add-ons: Email support, live Slack support, HIPAA compliance

Start at app.blaxel.ai or book a conversation with the Blaxel team to walk through your architecture.

FAQs

How do I design a human eval for an agent when infrastructure is part of the failure mode?

Treat human eval as an end-to-end system test. Review complete traces: prompts, tool calls, execution logs, and final outputs. Score both outcome quality and operational correctness. When you see failures, label the root cause explicitly. Separate reasoning failures from orchestration issues, tool integration problems, and execution constraints. This prevents model churn when the real issue is cold starts or missing states.

What should I ask vendors to prove about cold starts and state persistence?

Ask for tail latency data during sequential tool-call chains. Clarify what "state" means: filesystem only, or running processes and in-memory state across reconnects. If a platform claims fast resume, ask what mechanism provides it and what conditions break it. Timeouts, evictions, region moves, and zero-data-retention modes all affect state behavior.

What usually breaks first when scaling MCP in production?

Session state and auth flows break first. MCP's stateful sessions mean load balancers can't freely route requests across instances. You need a session strategy. OAuth 2.1 adds operational work for token lifecycles, scopes, consent flows, and incident response. Before scaling, design for sticky routing or session-aware sharding, rate limiting, and per-tool backpressure. The base MCP protocol doesn't standardize those concerns.

For compliance, what changes in the audit packet when you move from containers to microVMs?

In a SOC 2 audit, container-based multi-tenancy forces you to document compensating controls around shared-kernel risk. With microVMs, the isolation boundary is easier to describe and evidence. Separate guest kernels and hypervisor-enforced separation provide a cleaner narrative. You still need patching and monitoring either way. The "why isolation is adequate" argument is usually more straightforward with virtualization.