16
How engineering leaders should evaluate an AI agent infrastructure stack
Learn the eight layers of the AI agent infrastructure stack, where cold starts and isolation gaps hurt, and how to decide what to build or buy first.

For coding agents, most task time goes to the infrastructure around the model. In the AgentCgroup study of 144 coding tasks, OS-level execution accounted for 55% to 60% of end-to-end latency. LLM reasoning took the remaining 40% to 45%.
What does that mean for engineering leaders?
Container startup and tool execution shape coding-agent performance at least as much as the model does. Those same layers govern how you isolate untrusted code. They also set what you pay while agents sit idle between tasks.
So the stack deserves the same scrutiny teams already give model selection. Each layer brings its own tradeoffs, and the wrong early choice tends to surface later as latency or cost. The sections below break the stack into the eight layers buyers evaluate.
Key takeaways
- Treat the model provider choice as a business continuity call. A worldwide Claude outage on March 2, 2026 brought hours of elevated errors.
- Agent memory peaks reach up to 15.4x the average during tool call bursts. Capacity plans built on averages under-provision these workloads.
- Blaxel runs sandboxes in Firecracker microVMs that resume from standby in under 25ms. Standby costs nothing in compute.
- For AI-generated code, microVMs provide cleaner isolation evidence for auditors. Shared-kernel containers need more documented compensating controls.
What is an AI agent infrastructure stack?
An AI agent infrastructure stack is <><complete>
<3 paragraphs of words of 3 paragraphs>
Why AI agent infrastructure diverges from traditional ML infrastructure
Your existing ML infrastructure handles two workload types well. Batch training runs last for hours on persistent GPU clusters. Inference endpoints process stateless requests in predictable sub-second cycles. Agent workloads fit neither pattern.
Agent tasks are multi-step and non-deterministic. A single interaction might query a database, call APIs, generate code, and execute it in a sandbox. The agent evaluates the output, then repeats with different tools.
The AgentCgroup study found that OS-level execution accounts for 55% to 60% of end-to-end latency in coding-agent tasks. Tool calls, container initialization, and agent setup dominate. LLM reasoning accounts for 40% to 45%. For latency work, the execution layer is a bigger target than model serving.
Memory behaves differently too. Traditional inference has stable per-request memory footprints. Agent workloads can reach peak-to-average memory ratios of up to 15.4x during tool-call bursts, according to the same AgentCgroup research. Sizing memory to the average leads to out-of-memory kills during those spikes.
Traditional autoscaling reacts to metrics averaged over minutes. Tool calls create sub-second bursts that complete before autoscaling detects them. As a result, capacity plans built on averages under-provision agent workloads.
The eight layers of the AI agent infrastructure stack
Each layer is an architectural decision with its own tradeoffs. Getting one layer right doesn't compensate for getting another wrong. Latency adds up across layers. Security gaps at the execution layer undermine compliance work elsewhere. Poor observability makes debugging any other layer harder.
No industry-standard layer count exists. Published frameworks range from three to eight layers, and O'Reilly's 2026 edition uses six. Eight layers work better for buyers because each one maps to a separate buying decision:
| Layer | What it does | Example tool categories | Where Blaxel fits |
|---|---|---|---|
| Models and inference | Generates reasoning and code, and routes requests across providers | Foundation model APIs, dedicated inference, model gateways | Not covered. Bring your own model providers |
| Orchestration and frameworks | Plans multi-step work, manages handoffs, checkpoints progress | Graph-based frameworks, role-based frameworks, agent SDKs, durable workflow engines | Not covered. Pair with your preferred framework |
| Tools and MCP | Connects agents to APIs, data, and services | MCP servers, MCP gateways, integration platforms | Not covered. Pair with your MCP servers and gateways |
| Code execution sandboxes | Runs AI-generated or untrusted code in isolation | MicroVM sandboxes, gVisor sandboxes, batch compute | Sandboxes (Firecracker microVMs) and Batch Jobs for parallel fan-out |
| Memory and storage | Keeps session state, long-term memory, and shared artifacts | Session stores, vector databases, block storage, shared filesystems | Standby snapshots, Volumes, and Agent Drive (in private preview) |
| Networking and egress | Controls inbound exposure and outbound traffic | Egress proxies, domain allowlists, static IPs, custom domains | Custom domains, proxy secrets injection and domain filtering (public preview), dedicated egress gateways (in private preview) |
| Observability and evaluation | Traces reasoning, tool calls, and token spend, and scores outputs | Agent tracing platforms, eval frameworks, OpenTelemetry pipelines | Not covered. Pair with a tracing and evaluation tool |
| Security and compliance | Enforces permissions, identity, audit trails, and approvals | Policy engines, identity providers, guardrail tools, audit logging | Not covered. Pair with your policy and identity tooling |
Models and inference
Foundation models sit at the base of every agent system. The first decision is whether to use a single provider or multi-model routing.
Single-provider setups are operationally simpler. Teams manage one API key, one bill, and one set of rate limits. One provider also means one point of failure. A worldwide Claude outage on March 2, 2026, brought hours of elevated errors.
Multi-model routing adds complexity but limits the impact of a provider outage. A model gateway handles authentication, budget verification, rate limiting, caching, and routing. Track input and output tokens separately per request.
Attribute spend to specific users, teams, and endpoints. Token cost control at this layer limits runaway spending from reasoning loops. Treat the single-versus-multi choice as a business continuity decision. Size it against how long your product can tolerate a provider outage.
Teams with sustained production workloads should also evaluate reserved inference. Providers like Morph offer dedicated inference with reserved capacity and managed model serving. Compare the reservation against your actual usage, including idle time.
Orchestration and frameworks
Orchestration governs how agents plan, execute, and recover across multi-step tasks.
Production patterns fall into three categories:
- Graph-based frameworks: LangGraph and similar tools provide native cycle support and state checkpointing for complex workflows.
- Role-based frameworks: CrewAI and others use role abstractions to speed up multi-agent builds.
- Custom systems: In-house orchestration gives maximum control, but your team builds and maintains every component.
Agent SDKs now include code execution. In April 2026, OpenAI released native sandbox execution in the Agents SDK, with built-in support for several providers. Google ADK and the Claude Agent SDK each handle code execution differently. Check which sandbox types your framework supports before you choose one.
Whatever framework you use, checkpoint state explicitly and collect end-to-end telemetry. Use circuit breakers to isolate failures in production. Teams also own routing between agents and data governance at this layer.
Pick the framework whose failure behavior you can observe and test before rollout.
Tools and MCP
Before Model Context Protocol (MCP), integrating agents with external tools required custom code for every agent-tool pair. MCP standardizes how agents discover and call tools. The three-tier architecture separates hosts, clients, and servers.
Servers expose tools, resources, and prompts. The earlier 2025-11-25 specification ran as a stateful JSON-RPC session, which forced sticky routing in production.
The 2026-07-28 specification retires the initialize handshake and the Mcp-Session-Id header. Each request now carries its protocol version, client identity, and capabilities. Any request can land on any server instance behind a plain round-robin load balancer.
The same release reports close to half a billion monthly downloads across Tier 1 SDKs. Google Cloud also ships official MCP support. For teams building new tool layers, proprietary integration approaches now carry migration risk.
Operational work shifts from session routing to authorization. The 2026 release hardens authorization around OAuth requirements. Each tool server still needs its own auth implementation. Plan rate limiting, per-tool backpressure, and scoped credentials yourself.
Code execution sandboxes
Containers are the industry standard for running your own trusted software in multi-tenant settings. Agents that generate and execute code change the threat model. NVIDIA's AI red team states that teams must treat LLM-generated code as untrusted output.
The model writing that code follows instructions from untrusted input. For AI-generated or untrusted code, microVMs add a hardware boundary between the workload and the host kernel. Containers share that kernel.
gVisor improves on containers by intercepting system calls in a user-space kernel. MicroVMs go further, giving each workload its own guest kernel behind a hypervisor. The Firecracker USENIX paper shows how narrow that attack surface is.
Firecracker emulates only network, block, serial, and partial keyboard devices, and the implementation is roughly 50,000 lines of Rust. Breaking microVM isolation means crossing the guest-to-host security boundary.
State persistence is the second architectural choice. For coding agents that reuse an environment across sessions, ephemeral sandboxes add setup work to every session. Persistent sandboxes keep state between sessions and skip that setup work.
Blaxel runs sandboxes in Firecracker microVMs and keeps them in standby between sessions. They resume from standby in under 25ms with filesystem, memory, and running processes restored. They return to standby after approximately 15 seconds of network inactivity.
Standby costs nothing in compute, and teams pay only for snapshot storage. Higher quota tiers set no standby time limit. Tier 0 and Tier 1 cap standby at 7 and 30 days. Initial creation from a template takes under a second. Blaxel holds SOC 2 Type II, HIPAA, and ISO 27001.
For fan-out work, Blaxel Batch Jobs run thousands of parallel tasks in isolated sandboxes. Batch Jobs boot in roughly 30 seconds, which suits workloads that run from minutes to hours.
Memory and storage
Memory is three separate problems that teams often lump together. In-context memory lives in the prompt and disappears after the call. Session memory holds the working state of one task: files, processes, intermediate artifacts. Long-term memory holds what an agent should recall across sessions, often in a vector database or structured store.
Each type fails differently:
- In-context memory: Overflows the context window, or pushes token costs up as every turn reprocesses prior context.
- Session memory: Disappears when the sandbox resets, so the agent rebuilds its environment from scratch.
- Long-term memory: Degrades through stale or conflicting entries. Secrets written into memory can also leak into later sessions.
Standby preserves a working environment, but Blaxel doesn't guarantee durable persistence for standby snapshots. For data that must survive for months or longer, attach Volumes, which provide block storage for a single sandbox.
When several agents need the same files, Agent Drive (in private preview) gives them a shared filesystem. Multiple sandboxes can mount it at once, including sandboxes already running. Blaxel's storage primitives are filesystems and block storage, so retrieval indexes run as a separate service or inside the sandbox.
Map each data type to its storage tier before production. The mapping decides whether a crashed session costs seconds or a full repository clone.
Networking and egress
An agent with code execution does its damage through outbound traffic. A prompt-injected agent can exfiltrate data to any reachable domain. NVIDIA recommends running command-line tools in an isolated environment with strong network egress controls and a least-privilege allowlist of commands.
This layer covers four controls:
- Domain filtering: Restricts which hosts a sandbox can call, so injected instructions can't reach arbitrary endpoints.
- Secrets injection at the proxy: Adds API keys to outbound requests server-side. Credentials stay out of sandbox code and memory.
- Static outbound IPs: Lets customers allowlist your agents' traffic on their firewalls.
- Custom domains and preview URLs: Exposes agent output, such as a coding agent's live preview, on your own domain.
Blaxel covers all four. Proxy secrets injection and domain filtering are in public preview. Dedicated egress gateways for static IPs are in private preview. List every external host each agent needs, then deny everything else by default.
Observability and evaluation
Vendors designed traditional application performance monitoring (APM) tools for deterministic, linear request-response flows. Agent workloads break those assumptions. The same input produces different outputs across runs. Linear traces can't represent multi-step reasoning loops.
APM tools also miss cost. Token usage is both a cost signal and a behavioral signal. Excessive token consumption often indicates runaway reasoning loops. By the time CPU and memory metrics alert, unexpected costs have already accumulated.
Agent-specific observability needs hierarchical tracing that captures reasoning steps, tool calls and their purpose, decision points, and per-call token metrics. Evaluation lags tracing. LangChain's survey found that 89% of organizations have some observability, but only 52% run offline evaluation.
Teams without offline evaluation can see a failure but can't measure whether a fix improved outputs. Human evaluation remains essential for validating agent outputs when automated metrics can't capture reasoning quality.
Security and compliance
Sandbox isolation contains damage from bad code, but it doesn't stop an agent from misusing tools it's allowed to call. OWASP calls this Excessive Agency: damaging actions triggered by unexpected, ambiguous, or manipulated outputs. OWASP names three root causes: excessive functionality, permissions, and autonomy.
Plan these controls together:
- Per-tool permissions: Give each agent the minimum tool set for its task. Separate read tools from write tools.
- Scoped identity: Issue short-lived, per-task credentials instead of a long-lived key shared across agents.
- Approval gates: Pause before destructive actions, such as deleting data or deploying code, and resume on human approval.
- Audit trails: Log agent ID, sandbox ID, tool, input and output hashes, approver, and timestamp for every action.
NIST's draft Cyber AI Profile says most applications should curtail or sandbox arbitrary code execution. The alternatives are approval and monitoring, or disallowing it. Enterprise buyers will ask for evidence of each control during procurement. Tool permissions and approval logic stay with your team, whichever sandbox you run.
The biggest infra challenges in an AI agent infrastructure stack
Layer choices interact, so a weak decision in one layer adds cost or latency in others.
Cold start latency compounds across tool calls
Agents making sequential tool calls accumulate latency at each step. Standard serverless platforms add a cold start latency penalty whenever a new execution environment spins up. The penalty is worst at the tail.
A coding agent that reads a file, runs a test, and writes a patch can hit three cold starts. Five or ten sequential calls multiply the delay instead of averaging it out.
Always-on compute eliminates cold starts but means paying for idle time. Scale-to-zero cuts idle costs but introduces multi-second delays. Standby is a third option. Sandboxes suspend after inactivity and resume with state intact, without the always-on bill.
State loss forces expensive re-initialization
Agents that lose state between invocations repeat expensive operations. Context rebuilt from scratch creates quadratic token growth because each subsequent turn reprocesses all prior context.
Stevens Institute's analysis calls quadratic token growth in multi-turn loops the most dangerous agent cost trap. A dedicated memory layer cuts cost because the agent stops rebuilding what it already knew.
Multi-agent systems multiply the cost. An orchestrator spawning 50 workers that each re-clone the same repository pays for that operation 50 times per cycle. Cost and latency scale linearly with the number of agents.
A shared filesystem lets those 50 workers read one cloned repository instead of fetching 50 copies.
Security isolation gaps block enterprise adoption
Enterprise buyers of agents that execute untrusted code require tenant isolation that shared-kernel containers can't guarantee. In the American Institute of Certified Public Accountants' SOC 2 criteria, CC6 governs logical access controls.
Container architectures require extensive documentation of compensating controls around shared-kernel risk.
MicroVM architectures give auditors clearer evidence, because the hypervisor boundary itself shows the separation. Agents that process electronic protected health information must also meet the HIPAA technical safeguards in 45 CFR § 164.312.
HHS describes the Security Rule as technology neutral, so it doesn't prescribe an isolation model. Without that guidance, microVM boundaries leave auditors less to interpret.
How to choose your AI agent infrastructure stack
Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027. The firm cites escalating costs, unclear business value, and inadequate risk controls. Infrastructure choices drive two of those causes: cost and risk controls.
Use these criteria to decide layer by layer:
- Build vs buy per layer: Open-source frameworks are a reasonable default for orchestration. Buy where the work is undifferentiated and expensive to secure, which usually means the execution layer.
- Isolation model: If agents run AI-generated or customer-supplied code, require microVM isolation. Containers remain fine for your own trusted services.
- Cost at idle: Ask what you pay while agents wait on LLM responses or users. Compare always-on, scale-to-zero, and standby models on both bill and resume latency.
- State and storage: Confirm how long sandboxes persist and whether running processes survive. Check what storage carries data beyond that window.
- Network controls: Check for egress filtering, secrets injection, and static IPs before an enterprise security review asks for them.
Run a two-week proof of concept that replays a representative multi-step workload against each finalist. Measure tail latency across sequential tool calls, idle cost over a full day, and state restoration after resume.
Where to start with your AI agent infrastructure stack
The execution layer carries the most risk, because it sets tool-call latency and the isolation evidence enterprise buyers ask for.
Blaxel provides the infrastructure foundation for autonomous agents, covering execution, storage, networking, and batch. Models and orchestration stay with the providers and frameworks you already use.
FAQs about AI agent infrastructure stack
How many layers does an AI agent infrastructure stack have?
There's no standard count. Published frameworks use anywhere from three to eight layers. An eight-layer model gives each buying decision its own layer, from models and inference through security and compliance.
How do I design a human eval for an agent when infrastructure is part of the failure mode?
Treat human evaluation as an end-to-end system test. Review complete traces: prompts, tool calls, execution logs, and final outputs. Score both outcome quality and operational correctness. When you see failures, label the root cause explicitly. Separate reasoning failures from orchestration issues, tool integration problems, and execution constraints. Root-cause labels stop teams from swapping models when the cause is cold starts or missing state.
What should I ask vendors to prove about cold starts and state persistence?
Ask for tail latency data during sequential tool-call chains. Clarify what "state" means: filesystem only, or running processes and in-memory state across reconnects. Ask how long a sandbox can stay suspended before deletion. If a platform claims fast resume, ask what mechanism provides it and what conditions break it. Timeouts, evictions, region moves, and zero-data-retention modes all affect state behavior.
What usually breaks first when scaling MCP in production?
Auth flows and per-call authorization break first. Token lifecycles, scopes, consent flows, and incident response still add operational work for every tool server. Before scaling, put rate limits and scoped credentials on every tool server, and log each call for audit. Teams still on servers built for the earlier stateful specification should plan the migration. The biggest changes in the 2026-07-28 release are breaking ones.
For compliance, what changes in the audit packet when you move from containers to microVMs?
In a SOC 2 audit, container multi-tenancy for untrusted code needs written controls to offset the shared-host kernel. With microVMs, the isolation boundary is easier to describe and document. The packet can show a guest kernel per workload, enforced by the hypervisor. You still need patching and monitoring either way.
Related articles

Guides
What is a computer use agent? How AI agents automate GUI workflows
Learn how a computer use agent operates software through screenshots, clicks, and keystrokes.
13 min read

Guides
A guide to Bring Your Own Cloud (BYOC) for enterprise leaders
Learn what BYOC is and how Bring Your Own Cloud splits the control and data planes for enterprise workloads.
12 min read

Guides
How gVisor vs Firecracker compare for sandboxing AI agent workloads
Compare gVisor vs Firecracker on isolation, startup, I/O overhead, KVM needs, and Kubernetes fit to choose the right sandbox runtime.
12 min read