Sandbox performance optimization: a production playbook

Your agent prototype works in development. It parses documents, generates code, and executes tool calls without delay. Then concurrent users hit production, and response times drift past the second-mark where users notice the lag. For code-executing agents, infrastructure latency, not agent logic, now controls response time.

This gap matters more now than it did earlier in the adoption curve. As production volume scales, the sandbox layer controls both unit economics and SLA exposure for code-executing workloads. A sandbox that boots from cold rather than resuming in under 25 milliseconds compounds the cost. Every tool call, every concurrent user, and every invoice carries that delta. The sandbox layer becomes the dominant variable in production performance.

This guide covers the four performance dimensions that move together. It maps the architectural choices that cap your tuning ceiling. It then walks through a playbook your team can run this quarter.

Why sandbox performance defines production AI economics

Cost per interaction and SLA exposure don't sit in isolation. They compound, and the compounding is where most teams underestimate the bill.

Take cost first. A sandbox that takes seconds to boot forces a choice. Pay for idle warm pools, or accept cold starts that break the user experience. For low-duty-cycle workloads, always-on compute costs many multiples more than suspend-and-resume architectures. That multiplier hits every interaction your platform serves. AWS made this worse in August 2025. The Lambda billing change started charging cold-start initialization at invocation-duration rates, which increases per-cold-start cost for bursty workloads. Agent traffic is typically bursty.

SLA exposure compounds in a different way. A p99 cold start of 146 milliseconds for a single microVM, reported in the Firecracker paper, sounds acceptable in isolation. Stack inference latency, network round trips, and scheduling delay on top of it. Composite latency crosses the one-second mark where users lose their flow of thought. Local targets look healthy while the end-to-end request still misses its goal.

The four dimensions of sandbox performance optimization

Most teams optimize for one metric and accidentally regress on the others. Cold start improvements that increase memory footprint reduce tenant density. These four dimensions move together.

Cold start and resume latency

Cold start latency is the time between requesting a sandbox and having a usable execution environment. Coding assistants and PR review agents execute code in real time. For them, this number sets a hard floor on response time.

The range across architectures is wide.

The figures use different startup definitions, but architecture sets the floor either way.

Resume from standby skips kernel boot and runtime initialization entirely. Optimizing it pays off more than tuning fresh creation, since most production traffic comes from returning sessions. The perception threshold for instantaneous response sits at 100 milliseconds per published UX research. Resume latency above that breaks the feel of immediacy.

Memory footprint and density

Per-sandbox RAM allocation dictates how many concurrent tenants fit on each host. RAM drives unit cost more directly than CPU. Most agent workloads spend their time waiting on network I/O or LLM inference.

Unoptimized microVM overhead can match the workload's own memory allocation. A 128 MB container running with Kata-Firecracker carried 94 MB of memory overhead in the RunD study. That is roughly a 1:1 ratio of infrastructure cost to workload cost. Shared base layers via virtio-fs Direct Access (DAX) and template reuse drive that overhead down sharply.

The density math is straightforward. Each sandbox consumes memory for both the workload and runtime overhead. Lower per-VM overhead means more tenants per host. The density gain compounds across every host in your fleet. At production concurrency, the monthly cost difference between configurations reaches tens of thousands of dollars.

Network round-trip time

P90 round-trip time (RTT) between agent, sandbox, and external APIs compounds across multi-step tool calls. Averages obscure how badly. Assume each service has a 1% chance of exceeding its p99 threshold. Across five sequential calls, the chance that at least one is slow reaches roughly 4.9%. Tail latency stops being exceptional once workflows chain enough steps together.

Physical constraints matter too. Cross-AZ calls add single-digit milliseconds per hop. That extra distance accumulates into latency your application code cannot win back later. The gap between median and tail latency within a single service can be enormous.

One ACM Queue case study reported median latency of 26 milliseconds against p99 of 696 milliseconds. That spread is why tail behavior matters more than averages in tool-heavy agent systems. Multi-agent patterns across frameworks like LangChain, CrewAI, and AutoGen split into two latency profiles:

• Handoff architectures: Sequential execution across model calls. RTT compounds fully across the chain.
• Subagent and router patterns: Parallel calls. Latency is bounded by the longest sequential chain rather than the total chain length.

Pattern choice determines whether RTT optimization compounds across a workflow or caps at a single sequential step.

Concurrency and burst capacity

Average performance lies during peak fan-out workloads. When a frontend aggregates requests across N downstream services, end-user latency is bounded by the slowest response. Tail latency in one downstream service becomes routine at the aggregation layer, a pattern noted in the SRE Book.

Measure concurrency at p99, not average. Average-based autoscaling alarms can miss burst-induced degradation entirely. Their trigger sensitivity depends on configured periods and evaluation counts, not on raw spikes. Agent workflows often fan out one user request into multiple parallel sandbox executions. That makes p99 the dominant user experience metric.

Track concurrency metrics against their hard limits. Profile burst scenarios that mirror real production traffic. Production agent platforms bound conversation length and step counts based on observed p95 and p99 distributions of cost and runtime. These bounds prevent runaway loops from saturating concurrency headroom.

Architectural decisions that set your performance ceiling

Code-level optimization cannot overcome architectural choices made earlier in the stack. A sandbox built on containers for untrusted-code execution will not match microVM resume-path behavior through tuning alone. These three decisions set the floor and ceiling for everything that follows.

Isolation technology and the container versus microVM tradeoff

The isolation model determines your cold start floor, memory overhead floor, and security posture simultaneously.

Two of the highest-severity container escape CVEs, CVE-2019-5736 and CVE-2024-21626, exploit the container runtime itself. Hardened kernel configurations would not have prevented exploitation. That is the threat model that matters for sandboxing untrusted agent-generated code.

The performance cost of microVM isolation is smaller than most teams assume. Firecracker's specification enforces greater than 95% of bare-metal CPU performance, with VMM memory overhead under 5 megabytes per VM. For multi-tenant environments running untrusted agent-generated code, less-than-5% CPU overhead is a small price. Both AWS Lambda and AWS Fargate run on Firecracker, an open-source microVM technology. That deployment validates the approach at high volume.

Filesystem strategy and the OverlayFS advantage

OverlayFS presents a merged view of two directory trees: a read-only shared lower layer and a writable per-instance upper layer. The memory savings come from the kernel's page cache behavior. Sandboxes sharing the same lower layer read from a single set of kernel page cache entries. They don't maintain per-sandbox duplicates. Run a hundred sandboxes from the same base image and you get one copy in memory, not a hundred.

A critical subtlety exists in microVM environments. Without DAX mode, virtio-fs lets each guest maintain its own page cache, which eliminates the density benefit entirely. Turning on DAX bypasses the guest page cache. It maps the host page cache directly into guest address space. Multiple VMs share one set of pages instead of duplicating them.

Blaxel's sandbox filesystem uses a three-layer architecture on this principle:

• EROFS (Extendable Read-Only File System): Provides the read-only base on host storage.
• tmpfs: Provides the writable layer in the sandbox's RAM.
• OverlayFS: Orchestrates reads and writes between them.

The memory savings Blaxel measured show how much density this approach delivers compared with naive per-sandbox copies.

Co-location of agent and sandbox compute

The dominant source of tool-call latency in most architectures is network distance between the agent process and its sandbox. Agent logic often runs in one data center while the sandbox runs in another. Every tool call pays a network round trip.

Same-AZ co-location reduces per-call overhead to sub-millisecond latency, while cross-AZ routing adds single-digit milliseconds per hop. The physical New York to London round-trip floor sits near 59 milliseconds, set by the speed of light. That distance is not optimizable at the application layer.

When end-to-end latencies climb past two to three seconds per agentic cycle, agents start timing out or producing degraded decisions. Co-location is the architectural decision with the largest single impact on composite latency. For tool-call-heavy agents, even small per-call latency savings multiply across the full workflow.

How to optimize sandbox performance in production

These four steps, run in order, deliver compounding gains. Baseline first so you know what to fix. Then compress cold starts, reduce memory pressure, and validate under realistic burst conditions.

1. Establish a workload-specific baseline

Profile real production traces, not synthetic benchmarks. Synthetic load tests miss bursty arrival rates and long-tail latency spikes.

Capture p50, p90, and p99 across all four dimensions using your actual agent workloads. P50 captures typical latency, p90 the speed 90% of requests beat, and p99 the worst-case 1%. High p99 with normal p50 signals sporadic issues. Consistently elevated p50 signals systemic degradation.

Factor p99 latency into application SLAs from day one. Instrument your agent code with OpenTelemetry. Use separate spans for prompt construction, per-iteration LLM calls, and per-iteration tool calls. Tag every trace with the sandbox lifecycle state, whether cold start or warm resume. The tagging lets you attribute regressions to the right layer.

2. Compress the cold start path

Cold start latency typically splits into two phases. VM-level startup runs around 200 milliseconds for microVMs in non-edge configurations per published measurements. Code download plus runtime initialization ranges from tens of milliseconds to several seconds. Image size and dependencies decide which end of that range you hit. Which phase dominates depends on the platform and workload. The phase-based figures don't directly compare to the earlier Firecracker spec target. That spec measures a narrower step under different startup conditions.

Snapshot strategies address the runtime-initialization phase directly. Lambda SnapStart with invoke priming reduced Spring Boot Java cold starts dramatically. The p50 baseline of around 5,047 ms dropped to 781.68 ms at p99.9, per AWS-published measurements.

A 10% buffer above measured peak concurrency is a reasonable rule of thumb when sizing provisioned capacity. The tradeoff depends on traffic pattern. Provisioned concurrency delivers double-digit millisecond response times but creates idle cost during low-traffic periods. Snapshot resume avoids the cost of keeping instances warm. Resumed functions still incur startup latency in the hundreds of milliseconds to sub-second range. For bursty agent workloads, architectures with sub-100ms resume eliminate the warm pool entirely.

3. Reduce per-sandbox memory pressure

Shared base images deliver the highest-return memory optimization. With OverlayFS and a shared read-only lower layer, every sandbox reads from the same kernel page cache entries. There's no per-VM duplication. The optimization compounds. The more sandboxes you run from the same base, the lower your effective per-VM overhead becomes.

Image slimming reduces the base layer size itself. Audit your production images for bloat. Remove development tools, documentation, and unused libraries. Most sandbox images carry hundreds of megabytes of packages the agent never touches.

Kernel Same-page Merging (KSM) provides a complementary mechanism. It scans for identical memory pages across guest VMs at runtime and merges them. KSM works reactively on pages that are already allocated. That makes it useful as a second-line optimization on top of shared base images.

4. Validate behavior at peak concurrency

Load-test fan-out scenarios that mirror production bursts, not steady-state traffic. Test both steady and burst rates.

Watch for coordinated omission in your load testing tools. When a load generator pauses to wait for a slow response, it stops sending requests on schedule. It silently drops the latency measurements that should have been recorded for those queued requests. The result is systematic p99 underreporting under burst conditions. That's exactly when accuracy matters most.

During burst tests, track three signals:

• Queueing depth: Are requests stacking up waiting for sandbox creation? If so, your warm pool or resume path cannot keep pace with arrival rate.
• Scheduling latency: How long passes between a sandbox request and actual VM allocation? This isolates control-plane delay from data-plane work.
• Tail latency divergence: How far does p99 drift from p50 under burst load? A widening gap signals structural variance, which produces user-visible hangs during peak traffic.

Run these tests above your current peak load to validate headroom. Lambda's published capacity guidance provides a useful upper bound. The platform absorbs traffic doubling within five minutes without throttles. That sets a sensible burst-test target.

Make sandbox performance optimization a production discipline

Teams that treat sandbox performance as a quarterly architecture review ship faster and spend less. The compounding nature of these four dimensions means small regressions cascade into SLA misses and cost overruns. Build performance baselines into your CI pipeline and review p99 trends at the same cadence you review infrastructure spend.

For teams building coding agents, PR review agents, or data analysis agents, these architecture choices directly determine agent experience quality. The gap between sub-100ms resume and multi-second cold starts is the difference between a smooth experience and a frustrating one.

Perpetual sandbox platforms like Blaxel are built around this performance discipline:

• MicroVM isolation: Firecracker microVMs provide hardware-enforced tenant separation for executing untrusted and AI-generated code. Backed by SOC 2 Type II, HIPAA (BAA available), and ISO 27001.
• Sub-25-millisecond resume from standby: Restores the complete filesystem, memory, and running processes from a perpetual standby snapshot. Sandboxes stay in standby indefinitely with zero compute cost while idle. Competitors cap standby at 30 days or less.
• Initial creation in 200 to 600 milliseconds: Fast enough that warm pools become optional rather than mandatory for most workloads.
• P90 network RTT under 50 ms: Inside Blaxel's optimized networking layer, keeping per-call overhead well below typical cross-AZ floors.
• Network-based auto-shutdown: Sandboxes return to standby after 15 seconds of network inactivity. No manual lifecycle management required, and idle compute charges disappear.
• Agent Drive: Distributed filesystem for sharing data between agents and sessions without intermediary storage or network hops. Blaxel is now a first-class sandbox provider in the OpenAI Agents SDK.
• Batch Jobs: Handles parallel fan-out and asynchronous burst workloads when thousands of background tasks need to run concurrently.
• Volumes: Persistent block storage for long-lived data across sandbox sessions. Pairs with Agent Drive for workloads that need both shared context and raw I/O performance.

The result is straightforward. Agents that execute untrusted code get microVM isolation. Resume times stay well inside the perception threshold for instantaneous response.

Build0 cut sandbox costs by 80% running on this model.

Book a demo to benchmark Blaxel against your workload, or start free with $200 in credits.

Frequently asked questions

Why does resume latency matter more than cold start optimization?

Most production traffic comes from returning sessions, not first-time creations. Optimizing resume from standby affects a larger share of requests than tuning fresh sandbox creation. The perception threshold for instantaneous response sits at 100 milliseconds per published UX research. Resume latency above that mark breaks the feel of immediacy. Cold start tuning matters less when the architecture supports fast resume.

How does OverlayFS reduce per-sandbox memory consumption?

OverlayFS merges a shared read-only base layer with a writable per-instance layer. Sandboxes sharing the same base image read from a single set of kernel page cache entries. They don't maintain separate copies. Running a hundred sandboxes from the same image keeps one copy in memory. In microVM environments, DAX mode maps the host page cache directly into guest address space for the same benefit.

What is coordinated omission in sandbox load testing?

Coordinated omission occurs when a load generator pauses to wait for slow responses instead of sending requests on schedule. The tool silently drops latency measurements for queued requests that should have been recorded. The result is systematic p99 underreporting during burst conditions. Use load testing tools that account for this effect, or your benchmarks will mask real production tail latency.

Why should teams measure p99 latency instead of averages?

Agent workflows fan out one user request into multiple parallel sandbox executions. The end user waits for the slowest response. Even a 1% chance of exceeding a latency threshold compounds across chained calls. Across five sequential steps, roughly 5% of requests hit the tail. Average-based metrics hide this compounding effect entirely.

How does Blaxel's architecture address all four performance dimensions?

Blaxel's perpetual sandbox platform uses Firecracker microVMs with sub-25ms resume from standby, eliminating warm pool costs. OverlayFS with shared base layers reduces per-sandbox memory overhead. The optimized networking layer achieves p90 RTT under 50 milliseconds. Network-based auto-shutdown after 15 seconds of inactivity keeps idle compute costs at zero. Agent Drive shares data across sandboxes without network hops.