Back to blog posts

10 min

A production playbook to optimize sandbox performance for AI agents

Learn how to optimize sandbox performance for AI agents. Measure create, resume, and first command latency at p99.

Nicolas LecomteNico is a founder of Blaxel, who usually writes about AI, agentics, and the future of AI runtimes.

Sandbox performance optimization covers the whole life of a sandbox, from creation through standby and back. Agent tasks repeat that cycle constantly, so small delays compound across every step. Idle compute adds cost on top.

Earlier guides cover the startup side. One fixes cold start latency, the wait before a new sandbox can run commands. Another shows how to keep sandboxes warm. Resume time, tail latency, image size, sizing, region, and storage matter just as much.

For agents that execute code, Baseten Sandboxes create in under a second. They resume from standby in under 25ms.

Key takeaways

  • Measure the full path: Track create, resume, and first-command times separately, plus cost per active hour.
  • Benchmark the tail: Send requests on a fixed schedule and report p95 and p99, which shape user experience at fan-out.
  • Shrink the template first: Slimmed images with preinstalled dependencies cut startup work, so tune them before other levers.
  • Match lifecycle to session pattern: Standby suits returning sessions. Recreating with a Volume suits long-lived data.
  • Place sandboxes near the model API: Cross-region round trips add up on every step of a sequential agent loop.

How to measure sandbox performance

Track these metrics per workload, each broken out by template and region:

  • Time to create: The clock runs from the create call to a sandbox that accepts commands. It includes image work and any entrypoint setup.
  • Time to resume: Start timing at reconnect and stop once the sandbox is active with its prior state intact.
  • Time to first command: Count from the request to the first output byte of a typical command.
  • p50, p95, and p99 for each: Percentiles expose the slow tail that an average hides.
  • Cost per active hour: Multiply allocated memory (GB) by the per-GB-second rate and 3,600. Baseten Sandboxes bill by memory allocation and don't charge for CPU separately.

Time to first command usually runs higher than time to resume because snapshot restores load memory pages on demand. In the FaaSnap study (EuroSys 2022), a hello-world function took more than 200 ms when restored from Firecracker snapshots. 

A warm VM finished it within 4 ms. The gap shows that restore time alone is an incomplete measure of readiness. REAP (ASPLOS 2021) found that servicing page faults on the critical path took 95% of processing time on average. Time the first command alongside the resume call.

Tail latency costs more as agents fan out and run many tool calls in parallel. Dean and Barroso modeled a server where one request in 100 takes a full second. A user request waiting on 100 such servers exceeds one second 63% of the time. 

DoorDash runs a 95th-percentile objective of under five seconds for end-to-end sandbox setup. The budget covers starting the microVM (a lightweight virtual machine), cloning repositories, and installing build tools. The budget covers the whole path, with microVM startup as only one part.

How to improve sandbox performance

LeverWhat it improvesTradeoff
Slimmed template imageTime to createBuild tools are unavailable at runtime unless you set slim = false
Dependencies preinstalled in the templateTime to first commandLarger image and a rebuild when dependencies change
Template reuse and sharingBuild time and consistency across workspacesUnderlying storage stays in the source workspace
Standby and resumeTime to resume and idle costSnapshot storage cost, and external connections drop
Always-on with keep-alive (in alpha)Removes the resume stepActive memory billed the whole time
Recreate from template plus VolumeDurable data and clean stateProcesses restart and setup reruns each session
CPU and memory sizingExecution speedHigher cost per active hour
Region placement near the model APINetwork latency per agent stepLimited to available region codes
Concurrency and fan-out limitsThroughputQuota tier caps and wider tails at high fan-out
In-memory sandbox filesystemFastest file I/OConsumes sandbox memory and is lost on deletion
VolumesData that outlives the sandboxAttached only at creation, one per sandbox

Optimize image size, layering, and preinstalled dependencies

Reduce startup work by shrinking images and preinstalling dependencies. FaaSNet (USENIX ATC 2021) found over half of invocations in one region spent 80% or more of startup pulling images. Count image transfer in the startup budget alongside command execution. 

Starlight (NSDI 2022) found sample containers read less than 1% of their files during startup. Lazy loading fetches only the files a container reads, on demand. AWS suggests lazy loading for images over 250 MiB compressed. Smaller images are less likely to benefit.

Baseten Sandboxes templates run a default slimming pass after the Docker build, so keep it on. It strips package caches, docs, headers, and toolchains like make, gcc, and g++. Setting slim = false keeps them, at the cost of a larger image and slower boot.

Install dependencies during the image build, in RUN steps ordered from rarely changed to often changed. Keep installs like npm install out of the entrypoint script so new sandboxes skip that work. 

On Baseten Sandboxes, push once with bl push and share the image across workspaces. Sharing copies only metadata. Compare time to first command on the old and new templates to confirm the gain.

Choose standby, always-on, or recreate

Each lifecycle trades idle cost against resume speed and how much state survives:

  • Standby and resume: Memory, files, and running processes stay in a snapshot. You pay snapshot storage instead of active memory.
  • Always-on: Baseten Sandboxes offer keep-alive (in alpha), which keeps a sandbox active. So does an active connection, though idle ones time out after 15 minutes. Always-on fits long tasks inside a session, like waiting on a slow large language model (LLM) response.
  • Recreate from template plus Volume: Files survive on the Volume, but processes restart and setup reruns. Recreating suits sessions where durable data or a clean start matters more than warm state.

Use your client library's health check or reconnection option after resume. External connections, like database pools, can close during standby and fail on first use. For Baseten Sandboxes on higher quota tiers, set an expiration policy explicitly, since the platform never deletes sandboxes without one. 

Test each lifecycle against your workload's real idle pattern.

Size CPU, memory, and concurrency

In a Baseten Sandbox, billing follows the memory allocation, and CPU carries no separate charge. Run a CPU-bound task, such as your test suite, at two or three memory sizes. Pick the smallest size where the task's run time stops improving.

Agent fan-out sets peak concurrency. One study used synthetic traces from a production agent platform. Those traces showed up to 21 tool calls in a single agent iteration. Measure peak fan-out from your own traces, then set concurrency limits at that peak rather than at average load.

A limit below peak queues tool calls and widens tail latency. Quota tiers also cap concurrency, so check yours before a fan-out test.

Choose a filesystem: tmpfs or Volumes

Each Baseten Sandbox keeps its filesystem in RAM through tmpfs. It's the fastest option, with capacity of roughly half the memory allocation. Every file an agent writes consumes memory and grows the standby snapshot. 

A layered OverlayFS design cut a base Next.js image's memory needs from 4GB to 1GB in an internal measurement. Size memory for agent writes and running processes.

Volumes provide block storage that survives sandbox deletion. You attach one at creation, one per sandbox, and you can't detach it later. Volume sizes can only increase. Volume templates copy files up to 90% faster than cp -r. Leave 20 to 30% extra space when you use a Volume template. Keep scratch data in tmpfs and move durable, single-sandbox data to a Volume.

Agent Drive, in private preview, holds files shared across sandboxes or sessions. It supports concurrent read-write access from multiple sandboxes. Choose it for shared context and artifacts rather than single-sandbox block storage.

Place sandboxes near the model API

Sequential agent loops pay a network round trip on every step. Azure's July 30, 2026 dataset lists East US to East US 2 at 8 ms p50. East US to West Europe sits at 83 ms. Ten sequential model API calls across the Atlantic add 830 ms per loop at the median. The same calls between that US region pair add 80 ms.

Baseten Sandboxes run in documented US and EU region codes, such as us-was-1 and eu-fra-1. Pick the code closest to your model API endpoint, then measure per-step latency from inside the sandbox. 

Parallelize independent tool calls where you can, because a parallel batch takes only as long as its slowest call.

How to benchmark your own sandbox workload

Run this method against your own template and a production-shaped task. It produces percentiles you can compare across templates, sizes, and regions.

  1. Script the full path: Create from your template, run a typical command, and let the sandbox enter standby. Then resume and run the command again, recording each phase separately.
  2. Tag cold and warm runs: Fresh creations, first resumes after long idle, and repeat resumes behave differently. Keep them in separate series.
  3. Drive open-loop load: Send requests on a fixed schedule instead of waiting for each response. In the LANCET study, capping outstanding requests produced lower tail latency readings. The readings reflected the cap rather than the workload.
  4. Record histograms: You can't merge percentile summaries from separate hosts, but aligned histograms merge without losing precision. Track failed-request latency separately from successful requests.
  5. Ramp and burst: Step up to your peak fan-out, then test an impulse. Gradual and impulse loads can produce different results because of caching effects.
  6. Compare tail to median: Google's rule of thumb keeps p99 within three to five times the median. A wider gap usually means queuing, page faults, or a slow image path.

Rerun the script after every template, sizing, or region change.

Where Baseten Sandboxes fit

Baseten Sandboxes are the infrastructure foundation for autonomous agents. They fit coding agents and other agents that run generated or untrusted code. The microVM boundary isolates each sandbox.

  • Time to create: Creating a new sandbox takes under a second. Include that setup time in the budget for fresh sessions.
  • Time to resume: Sandboxes resume from standby in under 25ms, with processes preserved.
  • Idle cost: A sandbox enters standby after approximately 15 seconds of network inactivity. Active-memory billing stops during standby, but snapshot storage still accrues charges. Standby sandboxes also count against workspace quota limits. A workspace's tier updates continuously based on its 30-day top-up volume. Standby has no time limit on higher quota tiers. Tier 0 and Tier 1 cap standby at seven and 30 days. Check your workspace's tier before relying on a long idle session.
  • Persistent data: Volumes keep files through sandbox deletion, so a recreated sandbox starts with the same data.
  • Parallel runs: Batch Jobs run each execution as multiple tasks in parallel.

FAQs about sandbox performance optimization

Which sandbox provider has the fastest cold start?

Vendors measure cold starts differently, so published figures rarely compare directly. Some resume figures stop the clock when the restore call returns, before memory pages load. Compare providers on time to create, time to resume, and time to first command at p95 and p99. Creating a Baseten Sandbox takes under a second, and resuming from standby takes under 25ms. Test those figures against your own workload before you choose.

How does resuming from standby differ from creating from a template?

Resume restores a snapshot of memory, files, and running processes, so repo clones and dependency installs don't rerun. Creating from a template starts a fresh sandbox, then runs your entrypoint and any setup. External connections, like database pools, can drop during standby and need to reconnect. Resume suits returning sessions, while creating from a template suits sessions that need a clean start.

Should sandboxes stay always-on to avoid resume latency?

Idle sessions rarely need always-on. Always-on bills active memory the whole time, even while nothing runs. Standby stops compute charges and bills only snapshot and volume storage. Keep-alive, currently in alpha, suits sessions that must stay active through long tool calls. Reserve always-on for sessions where even a short resume delay is unacceptable.

When should data go on a Volume instead of the sandbox filesystem?

Keep scratch files and working directories in the in-memory filesystem. Those files are fastest to read and write, but capacity is roughly half the memory allocation, and deletion erases them. Data that's hard to recreate belongs on a Volume, which survives sandbox deletion. Attach it when you create the sandbox, because you can't add or detach it later. Use Agent Drive, in private preview, for files shared across sandboxes.

Related articles