How to handle agent code execution under heavy concurrency without cold starts

Your agent product reaches a large daily user base. Each session spins up a sandbox to execute code. At peak load, many sandboxes boot simultaneously. Every boot initializes the kernel and userspace before loading dependencies.

Inside production serverless frameworks, the microVM boot alone takes 700–1,300 milliseconds. Dependency loading stacks seconds on top. Warm standby and pre-warmed pools address execution latency, as do template snapshots. Independent orchestration and regional placement maintain throughput and reduce network latency.

Users stare at a spinner before the agent responds. The Lambda-based architecture that worked fine at low traffic now constrains every interaction.

Cold starts are the gap between receiving a request and having a sandbox ready to execute code. Serverless platforms accept that gap in exchange for stateless, short-lived workloads. Agent workloads break that trade. Users expect sub-second responses, and sessions carry state. Traffic also spikes without warning.

Handling agent code execution under heavy concurrency means avoiding cold starts for active and returning sessions. It also means minimizing the first-launch path.

Agent workloads suffer more than traditional serverless for reasons specific to how agents run. Removing the delay requires architectural control over sandbox state between requests.

TL;DR:

• Cold starts hit agents harder: Users wait in real time, sessions carry state, and dependency setup repeats on every boot. A multi-second pause reads as a broken product.
• Warm standby for returning sessions: Snapshot resume skips kernel boot and dependency loading entirely. Firecracker restores VM state in single-digit milliseconds.
• Pre-warmed pools for new sessions: Fully initialized sandboxes wait in a pool and absorb first-time demand. Size the pool against observed claim rate, not a fixed count.
• Template snapshots for the cold path: Prebuilt images collapse dependency installation into load or resume time. Combined with standby, they cover both new and returning users.
• Orchestration and placement at scale: Separate the orchestration layer from compute so traffic spikes don't starve routing. Place sandboxes near users to keep network latency below the threshold cold-start elimination already cleared.

Why cold starts hit agent workloads harder than serverless functions

Standard mitigations cap out well short of what agents need. Provisioned concurrency and slimmer images reduce the delay without removing it. Connection pooling has the same limitation.

Latency and state under bursty demand

Lambda cold starts range from under 100 milliseconds to over one second. For an API endpoint serving web requests, that fraction of a second rarely matters. Google Cloud Run bakes the same tolerance into its contract. Services must be stateless and cannot rely on persistent local state.

The same latency is far more damaging in agent sandboxes. A user waits in real time. A multi-second pause between prompt and execution reads as a broken product.

The sandbox needs the right packages and files for the session context. That setup repeats on every cold boot. Measured Python import overhead runs around 100 milliseconds for common libraries. Full package initialization adds further delay. Consequently, dependency setup can dominate the cold-start path.

A launch or batch operation can multiply concurrent sandbox demand within minutes. Lambda caps scale-out at 1,000 new environments per function every 10 seconds. A larger burst can leave requests waiting for environment creation.

Provisioned concurrency helps Lambda absorb API bursts. Lambda can still hit a cold start when it resets the execution environment. Traffic that spills past the provisioned pool cold-starts too. Agents need a warm, stateful environment ready for a returning user. They do not need a cold, empty function ready for any request.

How to architect agent execution without cold starts

Avoiding cold starts means preparing the sandbox before the user's request arrives whenever possible.

• Warm standby: Idle sandboxes stay paused with state preserved, so returning sessions never reboot.
• Pre-warmed pools: Initialized sandboxes wait in a pool and absorb demand from new sessions.
• Template snapshots: A prebuilt image reduces initialization time for genuinely new environments.

Warm standby needs a hypervisor or runtime that can snapshot and restore memory state. An orchestration layer must track pool depth and claim rate for pre-warmed pools, while template snapshots need a reusable image build step and a registry to serve it.

Keep warm standby for instant resume

Booting a new sandbox per request repeats work that snapshotting can skip entirely. Warm standby keeps idle sandboxes paused. Their filesystem and memory remain preserved, along with process state. When a request arrives, the hypervisor restores a memory snapshot. That restore skips kernel boot and userspace initialization.

Firecracker restores VM state from a snapshot in roughly 3 milliseconds on a tuned aarch64 host. That figure covers VM-state restore with guest memory pages already resident. With lazy paging, page faults dominate the post-restore path. They can stretch a resume well past the restore number.

Restore also skips runtime startup. In one published measurement, JVM startup and class loading took 128.8 and 79.2 milliseconds. A resumed snapshot never repeats that work.

The economics depend on the billing model. Platforms that charge for idle standby create a tradeoff between latency and idle spend. Platforms with no standby compute fees remove the compute portion of that tradeoff. Snapshot and storage charges can remain.

Perpetual sandbox platforms like Blaxel keep sandboxes in standby during idle periods, with compute charges reduced to zero while snapshot and volume storage costs may continue to accrue. Resume takes under 25 milliseconds, with complete filesystem and memory state preserved. That speed sits well inside Jakob Nielsen's 100-millisecond limit for a system to feel instantaneous.

Warm standby removes cold starts for users who return to the same sandbox across sessions. Audit your session mix first. The ratio of returning sessions to first-time boots shows how much latency standby can remove.

Maintain pre-warmed sandbox pools

Pre-warming maintains a pool of fully initialized sandboxes. The orchestration layer assigns them on arrival. Define a warm template with packages installed and dependencies loaded in a configured environment. Then replenish the pool in the background. Useful signals include time-of-day patterns and queue depth. Also track the balance between claims and additions.

Pre-warmed build fleets can reduce build queue wait. Set pool depth from measured demand. Replenish in the background against observed claim rate rather than a fixed machine count.

Size the pool against observed traffic. A production trace of Azure Functions quantifies the cost of guessing wrong. Under a fixed 10-minute keep-alive, the 75th-percentile application hit cold starts 50.3% of the time. That was substantially higher than a histogram-based policy using an equivalent memory footprint.

A fixed keep-alive window is a bet on traffic shape. The wrong bet costs both latency and memory.

Concurrent claims can degrade pool performance. The Cocoon sandbox operator measured p50 claim latency of 33 milliseconds from a 200-sandbox pool. That rose to 316 milliseconds when 20 concurrent claims competed with replenishment. Pool-capacity tests must therefore model concurrent claims.

Pre-warming fits workloads with predictable traffic shapes. For spiky, unpredictable load, pool sizing becomes a guessing game. Warm standby then has to carry more of the weight. Instrument claim rate against replenishment rate. Azure's hybrid histogram policy uses the fifth percentile of inter-invocation time for its pre-warming window. Mirror that approach instead of using a fixed keep-alive window.

Use template snapshots for fast initialization

Template snapshots capture a prepared sandbox state at build time. They can include the OS, packages, dependencies, and configuration. A disk or image template may still require a boot. A memory or VM snapshot can instead resume initialized state. Both approaches avoid installing dependencies from scratch and collapse the longest initialization work into loading or resume time.

AWS Lambda SnapStart shows the ceiling of resumable VM snapshots. It reduced a Java function's startup substantially by resuming from a publish-time microVM snapshot.

Snapshots pair naturally with warm standby. A user's first session starts from a prepared template or resumable snapshot. It is fast but not instant because no user-specific state exists to resume.

Every later session resumes from warm standby in milliseconds. The template covers the cold path for genuinely new sandboxes. Standby covers everything after.

Teams that combine both reduce the new-user path to the template's load or resume time. They avoid a full dependency install. Returning users get an effectively instant resume, without holding a running machine for either path. Compare snapshot load time with a full dependency install. A small gap indicates that the snapshot is doing its job.

How to scale large concurrent sandbox fleets

Latency and throughput fail in different ways. Warm sandboxes fix the wait a single user feels. A warm resume reveals nothing about the control plane's placement rate. Serving a large concurrent sandbox fleet is a separate throughput problem.

During a large wave of simultaneous session starts, placement decisions and network distance become the limiting factors instead of boot time. Two patterns keep latency flat as concurrency grows. Separate orchestration from compute, and place sandboxes near users.

Scale orchestration independently from compute

The orchestration layer assigns users to sandboxes and manages lifecycle. It must scale independently from the compute layer running the workloads. A spike in user requests stresses orchestration first. A spike in long-running agent sessions stresses compute.

Couple them on shared infrastructure, and a saturated compute fleet can starve the orchestrator. The orchestrator then lacks headroom to route around saturation. AWS's 2020 Kinesis outage illustrates this failure. Front-end servers shared responsibility for building routing state. Recovery became self-reinforcing because rebuilding caches competed with serving requests. AWS's remediation moved the cache to a dedicated fleet.

The fix is a stateless orchestrator that scales horizontally on request volume and holds no session state. It reads a global session registry. It then assigns users to available sandboxes across the fleet.

AWS Lambda's own architecture works this way. Any stateless frontend can handle traffic for any function. A separate Worker Manager routes millions of requests per second at sub-10-millisecond p99.9 latency. It leases slots so the data plane can keep deciding when placement lags. Independent scaling lets orchestration absorb a traffic spike. Compute can provision capacity in the background.

Check whether your orchestrator reads session state from a shared registry or the compute nodes themselves. The latter creates your next failure mode.

Distribute sandboxes regionally to reduce network latency

Once sandboxes resume in milliseconds, the network between user and sandbox becomes the dominant latency source. Azure's inter-region measurements show a P50 of 162 milliseconds from Japan East to East US. Sequential tool calls across that path compound the round-trip delay. That estimate comes from the P50 alone.

That's before any code runs. Only placement reduces that number, and cold start elimination has no effect on it.

Provision sandboxes in the region closest to the user. That requires a multi-region compute fleet. The orchestrator routes sandbox creation to the nearest available region. Multi-region operation adds state replication and failover handling. It also requires per-region capacity management.

Per-region capacity management creates much of the engineering cost. Each region needs its own standby depth and pool sizing. Claim-rate instrumentation must be rebuilt per region rather than once globally. A session pinned to one region cannot resume elsewhere without moving its snapshot. Failover therefore requires snapshot data movement.

Teams with a global user base can get latency reductions that justify the complexity. Concentrated user bases should use single-region deployments with warm standby. They should skip the overhead. Measure round-trip time from your primary user geographies to your current region. Do that before committing to a multi-region fleet.

How to avoid common mistakes when scaling agent code execution

Running agent sandboxes on raw serverless works for a demo with a handful of sessions. It breaks under a large burst of concurrent boots. Lambda's guidance is to assume the environment exists only for a single invocation. Agents needing state between tool calls must serialize everything to external storage.

A one KB round-trip from Lambda through S3 measured 303 milliseconds. Direct instance-to-instance messaging took 290 microseconds. Multiply that difference by every tool call in a session. The serialization tax then dwarfs the cold starts you set out to fix.

Over-provisioning pre-warmed pools trades the latency problem for a cost problem. Consider a team holding substantially more warm capacity than its peak requires. It pays for idle environments around the clock. Datadog's cost analysis tied 83% of container costs to idle resources. Flexera pegs 29% of cloud spend as waste. These figures make idle-capacity control a core cost requirement rather than a secondary optimization.

Idle capacity of that size is the default outcome of pool sizing by guesswork. It is not an unusual failure. On platforms without standby compute charges, suspending removes the compute charge. A deep standby pool then leaves only snapshot and volume storage to budget for.

Ignoring lifecycle management leaks resources until orphaned sandboxes crowd out legitimate demand. GitLab's CI fleet encountered exactly that failure. A single error-handling gap left 257 orphan VMs. It also left 10,200 stuck machines that cleanup loops retried for hours.

Define session-to-sandbox lifecycle rules before traffic forces the issue. Auto-suspend on inactivity, then auto-delete after a set idle period while monitoring for orphans. Set a maximum age and an idle timeout per sandbox. Make both adjustable after creation.

How to eliminate cold starts for large concurrent sandbox fleets

Cold starts at production scale determine whether your agent product feels responsive or broken as traffic grows. Every peak-time boot makes the user wait and doubt the product. Optimization tricks that trim a cold boot by milliseconds do not change that outcome.

The architecture must keep active and returning sandboxes warm while minimizing the genuinely new-sandbox path. Standby covers returning sessions, while pre-warmed pools cover predictable demand. Template snapshots reduce the genuinely cold path. The orchestration layer scales apart from compute.

For teams scaling agent code execution across large concurrent fleets, Blaxel keeps sandboxes in standby with zero compute cost. It is a perpetual sandbox platform, with separate pricing for applicable usage and storage.

• After 15 seconds of network inactivity, a sandbox returns to standby with no compute cost and stays there indefinitely. It then resumes with filesystem and memory state intact.
• To keep network latency low, you can deploy across US and European regions.
• TTL and expiration policies provide lifecycle controls by deleting sandboxes after a configured idle duration or maximum age. Expiry can be adjusted after creation.

Talk to the team or start building.

FAQ

What causes cold starts in AI agent code execution? Compare first-invocation and resumed-session traces, separating platform boot, runtime initialization, dependency loading, and session restoration. Large container images can push initial invocations to 10–12 seconds. The largest difference between the traces identifies whether to prioritize templates, snapshots, standby, or changes to dependency setup.

How do you verify that cold-start mitigation is working? Track returning-session share, first-launch time, resumed-session time, pool claim rate, replenishment rate, and snapshot load time. Compare snapshot loading with a full dependency install, and test concurrent claims rather than steady-state pool depth. These measurements show whether standby or pool sizing remains the limiting factor. They also expose problems with template preparation.

How do you validate scaling for a large concurrent sandbox fleet? Load-test orchestration separately from execution, then combine simultaneous placement and resume events. Watch registry latency, placement throughput, queue depth, and per-region capacity. Test regional exhaustion and verify that lifecycle cleanup continues outside the request path. Add regions only when measured network delay justifies the added state-replication, failover, and per-region capacity-management work.