In-memory filesystem architecture for AI sandboxes: how tmpfs and EROFS work together

Your agent runs a data pipeline in a sandbox. It reads many files, writes intermediate results, installs packages, and produces a final report. The task finishes quickly. Then the sandbox enters standby and resumes in under 25ms. Its installed packages and written files stay intact, as do its running processes. How does a system snapshot and restore that much state in that window?

Sandbox filesystem layering makes this possible. A sandbox separates its read-only base image from the agent's runtime writes. The base uses EROFS, or Enhanced Read-Only File System. Default runtime writes live in a tmpfs layer backed by virtual memory, with optional swap.

EROFS and tmpfs form the two layers of this microVM filesystem architecture, and OverlayFS composes them into one tree. Their interaction affects performance, isolation, snapshot size, and the need for persistent storage.

TL;DR

• Sandbox filesystems use two layers: A read-only EROFS layer holds the container image. A writable tmpfs layer receives default runtime writes.
• EROFS gives fast, immutable image access: The base uses a compact, read-optimized format. Every sandbox mounts the same image without copying it.
• tmpfs makes writes fast but volatile: Writes occupy virtual memory and may enter swap. Hibernation preserves them, but standby is not durable storage.
• Snapshots hold the in-memory state: The snapshot contains tmpfs contents and process memory. The EROFS base remains unchanged on the host.
• A thin in-memory delta supports fast resume: Restoring a small diff costs less than restoring a full disk image.

Why do sandbox filesystems need a layered architecture?

A sandbox filesystem must balance isolation and execution speed while maintaining session continuity.

The tension between isolation, speed, and state

Each sandbox needs an isolated filesystem. Reads and writes must stay fast enough to avoid constraining execution. The agent's work must also survive between sessions. That session state includes writable filesystem changes and process memory.

A flat, full-copy model satisfies isolation but sacrifices efficiency. It duplicates every multi-gigabyte image across every sandbox. Each new sandbox must wait for its full copy before starting. Every snapshot must also capture the complete filesystem.

Layering resolves the conflict. All sandboxes share the base image read-only. Each receives a thin writable layer for its own changes. Isolation holds because writable layers never overlap. Storage stays constant per base image because the base exists once on host storage. A new sandbox adds only the files its agent writes. Snapshots stay small because only the writable filesystem layer varies. Each sandbox's upper layer can therefore be created independently without cloning unchanged runtimes, libraries, and system files from the shared base.

Container image layering, extended to microVMs

Docker popularized the layered image model. It stacks read-only filesystem layers beneath a thin writable layer. Sandbox platforms retain that concept but change both parts.

In a container, the writable layer typically sits on disk through an overlay filesystem. That works well for trusted first-party code. Fast-resume sandbox platforms place the writable layer in virtual memory through tmpfs. File writes avoid direct block-device I/O, although tmpfs pages may enter swap.

The base format changes too. Open Container Initiative (OCI) images ship layers as tar archives. That format supports unpacking rather than seeking. Pulling container images consumed 76% of start time, yet only 6.4% of that data was read. Image pulls dominated startup while most bytes went unused. Random-access formats address that inefficiency. Speed-focused sandbox platforms store the base as EROFS. Many sandboxes reading one base pay only for the blocks they open.

EROFS: the read-only base layer

EROFS handles the filesystem content that never changes.

What EROFS is and why it's used for sandbox images

EROFS is a Linux kernel filesystem for immutable images. Its targets include container images and application sandbox images. It compresses data into fixed-size output blocks, so reading one file decompresses only the blocks containing that file.

EROFS uses four KiB physical clusters by default, while SquashFS uses 128 KiB clusters. This difference changes the data volume read during random access. A 16 MB random read cost 26.12 MB of I/O on EROFS. The same SquashFS read cost 165.27 MB with 128 KiB clusters. That is roughly six times the disk traffic. The gap repeats on every cold read when many sandboxes share one base.

EROFS also mounts without extraction. EROFS images mount natively without loopback devices. The image is shared and exists once on host storage. Every sandbox mounts it read-only and sees an ordinary directory tree.

Immutability as an architectural property

The EROFS layer never changes after sandbox creation. Golden images are "built once and mounted everywhere." They stay "bit-for-bit identical across all deployments." User writes go to another trusted filesystem, such as OverlayFS. Every agent write to the default OverlayFS tree lands in tmpfs; writes to mounted external storage bypass it.

A read-only base directly protects agents running generated or untrusted code. Neither the agent nor compromised code can modify base system files. The installed runtime and platform components also remain protected. Read-only root filesystems are recommended in NIST SP 800-190. It states, "Containers should also be run with their root filesystems in read-only mode."

Layering also makes resets cheap. Discarding the writable layer leaves the base untouched. Because the base remains unchanged on host storage, it likely needn't be recaptured during each hibernation.

Custom images and the EROFS packaging pipeline

Teams write a standard Dockerfile, and the platform stores the built image as EROFS. Every sandbox created from that image mounts the same copy. The same EROFS-plus-OverlayFS pattern appears in containerd's EROFS snapshotter. The pattern therefore is not platform-specific.

Each image version is built and stored once. Every sandbox created from it reuses that copy. The build becomes a one-time cost per version. The cost returns when a dependency changes. Bumping a runtime or framework version means rebuilding the image. That trade keeps dependency weight out of every sandbox's RAM. Rebuild after dependency changes instead of patching packages inside running sandboxes.

Treat the Dockerfile as the boundary between static dependencies and session-specific changes. If a package is required in every sandbox, placing it in the image avoids repeated installation, writable-layer growth, and larger standby snapshots.

Confirm that large runtimes and machine learning (ML) frameworks reside in the image, not startup scripts. Large, static image content consumes no writable-layer RAM at runtime.

tmpfs: the in-memory writable layer

tmpfs receives every byte the agent writes to the default OverlayFS writable layer; mounted external paths use their backing storage.

What tmpfs is and how it differs from disk-backed storage

tmpfs is a Linux filesystem that keeps files in virtual memory. It grows and shrinks with the files it contains. Unneeded pages can move to swap when the mount permits it. Swap is permitted by default, while the noswap mount option disables it.

tmpfs has no block device, so fsync cannot flush its files to one. However, swap may still write tmpfs pages to a block device. Avoiding direct file I/O drives the performance difference. In a small-file create-and-delete workload with fsync, ext4 was 10× slower than tmpfs. tmpfs can therefore remove substantial synchronous storage overhead from metadata-heavy package installations. Buffered writes without fsync showed only marginal differences among RAM disks, SSDs, and HDDs. That result came from 80-core runs, with XFS as the exception. tmpfs therefore has its largest advantage in synchronous, metadata-heavy workloads rather than buffered writes.

A package installation still creates thousands of small files. The next package alone unpacks to 7,455 files across 126 MB. Each file creates a metadata write in tmpfs and the next snapshot. The tradeoff is volatility. Unmounting tmpfs loses its contents. Hibernation copies that state into warm storage, but sandbox deletion discards it. Durable or shared data therefore needs external storage.

How tmpfs and EROFS compose as a unified filesystem

The agent sees one directory tree. OverlayFS creates it by mounting EROFS as the lower, read-only layer. It mounts tmpfs as the upper, writable layer. Reads of existing base files go to EROFS. New files in the default overlay tree go to tmpfs.

Writes to existing base files trigger copy-up. On the first write, OverlayFS copies the file's metadata and data into the upper layer. It then applies the write to that copy. Subsequent writes directly reach the tmpfs copy. EROFS is never modified. Copy-up works on whole files. Even a small configuration edit therefore copies the entire file into virtual memory.

Whole-file copy-up shapes runtime behavior. Overwriting CPU PyTorch's 482 MB torch.so copies nearly half a gigabyte into tmpfs. That data moves from shared EROFS storage into the sandbox's virtual memory. It may also enter swap when enabled. Agents should modify small files and leave large binaries in the base.

How this architecture supports fast resume

When a sandbox hibernates, its snapshot captures tmpfs contents and process memory. The EROFS base requires no redistribution during resume because it never changed.

Around 96% of restoration time in mWarp went to memory state. The same principle applies to sandbox resume. A sandbox installing one package and writing several files has a small diff. Gigabytes of runtime and system files in the base do not move.

Perpetual sandbox platforms like Blaxel use this layered stack. It contributes to their fast resume. Blaxel's OverlayFS implementation directs reads to EROFS and writes to in-memory tmpfs. When possible, Blaxel reserves roughly half of available sandbox memory for tmpfs. In Blaxel's Next.js example, the architecture reduced memory from four GB to one GB. A smaller footprint leaves proportionally less in-memory state to snapshot and restore.

Implications for sandbox performance and persistence strategy

This design determines what hibernation keeps, how the writable layer grows, and which external storage option fits each workload.

What survives hibernation vs. what doesn't

On platforms with automatic standby, a sandbox hibernates after losing its active connection. Hibernation preserves its writable layer and process state. Deletion discards both.

• Survives hibernation: Created or modified files remain available. Runtime-installed packages, outputs, and running processes also resume. External network connections do not survive. Database connections and HTTP pools may time out remotely and require reopening.
• Does not survive deletion: The entire tmpfs layer is discarded. A deleted sandbox therefore loses any dataset stored only in tmpfs.

Standby duration can be indefinite, but it should still be treated as resumable compute state rather than guaranteed durable storage. If losing a path would be unacceptable, place it on a Volume or Agent Drive according to its sharing needs.

Persistent data needs storage outside the standby snapshot. Start by listing every path your agent writes during a session. Tag paths that need durability by whether one sandbox or multiple sandboxes need them; treat the rest as resumable state. Use that list to choose the storage layout before the first production run.

Sizing the writable layer and managing snapshot growth

tmpfs consumes virtual memory, including RAM and any permitted swap. More agent writes mean greater sandbox memory consumption. The snapshot also grows. By default, tmpfs can use half of physical RAM. An oversized instance can deadlock the machine. The out-of-memory (OOM) handler cannot free tmpfs pages.

Runtime installations of large packages are the usual cause. A runtime pip install torch downloads an 887.9 MB Linux x86_64 wheel for torch 2.8.0. The installer then unpacks it into tmpfs. This inflates sandbox memory and its snapshot. The cost repeats on every fresh sandbox and slows every resume.

Large, static dependencies belong in the base image. Runtime writes should remain thin and session-specific. Agent code with small configuration edits fits that pattern, as do session outputs. Audit startup scripts for repeated pip install and npm install commands. Check for apt-get calls too. Move those commands into the Dockerfile.

When to add Volumes or Agent Drive for persistence

External storage covers data that snapshots should not carry. Blaxel Volumes provide block storage attached to one sandbox at creation. They suit raw I/O performance and durable, long-term data for that sandbox. Blaxel volumes persist files for weeks or years. A volume can currently grow but not shrink, so leave headroom when creating one.

Use Volumes for model weights and long-running datasets. They also fit outputs tied to one sandbox. Mount the volume at creation. Leave /tmp and the working directory in tmpfs. Use Agent Drive when files, artifacts, or context must span sandboxes or sessions. Agent Drive is a distributed filesystem available in private preview. Multiple sandboxes can mount it concurrently with read-write access. Unlike Volumes, it can also attach to already-running sandboxes.

If regenerable session state only needs fast resume, hibernation is sufficient. Use a Volume when one sandbox needs durable storage or raw I/O performance. Use Agent Drive when agents or sandboxes must share context, artifacts, datasets, or caches. For more on Volumes, see the companion article, "Persistent storage for AI agents."

Choosing the right storage layer

Blaxel implements this layered model natively. Custom Dockerfile images become EROFS, while writes to the default OverlayFS tree enter tmpfs virtual memory. Swap remains available unless disabled. Blaxel snapshots memory state during hibernation. Each sandbox runs inside a Firecracker microVM, the same technology as AWS Lambda.

After 15 seconds of network inactivity, sandboxes return to standby. They remain there indefinitely with zero compute cost and resume with filesystem and memory state intact. Use standby for regenerable session state, Volumes for durable single-sandbox storage, and Agent Drive for shared files.

Start building at app.blaxel.ai or talk to the team at blaxel.ai/contact.

FAQ

Why can a small edit to a base-image file consume much more tmpfs memory than expected?

OverlayFS copy-up operates on whole files rather than changed byte ranges. The first write copies the file's data and metadata from EROFS into tmpfs before applying the edit. A tiny change to a large binary can therefore create a large writable-layer delta, while repeated changes to the copied file continue using the existing tmpfs version.

Does calling fsync make a tmpfs file durable?

No. tmpfs has no block device to which fsync can durably flush the file. Its pages may move to swap, and hibernation may preserve them in a standby snapshot, but neither behavior turns the path into guaranteed long-term storage. Data that must survive sandbox deletion belongs on a Volume or Agent Drive.

Why can runtime package installation increase both memory use and resume work?

Package managers often unpack many small files into the writable layer. Those files consume tmpfs memory and become part of the changing state captured for standby. Installing shared, static dependencies in the Dockerfile keeps that weight in EROFS, where sandboxes can reuse the immutable image instead of recreating the files at runtime.

Do mounted Volumes and Agent Drive paths use tmpfs?

No. Writes under a mounted external-storage path go to that storage backend rather than the default OverlayFS upper layer. This lets a sandbox keep temporary working files in tmpfs while directing durable single-sandbox data to a Volume or shared cross-sandbox data to Agent Drive.