Source-linked AI summary
Crab: A Semantics-Aware Checkpoint/Restore Runtime for Agent Sandboxes
Tianyuan Wu, Chaokun Chang, Lunxi Cao, Wei Gao, Wei Wang
TL;DR
Sandboxed agents accumulate OS state that existing checkpoint/restore approaches either miss or capture too aggressively because of an agent–OS semantic gap. Crab observes OS-visible effects to choose checkpoint granularity and coordinate work, achieving 100% recovery correctness, up to 87% less checkpoint traffic, and execution within 1.9% of fault-free time.
Problem
Existing C/R methods either miss OS-side effects or checkpoint every turn, while neither agent nor OS layers alone can identify recovery-relevant state.
Method
Crab is a transparent host-side runtime that uses turn boundaries, OS-visible effects, asynchronous overlap, and host-scoped scheduling to coordinate selective C/R.
Results
Crab achieves 100% recovery correctness, cuts checkpoint traffic by up to 87%, and stays within 1.9% of fault-free execution time.
Takeaways & Limitations
OS-visible effects can support selective checkpointing for opaque agent workloads while preserving recovery correctness and low execution overhead.
Takeaways & Limitations
When the agent itself runs inside the sandbox, tracking it as an ordinary process can make nearly every turn appear stateful and force expensive process checkpoints.
Abstract
from arXiv · showhide
Autonomous agents act through sandboxed containers and microVMs whose state spans filesystems, processes, and runtime artifacts. Checkpoint and restore (C/R) of this state is needed for fault tolerance, spot execution, RL rollout branching, and safe rollback-yet existing approaches fall into two extremes: application-level recovery preserves chat history but misses OS-side effects, while full per-turn checkpointing is correct but too expensive under dense co-location. The root cause is an agent-OS semantic gap: agent frameworks see tool calls but not their OS effects; the OS sees state changes but lacks turn-level context to judge recovery relevance. This gap hides massive sparsity: over 75% of agent turns produce no recovery-relevant state, so most checkpoints are unnecessary. Crab (Checkpoint-and-Restore for Agent SandBoxes) is a transparent host-side runtime that bridges this gap without modifying agents or C/R backends. An eBPF-based inspector classifies each turn's OS-visible effects to decide checkpoint granularity; a coordinator aligns checkpoints with turn boundaries and overlaps C/R with LLM wait time; and a host-scoped engine schedules checkpoint traffic across co-located sandboxes. On shell-intensive and code-repair workloads, Crab raises recovery correctness from 8% (chat-only) to 100%, cuts checkpoint traffic by up to 87%, and stays within 1.9% of fault-free execution time.
1 Introduction
Agent sandboxes need checkpoint/restore for fault tolerance, preemption, rollout branching, and rollback, but existing methods are either incomplete or too costly. Crab bridges this gap with semantics-aware, host-side coordination and achieves correct recovery with low overhead.
- Motivation: Checkpoint/restore supports fault tolerance, spot execution, RL rollout branching, and safe rollback for long-running sandboxed agents.These use cases depend on preserving or branching accumulated filesystem, process, and runtime state.
- The agent–OS semantic gap: Application-level recovery misses OS-side effects, while OS- and VM-level recovery captures state expensively under dense deployment.The two approaches preserve too little or checkpoint too aggressively because neither layer has complete agent–OS context.
- Crab: Crab infers recovery-relevant state from OS-visible effects, aligns checkpoints with turn boundaries, overlaps work with LLM waits, and schedules traffic across sandboxes.Its host-side runtime uses a Coordinator, eBPF-based Inspector, and C/R Engine without modifying agents or existing backends.
- Results: Crab raises recovery correctness from 8–13% for chat-only baselines to 100% and reduces checkpoint traffic by up to 87%.Every-turn full checkpointing can add up to 3.78× execution time at high density, whereas Crab avoids unnecessary checkpoint work.
- Agent-facing uses: Agent-facing rollback reduces wall-clock time by up to 29% and rollback tokens by 36% in case studies.Rollback is exposed as a tool that replaces brittle shell-level self-recovery with a sandbox restore.
- Results: 100% recovery correctness is achieved while execution remains within 1.9% of no-fault, checkpoint-free time, even at 96-sandbox density.The evaluation covers Terminal-Bench and SWE-Bench workloads.
2 Background
Agents interact with real sandboxed operating systems through repeated model–environment turns, accumulating mutable state that motivates efficient checkpoint/restore. Although use cases differ, they all require efficient snapshotting and recovery.
- Agent execution: An interaction turn exchanges an LLM response, tool actions, observations, and the next model response.This loop lets agents read and edit files, run commands, and coordinate multi-step workflows.
- Agent execution: Sandboxes are isolated containers or VMs exposing filesystem, shell, and process interfaces, with agents either inside or outside the sandbox.Both deployment patterns mutate conversation history and sandbox OS state across turns.
- Why C/R matters: Checkpoint/restore enables crash recovery, spot-instance continuity, and shared-prefix branching for reinforcement-learning rollouts.These use cases require preserving accumulated sandbox state rather than only conversational context.
- C/R use cases: Fault recovery, preemption, tree-based RL, and speculative execution differ in timing and persistence needs but share efficient sandbox snapshot and recovery.The shared requirement is stated across the use-case comparison.
3 Gaps and Challenges
Existing recovery strategies either miss OS-side effects or impose prohibitive full-state checkpointing costs. The agent–OS semantic gap makes recovery-relevant changes difficult to infer, despite most turns being stateless.
- Lightweight Recovery: Application-level recovery preserves conversational or file state but can miss filesystem and live process effects required by later tool invocations.This can produce inconsistent environments after recovery.
- Lightweight Recovery: 1.81× and 1.55× median time-to-solve slowdowns accompany restart recovery under replay and live-LLM settings, respectively.Restart preserves correctness by re-executing prior work.
- Lightweight Recovery: 6% and 28% task success rates for chat-only recovery under replay and live-LLM settings show lightweight recovery can be fundamentally incomplete.Chat+FS reaches 48% under replay and 34% with a live LLM.
- Full-State Checkpointing: 17 requests/s median and 26 requests/s p90 checkpoint arrivals arise from every-turn checkpointing with 100 sandboxes per host.Terminal-Bench traces have a 3.34 s median turn duration and 117 expected turns per task.
- Full-State Checkpointing: 22 ms per-snapshot overhead for ZFS contrasts with 47 s for 64 concurrent 1 GB CRIU dumps, exposing process checkpointing as the scalability bottleneck.The experiment used local AWS NVMe storage; realistic co-location densities can overwhelm host I/O.
- Agent–OS Semantic Gap: More than 75% of turns are stateless, but neither agent frameworks nor the OS alone can identify which changes matter for recovery.Tool signatures such as shell commands do not reliably reveal whether execution modifies files, spawns processes, or only produces output.
4 Design Overview
Crab is a host-side design that combines agent turn semantics with OS-visible state changes to choose checkpoint granularity, hide latency, and coordinate co-located sandboxes. Its workflow dispatches checkpoint work after each completed turn and gates the next turn on durable completion.
- Design Goals: Crab targets correctness, low exposed overhead, and scalability under dense sandbox co-location.These requirements cover reconstructing filesystem and process state, avoiding critical-path pauses, and controlling shared-resource contention.
- Architecture: The Coordinator identifies turn boundaries, the Inspector classifies recovery-relevant OS changes, and the C/R Engine schedules checkpoint traffic.Together, the components couple turn-level control with host-side state observation and scheduling.
- Workflow: At turn end, Crab chooses Skip, Filesystem-only, Process-only, or Filesystem+Process checkpointing based on Inspector output.The decision is made after the agent request to the LLM is forwarded, opening an overlap window.
- Workflow: When the LLM response returns before checkpoint completion, the Coordinator delays release until the checkpoint finishes, ensuring durable recovery state before the next turn.Completed checkpoints allow immediate response forwarding; unfinished work is gated.
5 Runtime and Mechanisms
Crab implements semantics-aware checkpointing through an HTTP Coordinator, an eBPF-based Inspector, and a host-scoped C/R Engine. The system tracks persistent net changes, composes partial artifacts into versioned manifests, and publishes only complete recovery points.
- Coordinator: The Coordinator detects turn boundaries through HTTP interposition, dispatches checkpoint jobs asynchronously, and regulates agent progress.Checkpoint work is submitted after the outbound LLM request and can overlap inference.
- Inspector: The Inspector distinguishes no change, filesystem-only, process-only, and full recovery-relevant state using OS-visible effects.Its eBPF monitor and userspace daemon expose results to the Coordinator.
- Inspector: Persistent net changes are reported while transient effects that disappear before inspection are ignored.Examples include surviving files, live processes, and memory pages dirtied by long-running processes.
- C/R Engine: The Scheduler separates hidden-latency jobs from exposed-latency jobs using normal and high-priority FIFO queues.Jobs whose matching LLM response returns first are promoted so workers prioritize exposed checkpoint latency.
- C/R Engine: CRIU handles process state and OpenZFS handles filesystem state, while the Manager combines compatible artifacts into versioned recoverable manifests.A later filesystem-only artifact can pair with the latest valid process artifact, and vice versa.
- C/R Engine: Only checkpoints reaching the done state after artifact combination are published; interrupted or failed operations remain unavailable for recovery.The lifecycle proceeds through pending, dumping, versioning, and done or failed states.
6 Deployment Refinement
Crab refines recovery for two deployment models by masking interrupted commands and reconciling stale agent memory with restored filesystem state. In agent-in-a-sandbox deployments, fast-forwarding prevents inconsistent replay after selective restoration.
- Agent-with-a-Sandbox: In agent-with-a-sandbox deployments, the Coordinator logs commands and reissues an outstanding command after restoration if it had not completed.This masks the sandbox failure from the external agent.
- Agent-in-a-Sandbox: Tracking the long-lived agent process would make nearly every turn appear stateful and force expensive process checkpoints at turn frequency.Crab excludes the agent process from process-state tracking to preserve selective checkpointing.
- Agent-in-a-Sandbox: A manifest pairing older process state with newer filesystem state can restore an agent whose logical progress lags behind its filesystem.Naively continuing from this mismatch can make the agent and sandbox inconsistent.
- Agent-in-a-Sandbox: Fast-forwarding replays cached request–response interactions with synthetic responses until the stale agent reaches the checkpoint head.The mechanism avoids forwarding repeated interactions to the LLM or re-executing actual work.
7 Evaluation
The evaluation measures Crab’s correctness, overhead, scalability, and agent-facing benefits across code-repair and shell-intensive workloads. Crab preserves recovery correctness while reducing checkpoint cost and remaining close to fault-free execution.
- Evaluation scope: The evaluation covers correctness, component overhead, and agent-facing case studies across SWE-Bench and Terminal-Bench configurations.Experiments include Claude-code, iFlow-cli, and SWE-agent deployments.
- Recovery correctness: 100% recovery correctness is achieved by Crab and FullCkpt, while Chat+FS reaches only 28% and 42% on Terminal-Bench.Chat+FS reaches 100% on SWE-bench, where persistent process state is less central.
- Recovery correctness: The Inspector reports 100% process-change detection and 98.3% filesystem-detection accuracy with zero false negatives over 2,063 labeled turns.Filesystem detection has a 2.3% false positive rate.
- Overhead and scalability: Up to 87% of turns require no checkpoint, reducing unnecessary checkpoint traffic under dense co-location.This sparsity supports selective checkpointing instead of full per-turn checkpointing.
- Overhead and scalability: The Coordinator adds tens of microseconds per turn at 96-sandbox density, while asynchronous checkpointing keeps median exposed delay at zero.The reported p95 exposed delays are 0.00%, 0.37%, 0.44%, and 3.65% at densities 16, 32, 64, and 96.
- Agent-facing case studies: Crab reduces QEMU-startup wall-clock time by 29%, improves speculative median task time by 7.9%, and reduces RL rollout tokens by 40.0–64.2%.The QEMU case also avoids partial-cleanup stalls through sandbox-level rollback.
8 Related Work
Crab differs from prior agent sandbox and persistent-state recovery systems by inferring checkpoint needs from OS-visible effects at turn boundaries. It targets the choice of when and what to checkpoint rather than sandbox provisioning or isolation.
- Sandboxed environments for agents: Agent sandbox platforms primarily provide isolated containers or microVMs for provisioning, isolation, and reuse.
- Persistent-state recovery: Existing checkpoint/restore systems span process, container, VM, and whole-system recovery mechanisms.
- Crab’s position: Crab addresses the orthogonal problem of inferring when and what to checkpoint from OS-visible effects at agent turn boundaries.
9 Conclusion
Crab addresses the agent–OS semantic gap by using OS-visible effects to infer checkpoint granularity and exploit sparse recovery needs. Across shell-intensive and code-repair workloads, it improves correctness and efficiency while remaining near fault-free performance.
- Conclusion: Existing C/R approaches either miss accumulated OS state in agent sandboxes or capture it too aggressively.
- Conclusion: Crab observes recovery-relevant OS effects, overlaps C/R with LLM wait time, and schedules checkpoint traffic across co-located sandboxes.
- Conclusion: Crab raises recovery correctness from 8% to 100%, cuts checkpoint traffic by up to 87%, and stays within 1.9% of fault-free execution time.