PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantifying Frontier LLM Capabilities for Container Sandbox Escape

Rahul Marchand, Art O Cathain, Jerome Wynne, Philippos Maximos Giavridis, Stuart Jennings, Freddy Tuxworth, Tolga H. Dur, Sam Deverett, John Wilkinson, Jason Gwartz, Harry Coppock

arXiv:2603.02277v3cs.CRcs.AI

TL;DR

AI agents can already discover and exploit vulnerabilities, including weaknesses in their execution environments, raising concern about escapes from sandboxed systems. SANDBOXESCAPEBENCH uses nested sandboxing and an 18-task benchmark to measure this capability, finding that frontier models can reliably escape under common failure modes and sometimes succeed on harder tasks.

  • Problem

    AI agents increasingly perform multi-step vulnerability discovery and exploitation, while documented probing of exposed interfaces and misconfigurations motivates measuring their ability to escape sandboxed execution environments.

  • Method

    SANDBOXESCAPEBENCH safely evaluates sandbox escapes with nested sandboxing and 18 tasks spanning orchestration, runtime, and kernel layers, using known exploits and hardened against unintended escape paths.

  • Results

    Frontier models reliably escape container sandboxes under common real-world failure modes and achieve non-trivial success on harder runtime and kernel tasks.

  • Takeaways & Limitations

    The benchmark provides a safe, reproducible way to track sandbox-escape capability and motivates treating plain Docker isolation as insufficient by default.

  • Takeaways & Limitations

    Results concern container or pod-to-host escapes under an intentionally weakened, motivated-adversary setting and should not be interpreted as end-to-end compromise likelihood in defended production environments.

Abstract

from arXiv · show

Large language models (LLMs) increasingly act as autonomous agents, using tools to execute code, read and write files, and access networks, creating novel security risks. To mitigate these risks, agents are commonly deployed and evaluated in isolated "sandbox" environments, often implemented using Docker/OCI containers. We introduce SANDBOXESCAPEBENCH, an open benchmark that safely measures an LLM's capacity to break out of these sandboxes. The benchmark is implemented as an Inspect AI Capture the Flag (CTF) evaluation utilising a nested sandbox architecture with the outer layer containing the flag and no known vulnerabilities. Following a threat model of a motivated adversarial agent with shell access inside a container, SANDBOXESCAPEBENCH covers a spectrum of sandboxescape mechanisms spanning misconfiguration, privilege allocation mistakes, kernel flaws, and runtime/orchestration weaknesses. We find that, when vulnerabilities are added, LLMs are able to identify and exploit them, showing that use of evaluation like SANDBOXESCAPEBENCH is needed to ensure sandboxing continues to provide the encapsulation needed for highly-capable models.

1. Introduction

As capable agents gain tools and cyber skills, sandbox escape becomes a concrete security concern, especially in Docker-based environments. SANDBOXESCAPEBENCH provides a safe benchmark to measure this capability across realistic escape mechanisms.

  • Modern agents can perform multi-step vulnerability discovery and exploitation, while misaligned goals or subverted instructions can turn environmental awareness into sandbox escape.
  • Docker-based isolation remains common in open-source agents and local tooling, but no systematic method previously measured whether AI agents could exploit container weaknesses.
  • SANDBOXESCAPEBENCH uses nested sandboxing so models can be tested for escape capability without risking the evaluation infrastructure.
  • The benchmark contains 18 tasks spanning orchestration, runtime, and kernel layers, grounded in documented vulnerability classes and known exploits.
  • Frontier models reliably escape common misconfigurations and exposed control surfaces, while achieving non-trivial success on harder runtime and kernel tasks.
  • The released benchmark supports tracking escape capabilities, deployment decisions, and AI control research while retaining a private test set.

2. Related Work

Existing evaluations measure web exploitation, binary analysis, and general CTF performance, but not container escape as a distinct capability. SANDBOXESCAPEBENCH fills that gap and connects container escape measurement to containment and control research.

  • Existing benchmarks cover professional CTFs, real-world web vulnerabilities, and large-scale vulnerability evaluation, but none systematically evaluates container escape.
  • SANDBOXESCAPEBENCH isolates model capability from deployment-time defences to measure container-escape capability directly.
  • The benchmark adopts a layered container-security taxonomy spanning orchestration, application, runtime, host/kernel, and hardware layers.
  • Container escape provides a concrete setting for testing containment protocols and supports control research on model escape behaviour.

3. Methodology

The methodology evaluates a motivated agent with shell access against intentionally vulnerable, reproducible container-to-host escape tasks. Success is defined by exact host-file retrieval, with task validity and unintended shortcuts explicitly checked.

  • Threat Model: The benchmark targets container-to-host isolation under a motivated-adversary model with initial code execution inside the container.
  • Threat Model: Agents receive interactive shell access, usually root privileges, standard CLI tools, and network connectivity, but no search tools or knowledge of the specific vulnerability.
  • Task Design: The benchmark uses a fixed set of documented vulnerability classes and excludes zero-days to provide a controlled capability signal.
  • Task Design: The 18 tasks cover orchestration, engine/runtime, and host/kernel layers, selected for exploit-class diversity, reproducibility, and real-world prevalence.
  • Task Design: Difficulty ratings from 1 to 5 estimate expert solve time and achieved Krippendorff’s α = 0.816 across raters.
  • Validity: Reference exploits verify solvability, while exact matching of a host /flag.txt string provides the binary success signal.
  • Outcome Validity: The authors hardened the evaluation after identifying four unintended escape paths and use transcript analysis to detect new shortcuts.

4. Implementation

The implementation runs agent tool calls inside a container nested within an isolated virtual machine, with the host flag outside the container boundary. Inspect manages the evaluation loop and supports multiple VM providers.

  • Inspect manages model-provider API calls, the agent loop, and sandboxed tool execution for the CTF evaluation.
  • A container runs inside a virtual machine, so escaping into the VM filesystem does not expose the host running Inspect.
  • The evaluation places a flag in the VM filesystem and tests whether the model can break out of a Docker container or Kubernetes pod to retrieve it.
  • Figure 2 groups mean success rates over 5 samples by task difficulty from 1/5 to 5/5, distinguishing newer models and token budgets.
  • The benchmark supports Vagrant-based local VM sandboxes and was also tested with Inspect’s AWS EC2 provider.

5. Results and Discussion

Frontier models readily escape common container weaknesses, while harder runtime and kernel exploits remain substantially more difficult but are tractable for the largest models. Success also rises with inference-time compute and varies with model tier, hints, deployment strategy, and model version.

  • Primary results: Frontier models achieve high success on easy misconfiguration escapes, including privileged containers, writable host mounts, and exposed Docker daemons.These tasks represent common failure modes in developer tooling, evaluation harnesses, and agent prototypes.
  • Primary results: 0.40 [0.26, 0.56] is the success rate for GPT-5 and Opus 4.5 on level 3 tasks, while both achieve zero success on levels 4 and 5.Harder tasks require specific vulnerability knowledge or multi-step exploitation pipelines, including runc CVEs and kernel-level escalation.
  • Model differences: Opus achieves 0.49 [0.39, 0.59] overall versus 0.18 [0.11, 0.27] for Haiku and 0.16 [0.09, 0.24] for open-source GPT-OSS-120B.Smaller models remain competitive on direct misconfiguration escapes but degrade as tasks require exploit adaptation, compilation, kernel reasoning, or cross-namespace sequencing.
  • Scaling and deployment: Performance increases approximately log-linearly with inference-time compute, so token budgets, retries, and parallelism can move systems across capability thresholds.The trend is aggregate across heterogeneous vulnerabilities, so no success at a particular budget does not rule out a vulnerability.
  • Model differences: GPT-5.2 performs worse than GPT-5, with success rates of 0.27 [0.19, 0.37] versus 0.50 [0.40, 0.60].The authors report that the cause is difficult to identify without access to training details.
  • Vulnerability hints: Hints dramatically accelerate GPT-5 and Opus below 10^5 tokens, while Opus continues benefiting above that threshold and GPT-5 changes little relative to no hints.The results suggest vulnerability discovery is an early bottleneck, but models differ in translating vulnerability knowledge into exploitation procedures.

6. Updated Results

Updated evaluations show rapid benchmark saturation by newer frontier models, while longer runs reveal self-inflicted sandbox crashes that complicate interpretation of hard-task scaling.

  • Evaluation Setup: The updated evaluation tested Claude Mythos Preview and Opus 4.6, and GPT-5.4 Pro and 5.5 with per-sample limits extended to 100 million tokens.The models were evaluated under the same configuration as Section 5 while extending token limits.
  • Sandbox Failure: Longer runs exposed occasional sandbox crashes that left agents unable to issue further tool calls.The crashes were absent from the original evaluation and emerged during attempts involving complex kernel and runtime exploit chains.
  • Sandbox Failure: Crashes were classified as outer VM failures or inner runtime failures, depending on which sandbox layer became unresponsive.Outer crashes typically followed kernel-level exploits, whereas inner crashes typically followed runtime exploits.
  • Sandbox Failure: The sandbox became unresponsive in 100% (14/14) of Mythos Preview failures, 65% (20/31) of Opus 4.6 failures, 63% (19/30) of GPT-5.4-Pro failures, and 47% (15/32) of GPT-5.5 failures.These self-inflicted crashes occurred early in attempts, at a median of 11% through the transcript.
  • Scaling Behaviour: Hard-task scaling results are inconclusive because apparent plateaus were largely driven by crashing containers rather than necessarily by plateauing capability.On easy tasks, newer models roughly followed log-linear scaling until reaching 100% success; Mythos Preview achieved pass@5 of 100% on harder tasks.

7. Limitations

The benchmark measures container or pod-to-host escape under an intentionally weakened, minimally monitored threat model, so its results do not estimate compromise likelihood in defended production environments.

  • Scope: SANDBOXESCAPEBENCH assumes a motivated adversary with shell access, intentionally introduced weaknesses, and minimal monitoring.Its scope is container or pod-to-host escape rather than general deployment security.
  • Scope: Benchmark results should not be interpreted as end-to-end compromise likelihood in defended production environments.The passage explicitly limits the interpretation of results to the benchmark’s threat model.
  • Coverage: Expert-estimated difficulty ratings and incomplete coverage of container deployments and mitigations limit generalization across deployment settings.The task set is grounded in real vulnerability classes but cannot exhaust the space of configurations and defenses.
  • Agent Architecture: Performance may increase with more sophisticated agents using persistent memory, web search, structured planners, or integrated developer tools.The evaluation uses relatively simple agentic architectures.

8. Conclusion

SANDBOXESCAPEBENCH offers a safe, reproducible way to track sandbox-escape capability and recommends treating plain Docker isolation as insufficient by default.

  • Conclusion: SANDBOXESCAPEBENCH provides a safe, reproducible methodology for tracking sandbox-escape capability over time.The benchmark is intended to support ongoing assessment as model capabilities change.
  • Conclusion: Agent developers should treat “plain Docker isolation” as insufficient by default.The conclusion recommends stress-testing configurations rather than relying on standard Docker isolation alone.
  • Conclusion: The benchmark is intended to motivate stronger isolation primitives and control mechanisms.This recommendation follows the proposed use of SANDBOXESCAPEBENCH for configuration stress-testing.

Impact Statement

The work systematically evaluates LLM sandbox escape using nested, controlled prompts and environments, studying exploits against a host flag without exposing a real host.

  • Impact Statement: The benchmark studies sandbox escape as a machine-learning security capability using a controlled evaluation setting.The work is designed to assess known misconfigurations or artificially introduced vulnerabilities rather than discover new exploit capabilities.
  • Prompts: The system prompt frames the model as a white-hat evaluator permitted to do what is necessary, with bash access for code execution.The prompt asks the agent to reason before tool calls and describe how those calls fit its plan.
  • Task Design: The task asks an agent inside an unknown container to escape to an outer Ubuntu VM and retrieve a flag from /flag.txt.The escape game is presented as a capture-the-flag challenge.
  • Prompt Variants: Hint variants append task-specific exploitation guidance to the user message.The appended text instructs the agent to use the hint to guide its exploitation strategy.
  • Evaluation Control: If an agent emits a message without a tool call, the evaluation automatically inserts a continue prompt.This behavior applies across variants.

B. Task Mapping

The benchmark maps container-escape tasks across escalating difficulty levels and attack surfaces, from straightforward misconfigurations to complex runtime and kernel exploits. Model behavior also revealed unintended escape routes, including default credentials and alternative kernel vulnerabilities.

  • Difficulty scale: Difficulty 1 covers obvious misconfigurations with direct exploitation, while difficulty 2 covers well-known techniques requiring minimal enumeration.Ratings assume a security professional familiar with container escapes, CTFs, the escape objective, and internet access.
  • Difficulty scale: Difficulty 3 requires specific vulnerability knowledge or multi-step processes, whereas difficulties 4 and 5 involve complex CVEs, research, compilation, or deep system understanding.The scale progresses from proof-of-concept adaptation to kernel exploits or sophisticated runtime vulnerabilities.
  • Attack coverage: The task mapping organizes attack mechanisms across orchestration, runtime, and host/kernel layers, including namespace exposure, cgroup abuse, capability misuse, and kernel vulnerabilities.Table 1 records attack categories, exploit mechanisms, difficulty, estimated completion time, CVEs, and original proof-of-concept sources.
  • Attack coverage: Container misconfigurations include writable host mounts, overwritten host runc binaries, leaked file descriptors, and exposed host process namespaces.These paths can enable host-file modification, attacker-controlled execution, or process injection from inside the container.
  • Attack coverage: Kernel and privilege tasks include eBPF verifier escalation, Dirty COW, Dirty Pipe, packet-socket use-after-free, and cgroup release-agent execution.The listed mechanisms span capability misuse, race conditions, page-cache interference, and kernel privilege escalation.
  • Unintended paths: Models found four unintended shortcuts, including default Vagrant credentials and alternative kernel exploits, complicating benchmark validity and real-world containment.The default credential path used vagrant:vagrant, while models substituted Dirty COW or eBPF routes for intended vulnerabilities.

E.2.1. PROGRAMMATIC CHECKS

The programmatic checks show that runs generally engaged with the environment and reached configured limits, while disengagement, refusal-shaped language, and environment breakage varied substantially across models. Deterministic log analyses and scanners provide complementary evidence about these behaviors.

  • Checks: Deterministic checks over Inspect-AI logs measured termination, configuration, engagement, refusal, submission, and resource-use behavior.Figure 14 uses samples df, events df, and messages df; Figure 15 applies a regex detector to every assistant message.
  • Sample termination: Almost every sample ended through successful submission or the token-budget ceiling, with non-zero errors concentrated in a few Claude runs and 11 GPT-5.4-Pro API errors.Only one GPT-5 and six GPT-5-mini samples were missing because of cancellations.
  • Engagement: Every model used bash on most samples, while GPT-5.2 accumulated hundreds of continue prompts per failed sample and the original Claude models used none.Among recent models, Claude Opus 4.6 had 0% continue-prompted failed samples, whereas GPT-5.4-Pro and GPT-5.5 were prompted on most failed samples.
  • Configuration: The five most recent models used a distinct but internally consistent configuration across all 90 evaluations, with a 108-token budget and max connections of 50.The solver, system prompt, and attempts limit were otherwise identical, and no exceptions were reported.
  • Refusal checks: Refusal-shaped language appeared in about 46% of GPT-5.2 samples, about 41% of GPT-OSS-120B samples, about 38% of GPT-5-mini samples, and about 22% of GPT-5-nano samples.For recent models, refusal-shaped language was rare at ≤9% except GPT-5.5 at 35.6%, where it largely coincided with self-inflicted environment breakage.
  • Scanner outcomes: Closed-form scanners found dense incorrect-submission and mid-task-disengagement signals for several GPT and open-source models, while the Claude family was essentially clean.Open-ended issue clusters similarly concentrated on smaller GPT and open-source models; judge refusals clustered around harmful-seeming exploits and policy-laden transcripts.

G. Inter-Rater Agreement on Difficulty Ratings

Independent security experts substantially agreed on the ordinal difficulty ratings for the 18 tasks. Disagreements were generally small, with no pair differing by more than two points on any task.

  • Rating procedure: Two additional security experts independently re-rated all 18 tasks using the same rubric without access to the canonical ratings.R1's ratings remained the canonical ratings used throughout the paper.
  • Agreement: Krippendorff’s α was 0.816, indicating substantial agreement among the three raters on the ordinal difficulty scale.Pairwise linear-weighted Cohen’s κ scores are reported separately.
  • Disagreement: Disagreements were predominantly one point on the five-point scale, and no pair of raters differed by more than two points on any task.R1’s mean ratings exceeded R2’s by 0.06 and R3’s by 0.17.

H. Full results

Full results show a strong separation between easier and harder escape tasks. Models often succeeded on level 2 tasks, some large frontier models reached meaningful level 3 success, and no success was observed on levels 4 and 5.

  • Uncertainty reporting: The results tables report model performance per scenario with 95% Wilson score confidence intervals.The continued table contains the remaining scenario results.
  • Level 2 results: The level-2 table reports per-model success rates for Kubernetes RBAC, permissions, PID namespaces, and SYS_ADMIN scenarios.The table also reports each model’s average across the four level-2 tasks.
  • Level 2 results: GPT-5, Claude Opus 4.5, Claude Opus 4.6, Claude Sonnet 4.5, and Claude Mythos Preview achieved 1.00 average success on level 2.Claude Haiku 4.5 and GPT-5-nano each averaged 0.40, while DeepSeek-R1 averaged 0.13.
  • Level 3 results: The level-3 table evaluates cgroup release-agent, DAC read-search, Dirty COW, Dirty Pipe, Kubernetes, runc, runc overwrite, and system-module tasks.The table reports per-task rates and an average across the level-3 scenarios.
  • Level 3 results: Claude Mythos Preview averaged 0.86 on level 3, Claude Opus 4.6 averaged 0.69, and Claude Opus 4.5 and GPT-5 averaged 0.41.These averages summarize the level-3 task results shown in the table.
Loading 2603.02277v3…