PaperlayerView exact revision on arXiv

Source-linked AI summary

Sandman: Building and Operating Sandboxes for Agentic Reinforcement Learning on Shared Kubernetes

Yiming Yan, Hongyang Wang, Wenbo Chen, Xuan Wang, Jingguo Liu, Xian Li, Chen Qi, Leo Fan, Siwei Li, Qianli Wu, Yichuan Li, Ruochen Wen, Qian Qu, Zheshen Wang, Benoit Dumoulin

arXiv:2610.39711v1cs.DCcs.AIcs.LG

TL;DR

Agentic RL needs persistent task environments for repeated tool calls, but training can create thousands in bursts on shared clusters. Sandman manages these environments as Kubernetes pods; it held 11,858 concurrent tiny sandboxes in a scale test and supported 3.13 million sandbox creations across four production campaigns. The paper reports operational lessons and limits its detailed measurements to a specific cluster and workload setting.

  • Problem

    Agentic RL training creates task-specific sandboxes in bursts on shared clusters, while the managed service tried by the authors could not run task images and throttled calls during session recycling.

    Source: §1 · p. 1

  • Method

    Sandman manages task-image sandboxes as ordinary Kubernetes pods, handling creation, readiness, and cleanup while training frameworks use their persistent state across tool calls.

    Source: §1 · p. 1

  • Results

    11,858 concurrent tiny sandboxes were held in a scale test, and four production SWE campaigns created 3.13 million sandboxes in October 2026.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Sandman’s production experience indicates that sandbox services must handle paced, spread creation, provider-owned cleanup, and grading checked against declared run conditions.

    Source: §8 · p. 10

    The study covers one shared EKS cluster, with detailed training measurements limited to SWE-bench-style Python tasks and one agent harness.

    Source: Most recorded cases had a root in our… · p. 10

Abstract

from arXiv · show

Agentic reinforcement learning (RL) often requires repeated tool calls in task-specific environments. A training job may create thousands of these environments in bursts while sharing a cluster with other jobs. We present Sandman, a sandbox service built and operated at Amazon for agent tool execution and evaluation. It serves multiple production agentic RL workloads on shared Kubernetes. A training framework supplies a task image and uses commands and file operations in the returned sandbox. Files and processes persist across calls until the framework releases the environment. Sandman manages creation, readiness and cleanup. Each sandbox is a Kubernetes pod with a small daemon mounted into the unmodified image. In a scale test, Sandman held 11,858 concurrent tiny sandboxes. Four production SWE-bench Verified training campaigns created 3.1 million sandboxes in October 2026. We report the design trade-offs and operational lessons from handling creation bursts, preparing task environments, reclaiming abandoned sandboxes, and running evaluation under controlled conditions. We will open-source Sandman's daemon, provider and pool.

1 INTRODUCTION

Agentic RL needs persistent task environments that can absorb bursty demand on shared clusters. Sandman provides that service on ordinary Kubernetes pods and reports both large-scale use and operational lessons.

1 INTRODUCTION

  • Bursty shared-cluster demand and limits of a managed code-interpreter service motivated a service able to run task images and handle session-recycling bursts.The managed service could not run task images, and throttled calls during a batch recycle timed out trajectories.Source: §1 · p. 1
  • Sandman serves agentic RL workloads by managing sandbox creation, readiness, and cleanup independently of the model.Training frameworks provide task images and use commands and file operations across calls in the returned sandbox.Source: §1 · p. 1
  • Sandman uses ordinary pods with a mounted daemon, provider-owned creation and deletion, and an optional per-job pool; the paper reports design trade-offs and operational lessons.The daemon, provider, and pool are planned for open-source release.Source: §1 · pp. 1–2
  • 11,858 concurrent tiny sandboxes were held in a scale test, while four October 2026 production SWE campaigns created 3.13 million sandboxes.The campaigns comprised 122 runs on up to 432 B200 GPUs.Source: §1 · pp. 1–2

2 BACKGROUND AND MOTIVATION

Agentic RL rollouts use stateful sandboxes throughout repeated model and tool interactions, with patches graded separately. Asynchronous training and bursty, image-specific demand shape sandbox provisioning and network requirements.

2.1 Rollouts and sandboxes in agentic RL

  • A rollout acquires a task-image sandbox, alternates model actions with shell and file tools, then has its patch tested in a separate sandbox and converted into a GRPO reward.The trainer releases both sandboxes after grading.Source: §2.1 · p. 2
  • Persistent files and processes prevent swapping a sandbox mid-rollout, while asynchronous training can hide some acquisition latency but slower acquisition may hold in-flight slots longer.The paper defines a trajectory as a rollout’s record and an attempt as one agent-session start.Source: §2.1 · p. 2

2.2 Motivation

  • 132 creates arrived in one second when two campaign-A runs started together, versus a campaign mean of 1.8 per second, illustrating bursts that average rates and live-pod counts can hide.The 122 October runs averaged about 26,000 sandboxes each.Source: §2.2 · p. 2
  • Rollout sandboxes spent 74–91% of their lifetime waiting in production traces, and about 80% of requests for an image arrived within ten seconds of another request for it.Prewarming may serve repeated image requests, but reserves resources and serves only compatible requests.Source: §2.2 · p. 2
  • Of 15 reference patches whose verdicts flipped under changed run conditions, 11 flipped when outbound traffic was denied, motivating per-job network rules and checks before grading.The SWE rollout path also separates patch grading from agent execution.Source: §2.2 · p. 3, Figure 2 · p. 3

3 SANDMAN OVERVIEW

Sandman combines a mounted command daemon, a pod provider, and an optional pool to support stateful SWE rollouts on shared Kubernetes. The overview also defines ownership boundaries and the workloads the system trusts.

3.1 Components and the path of a rollout

  • execd runs commands and file operations inside the unmodified task image, using one persistent bash process to retain the working directory, environment, and background processes across calls.Jobs mount the small daemon read-only; it needs only bash and Python 3.Source: §3.1 · p. 3

The provider turns one sandbox request into one

  • The provider turns a request naming an image, CPU, memory, and workspace into a pod, and marks it ready after the readiness probe passes and execd answers its poll.The provider paces and spreads creates, installs network guards, and owns each create until it settles.Source: The provider turns one sandbox request… · p. 3
  • The optional Ray actor pool keeps never-used pods by image and base commit, caps live pods and concurrent creates, and paces its creates with a token bucket.The pool is optional; the SWE jobs described run it with no warm pods.Source: The provider turns one sandbox request… · p. 3
  • The SWE worker acquires a rollout sandbox, captures the patch, sends it for grading in a separate sandbox, releases both, and returns the reward to the learner.The harness calls execd over HTTP while model calls pass through a local proxy.Source: The provider turns one sandbox request… · p. 3

3.2 Responsibilities and the trust boundary

  • Sandman trusts the trainer, graders, and cluster control plane, but not model-written code or other workloads’ pods; model-written code runs as root in containers sharing each node’s kernel.The service account can create and delete pods in its namespace but cannot read webhooks or create network policies.Source: §3.2 · pp. 3–4

4 DESIGN CONSIDERATIONS

Sandman runs task-specific sandboxes as ordinary Kubernetes pods, balancing create pacing, image preparation, cleanup, and controlled grading against cluster capacity and security constraints. Its design assigns each create to a provider and uses optional pooling, reuse, and layered grading safeguards.

4.1 Where sandboxes run: managed service or self-managed

  • Managed sessions could not run task images and had fixed resources; self-managed pods allow per-job sizing but share the training cluster’s creation path and infrastructure.Sandman mounts a small daemon into unmodified images without requiring a node daemon or container runtime.Source: §4.1 · p. 4

4.2 Sandbox creation and placement

  • A pool’s shared token bucket paces its on-demand, warm, and replacement creates; spreading trades caller wait and image pulls for a shorter startup tail.Agent workers address pods directly rather than through Services, avoiding Service object creation and route-programming waits.Source: §4.2 · p. 4

4.3 Sandbox provisioning: on demand, prewarmed or reused

  • On-demand creation is the default; warm pods are health-checked and matched to requested resources, while reused grading pods are reset against a recorded baseline or deleted.Warm pods are keyed by image reference; campaigns used digests to keep rollout and grading images fixed, while warm capacity is reserved image by image.Source: §4.3 · p. 5, Only a digest reference fixes the… · p. 5

4.4 Cleanup when the caller gives up

  • The provider owns each submitted create through completion and cleanup, preventing tested cancellation failures from leaving unowned pods, but callers may wait and late warm pods consume capacity.Commands that time out or reset have their process groups killed; there is no periodic sweep for ownerless labelled pods.Source: §4.4 · p. 5

4.5 SWE grading: files, patches and verdicts

  • Patch extraction stages only agent changes against the task baseline, while reference-patch and empty-patch preflights filter tasks whose tests do not distinguish a fix.The file editor verifies length and SHA-256 and refuses files it cannot write back byte-exactly, directing the agent to use the shell instead.Source: §4.5 · p. 5
  • Before grading, an in-pod probe checks readable declared run conditions and rejects contradictions, but unreadable fields can pass and checks cost 0.06–0.14 pod-hours per task.Each test suite runs in a separate grading pod, never-used by default; infrastructure failures are retried or resampled rather than scored zero.Source: §4.5 · pp. 5–6

4.6 Isolation and access control

  • A per-sandbox token and cluster-enforced network policy restrict access, since outbound blocking alone did not prevent other pods from invoking sandbox commands.A provider refuses an egress-denied label unless it installs the guard; in one check, all six labelled pods reached the internet.Source: §4.6 · p. 6
  • Sandman accepts a shared kernel for code from its own models and does not claim isolation against malicious co-tenants; the runtime default’s effect on verdicts is untested.The design also reports no pod seccomp profile.Source: §4.6 · p. 6

5 EVALUATION

The evaluation combines production logs with controlled experiments to measure Sandman’s scale, startup, prewarming, reuse, and training effects. Its results show both operational trade-offs and limits on what the experiments establish.

5.1 Sandman in production

  • 3.13 million sandboxes were created across four SWE campaigns in two weeks, with average campaign rates of 1.8–6.3 creates per second.The campaigns ran 122 training runs; rollout sandboxes had median lifetimes of 136–207 s.Source: §5.1 · pp. 6–7

5.2 Creation at scale

  • 11,858 concurrent tiny sandboxes were held at a 5–7 s median create-to-Ready time, but the ramp did not test sustained churn.The 38,377 creates had no admission errors; the median was 2 s with 2-vCPU pods.Source: §5.2 · p. 7
  • Spreading a 1,000-pod batch reduced create-to-Ready p99 from 57 s to 8 s, while the default batch had 254 first-attempt address failures.This was one sequential pair; other workloads’ failure rates changed during the comparison.Source: §5.2 · pp. 7–8, Figure 3 · p. 7

5.3 Prewarming and reuse

  • 17.4 s was the median create time when pulling a 1.3–8.4 GB image, versus 4.1 s with the image already on the node.A 120 s readiness budget that included image pulls rejected 9 admissible tasks from a 280-task set.Source: §5.3 · p. 8
  • A warm pool cut grading acquisition from 6.2 s to 0.12 s, but pool-on runs used 2.46× the sandbox vCPU-hours at a 2,100 s idle timeout.In replays, a 300 s timeout retained a 94.5–97.7% warm-hit rate at about half the idle cost.Source: §5.3 · p. 8, Figure 4 · p. 8
  • 29% fewer creates per grading were achieved with checked reuse against a warm pool, but on-demand creation used the fewest pod-hours in all eight replayed traces.In the pre-registered configuration, queue-inclusive acquisition averaged 20.3 s with the pool and 7.2 s with direct creation.Source: §5.3 · p. 8

5.4 Sandboxes and training

  • Grading acquisition and tests lay on the critical path: fresh-pod acquisition took 4.5–6.8% of step time, while rollout acquisition took at most 0.9%.The fixed-concurrency model estimated, but did not measure, 6–14% shorter steps per campaign; changing one run condition altered at most 12 of 11,399 verdicts on a robust task set.Source: §5.4 · p. 9, Figure 5 · p. 9
  • 46.1% of held-out tasks were resolved with the combined grading change, versus 48.4% without it; the six-pair difference was −2.33 points.The change altered three things at once, so the comparison cannot isolate a cause.Source: §5.4 · p. 9

6 LESSONS FROM OPERATING SANDMAN

The operating lessons focus on managing shared-cluster pressure, preventing leaked pods, validating grading conditions, and interpreting incident evidence within its limits.

6 LESSONS FROM OPERATING SANDMAN

  • Pace bursts through a shared pool: an unpaced 4,096-trajectory evaluation coincided with webhook errors affecting 84.9% of its trajectories and 6,126 other-job creates.The authors attribute the cause to webhook configuration and the burst as the trigger; later paced creates are consistent with the fix but do not prove it.Source: §6 · p. 9
  • Turn off Service links: with 3,255 namespace Services, Kubernetes injected about 23,400 variables per new container, while disabling links yielded 16–22 ms median bash calls.With Service creates unpaced, 241 of 1,000 sandboxes failed in a Service-routing test.Source: §6 · p. 9
  • Select across the node pool and monitor free addresses; workers pinned to one instance type found its 115 nodes full, and the ramp reduced free pod addresses from 33.6k to 26.5k.The ramp stopped at 12,000 sandboxes because the tightest subnet was shared with GPU nodes.Source: §6 · p. 10
  • Keep ownership of each submitted create in the provider: acquire timeouts left 5,904 of 13,516 rollout pods for teardown garbage collection, while provider ownership left no tested failure with an unowned pod.The ownership result was tested across API failure windows and 25 cluster trials.Source: §6 · p. 10
  • Preflight reference and empty patches under each job’s configuration to flag tasks whose reference fails or whose empty patch passes.This misses condition effects triggered only by agent patches; under 16 configurations, 15 of 500 Verified reference patches flipped.Source: §6 · p. 10

Most recorded cases had a root in our own implemen-

  • The 41 recorded operations cases are not a census; 23 had a root in the sandbox implementation, and accounted GPU-hour upper bounds totaled 7.1–7.8k.At most 0.47k of those hours fell within the campaigns’ 21.4k B200-hours; losses without an account were not estimated.Source: Most recorded cases had a root in our… · p. 10

7 RELATED WORK

Related work spans sandbox platforms, environment preparation and reuse, training on environment outcomes, grading validity, and infrastructure reporting. Sandman is distinguished by its use of ordinary pods on a shared training cluster and its measurements of effects on training.

7 RELATED WORK

  • Sandbox platforms include purpose-built fleets, hosted runtimes, Kubernetes sandboxes, and microVM-based systems; Sandman instead uses ordinary pods on a shared training cluster.The comparison also notes that the Kubernetes agent-sandbox project mounts a daemon into unmodified images, as execd does.Source: §7 · p. 10
  • Prior work addresses environment preparation through locality-aware warm pools and checkpointed sandboxes, while rollout systems often treat sandboxes as black boxes.Some agent-training systems drop samples lost to environment failures or zero-variance groups.Source: §7 · p. 10
  • Related grading and infrastructure studies examine score shifts, test validation, fleet goodput, reliability, and cluster workloads; Sandman revalidates tests under each job configuration and reports sandbox evidence reaching training.The passages cite work on resource settings and image rebuilds, as well as infrastructure reports on failures and stragglers.Source: §7 · p. 10

8 CONCLUSION

Sandman turns task images into stateful sandboxes on a shared cluster, and its operating studies identify design choices for creation, cleanup, and grading. Faster starts should be judged by training progress and cost.

8 CONCLUSION

  • Sandman uses paced and spread creates, provider-owned cleanup, and grading checks against declared run conditions; faster starts should be judged by training-step time and cost.Source: §8 · p. 10

AI USAGE DISCLOSURE

The supplement documents extensive AI assistance under author direction and review, defines the evidence and workload scope, and details the production workload patterns behind Sandman’s measurements.

AI USAGE DISCLOSURE

  • AI tools assisted with experiment design, code, literature search, cluster operations, and most writing, while authors directed, reviewed, and checked the work.The authors report no AI-generated data beyond specified experimental artifacts and trajectories, and take full responsibility for the submission.Source: AI USAGE DISCLOSURE · p. 14

Appendix

  • The supplement organizes reproducibility details across evidence and workloads, workload distributions, the paper’s section order, failure-path fixes, and recorded operations cases.It defines abbreviations including EKS, GRPO, RL, SWE, and VPC CNI.Source: Appendix · p. 15

A EVIDENCE, WORKLOADS, CODE VERSIONS AND ENDPOINTS

  • Measurements distinguish production campaigns, controlled experiments, virtual-time replays, and reproductions; workload settings specify tasks, resource requests, timeouts, and differing outbound-traffic guards.Endpoints define create, acquisition, waiting share, availability, and sandbox-hours per GPU-hour; intervals and code versions are also specified.Source: A EVIDENCE, WORKLOADS, CODE VERSIONS… · p. 15
  • At list prices, sandbox costs were about 1.7% of GPU cost by CPU allocation or 3.5% by memory allocation, rising to 1.8–3.7% with re-execution and evaluation.The campaigns used about 21,407 B200-hours; the seccomp check found the managed default unconfined, but its effect on verdicts was not measured.Source: A EVIDENCE, WORKLOADS, CODE VERSIONS… · p. 15

B WORKLOAD DISTRIBUTIONS

  • Two concurrent campaign runs produced 132 creates in one second, while rollout sandboxes spent 74–91% of their lifetimes idle during model generation.Rollouts used 8.8–12.4 tool calls per minute; task images averaged 1.2–1.4 GB on Verified and 2.85–3.00 GB on SWE-Gym, and some tasks needed more than 8 GiB.Source: B WORKLOAD DISTRIBUTIONS · pp. 18, 20, Figure A1 · p. 19, Table A9 · p. 19

C SCALE RAMP AND ADDRESS-ASSIGNMENT FOLLOW-UP

The scale and placement studies examine burst handling, address assignment, Services, and prewarming; saved records then characterize image preparation and its limits.

C SCALE RAMP AND ADDRESS-ASSIGNMENT FOLLOW-UP

  • At the 12,000-acquire step, 11,858 sandboxes were live at peak, while 83 acquires timed out behind the pacer and no step left a pod.The ramp was stopped as available subnet addresses declined; the observed tail lay between scheduling and network readiness.Source: C SCALE RAMP AND ADDRESS-ASSIGNMENT… · p. 21
  • Pacing Service creation with pod creates raised success from 759 of 1,000 sandboxes to 1,000 of 1,000; the warm pool shifted queueing into prewarm and lowered acquire p50.Warm-hit acquisition p50 was 0.04–0.51 s at 99.2–100% warm hits, compared with 8.5–61.9 s without the pool.Source: C SCALE RAMP AND ADDRESS-ASSIGNMENT… · p. 21
  • Spreading 1,000 pods across 375 nodes yielded no address-assignment failures and an 8 s create-to-Ready p99, versus 254 failed pods and 57 s under default placement.The back-to-back comparison was one pair with no rate contrast or repetition; other tenants’ failure rates also differed across the experiment’s phases.Source: C SCALE RAMP AND ADDRESS-ASSIGNMENT… · pp. 21–22

D IMAGE PREPARATION FROM RECORDS

  • The image analysis uses records from earlier controlled runs and cannot separate download from unpack or time the first command on a SWE image.Its cold tail is slightly optimistic because 359 re-grading pods lacked grading records and had longer pulls.Source: D IMAGE PREPARATION FROM RECORDS · pp. 22–23
  • Saved records show a 51 ms median request-to-first-command time for a never-used warm pod, versus 8.30–9.11 s for on-demand pods with the image present or pulled.The image pull itself took 0.17 s for the 43 MB image; warm pods idled 121–256 s at the median before first use.Source: D IMAGE PREPARATION FROM RECORDS · p. 22

E WARM POOL AND REUSE

The warm-pool and checked-reuse studies trade faster grading acquisition or fewer creates against idle resource use, with steady-state training effects unresolved.

E WARM POOL AND REUSE

  • The warm pool cut grading acquisition by 97–99.5% at the median in every pair, while using 2.46 times the pool-off runs’ sandbox vCPU-hours.Most additional vCPU-hours came from idle warm pods.Source: Table A17 · p. 24
  • The warm pool reached step 24 sooner in all three pairs, with a 0.906 ratio and 9.4% less wall clock, but its effect on step interval remained inconclusive.The time was gained in different phases across pairs, and whether faster acquisition caused the gain was unresolved.Source: E WARM POOL AND REUSE · p. 24, Table A17 · p. 24
  • Checked reuse cut pod creates per grading by 26.5–32.0% in every pair, with no detectable hit-rate change and an inconclusive step-time effect.The budget discarded most checked pods unused, and reused pods idled in the same buffer.Source: Table A18 · p. 24
  • A per-image pool with a 300 s idle timeout retained 94.5–97.7% hits at about half the idle pod-hours, while global budgets of 16–64 pods performed worse.The pool held no task known to be reuse-sensitive, so the A/B found no harm from residue in that setting, not a need for reuse.Source: Table A19 · p. 24, E WARM POOL AND REUSE · p. 24
  • Across eight replay traces, checked reuse cut creates per grading by 26–46% on five Verified and natural-pool traces and by 39–41% on two SWE-Gym runs, but raised pod-hours on every trace.The increase in pod-hours ranged from 2–60%; no single-use setting reached 95% hits on the SWE-Gym traces.Source: E WARM POOL AND REUSE · p. 24, Table A20 · p. 25

F CANCELLATION WINDOWS

Cancellation and error-path studies trace pod ownership through failure windows, then test fixes in a stand-in API and on the shared cluster.

F CANCELLATION WINDOWS

  • Without the cancellation fix, stored pods could become ownerless; the fix makes cancelled callers wait for an in-flight create and its delete.A cancel before the request is sent still sends it, and an abandoned rollout create is deleted only at readiness or budget expiry.Source: F CANCELLATION WINDOWS · p. 25
  • The alternative implementation closed two create-error windows but still lost failed releases, lost identity across a process boundary, and deleted late grading pods rather than reusing them.Its retrying caller removed a failed-cleanup pod in 20 of 20 trials.Source: F CANCELLATION WINDOWS · p. 25
  • The error-path fix left no pod without an owner in any tested window and never deleted another tenant’s sentinel pod in 1,000 trials.It reads back and deletes a pod after 5xx or transport errors only when its nonce matches; failed deletes enter a retried pending set.Source: F CANCELLATION WINDOWS · p. 25, Table A21 · p. 26
  • In 16 shared-cluster cancellation trials, no pod remained; late grading pods were reused, late rollout pods deleted once Ready, and cancels or drains sent deletes within 0.05 s.The client-injected faults’ frequency in the real API server was not measured, and the stand-in omitted admission, scheduling, pulls, contention, and real retry behavior.Source: F CANCELLATION WINDOWS · p. 25

G REPRODUCTIONS OF EXECUTION DEFECTS

Controlled reproductions on task images exposed defects in command handling, file edits, line endings, exit-status parsing, and output draining. Several fixes were verified through observed reads, writes, process states, or parser outcomes.

G REPRODUCTIONS OF EXECUTION DEFECTS

H GRADING-CONDITION STUDY

The study tests whether grading verdicts depend on declared run conditions, checks whether grading changes leave reusable pods clean, and measures sandbox waits in training. It finds condition-dependent labels and reuse hazards, while the observed learner-path effects are difficult to attribute to sandbox changes alone.

H GRADING-CONDITION STUDY

  • 15 of 500 Verified reference patches (3.0%) were condition-dependent; 481 were stable, two failed everywhere, and two were flaky within a configuration.The two failures everywhere were image-provenance flips caused by newer time-zone data in a rebuilt image.Source: H GRADING-CONDITION STUDY · p. 28
  • 115 of 2,401 SWE-Gym tasks (4.79%, 95% CI 4.01–5.72%) were condition-dependent across four patterns, though the visible-CPU-only class may be noise.Without that class, the share was 4.62% (Wilson 3.85–5.54%).Source: H GRADING-CONDITION STUDY · p. 29, Table A30 · p. 29
  • On two vacuous-test tasks, restored or hardened tests showed that 19.0–22.9% of campaign D’s rewards went to wrong patches.The reference/empty-patch check detects tests that cannot be validated at the checked configuration, not wrong bytes or effects triggered only by agent patches; it cost 0.06–0.14 pod-hours per task.Source: Table A31 · p. 29, H GRADING-CONDITION STUDY · pp. 29–30

I STATE CHECK SCOPE

  • On 240 admitted SWE-Gym tasks, the probe refused all pods for two wrong declarations: a 1,800 s limit with a grader sending no timeout, and a pinned hash seed the pods did not set.In the frozen deployment, the memory check refused 116 of 7,514 readings on 85 pods from nine tasks with 16 GiB under an 8 GiB declaration.Source: H GRADING-CONDITION STUDY · p. 30
  • After release, execd compares each pod against a create-time baseline, restores exact undoable differences, and deletes pods with other differences; it does not watch several state areas and reuses pods at most eight times.The baseline took 0.4–0.8 s at create, and the post-release check took 0.23–0.51 s at the median, off the grading path.Source: I STATE CHECK SCOPE · p. 30
  • Plain reuse was wrong on 64 of 787 gradings, including all 60 gradings after a poisoned tenant; checked reuse was wrong on none of the same gradings.The contradiction rule retired two of three defective pods early, but also retired three healthy pods after a flaky test; it never triggered across 784 validation releases.Source: I STATE CHECK SCOPE · p. 30, Table A34 · p. 31, Table A35 · p. 31

J SANDBOXES ON THE LEARNER’S CRITICAL PATH

  • Rollout-sandbox creation was almost never on the learner’s critical path; grading acquisition and test runs at the slowest trajectory’s end were the relevant waits.The accounting model assumes a generation-bound run and unchanged generation speed; it reproduced mean step intervals within 3% for workloads with records.Source: Table A36 · p. 31, J SANDBOXES ON THE LEARNER’S CRITICAL… · p. 31
  • With the warm pool, last-group grading acquisition ended the learner’s wait in 0.3 s rather than 8.5 s, but records do not isolate the pool’s share of the 9.4% shorter time to step 24.The pool’s predicted step effect was −4.5 to −8.2 s with only 7–11% power at n = 3; checked reuse left acquisition latency essentially unchanged.Source: J SANDBOXES ON THE LEARNER’S CRITICAL… · pp. 31–32

K PRE-REGISTERED COMPARISON

The pre-registered comparison found no evidence that the grading package improved held-out resolve rate, while its bundled changes and operational costs constrain interpretation. The paper also reports failure-path measurements and cautions that its incident records are neither complete nor representative.

K PRE-REGISTERED COMPARISON

  • The comparison tested a package, not the task set alone: arms differed in task set, grading path, and token, with the package arm training on 86 rather than 89 tasks.The check rejected two tasks whose empty patches passed and one whose link checks failed with egress denied.Source: K PRE-REGISTERED COMPARISON · p. 32
  • Across six pairs, mean held-out resolve-rate difference at step 150 on 112 tasks was −2.33 points (95% CI −4.22 to −0.43); the registered reading was “no package higher”.The exact sign-flip test gives p = 0.0625; the package scored lower in five of six pairs and tied in one.Source: Table A38 · p. 32, K PRE-REGISTERED COMPARISON · p. 32
  • 50.9% of package-arm gradings waited for a pool-paced create, with a 38.6 s median; warm and reused hits took 12 ms at the median, while reuse cost 1.21 creates per grading.The package arm’s acquisition median was 7.8 s versus 5.5 s for direct creates, and the design cannot identify why its score was lower.Source: K PRE-REGISTERED COMPARISON · p. 33

L FAILURE PATHS AND THEIR FIXES

  • 15 of 500 Verified reference patches flipped under grading conditions, and 4.79% of 2,401 SWE-Gym tasks depended on run conditions; two campaign D tasks also had tests that did not check the fix.With missing tests restored, 19% of D’s positive rewards on those tasks went to wrong patches; one published hardened suite yielded 23%.Source: L FAILURE PATHS AND THEIR FIXES · p. 33

M FULL INCIDENT LIST

  • The incident appendix lists 41 recorded cases, but explicitly says they are neither complete nor representative, so case counts are not failure rates or cost weights.GPU-hour accounts bound losses for ten cases, with each job assigned to one case to avoid double counting.Source: M FULL INCIDENT LIST · p. 34, Table A49 · p. 42

Related papers

Loading 2610.39711v1…