PaperlayerView exact revision on arXiv

Source-linked AI summary

Shepherd: Enabling Programmable Meta-Agents via Reversible Agentic Execution Traces

Simon Yu, Derek Chong, Ananjan Nandi, Dilara Soylu, Jiuding Sun, Christopher D Manning, Weiyan Shi

arXiv:2605.10913v3cs.AIcs.PLcs.SE

TL;DR

Existing agentic substrates expose limited execution state, making runtime coordination, intervention, and repair difficult for meta-agents. Shepherd treats execution as a reversible, programmable object and demonstrates meaningful uplift across runtime supervision, workflow optimization, and long-horizon reinforcement learning.

  • Problem

    Existing substrates expose limited execution state, making it difficult for meta-agents to inspect, intercept, fork, revert, and modify agentic workflows.

    Source: SHEPHERD Programming Model · p. 4

  • Method

    Shepherd represents agent execution as a reversible, Git-like trace that meta-agents can observe, intercept, revert, fork, and rewrite.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Across three applications, Shepherd-supported meta-agents produced meaningful uplift, while state forking ran 5× faster than docker commit for a 5.8 GB image.

    Source: §1 · p. 3, §6 · p. 11

  • Takeaways & Limitations

    Shepherd provides a framework for building higher-order agents that operate over execution across runtime supervision, workflow optimization, and long-horizon reinforcement learning.

    Source: §6 · p. 11

    The case studies are proof-of-existence demonstrations and do not establish policy optimality, broad robustness, or superiority over all alternative substrates.

    Source: §A.1 · p. 15

Abstract

from arXiv · show

As LLM agent systems take on more complex tasks, they increasingly rely on meta-agents: higher-order agents that create, operate on and manage other agents. Meta-agent operations such as coordinating agents, halting risky actions before execution, or repairing failed runs, require runtime manipulation of agentic execution. Yet existing agentic substrates make this difficult: they expose only transcripts and environment snapshots, forcing meta-agents to build ad hoc tooling to reconstruct and operate over full execution state. Therefore, we introduce Shepherd, a Python substrate grounded in functional programming principles, where an agent's execution is itself a first-class object that a meta-agent can easily inspect and transform. Every model action, tool call, and environment change becomes a structured event in a reversible, Git-like execution trace, where any past state can be reverted 5x faster than docker commit and fork. Three example use cases show Shepherd's versatility: (1) a supervisor meta-agent prevents conflicts among parallel coding agents, lifting pair-coding pass rate from 28.8% to 54.7% on CooperBench; (2) a counterfactual optimization meta-agent repairs agent workflows by proposing edits and replaying runs from the point of changed behavior, outperforming MetaHarness on Terminal-Bench 2.0 by 12.8% with 58% lower wall-clock; (3) a training meta-agent picks fork points during rollouts to improve credit assignment in long-horizon agentic RL, doubling GRPO's uplift on Terminal-Bench 2.0. We open-source Shepherd to enable principled and efficient operations over agentic execution for both users and meta-agents.

1 Introduction

Shepherd treats agents and their executions as first-class, reversible structured objects, enabling meta-agents to observe, intervene, revert, fork, and rewrite agentic runs. Its Python substrate supports efficient, formally grounded execution manipulation and three lifecycle-spanning applications that improve coding, optimization, and training outcomes.

Motivation

  • Meta-agents are higher-order agents that operate over other agents and execution traces during or after execution, including advisors, workflow optimizers, and tree-search trainers.Shepherd targets these increasingly central systems by making executions structured data that meta-agents can hold, execute, copy, and rewrite.Source: §1 · p. 2

Contributions

  • Shepherd contributes a functional programming model for higher-order agents, an efficient Python framework, and three lifecycle-spanning meta-agent implementations.The model is instantiated as an intuitive Python substrate and grounded in mechanized formal semantics.Source: §1 · pp. 2–3

Execution model

  • Shepherd records every agent action and environment change as a commit in a reversible, Git-like trace supporting observation, interception, checkout, forking, and task rewriting.Agents are typed tasks, and meta-agents can manipulate execution state or task definitions through structured operations.Source: §1 · p. 2

Foundations

  • Functional-programming foundations enable non-perturbing observation, isolated parallel execution, byte-identical reversion, and pause-inspect-decide-resume control.These properties are formalized in a Lean-mechanized algebraic-effects calculus with a precise execution-trace semantics.Source: §1 · p. 2

Efficiency

  • 5× faster: Shepherd forks a 5.8 GB dockerized agent-environment state than docker commit while reusing over 95% of the LLM provider’s KV cache.The implementation is model-agnostic and lightweight.Source: §1 · p. 3

Applications

  • 28.8% to 54.7%: A runtime supervisor raises CooperBench joint pass rate by intercepting conflicts among parallel coding workers.A counterfactual meta-optimizer outperforms MetaHarness [20] by up to 27.5% while cutting wall-clock time by up to 58%, and tree-search RL improves GRPO [33] training by 5.2 points on Terminal-Bench 2.0.Source: §1 · p. 3

2 Related Work

Prior meta-agent and runtime systems build task-specific machinery or treat execution as a black box. SHEPHERD unifies observation, coupled state forking, exact replay, and runtime-to-training operations through a shared substrate.

Meta-Agents

  • Meta-agent methods [49] [52] [44] [53] [20] [26] [42] [55] [22] [56] [47] [43] [34] reinvent runtime machinery, whereas SHEPHERD enables nonperturbing observation, coupled-state forking, and byte-identical replay.These prior approaches maintain code archives, optimize strategies, or refine context and memory through evolutionary or memory-augmented methods.Source: §2 · p. 3

Agentic Meta-Optimization

  • Pipeline optimizers and test-time scaling methods [16] [6] [57] [18] [21], along with GEPA [2], re-execute candidates from scratch, while SHEPHERD replays only the affected suffix after branching at the first behavior-changing commit.This evaluation primitive lets candidates reuse computation effectively.Source: §2 · p. 3

Agentic Runtime and Infrastructure

  • Unlike AgentGit [23], BranchFS [38], and AgentSPEX [39], SHEPHERD couples environment and agent execution states without worker rewrites, supporting supervision, trajectory optimization, meta-optimization, and stateful RL.Existing systems place checkpointing at different stack layers, trading off autonomy, transparency, and language-level integration.Source: §2 · p. 3

SHEPHERD Programming Model

SHEPHERD treats agentic execution as a first-class, typed, reversible object, enabling meta-agents to inspect, modify, fork, replay, and revert agents through functional-programming-inspired primitives. Its model comprises tasks, effects, scopes, and persistent execution traces.

SHEPHERD Programming Model

  • SHEPHERD elevates tasks, effects, scopes, and execution traces to first-class concepts, making agentic execution observable, substitutable, branchable, and replayable.The design is grounded in functional-programming constructs and supported by Lean-mechanized semantics for typed effect traces.Source: SHEPHERD Programming Model · p. 4

3.1 Task: Agent Definition

  • Tasks are typed agent functions that meta-agents can substitute, compose, modify at runtime, or synthesize as new sub-agents.Tasks may combine deterministic and LLM-driven computation, while meta-agents are tasks whose arguments can be other tasks.Source: §3.1 · p. 4

3.2 Effects: Agent Actions

  • Effects reify each worker action as a typed event, allowing non-perturbing observation, pre-outcome interception, replay under alternative handlers, and rollback when reversible.Actions emit separate intent and outcome events; irreversible effects are recorded for audit rather than rolled back.Source: §3.2 · p. 5

3.3 Scopes: Agent Environments

  • Scopes provide isolated, copy-on-write environments whose fork, merge, discard, and emit operations atomically control agent execution and prevent branch effects from contaminating parents.Scopes can be parallelized, nested, reverted, and resumed while preserving each worker’s original environment bindings.Source: §3.3 · pp. 5–6

3.4 Execution Trace: Agent Execution History

  • The execution trace is a persistent Git-like commit graph that makes every past state reachable, replayable, and efficiently shareable across divergent branches.Replaying from a prior commit pays only for the executed suffix, while branches share their common prefix by content hash.Source: §3.4 · p. 6

4 Framework Performance

Shepherd’s framework performance depends on image-size-independent forking, write-scaled traces, and prompt-cache-preserving replay. Experiments show fast copy-on-write branching and approximately 95% prompt-cache hit rates during replay.

Performance requirements

  • Shepherd evaluates three cost requirements on real Terminal-Bench 2.0 images: image-size-independent forks, write-scaled traces, and replay that reaches the LLM provider’s prompt cache.These properties are identified as crucial for performant meta-agents; full measurements appear in Appendix C.Source: §4 · p. 6

Fork and revert

  • Across three Terminal-Bench 2.0 images, at K=4 concurrent branches, SHEPHERD’s fork latency is 134–143 ms, equal to 2–3% of one agent turn.Source: Table 3 · p. 7

Replay

  • ∼95% prompt-cache hit rates plateau from K=2 onward across 8 Terminal-Bench 2.0 tasks, staying within 5% of the byte-identical ceiling.Fork preserves the parent’s byte-identical LLM message prefix, allowing provider-side cache reuse during fan-out and trajectory replay.Source: §4 · p. 7

5 Experiments

Shepherd supports three meta-agent applications spanning runtime coordination, counterfactual workflow optimization, and agentic-RL training. Across these applications, it improves benchmark performance while exploiting non-perturbing observation, trace replay, and cheap state forking.

5 Experiments

  • Shepherd’s three applications raise CooperBench pair pass rate from 28.8% to 54.7%, outperform MetaHarness on four of five datasets with up to 58% lower wall-clock, and lift Terminal-Bench 2.0 by 5.2 points.These applications span runtime supervision, post-hoc workflow optimization, and agentic-RL training.Source: §5 · p. 7, Figure 2 · p. 7

5.1 Meta-Agent for Multi-Agent Coordination: Runtime Supervisor

  • Opus runtime supervision reaches 54.7% pair pass rate on CooperBench, closing 91% of the coordination gap from the 28.8% coop baseline to the 57.2% solo ceiling.The supervisor observes both workers’ effect streams without perturbing them; Opus averages 24.2 minutes per pair versus 28.4 minutes for solo.Source: Figure 3 · p. 8, §5.1 · p. 8

5.2 Meta-Agent for Meta-Optimization: Counterfactual Replay Optimization (CRO)

  • CRO achieves the best performance on four of five datasets and improves Terminal-Bench 2.0 by 4 pts while requiring the least wall-clock.Compared with MetaHarness, CRO is simultaneously higher-scoring and faster across four datasets, with 27–58% wall-clock savings; it replays only affected trace suffixes.Source: Table 4 · p. 9, §5.2 · pp. 9–10

5.3 Meta-Agent for Training: Meta-Agent-Guided Tree-RL

  • Tree-GRPO improves Flat GRPO by 5.2 points for Qwen3.5-35B-A3B and 3.4 points for Nemotron-3-Super-120B-A12B on held-out Terminal-Bench 2.0.A meta-agent chooses fork turns, and intra-tree sibling comparisons provide suffix-specific advantages without a learned value function or process reward model.Source: Table 5 · p. 10, §5.3 · pp. 10–11

6 Conclusion · A Limitations and Future Works

SHEPHERD makes agent execution a programmable first-class object for meta-agents, supporting supervision, counterfactual replay, and finer-grained reinforcement-learning credit assignment. The paper positions execution-level control as a foundation for future systems that manage increasingly long-lived, stateful, and consequential agents.

6 Conclusion

  • SHEPHERD lets meta-agents hold, inspect, fork, and modify another agent’s execution as a first-class object.The design is grounded in functional-programming abstractions such as functions.Source: §6 · p. 11
  • SHEPHERD demonstrates higher-order meta-agents through runtime supervision, counterfactual workflow replay, and Tree-GRPO credit assignment.These applications span parallel coding, workflow optimization, and long-horizon reinforcement learning.Source: §6 · p. 11
  • The substrate enables meta-agents to act as active operators over agentic execution rather than merely observe transcripts or outcomes.This broadens meta-agent functionality to direct execution-level intervention.Source: §6 · p. 11

A Limitations and Future Works

  • Future systems could compress execution traces into reusable workflows and conduct counterfactual probes for agentic interpretability.Both directions use stored and manipulable execution histories.Source: §6 · p. 11
  • Future meta-agents could safely gate irreversible actions or learn policies for managing agentic execution.These uses extend SHEPHERD from retrospective analysis toward runtime control and learned execution management.Source: §6 · p. 11
  • The paper argues that execution-level control will become a core abstraction as agentic systems grow longer-lived, more stateful, and more consequential.SHEPHERD is presented as an initial step toward making this abstraction readily available and programmable.Source: §6 · p. 11

A.1 Limitations · A.2 Future Works

The case studies establish Shepherd as a useful proof-of-existence framework, not an optimal or broadly validated solution, with costs and replay limits that depend on task and edit structure. Future work extends its reversible execution primitives toward interpretability, continual learning, native agent control, and safety-critical deployment.

A.1 Limitations

  • Shepherd’s three case studies demonstrate substrate sufficiency for meaningful uplift, but do not establish optimal policies, broad robustness, or superiority over alternative substrates.The paper frames these results as proof-of-existence demonstrations rather than a full burden-of-proof comparison.Source: §A.1 · p. 15
  • Meta-agent supervision can cost more than worker execution on short tasks, with favorable economics depending on task length and worker-to-meta-agent cost ratios.The reported supervision and CRO results assume strong models such as Sonnet 4.6, Opus 4.7, or GPT-5.4.Source: §A.1 · p. 15
  • CRO’s replay savings disappear when edits have widely propagating side effects, although the cold first-session regime amortizes within two to three benchmark sessions.In such cases, replay begins at the first affected event and can encompass the entire trajectory.Source: §A.1 · p. 15

A.2 Future Works

  • Future interpretability work could use coupled forks and suffix replay to isolate component contributions and identify minimal edits that change decisions or preserve task success.This would repurpose CRO’s propose-and-replay loop from optimization toward explanation while holding other variance fixed.Source: §A.2 · p. 15
  • Cheap forks could support continual learning across computer-use, browsing, and large-codebase tasks by scheduling reads over coherent multi-week execution graphs.The trace would preserve revisits, backtracks, and retried tool calls across extended interaction rather than a single rollout episode.Source: §A.2 · p. 16
  • A future direction is post-training workers to natively fork, discard, and replay their typed effect streams, learning when to back up, retry, or spawn siblings.Shepherd supplies the typed state needed to define these actions and compute exact returns.Source: §A.2 · p. 16
  • Reversible sandboxes could support safety-critical deployment by preempting irreversible effects, rolling back to materialization points, and providing auditors with content-addressed execution records.The substrate offers differentiated rollback contracts for reversible, compensable, and irreversible effects, but does not solve alignment.Source: §A.2 · p. 16

B Mechanized Core and Proof Envelopes

Shepherd mechanizes a small algebraic-effects trace machine rather than the production runtime, using proof envelopes to identify static fragments whose lowered traces satisfy explicit validation and Lean-side conditions. The envelopes enforce inspectable claim tiers while leaving arbitrary Python execution and most operational behavior outside the formal guarantee.

Mechanized boundary

  • Shepherd verifies a small algebraic-effects trace machine and proof-backed static fragments, while ordinary Python tasks, SDK calls, shell commands, retries, scheduling, and storage remain unverified.The production runtime is separated from the mechanized semantic object; only lowered traces within the formal boundary receive proof-backed treatment.Source: B Mechanized Core and Proof Envelopes · p. 16

Claim tiers

  • Proof envelopes classify runs from runtime_only through reference validation to semantic adequacy, with Core-0/Core-A requiring matching lowering evidence, generated-trace validation, and Lean-side assumptions.Core-0H additionally requires a validated content-addressed sidecar manifest, while Core-A currently requires one direct abort capture and excludes selected abort-path resume/return.Source: B Mechanized Core and Proof Envelopes · p. 16

Executable reference boundary

  • The reference validator reruns static programs for exact generated-trace agreement, emits explicit envelopes, and records content-addressed proof evidence tied to the proof authority, validator, and trace.A Lean ABI table checks theorem identifiers and signatures during the proof-artifact build, while public metadata distinguishes claimed proof backing from runtime-verified authority.Source: B Mechanized Core and Proof Envelopes · pp. 16–17

Non-claims

  • The proof envelope does not verify arbitrary Python control flow, external services, model outputs, filesystem or sandbox correctness, scheduling, recovery, or multi-branch replay.The Lean artifact supplies the semantic core and proof-boundary discipline; Shepherd’s meta-agent applications still depend on the production substrate and empirical validation.Source: B Mechanized Core and Proof Envelopes · p. 17

C Framework Performance: Extended Results · C.1 Measurement Protocol

The extended-results appendix documents the measurement protocol and supporting analyses for Shepherd’s framework-performance claims. Appendix C.1 specifies hardware, workloads, mutation patterns, operation timing, resource accounting, and replication procedures.

C Framework Performance: Extended Results

  • The appendix covers measurement protocol, mutation-size, latency, depth-scaling, overhead, KV-cache, and cross-backend analyses supporting Section 4.These analyses span Docker, gVisor, Firecracker, and managed Linux-container backends.Source: C Framework Performance: Extended… · p. 17

C.1 Measurement Protocol

  • Measurements use a common Vultr setup for Shepherd, docker commit, and full-rootfs copy, while Modal, E2B, and Daytona use separate hosted runtimes.Modal timings include host-to-runtime network round-trip; Table 8 uses E2B Firecracker micro-VMs.Source: §C.1 · p. 17
  • Workloads span 42 MB, 200 MB, and 5.8 GB Terminal-Bench 2.0 images, plus synthetic 30 KB, 100 MB, and 1 GB working directories.Full copy is O(n) in image size, whereas docker commit, Modal snapshots, and Shepherd overlays are O(1) on overlay2.Source: §C.1 · pp. 17–18
  • The default mutation pattern establishes five baseline writes, then adds one 10 KB write per sibling branch across K−1=3 branches.Table 3 counts only post-fork deltas, while Table 7 includes baseline and branch data and sweeps writes from 1 KB to 100 MB.Source: §C.1 · p. 18
  • Fork and revert latency is measured as agent-perceived wall-clock from operation issuance until a tmux readiness probe returns under the production-like host-agent/container-sandbox pattern.Method-specific timings include container restart costs for docker commit and Modal, in-container overlay remounts for Shepherd, and computed rather than directly measured full-copy values.Source: §C.1 · p. 18
  • Disk @ K=4 sums added branch artifacts, while RAM @ K=4 aggregates live-container RSS with Shepherd reusing one host container across forked workers.Modal’s host RAM and disk are unavailable and therefore reported as missing cells.Source: §C.1 · p. 18
  • Each benchmark discards 2–3 warm-up iterations, uses monotonic wall-clock timing, repeats latency cells 10 times, and reports error bars as ±1σ.Resource columns use three repetitions, while full-copy storage uses five; storage is measured with du -sb on the relevant artifact.Source: §C.1 · p. 18

C.2 Substrate scaling under varying mutation size … C.7 Realization across Sandbox Backends

Across mutation sizes and execution depths, SHEPHERD preserves low, mostly metadata-level overhead while maintaining exact revert correctness and zero context inflation for supervisors. Its primitives transfer across sandbox backends, though unsupported OverlayFS environments require slower copy-based fallbacks.

C.2 Substrate scaling under varying mutation size

  • 12/12 revert checks pass across mutation sizes, while fork and revert latency remain approximately 340 ms and disk delta scales 1:1 with agent writes.The sweep ranges from 1 KB to 100 MB per step, producing 8 KB to 800 MB deltas without substrate overhead beyond emitted data.Source: Table 7 · p. 19, §C.2 · p. 19

C.3 Agent per-turn latency reference

  • SHEPHERD’s 134–143 ms fork adds around 2–3% to a typical Haiku 4.5 agent turn, whose mean per-turn total latency is 5.50 s.The LLM call accounts for 98% of per-turn wall-clock here; heavier tool actions such as compilation or training push per-turn time well above the median.Source: §C.3 · p. 19

C.4 Scaling Behaviour

  • 157–252 ms checkpoint/revert latency stays stable through 50 overlay layers, versus 451–558 ms for docker commit, while deeper chains require compaction near 60 layers.The overlay lower-directory chain is bounded by the kernel’s page-size limit.Source: §C.4 · p. 19, Table 8 · p. 20
  • Approximately 120 ms of per-event effect-stream overhead remains constant through 200 steps, while stream size grows linearly at approximately 130 B/event.The measured overhead is network-dominated on E2B.Source: §C.4 · p. 19

C.5 Observe Overhead Detail

  • Supervisor subscription adds exactly 0 tokens to worker context, while effect-stream recording costs 3.1 ms per event locally and 113 ms remotely.The remote 113 ms figure is dominated by network roundtrips rather than framework serialization, and worker message lists are byte-identical with or without supervision.Source: §C.5 · pp. 19–20, Table 9 · p. 20

C.7 Realization across Sandbox Backends

  • The common scope API runs unchanged across Docker, E2B Firecracker, Modal, Daytona, and Prime Intellect, with measured or estimated backend-specific fork behavior.Docker measures 72 ms median, E2B 159–169 ms, Modal snapshot checkpointing 935–1137 ms, Modal revert 75–79 ms, and Daytona is estimated at approximately 150 ms.Source: §C.7 · pp. 20–21, Table 11 · p. 21

C.6 KV-Cache Reuse Detail

  • Prompt-cache savings rise sharply from K=1 to K=2 branches and then stabilize, with approximately 95% hit rates from K=2 onward.The setup reverts to saved checkpoints and restores the cached LLM message prefix; tasks below Haiku 4.5’s 4,096-token cache threshold are excluded.Source: §C.6 · p. 20, Table 10 · p. 21

C.7 Realization across Sandbox Backends

  • Prime Intellect’s copy-based fallback scales from approximately 100 ms for ≤5 MB workdirs to 57 s for 6 GB rootfs, making it unsuitable for large repositories.The fallback is O(n) in working-directory size because gVisor blocks unmount despite allowing mount.Source: §C.7 · p. 21

D Trajectory Compression: Extended Results

The meta-agent compresses completed agent trajectories by selecting hindsight fork points and hints, enabling shorter reruns that preserve verification success across two benchmarks and worker model families.

Motivation and controls

  • Shepherd’s per-step snapshots let the meta-agent choose fork points after observing redundant exploration, while a best-of-five no-meta-agent control recovered only a small fraction of the savings.The study leaves generalization from compressed trajectories to reusable workflows for future task instances open.Source: D Trajectory Compression: Extended… · p. 22

Results

  • Compression rates were 68% and 82% on SWE-Bench Verified, and 77% and 68% on Terminal-Bench v2.0, for Sonnet and GPT-5.4 respectively.A compressed rerun had to pass verification while using strictly fewer model calls than its baseline.Source: Figure 6 · p. 22, D Trajectory Compression: Extended… · p. 22
  • Mean compressed-trajectory lengths fell from 21.4 to 8.9 and 19.0 to 8.8 calls on SWE-Bench, and from 15.8 to 7.1 and 11.4 to 5.2 on Terminal-Bench.The largest individual compression shortened sphinx-doc/sphinx-8459 from 80 to 7 calls; Sonnet saved 12.5 calls on average versus 10.2 for GPT-5.4 on SWE-Bench.Source: Figure 6 · p. 22, D Trajectory Compression: Extended… · pp. 22–23

Setup

  • The evaluation used 88 Terminal-Bench v2.0 tasks and 500 SWE-Bench Verified instances, with a GPT-5.4 xhigh meta-agent rerunning Sonnet 4.6 and GPT-5.4 workers from forked snapshots.The meta-agent emitted a fork step and hint from the completed effect stream, which was prepended to the worker’s system prompt before resumption.Source: D Trajectory Compression: Extended… · p. 22

D.1 Aggregate results … F.2.6 Handoff manifest example

This section presents Shepherd’s runtime supervision and CRO’s trace-grounded workflow optimization, covering selective intervention, counterfactual replay, and localized repair. It details the proposal loop, tool and prompt controls, validation gates, caching, and host-handoff mechanisms that constrain edits and preserve execution context.

D.1 Aggregate results

  • Compression rates exceed 60% in every worker–substrate cell, with mean compressed trajectories roughly halved and Sonnet on SWE-Bench losing 12.5 model calls on average.The reduction is computed only over compressed trajectories; baseline failures from sandbox exhaustion cannot contribute to compression rates.Source: §D.1 · p. 23

D.2 Top compressions

  • Top compressions either restart from scratch when the baseline path is discardable or retain a useful prefix before rewriting later exploratory thrashing.Examples include full restarts for sphinx-8459, xarray-2905, and cobol-modernization, and mid-trajectory cuts at f=8 for pylint-7277 and xarray-6599.Source: §D.2 · p. 23, Table 13 · p. 24

D.3 Hint examples

  • Hints target specific files and code-level intent while pruning detours, but leave the rerun worker to translate guidance into edits rather than providing code solutions.The examples disambiguate target files, correct library details, and explicitly forbid unproductive environment or testing detours.Source: §D.3 · pp. 24–25

D.4 Best-of-N control

  • On seven Terminal-Bench v2.0 tasks, shortest-of-five passing rollouts averaged 5.6 calls versus 4.4 for meta-agent reruns, leaving the compression gap substantially open.The control covered only eight comparable SWE-Bench tasks and seven Terminal-Bench tasks because fivefold resampling was costly.Source: §D.4 · p. 25

D.5 Fork-step distribution

  • Roughly one third of compressed trajectories fork at f=0, one third in the first half, and one third at or beyond the midpoint across both substrates and worker cells.Fork placement matches whether the baseline path was entirely discardable, initially useful, or diagnostically correct before later thrashing; Sonnet’s median rerun cost is roughly one-third of baseline cost.Source: §D.5 · p. 25

D.6 Meta-agent prompt and output schema

  • The pruning meta-agent reads a completed trajectory and emits one JSON object containing a fork step and concise hint, or no_prune when the trajectory is already efficient.The hint names the right file and approach while identifying a dead end to skip; fork_step=0 restarts from scratch, whereas later values restore the state before that step.Source: §D.6 · pp. 25–26

E Runtime supervision: protocol, tools, and meta-agent prompt

  • Opus intervenes more often than Sonnet—inject 39.2%, handoff 31.5%, discard 4.6% versus 26.4%, 18.7%, and 7.9%—while supervision uses escalating actions from nudges to resets.Inject preserves conversation and cache, handoff starts a fresh session while retaining files, and discard additionally restores the pre-run filesystem snapshot.Source: E Runtime supervision: protocol,… · p. 27

F.1 The CRO algorithm

  • CRO maintains a trace graph of workflow variants and outcomes, selecting parents, proposing localized repairs with fix and guard sets, and returning the highest-scoring dev candidate.Failed candidates remain available as evidence but cannot become parents; candidates improving on their fix sets are evaluated on dev and added to the parent pool.Source: §F.1 · pp. 28–29
  • Counterfactual replay forks a parent trace at the first edit-affected event, preserving all other conditions while executing only the changed suffix.This controls stochastic and environmental variation and reduces evaluation cost relative to full rollouts, enabling more candidate edits per unit wall-clock.Source: §F.1 · p. 29

F.2 Implementation Details

  • CRO’s proposer uses typed, queryable execution records—including per-example inputs and outputs, source snapshots, candidate decisions, and prior ledgers—to ground proposals in observed runs.The proposer is a coding agent operating on a real Python source tree rather than a fixed-schema prompt-rewriting model.Source: §F.2 · p. 30

F.2.1 Scratchpad, Host Handoff, and the Proposer Loop

  • Each proposer session reads file-mediated state, creates variants with targeted improve, protect, and invariant examples, writes hypotheses and a batch manifest, and then ends through finish_session.The host performs preflight and dev evaluation, archives failures, merges journals, regenerates state, and exclusively mutates authoritative history and indices.Source: §F.2.1 · pp. 30–31

F.2.2 Controlling prompts

  • The controlling prompt fixes a cacheable session protocol that directs proposers to read brief.md, choose prior bases, create sibling variants, run counterfactual batches, inspect outcomes, and finish the session.Variants must specify improve, protect, and invariant checks before hidden aggregate dev scoring, while the prompt directs agents to fix failure classes rather than literal training examples.Source: §F.2.2 · pp. 31–32
  • CRO’s proposer is instructed to fix error classes rather than instances, use dual semantic-and-structural guards, and iterate through analyze, hypothesize, branch, edit, batch, and observe stages.The prompt also treats aggregate DEV performance as the generalization signal and requires switching to structurally different moves when progress plateaus.Source: §F.2.2 · p. 32

F.2.3 Proposer tool surface

  • The proposer tool surface combines filesystem and ledger inspection, effect-level trace introspection, workflow editing, branching, staged variants, and host-mediated counterfactual evaluation.stage_variant creates sibling variants from a common parent, while run_counterfactual_batch evaluates them with targeted examples and an expected base-hash guard.Source: §F.2.3 · p. 33

F.2.4 Counterfactual replay

  • Counterfactual replay re-executes only the DAG subtree affected by edits using typed cache keys based on source and input hashes, while unaffected sibling subtasks remain reusable.Editing one subtask invalidates that subtask and pipeline caches but can preserve sibling hits; adding an agent between existing ones is described as the cheapest structural move because neighboring caches survive.Source: §F.2.4 · pp. 33–34

F.2.5 Reflection contract and targeted-eval gate

  • Every batch is gated by enforced reflection rules and targeted evaluation requiring cited prior runs, nonredundant mechanisms, at least twelve hypothesized mechanisms, and improve/protect/invariant examples.Variants need aggregate targeted non-regression, limited per-example drops, and improvement on at least 67% of improve examples; failures are archived before full-dev evaluation.Source: §F.2.5 · p. 34

F.2.6 Handoff manifest example

  • The pending_batches/session_NNN.json manifest hands the host a base reference, optional hash guard, staged variants, rationales, mechanism axes, and explicit targeted-example triples.The representative IFBench session-1 listing shows two of four variants, while Table 15 summarizes CRO benchmark metadata without supplying result values here.Source: §F.2.6 · pp. 34–35, Table 15 · p. 35

F.3 Per-dataset settings and optimised workflows

This section specifies dataset splits, metrics, baseline workflows, and optimised CRO workflows across five benchmarks plus Terminal-Bench 2.0. The optimisations add structural branching, auditing, retrieval expansion, repair, or targeted edits while preserving benchmark-specific evaluation protocols.

HoVer

  • HoVer uses a binary FullCoverage metric over 150/300/300 splits, while CRO adds summarisation, bridge resolution, deterministic grounding, recursive expansion, and a third hop.The baseline issues two generated queries and deterministically reranks TF-IDF top-100 candidates; CRO targets missing evidence and bridge-page failures.Source: §F.3 · pp. 35–37

MATH

  • MATH evaluates exact match on a uniformly sampled 100/50/50 L5 split, replacing one solver branch with plan-conditioned dual solvers, repair, and LLM selection.The baseline is a solver–runner–verifier loop with up to two revisions; the structural change shares a planner prefix and repairs only non-compliant or errored branches.Source: §F.3 · pp. 37–38

LiveCodeBench

  • CRO replaces the single-solver workflow with a public-test-graded fan-out and one repair-planned retry, while separating problem understanding from generation and grounding selection in deterministic public-test pass counts.The optimised pipeline runs at most one revision attempt, with the entire fan-out executed concurrently.Source: §F.3 · pp. 38–39

IFBench

  • IFBench measures per-constraint pass rate on 150/300/294 examples, and CRO adds gated auditing, repair after failure, and a final constrained rewrite after a second failure.The audit reads the original prompt’s constraints, emits a structured verdict, and conditions subsequent rewrites on audit results and history.Source: §F.3 · pp. 39–40

Terminal-Bench 2.0

  • Terminal-Bench 2.0 reports avg@5 on the overlapping 25-task Stable25 optimisation and reporting split, enabling apples-to-apples comparison with MetaHarness.The seed exposes seven mutable Terminus-2 surfaces, while CRO keeps that graph unchanged and edits the prompt template, completion checklist, and bootstrap context to address five failure clusters.Source: §F.3 · p. 40

F.4 Per-dataset CRO results … F.5.3 Case 3: Recovered Bridge Pages Needed to Become New Evidence Sources

CRO’s per-dataset results show faster or stronger optimization across several benchmarks, while HoVer case studies reveal progressively more general workflow edits that improve dev coverage from 0.447 to 0.797.

F.4 Per-dataset CRO results

  • 0.523 versus 0.512 gives MetaHarness a 1.0-point IFBench held-out lead, but CRO reaches that frontier in 82 minutes versus 126, with cache reuse near 50% by session 5.Cache reuse starts at 0% during the cold first session.Source: §F.4 · p. 41
  • 0.510 held-out pass-rate on LiveCodeBench puts CRO 11 points above MetaHarness and 2.3 points above GEPA at roughly half MetaHarness’s wall-clock.Both CRO and GEPA produce non-trivial dev trajectories.Source: §F.4 · p. 42
  • 0.80 MATH Level 5 dev pass-rate is matched by CRO in approximately 42 minutes versus MetaHarness’s approximately 100, although GEPA history was empty for this run.This dataset is reported for completeness and excluded from the main results table.Source: §F.4 · p. 43
  • 0.352 avg@5 held-out performance on Terminal-Bench 2.0 is achieved only by CRO, while MetaHarness, GEPA, and the baseline reach 0.312 despite identical 0.40 single-pass dev rates.Cache reuse saturates near 100% within three proposer sessions because each evaluation set contains 25 tasks.Source: §F.4 · pp. 43–44

F.5 Case Studies: Interpretable Counterfactual Workflow Edits on HoVer

  • HoVer case studies evaluate counterfactual workflow edits using dev-set gold-document coverage over 300 examples, with the baseline scoring 0.447 accuracy (134/300).The cases trace how CRO diagnoses failures, generates patches, and measures successive coverage improvements.Source: §F.5 · p. 44

F.5.1 Case 1: The Workflow Was Accidentally Candidate-Closed

  • 0.693 dev coverage, up from 0.447, followed a patch that preserved grounded non-candidate bridge titles as first-class evidence candidates instead of discarding them.The improvement was from 134/300 to 208/300, or +24.7 percentage points.Source: §F.5.1 · pp. 44–45

F.5.2 Case 2: The Workflow Could Diagnose Its Own Missing Evidence

  • 0.787 dev coverage, up from 0.737, resulted from a late missing-evidence resolver that grounded short surface forms to exact local Wikipedia pages.This was +34.0% relative to the original baseline.Source: §F.5.2 · p. 45

F.5.3 Case 3: Recovered Bridge Pages Needed to Become New Evidence Sources

  • 0.797 dev coverage, up from 0.787, came from recursively inspecting recovered bridge pages for relation-bearing sentences and next bridges, despite a lower targeted-train score of 0.833 versus 1.000.CRO selected this more general mechanism, yielding +35.0% over baseline and +1.0% over the previous frontier.Source: §F.5.3 · p. 46

F.6 Meta-Agent Guided Tree-RL: full training configuration

Meta-Agent Guided Tree-RL trains a policy with GRPO by branching trajectories at meta-agent-selected fork points and assigning credit separately across shared prefixes and tree suffixes. The full configuration specifies rollout, optimization, inference, filtering, and evaluation procedures.

Full training configuration

  • Each training step processes 128 rollouts from 16 prompts with 8 samples each, while rollouts allow up to 8 turns, 1,024 generated tokens per turn, and 16,384 input tokens.Training runs on Modal-managed 8×H100 nodes using SkyRL’s GRPO recipe with FSDP2, gradient checkpointing, and torch.compile; overlong trajectories are filtered.Source: §F.6 · p. 46

Meta-Agent Guided Tree-RL

  • Meta-agent branching rolls back each root rollout to a selected fork step, then creates K isolated branches sharing the preceding trajectory prefix.The procedure samples root trajectories, selects fork points, restores the environment, and forks isolated environments for parallel continuation.Source: §F.6 · p. 47
  • Credit assignment uses inter-root advantages for shared prefix actions and intra-tree advantages with a sibling baseline for suffix actions, followed by clipped GRPO updates.Advantages from all prompts in the batch are pooled for the policy update.Source: §F.6 · p. 47

Full training configuration

  • Optimization uses Adam at 5×10^-7 learning rate with 0.01 weight decay, 0.1 gradient clipping, 20 warm-up steps, and ten constant-schedule epochs totaling 1,120 steps.KL loss is disabled, advantages are GRPO-normalized and standardized per group, and checkpoints and validation occur every 10 steps.Source: §F.6 · p. 47
  • Inference uses four vLLM engines at tensor-parallel size 2 with NCCL synchronization and gpu_memory_utilization=0.80.The released launch script is experiments/a4-reversible-rl/modal_smoke_train.py.Source: §F.6 · p. 47

F.7 Meta-Agent Guided Tree-RL: meta-agent qualitative examples

The meta-agent selects causally important fork points from full trajectories and emits executable bash branches, outperforming uniform sampling in early, ambiguous, and deeply nested failures. Its commands are rendered through the policy’s renderer so injected branches match the policy’s own token format.

F.7 Meta-Agent Guided Tree-RL: meta-agent qualitative examples

  • The system receives a numbered inner-agent transcript and final reward, then returns a branch turn, bash tool call, and reason for injection as a rollout branch.The selected command is re-rendered through the policy’s renderer, making its tokens indistinguishable from the policy’s own emission at parse time.Source: §F.7 · pp. 47–49
  • The meta-agent favors the earliest causal mistake, traces symptoms back to their causes, and proposes executable repairs rather than merely identifying turns.It selects non-obvious causes such as a package typo or wrong target file, while tracing a defunct process from turn 5 back to a launch error at turn 3.Source: §F.7 · pp. 48–49
  • In the early-failure case, the meta-agent picks the turn-1 package-name typo and writes one command that installs the correct package and verifies its import.The trajectory continues with failed imports after installing request instead of requests, so the initial typo is the causal fork point.Source: Figure 20 · p. 49
  • In the ambiguous case, the meta-agent traces a turn-5 defunct process to the turn-3 background launch and proposes nohup with logging and explicit host binding.The runnable command converts a hand-wavy diagnosis into a specific hypothesis that can be executed.Source: §F.7 · p. 49, Figure 21 · p. 50
  • In the long trajectory, the meta-agent selects the turn-4 wrong-file edit and replaces recovery thrashing with a clean heredoc rewrite of utils.py.The nine-turn rollout shows five subsequent recovery turns, which uniform sampling cannot distinguish from the consequential earlier mistake.Source: Figure 22 · p. 50

Related papers

Loading 2605.10913v3…