PaperlayerView exact revision on arXiv

Source-linked AI summary

ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks

Jeonghye Kim, Minseon Kim, Young Jin Kim, Matheus Pereira, Marc-Alexandre Côté, Alessandro Sordoni, Xingdi Yuan, Zhengyan Shi

arXiv:2609.18805v1cs.SEcs.AI

TL;DR

Coding agents often receive specified behavior, but practical web development can require inferring it from working software. ProgramDistill turns interactive applications into replay-verified repair tasks and evaluates agents across reconstruction settings and restoration depths. Its results show limited full-application workflow recovery and declining partial-application success as restoration depth increases.

  • Problem

    Agents are often evaluated against specified behavior, while practical web development may require inferring behavior from a working reference application.

    Source: §1 · p. 1

  • Method

    ProgramDistill uses an automated mine–craft–patch pipeline to turn interactive applications into replay-verified tasks spanning feature repair and full reconstruction.

    Source: §1 · p. 2

  • Results

    In full-application reconstruction, GPT-6 Astra and Claude Opus 5 recover 49.2% and 28.8% of evaluated workflows, respectively, while partial-application success declines as restoration depth increases.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ProgramDistill provides a scalable benchmark with controlled restoration difficulty for evaluating reconstruction performance and interactive information-seeking strategies.

    Source: §1 · p. 2

    The agents receive structured browser observations without screenshots, and visual fidelity is outside the current replay-based behavioral checks.

    Source: §5 · p. 14

Abstract

from arXiv · show

Coding agents are typically evaluated with desired behavior specified through issues or instructions. In practical web development, however, agents may need to infer behavior from working software and implement it in an incomplete application. We introduce ProgramDistill, a benchmark evaluating coding agents on features discovered through interaction with fully functional reference applications. We build ProgramDistill by factorizing applications into features of different granularities, each associated with replayable behaviors executable via its gold patch. Our pipeline, mine-craft-patch, discovers 1,975 replay-verified behaviors across 26 applications and constructs 4,063 tasks without human intervention. Across nine frontier coding agents, GPT-6 Astra and Claude Opus 5 achieve 49.2% and 28.8% success on cumulative workflows in full-application reconstruction. In partial-application reconstruction, success falls from 100% to 64.0% and from 96% to 32% as restoration depth increases from 1 to 8. ProgramDistill thus provides a scalable benchmark with controlled difficulty for evaluating and diagnosing coding agents, and a natural basis for future curriculum-based training.

1 Introduction

ProgramDistill turns interactive web applications into replayable software-engineering tasks, asking agents to infer behavior from a working reference and restore it in an incomplete application. Its benchmark spans feature-level repair through full reconstruction and measures recovery along prerequisite lineages.

1 Introduction

  • In practical web development, agents may need to infer intended behavior from a working application rather than receive it as an issue or instruction.Even dragging a card can require inferring both its visible effect and the state that persists afterward.Source: §1 · p. 1
  • Application-to-task factorization turns features at varying granularities into SWE tasks with replayable interactions and gold patches, composable along prerequisite lineages.This yields a controllable restoration-depth axis rather than treating an application as one reconstruction problem.Source: §1 · p. 2
  • Reference-to-current distillation asks agents to infer behavior from a working reference and implement it in an incomplete application, with success measured by replaying the interaction.ProgramDistill uses “distillation” for extracting and transferring executable behavior across the task and application transformations.Source: §1 · p. 2
  • The fully automated mine–craft–patch pipeline mines 1,975 replay-verified behaviors across 26 applications and constructs 4,063 tasks without human intervention.The benchmark spans atomic repair, cumulative repair, and full-application reconstruction.Source: §1 · p. 2
  • 49.2% and 28.8% of evaluated workflows are recovered by GPT-6 Astra and Claude Opus 5, respectively, in full-application reconstruction; partial-application success declines as restoration depth rises from 1 to 8.In partial reconstruction, success falls from 100% to 64.0% for Astra and from 96% to 32% for Opus 5.Source: §1 · p. 2
  • Prerequisite lineages provide a progression from atomic repair to full reconstruction, exposing failures as restoration depth increases and offering a basis for future curriculum construction.The paper identifies trajectory distillation or reinforcement learning as possible future training approaches.Source: §1 · p. 2

2 Application-to-Task Factorization via the Mine–Craft–Patch Pipeline

The mine–craft–patch pipeline converts interactive web applications into repair tasks by mining replayable behaviors, masking their implementations, and evaluating repairs against the reference. Prerequisite lineages support both isolated repairs and cumulative tasks of increasing restoration depth.

2 Application-to-Task Factorization via the Mine–Craft–Patch Pipeline

  • Mining, crafting, and patching respectively discover replayable behaviors, mask one or more behaviors to construct tasks, and replay lineages to measure recovered behavior.The pipeline is model-agnostic and was orchestrated with multiple LLM agents in the experiments.Source: §2 · p. 2, Figure 2 · p. 3

2.1 Setup: Deterministic and Observable Application Instances

  • Each application runs in a live, self-contained environment with a stable reference, an editable implementation, and reproducible replay verification.The setup provides a common browser interface for observing behavior, constructing tasks, and repairing the current application.Source: §2.1 · p. 3

2.1.1 Applications and Runtime Instances

  • The 26-application corpus combines controlled OSWorld applications with open-source projects and SaaS clones that contribute larger codebases and deeper workflows.Fixed production builds serve as references while development-server versions are editable; resetting state and using a shared deterministic clock reduces time-dependent variation.Source: §2.1.1 · p. 3

2.1.2 Browser Helper

  • A shared Playwright-based helper executes browser actions and returns structured observations, using stable observable attributes and settled application state for reliable replay.Although it records screenshots for analysis, evaluated reconstruction agents receive structured observations only.Source: §2.1.2 · p. 3

2.2 Mining: Discovering and Verifying Interactive Behaviors

  • Mining proposes behavior goals, explores the live application from observations, then recollects concise traces with expected signals and parent references for replay.Only traces passing replay verification are admitted, and parent links preserve prerequisite context as mining expands to deeper behaviors.Source: §2.2 · p. 4

2.3 Crafting: From Verified Behaviors to Repair Tasks

  • Atomic masks are accepted only when the application builds, prerequisite traces still pass, and the target trace fails; gold-patch replay then verifies recovery of the complete lineage.A mask-depth critic rejects superficial changes that leave the substantive implementation intact.Source: §2.3 · pp. 5–6
  • Cumulative tasks combine validated masks along a lineage, with unmasked traces serving as replay bridges rather than repair targets.Of 1,201 cumulative tasks, 629 masks were composed deterministically and 572 required an LLM merge agent; the agent-assisted merges succeeded without omitting intended mask components.Source: §2.3 · pp. 5–6

2.4 Patching: Restoring Behavior from a Working Reference

  • Patching compares a repaired application against a working reference and scores it by replaying recorded actions, selectors, and expected signals, without requiring source-level equivalence to the gold patch.Partial reconstruction starts from a masked repository, while full reconstruction starts from a minimal scaffold; chain scoring gives partial credit for a recovered lineage prefix.Source: §2.4 · p. 6

3 Experiments

ProgramDistill yields a broad benchmark of replay-verified behaviors and repair tasks, then evaluates agents on partial- and full-application reconstruction. Success declines as repairs combine dependent behaviors, while analysis identifies gaps in observation, implementation, and validation.

3.1 Benchmark Generation Statistics

  • 1,975 verified behavior traces produce 4,063 repair tasks: 2,862 atomic and 1,201 cumulative, split between logic-only and logic-and-UI masks.The evaluation suite samples 300 tasks across all 26 applications, stratified by restoration depth and covering 269 distinct lineages.Source: §3.1 · pp. 7–8, Figure 5 · p. 7

3.2.1 Model Performance Comparison

  • Astra leads partial-repair scores at 84.3%, followed by Opus 5 at 68.7% and GPT-5.6 Sol at 60.7%, at mean trajectory costs of $33.99, $28.86, and $16.01.Astra’s 15.7-point gain over Opus 5 comes at about 18% higher mean trajectory cost; higher expenditure does not always yield better performance.Source: §3.2.1 · p. 8
  • 100% to 64.0% for Astra and 96% to 32.0% for Opus 5: binary success falls from restoration depth 1 to 8.Astra solves all depth-1 tasks; every other model retains less than half its depth-1 performance at depth 8.Source: §3.2.1 · pp. 8–9, Figure 7 · p. 8
  • A successful depth-8 Trello repair required repeated observe–edit–validate cycles across multiple dependent behaviors.Claude Opus 5 repeatedly inspected the reference and current application to identify residual mismatches and apply targeted fixes.Source: §3.2.1 · p. 9, Figure 8 · p. 9

3.2.2 Analysis

  • From depth 1 to 8, repair burden grows 9.3× in code lines and 10.7× in browser actions, while per-target observation effort falls roughly 75%.Reference observations per target drop from 34.60 to 8.46 and current-app observations from 27.69 to 6.81; unchanged target files and retained stub lines also increase.Source: §3.2.2 · pp. 9–10, Figure 10 · p. 10

3.3 Full-Application Reconstruction

  • Across 12 applications, evaluation covers 590 atomic behavior tests and 413 cumulative workflow tests, with cumulative tests requiring multiple dependent behaviors to pass together.Full-application reconstruction starts from a minimal executable scaffold and uses a product-level capability description and live reference.Source: §3.3 · pp. 10–11

3.3.1 Model Performance Comparison

  • 58.98% atomic and 49.15% cumulative recovery for GPT-6 Astra lead full-application results, ahead of Opus 5 and GPT-5.6 Sol.Opus 5 reaches 42.03% atomic and 28.81% cumulative recovery; Sol reaches 33.39% and 21.07%, respectively.Source: Table 1 · p. 11, §3.3.1 · p. 11

3.3.2 Analysis

  • In full-application failures, 59.2% were behaviors not observed; among the rest, 1.8% were not implemented, 11.1% had the wrong observable form, and 27.9% the wrong state, route, or result.Validation sometimes missed errors because agents did not recheck the exact failing workflow after the final relevant edit.Source: §3.3.2 · p. 12, Figure 12 · p. 12

4 Related Works

Prior benchmarks cover repository editing, visual reconstruction, reverse engineering, and browser use, while ProgramDistill combines behavior discovery, implementation, and replay validation against a working application. Its factorized tasks support localized repair, dependent workflows, and full-application reconstruction.

Repository-level software-engineering benchmarks

  • ProgramDistill uses a working application to supply both repair targets and replayable verifiers, rather than relying on an issue or other supplied specification.This contrasts with repository-level benchmarks whose desired behavior is provided through issues, tests, visual artifacts, or user interactions.Source: §4 · pp. 13–14

Visual and interactive web development

  • Visual-reference systems provide a fixed specification, whereas ProgramDistill requires agents to discover behavior by exercising a live reference and validating the implementation.The paper characterizes this as a shift from specification consumption to specification discovery.Source: §4 · p. 14

Behavioral reconstruction and reverse engineering

  • ProgramDistill factorizes applications into replayable behaviors with prerequisite relationships, unlike ProgramBench [37] and MirrorCode [1], which primarily target holistic reimplementation.The shared behavioral substrate supports atomic repair, cumulative workflows, and full-application reconstruction under one verifier.Source: §4 · p. 14

Browser-use agents and environments

  • Browser-use agents operate websites, while ProgramDistill agents modify application source to recover behavior observed from a reference.Discovered interactions serve both as software-engineering tasks and replayable verifiers.Source: §4 · p. 14

Synthesis

  • ProgramDistill brings observe, infer, implement, and validate into one closed-loop setting that mirrors reference-guided web development.Behaviors uncovered from applications provide a common substrate for localized repair, compositional workflows, and full-application reconstruction.Source: §4 · p. 14

5 Discussion

ProgramDistill exposes reconstruction gaps as restoration depth increases and substantial behavior remains unrecovered in full-application reconstruction. The discussion identifies scope and infrastructure boundaries, and points to training applications and broader uses of executable software as future directions.

Limitations and future work

  • Agents receive structured browser observations, not multimodal screenshot inputs; visual-fidelity evaluation would require criteria beyond replay-based behavioral checks.The authors identify screenshots as a possible extension for visually grounded interaction.Source: §5 · p. 14
  • Experiments focus on self-contained web applications, leaving external services, nondeterministic state, and desktop or mobile applications for future work.Source: §5 · p. 14
  • Construction uses GPT-5.6 Sol throughout, so mined-behavior and task quality may depend on construction-model capability.More capable models could improve discovery, masking, and task construction while increasing generation cost.Source: §5 · p. 15
  • ProgramDistill depends on browser and agent infrastructure for exposing, recording, and replaying behavior, and measured performance may vary with those choices.The authors identify replay robustness, observation quality, and agent tooling as infrastructure dimensions for future sensitivity analysis.Source: §5 · p. 15

Training coding agents

  • Replay-verifiable tasks and agent trajectories could support training through trajectory distillation or reinforcement learning, with restoration depth providing a curriculum axis.The use of ProgramDistill for training coding agents is left to future work.Source: §5 · p. 15

Potential misuse

  • The framework is intended for research and authorized software development; experiments use self-contained applications derived from public or open-source repositories.Source: §5 · p. 15

Conclusion

  • Across frontier coding agents, performance declines with restoration depth, while full-application reconstruction leaves substantial behavior unrecovered.These settings expose gaps between observing reference behavior, implementing it, and validating the resulting application.Source: §5 · p. 15

Broader implications

  • Executable software can serve as a scalable source of specifications, tasks, verifiers, and learning signals, not only as an evaluation target.The authors describe a possible continual cycle of increasingly challenging tasks that reveal agent limits and provide experience for improvement.Source: §5 · p. 15

A Application Corpus

The corpus combines controlled and real-world interactive web applications, pinned to reproducible builds for task construction and replay. Its execution setup also synchronizes application time so time-dependent behavior remains consistent across replays.

A Application Corpus

  • The 26-application corpus combines 18 OSWorld-adapted apps with eight public-repository projects and SaaS clones spanning diverse stacks and deeper workflows.The real-world applications include Django, Angular, React, and Next.js with SQL and document back ends.Source: A Application Corpus · p. 19, Figure 14 · p. 20
  • Applications are pinned to upstream commits and rebuilt to match vendored source trees, keeping mining, masking, repair, and replay on byte-identical versions.Task counts vary by application according to how many behaviors can be reliably discovered and replay-verified.Source: A Application Corpus · p. 19, Table 3 · p. 19

A.1 Deterministic Execution

  • Each trace starts from a seeded state and recorded application time; replay shifts the application clock by a constant offset while time advances normally.The synchronized frontend, server, interpreter, and database preserve time-dependent behavior and temporal ordering across replays.Source: §A.1 · pp. 19–20

B Browser Helper Implementation Details

A shared Chromium browser helper supports mining, replay verification, and repair evaluation by carrying out actions and returning structured observations. Replay reliability relies on stable element selectors and observations recorded after browser activity settles.

B Browser Helper Implementation Details

  • The shared browser helper executes actions in Chromium and returns structured observations for mining, replay verification, and repair evaluation.The solver-facing observation view is filtered, and repair can observe reference and editable applications jointly.Source: B Browser Helper Implementation Details · p. 21, §B.1 · p. 21

B.1 Action Vocabulary

  • Mining and repair share observation, navigation, and interaction actions, while repair adds side-by-side application observation and mining adds trace-construction actions.The complete action vocabulary is summarized in Table 4.Source: §B.1 · p. 21, Table 4 · p. 21

B.2 Replay-Stable Element Addressing

  • Recorded actions store selectors built from browser-observable attributes rather than build-dependent DOM identifiers, and ambiguous targets are refused.Live actions use transient element IDs; retained element-targeted actions must have selectors resolvable during replay.Source: §B.2 · pp. 21–22, Figure 15 · p. 22

B.3 Settled Observations

  • After each action, the helper waits for network activity and DOM mutations to settle before recording an observation, excluding traces that fail to settle within the limit.The default settled condition requires no tracked requests in flight and an unchanged DOM for 400 ms, with a 10 s cap.Source: §B.3 · p. 22, Figure 16 · p. 22

C Agent Harness and Execution Settings

  • The agent harness builds on R2E-Gym [14], retaining its coding-tool interface and adding live browser interaction through a browser CLI.The section covers runtime boundaries, communication, observation handling, history management, and execution settings.Source: C Agent Harness and Execution Settings · p. 23

C.1 Runtime Architecture and Isolation

The harness separates the editable application, reference-side browser service, and verifier, allowing agents to observe reference behavior without access to its source. Isolated browser contexts, restricted network paths, and fresh verification preserve that separation.

C.1 Runtime Architecture and Isolation

  • The runtime separates the editable application, reference-side browser service, and verifier, with the agent observing reference behavior but not receiving its source or a source checkout.The verifier separately evaluates submitted source.Source: §C.1 · p. 23, Figure 17 · p. 23
  • The agent’s shell and file tools operate only on the current application, while its browser command acts as a client of the reference-side browser service.The helper exposes browser interactions through a constrained API and returns structured observations.Source: §C.1 · p. 23
  • The browser helper maintains isolated reference and current contexts; only the current context connects across the runtime boundary to the editable application.The contexts use separate cookies, storage, and page state.Source: §C.1 · p. 23, Figure 17 · p. 23
  • Network filtering blocks direct agent connections to reference-application ports while allowing replies associated with browser connections to the editable application.The same policy applies in both evaluation settings.Source: §C.1 · p. 24
  • A separate verifier starts from a clean task baseline, applies the submission, rebuilds the application, resets persistent state, and replays workflows; grading traces and gold patches remain private.The boundaries are not designed as a general-purpose sandbox for arbitrary hostile code.Source: §C.1 · p. 24

C.2 Observations and Execution Settings

This appendix specifies the harness, prompts, trace representations, task-generation procedures, and evaluation settings used to mine replay-verified behaviors and create feature-restoration tasks. It also defines reconstruction contracts and execution conditions, including feature masking and model-cost accounting, while directing agents to restore and verify observable behavior against reference applications.

C.2 Observations and Execution Settings

  • Agents receive text-based tool observations, can revisit paginated outputs, and retain task instructions and recent context as older complete turns are removed; no run reached the token stop threshold.Full reconstruction also exposes selectors for observed targets; screenshots, raw HTML, API-request logs, and internal artifact paths are excluded. The maximum agent-step limit was reached only once, by Opus 5 in partial-application reconstruction.Source: §C.2 · p. 24

D.1 Mining-Agent Orchestration

  • Mining proceeds in rounds: planners propose goals, collectors discover and cleanly re-collect behaviors, replay verification admits stable traces, and reflection guides the next round.The Collector handles both discovery and clean re-collection, using a guided prompt variant for the latter; admitted traces and parent links are then inputs to crafting.Source: §D.1 · p. 25, D Mining Agent Prompts and Trace… · p. 25, Figure 18 · p. 25

D.2 Planner

  • The Planner proposes independently valuable, diverse goals grounded in source code and observed UI, prioritizing uncovered capabilities and relevant parent traces.It receives coverage, failures, app capabilities, available parents, and prior reflections, and must avoid redundant workflows and unsupported evidence.Source: §D.2 · pp. 25–26

D.3 Collector

  • Collectors record browser actions and select observation-derived signals that prove the achieved outcome; parent chains establish prerequisite state, while guided clean re-collection removes unnecessary exploration.Signals must prove material outcomes rather than merely reflect editable form values; child traces record only novel actions after the parent state is replayed.Source: §D.3 · pp. 26–27

D.4 Relabeler; D.5 Round Reflector

  • The Relabeler names only behavior demonstrated by actions and evidence, while the Reflector reviews actual outcomes and coverage gaps to advise the next planning round.The Relabeler does not see the original goal; the Reflector flags consequential gaps, repeated failures, and opportunities for deeper exploration.Source: §D.4 · p. 28, §D.5 · p. 28

D.6 Admitted Trace Representation; D.7 Trace Evolution Across Mining Stages

  • An admitted TraceSpec stores a clean-run action sequence, replay-stable selectors, expected signals, and an optional parent; replay verification ignores its route hint.In the example, a child trace replays a parent-created board, renames a list with three actions, and verifies the new accessible name and absence of the old text.Source: §D.6 · pp. 28–29, Table 7 · p. 29

E Crafting Agent Prompts; E.1 Task Generator

  • Crafting uses a task generator to remove substantive feature logic and a critic to reject superficial masks; the resulting app must still boot while the verified behavior fails.The generator receives the app description, trace actions, expected signals, and any prior verification failure; solver-facing statements describe the feature and workflow without revealing verification outcomes or masked paths.Source: E Crafting Agent Prompts · p. 30, §E.1 · p. 30

E.1 Task Generator

  • Task descriptions avoid function names, code, and step-by-step fixes, with length scaled to feature complexity.Source: §E.1 · p. 30

E.2 Mask-Depth Critic

  • The mask-depth critic rejects deletions that leave substantive logic intact, and UI masks must remove both rendered markup and logic.It receives the feature description, success signals, solver-facing statement, and production-to-masked diff; a deep mask requires rewriting substantive logic.Source: §E.2 · p. 31

E.3 Cumulative Mask Merger

  • When atomic masks conflict on source lines, the cumulative merger applies nonconflicting masks first, then resolves each conflict and retries rejected submissions.Source: §E.3 · p. 31

F Partial- and Full-Application Reconstruction Instructions

  • Partial- and full-application tasks use a fixed system message and templated user message, with task instructions and tool-call variants inserted into designated slots.Source: F Partial- and Full-Application… · p. 33

F.1 Partial-Application Reconstruction

  • In partial reconstruction, agents inspect the live reference, repair only relevant application source, and replay the workflow to verify restored behavior.Evaluation replays the recorded workflow and checks whether the reference app’s observable behavior is restored completely and accurately.Source: §F.1 · pp. 33–34

F.2 Full-Application Reconstruction

  • Full reconstruction requires systematic reference exploration and matching observable controls, selectors, intermediate states, validation, navigation, data changes, and side effects.Agents implement in a blank scaffold and compare equivalent starting states and interaction sequences across the reference and current apps.Source: §F.2 · pp. 35–36
  • The full-application contract required an exported async handle(req, res, context), state in context.state, and faithful reproduction of observed routes, forms, transitions, downloads, and validation.The runtime owned the ports and hot-reloaded modules; implementations used context.readBody(), context.json(), and context.html() as needed.Source: §F.2 · p. 37

G.1 Model Pricing

  • Model list prices came from LiteLLM’s model_cost table under harness model identifiers, while trajectory costs summed uncached-input, cached-input, and output-token charges.Each token category was multiplied by its per-token price before summing; Table 8 lists input and output prices per 1M tokens.Source: §G.1 · p. 37, Table 8 · p. 37

G.2 Execution Time

  • One GPT-6 Astra high-reasoning run completed 300 tasks in approximately 557 minutes (9 hours 17 minutes) across four CPU servers.Trials ran up to eight concurrently per server, for a maximum of 32. Wall-clock time included setup, agent execution, verification, and API waiting, but excluded one-time provisioning; trials averaged 48.2 minutes, with a 43.1-minute median. This is a practical runtime estimate, not standalone inference speed.Source: §G.2 · p. 37

H.1 Results by Mask Scope

Mask scope distinguishes restoring behavior behind an unchanged interface from rebuilding both the interface and behavior. Astra scores higher on logic-only than logic-and-UI tasks under both binary and chain scoring.

H.1 Results by Mask Scope

  • Logic-only tasks retain the interface while masking behavior; logic-and-UI tasks also remove the feature’s UI.The latter setting requires reconstructing both the observed interface and its underlying behavior.Source: §H.1 · p. 38, Table 9 · p. 38
  • Astra’s binary success is 92.9% on logic-only tasks and 76.9% on logic-and-UI tasks, a 16.0-point gap.The suite contains 140 logic-only and 160 logic-and-UI tasks.Source: §H.1 · p. 38, Table 9 · p. 38
  • Astra’s chain score is 96.2% on logic-only tasks and 84.9% on logic-and-UI tasks, an 11.3-point gap.The chain score credits the fraction of behaviors recovered before the first failure in a cumulative task.Source: §H.1 · p. 38

H.2 Results Under the Chain Score

Chain scoring gives partial credit for recovered prefixes of cumulative tasks. Astra leads on cumulative repair and full-application reconstruction, while chain scores exceed binary scores in both settings.

H.2 Results Under the Chain Score

  • The chain score credits the fraction of behaviors recovered before the first failure, unlike binary scoring, which requires all targets to pass.Figure 19 plots mean chain score against trajectory cost and restoration depth across all 300 tasks; its depth-related decline is shallower than under binary scoring, but remains.Source: §H.2 · p. 38, Figure 19 · p. 38
  • On cumulative repair tasks, Astra leads with 81.2% binary and 88.2% chain score.Opus 5 scores 63.2% binary and 71.1% chain; Sol scores 54.4% binary and 63.4% chain.Source: §H.2 · p. 39
  • Astra’s mean full-application chain score is 64.58%, versus 47.42% for Opus 5 and 34.22% for Sol.The corresponding mean binary scores are 49.15%, 28.81%, and 21.07%, respectively.Source: §H.2 · p. 39, Table 11 · p. 39

H.3 Does Context Length Explain the Depth Effect?

Performance continues to decline with restoration depth even among runs with similar prompt lengths. The analyses indicate that longer contexts alone cannot explain the depth effect.

H.3 Does Context Length Explain the Depth Effect?

  • Mean binary score falls from 83.4% at depth 1 to 23.1% at depth 8, while median prompt size per step rises 2.3-fold, from 47.5k to 107.5k tokens.Most context growth occurs by depth 4; beyond that, context length changes little while performance continues to decline.Source: §H.3 · p. 39
  • Binary score drops by 58.4, 54.5, and 31.5 points from depth 1 to 8 in the 40–80k, 80–120k, and 120–200k context bands, respectively.Within each context band, scores show an overall downward trend as depth increases.Source: §H.3 · p. 39, Table 12 · p. 39
  • Within each depth, prompt size has only a modest negative correlation with binary score (−0.20 to −0.05); task-centered pooled correlation is weakly positive (+0.11).Together, the analyses are consistent with increasing difficulty from composing multiple dependent repairs, not longer contexts alone.Source: §H.3 · p. 40

H.4 Agent Interaction Statistics

The interaction statistics report counts of agent, observation, and edit/write steps by model and restoration depth. Full-application runs show similar average step counts for Astra and Sol, with fewer for Opus 5.

H.4 Agent Interaction Statistics

  • Table 13 reports minimum, mean, and maximum counts of agent, observation, and edit/write steps for each model on ProgramDistill-300.An agent step counts at most once even if it targets both applications.Source: Table 13 · p. 40, §H.4 · p. 40
  • Its lower panel groups trajectories by restoration depth and reports mean interaction counts pooled across all nine models.The upper panel gives per-model interaction statistics.Source: Table 13 · p. 40
  • Astra averages 719.4 agent steps per full-application run, compared with 627.5 for Opus 5 and 719.0 for Sol.Table 14 reports steps by application for the full-application runs.Source: §H.4 · p. 40, Table 14 · p. 40
Loading 2609.18805v1…