PaperlayerView exact revision on arXiv

Source-linked AI summary

Externalizing Requirement-to-Repair Artifacts as Observable Traces for LLM-Based Program Repair

Zewen Tao, Shin-nosuke Ishikawa

arXiv:2609.14913v1cs.AIcs.SE

TL;DR

Repository-level repair must connect issue requirements to code edits and post-edit evidence in records that can be inspected across stages. THEMIS externalizes selected transitions through semantic interpretation, requirement-code relations, Developer guidance and execution records, and post-edit audit; a 300-case audit and paired workflow comparison report retained-field coverage and resolution outcomes.

  • Problem

    Repository-level repair requires systems to interpret issue reports, localize relevant code, infer intended behavior, and produce minimal patches, while preserving inspectable transitions from requirements through edits to post-edit audit.

    Source: §1 · p. 1

  • Method

    THEMIS builds repair context from a requirement-code graph and retains selected graph-derived summaries, Developer rationale and patches, and post-edit diagnostics as an inspectable, workflow-linked trace.

    Source: §1 · p. 2, §3.3 · p. 3, §3.6 · p. 4

  • Results

    214/300 cases (71.3%) retained a complete selected-stage field chain, and the paired workflow comparison resolved 19/100 cases versus 9/100 for the direct condition.

    Source: §7 · p. 8, §6 · p. 8

  • Takeaways & Limitations

    The retained artifacts support systematic inspection of how repair targets and decisions persist or evolve across stages, rather than leaving those transitions implicit behind the final patch.

    Source: §7 · p. 8

    The paired workflows differ in Analyzer output, graph-based distillation and relations, Judge records, token budget, and interface, so their outcome comparison is workflow-level rather than isolating graph effects.

    Source: §8 · pp. 8–9

Abstract

from arXiv · show

Repository-level repair requires not only correct patches but also inspectable records that explain how issue requirements are translated into code changes and post-edit evidence. We contribute THEMIS, a stage-aware repair workflow that externalizes this requirement-to-repair process through semantic interpretation, a runtime requirement-code graph, graph-derived Developer guidance, retained repair rationale and patches, and post-edit audit records. A retrospective audit of 300 SWE-bench Lite cases demonstrates that these artifacts provide broad support for cross-stage inspection: a complete Developer rationale is available for 288 cases, and 214 cases (71.3%) retain a complete audited field set connecting the selected stages. The retained records further enable systematic measurement of cross-stage correspondence: target symbols recur in 62.6% of Developer rationales and in 62.8% of patches, rising to 75.8% when related symbols are included. In a paired 100-case comparison, the relational workflow resolves 19 cases versus 9 for the direct same-input condition; because the two conditions also differ in Analyzer output, graph-derived distillation, and Judge records, we report this as preliminary, workflow-level evidence rather than a causal effect of the graph component. Together, these results show that THEMIS makes otherwise implicit requirement-to-repair transitions inspectable, enabling systematic examination of how repair decisions persist, align, and evolve across stages.

1 Introduction

Repository-level repair must translate issue requirements into code edits and post-edit audit records while preserving inspectable links between stages. THEMIS externalizes that process, audits retained artifacts, and compares its relational workflow with a direct same-input condition.

Motivation

  • Repository-level repair must jointly interpret issue reports, localize relevant code, infer intended behavior, and produce minimal patches satisfying test oracles.Issues may combine observed and expected behavior, failing tests, examples, and implicit domain constraints.Source: §1 · p. 1

Research questions

  • The paper asks what stage information can be externalized, what cross-stage correspondences or disagreements appear, and what workflow-level outcome differences are observed.These questions are operationalized through retained-field availability, lexical comparisons, and a paired 100-case workflow comparison.Source: §1 · pp. 1–2

Contribution

  • THEMIS externalizes requirement-to-repair transitions through semantic interpretation, requirement-code relations, graph-derived guidance, retained rationale and patches, and post-edit diagnostics.The study audits selected serialized records in an augmented-input, single-revision configuration.Source: §1 · p. 2

Contributions

  • THEMIS contributes an audit of selected stage-aware fields and a characterization of lexical correspondences and disagreements among targets, rationale, and patches.The contributions report retained-field availability and continuity gaps, plus correspondence analyses and illustrative cases.Source: §1 · p. 2

RQ3

  • 19/100 versus 9/100 paired cases were officially resolved by the relational workflow and direct same-input condition, respectively.The comparison spans conditions with distinct Analyzer output, graph-derived distillation, and Judge records, so it is workflow-level evidence.Source: §1 · p. 2

2 Related Work

The related work situates THEMIS among automated repair, LLM-based repository repair, graph-based code representations, and software traceability and agent observability. It distinguishes its focus on issue-derived requirement-code relations and requirement-to-repair audit traces.

Automated program repair

  • Classical automated repair spans generate-and-validate search, search-based repair, test-suite studies [Le Goues et al., 2012], learned patch priors, and semantic or symbolic methods [Xuan et al., 2017].THEMIS follows constraint-guided repair, deriving constraints from issue text and representing them as requirement-code graph relations.Source: §2 · p. 2

LLM-based repository repair

Graphs and semantic guidance for code

  • Prior graph work represents code structure through code property graphs, neural code relations [Allamanis et al., 2018; Alon et al., 2019a,b; Guo et al., 2021], and other structural information.THEMIS adds issue-derived requirement nodes and violation or advisory edges, using the graph as code representation and repair-context state.Source: §2 · p. 2

Software traceability and agent observability

  • Traceability research recovers links between requirements and downstream artifacts, while agent-observability surveys call for schemas recording evidence, tool calls, and intermediate reasoning [Wang et al., 2026].THEMIS targets repository repair with an externalized requirement-to-repair trace.Source: §2 · p. 2

3 System Architecture

THEMIS separates semantic interpretation, explicit requirement-code relations, Developer execution, and post-edit audit into selected records and workflow-asserted links. Its reported architecture makes each stage inspectable across a single repair revision.

3 System Architecture

  • Four modules separate semantic hypotheses, structural relations, Developer execution, and post-edit evidence across one repair revision.The Analyzer and graph manager distinguish issue-level hypotheses from relation claims; the Developer records guidance-to-patch execution, and the rebuilt graph and Judge form a distinct audit stage.Source: §3 · p. 3

3.1 Advanced Code Analyzer

  • The Advanced Code Analyzer records issue-level semantic findings and recommendations as pre-edit priors, separate from structural graph claims.It receives issue text and selected files, and produces findings, confidence scores, strategy metadata, and usage information when available.Source: §3.1 · p. 3

3.2 Enhanced Graph Manager

  • The Enhanced Graph Manager makes selected requirement-target-constraint relations inspectable by mapping decomposed issue requirements to code nodes and adding dependency neighborhoods and violation edges.High-signal blocking mismatches are VIOLATES edges; lower-confidence or non-blocking signals are ADVISORY edges.Source: §3.2 · p. 3

3.3 Developer

  • The Developer receives selected requirements, Analyzer findings, graph-derived priorities, target symbols, plans, and local code ingredients as readable guidance for one bounded edit.Its rationale and patch preserve the transition from guidance to execution; edits use search/replace or symbol-rewrite forms and pass file, syntax, and substantive-change checks.Source: §3.3 · p. 3

3.4 Judge

  • After the single edit and graph rebuild, the Judge separates remaining blocking conflicts from advisory findings and creates a repair brief for post-edit audit.The brief includes a target symbol, related symbols, issue summary, expected behavior, and minimal-change hint.Source: §3.4 · pp. 3–4

3 System Architecture

  • Figure 1 distinguishes runtime-only graph states from selected persisted artifacts and places Judge outputs after the edit as post-edit audit records.Solid arrows show reported data flow; persisted artifacts and dashed links denote recorded, workflow-asserted relations.Source: Figure 1 · p. 4

3.5 First-Pass Repair Context

  • The first Developer pass combines raw requirements and files, soft Analyzer clues, and graph-derived violations, target symbols, repair plans, and local ingredients.Judge conflict reports and repair briefs are generated only after the edit and graph rebuild.Source: §3.5 · p. 4

3.6 Inspectable Trace Artifacts

  • Selected serialization retains requirement and target summaries, Developer rationale, patch, and post-edit diagnostics, while the full runtime graph remains runtime material.The observable trace schema is τ = (r, t, d, p, q): requirement summary, target summary, Developer rationale, patch record, and post-edit diagnostic; inspectability means readable stored artifacts, while traceability means asserted links.Source: §3.6 · p. 4

4 Experimental Setup

The experiments audit THEMIS on SWE-bench Lite using an augmented-input, single-revision configuration and retained-record measures. A paired direct same-input comparison holds several inputs constant but differs in workflow components and resources.

Benchmark and task

  • THEMIS is evaluated on 300 SWE-bench Lite Python repository issues, consuming issue text and selected source files and producing patches for the official harness.Instances provide a base commit, issue text, optional hints, and tests.Source: §4 · p. 4

Run configuration

  • The main run uses gpt-5.1-codex-mini for one Developer revision, followed by graph rebuild and post-edit Judge audit.The repair brief and Judge diagnostics are post-edit records in the reported single-revision run.Source: §4 · p. 5, Table 1 · p. 5

Input condition

  • The reported condition uses FAIL_TO_PASS identifiers in file selection and requirement construction, so its evidence applies to augmented-input SWE-bench rather than issue-only input.The requirement string combines the problem statement, available hints, FAIL_TO_PASS identifiers, and any manually supplied semantic contracts.Source: §4 · p. 5

Paired comparison

  • The paired 100-case comparison matches instance IDs, model, seed, revision count, requirement builder, selected-file strategy, and actual files, but the workflows differ in Analyzer output, graph-derived distillation, Judge records, token budgets, and interfaces.Comparison lineage is reconstructed retrospectively from experiment metadata and baseline declarations.Source: §4 · p. 5

Audit measures

  • The audit measures retained-record availability, surface overlap, and quality flags, with rationale and patch cohorts defined by required non-empty fields.The matching rule casefolds text, replaces :: with ., and checks qualified or leaf symbol forms as substrings.Source: §4 · pp. 5–6

5 Results

The audit reports retained-field coverage and lexical correspondence, with illustrative matching and mismatching cases, alongside single-run outcomes, condition-stratified associations, and a paired workflow comparison.

5.1 Availability of Stage-Specific Repair Records

  • 214/300 cases (71.3%) retain a complete audited field chain across the selected stages.Completeness requires all three retained field groups: conflict history, Developer rationale, and repair-brief core.Source: §5.1 · p. 6

5.2 Surface Correspondence and Record-Quality Flags

  • 134/214 rationales and 130/207 patches contain recorded targets; related-symbol matching raises patch overlap to 157/207.Within 205 joint cases, both rationale and patch match in 110; 18 are rationale-only, 19 patch-only, and 58 have zero overlap. Across repair-brief-core cases, 131/222 briefs are low confidence and 138/222 have generic issue summaries; resolved counts across the four overlap groups are 29, 8, 2, and 8, respectively.Source: Table 3 · p. 6, §5.2 · p. 6

5.3 Two Illustrative Stage-Specific Record Cases

  • Resolved django-11620 is a correspondence case: its repair brief identifies REQ-029 and targets resolve, and its patch adds Http404 alongside Resolver404; unresolved django-16139 has a target mismatch.For django-16139, the graph repair brief reports confidence 0.2, and the official harness outcome is unresolved.Source: §5.3 · pp. 6–7

5.5 Descriptive Outcome Context by Recorded Condition

  • Official resolution is 22.0% versus 12.8% for complete/incomplete chains, 27.6% versus 12.5% for rationale-target presence/absence, and 23.8% versus 22.1% for patch-target presence/absence.These are correlational strata and may be confounded by case difficulty.Source: Table 4 · p. 7, §5.5 · p. 7

5.4 Patch Production and Official Outcomes

  • 279/300 cases produce non-empty patches, while 58/300 resolve officially; 58/279 non-empty-patch cases are resolved.These are descriptive outcomes for the 300-instance augmented-input, single-revision run; patch production and benchmark-passing outcomes are distinct measures.Source: §5.4 · p. 7

5.6 Paired Same-Input Workflow Comparison

  • 19/100 versus 9/100 cases resolve, and 89/100 versus 43/100 yield non-empty patches for THEMIS versus the direct condition.The paired cells are 9 resolved in both conditions, 10 THEMIS-only, 0 direct-only, and 81 unresolved in both. The conditions also differ in Analyzer output, graph-derived distillation and relations, Judge records, interfaces, and token budget, so this is preliminary workflow-level evidence.Source: Table 5 · p. 7, §5.6 · p. 7

6 Discussion

The discussion reports how retained fields support joint stage inspection, make correspondence and disagreement observable, and permit a bounded comparison of two repair workflows.

RQ1

  • 214/300 cases jointly retain all three field groups, while complete Developer rationales are available for 288/300 and repair-brief core fields for 222/300.The remaining 86 cases identify continuity gaps within the selected serialization.Source: §6 · p. 7

RQ2

  • 134/214 recorded targets recur in Developer rationales and 130/207 in patches; 95/205 joint-cohort cases fall into rationale-only, patch-only, or zero-overlap cells.Among 222 repair-brief cores, 131 are low confidence and 138 have generic issue summaries; these are record-surface observations under a lexical proxy, not validated quality measures.Source: §6 · p. 8

RQ3

  • 19/100 paired cases resolve under THEMIS versus 9/100 under the direct condition, a workflow-level comparison across conditions that differ in other components.The discordant outcomes are 10 THEMIS-only and 0 direct-only resolutions; the comparison is preliminary and does not isolate a graph effect.Source: §6 · p. 8

7 Conclusion

The conclusion presents THEMIS as a workflow for making requirement-to-repair transitions inspectable and reports broad artifact availability, measurable cross-stage correspondence, and preliminary workflow-level outcomes.

Contribution

  • THEMIS externalizes repair stages through semantic interpretation, explicit requirement-code relations, graph-derived Developer guidance, retained rationale and patch records, and post-edit diagnostics.The paper concludes that observable traces support systematic analysis of how requirements are interpreted, translated into edits, and reflected in post-edit evidence.Source: §7 · p. 8

Results

  • 214/300 SWE-bench Lite cases retain a complete field chain across selected stages, and complete Developer rationales are retained for 288 cases.The results support joint inspection of the requirement-to-repair process at benchmark scale.Source: §7 · p. 8
  • Target symbols recur in 62.6% of Developer rationales and 62.8% of patches, rising to 75.8% when related symbols are included.The retained artifacts make cross-stage correspondence measurable and let researchers examine how repair targets and decisions persist or evolve.Source: §7 · p. 8
  • 19 paired cases resolve under THEMIS versus 9 under the direct same-input condition, providing preliminary workflow-level evidence rather than isolating a graph effect.The conditions also differ in Analyzer output, distillation, and audit records.Source: §7 · p. 8

8 Limitations

The limitations concern workflow comparability, what the selected records and lexical proxy can establish, and the evaluation’s scope and unvalidated human utility.

Workflow and comparison boundaries

  • The paired workflows differ in Analyzer output, graph-based distillation and relations, Judge records, token budget, and interface, so results are limited to workflow-level outcomes.Lineage is reconstructed from experiment metadata and baseline declarations; causal and general-superiority questions remain for future work.Source: §8 · p. 8

Record and proxy boundaries

  • The selected serialization omits the full runtime graph, complete requirements, and complete reasoning; requirement decomposition, mapping, and violation flagging are heuristic.Lexical matching measures surface correspondence, while semantic correctness, trace correctness, and target-to-patch causal validation remain future targets. The retrospective target is Judge-generated after editing, not an independent pre-edit reference.Source: §8 · p. 9

Utility, scope, and model dependence

  • Human utility remains unvalidated, and evaluation covers Python repositories in SWE-bench Lite with one model configuration, gpt-5.1-codex-mini.Validation across languages, benchmarks, models, model scales, and repair settings is left for future work.Source: §8 · p. 9
Loading 2609.14913v1…