Source-linked AI summary
RepoTrace: Improving Requirement Tracing in Multi-Round Agent Interactions
Huanyu Liu, Yezheng Cheng, Jinliang You, Tianxiao Huang, Linyu Li, Zheng Fang, Peng Cao, Ge Li
TL;DR
In multi-round software tasks, agents degrade and often misidentify requirement targets, complicating maintenance of existing software. RepoTrace combines repository structure with requirement semantics to provide agents requirement-code correspondence. It improves per-round reward across models and harnesses and reduces regression errors.
Problem
In multi-round software tasks, agents degrade and struggle to distinguish primary requirement targets from related code, making iterative maintenance of existing software difficult.
Source: §1 · pp. 1–2Method
Results
RepoTrace improves mean per-round reward across models and harnesses, including +19.99 on GLM-5.2 and +10.22 on Claude Code.
Source: §1 · p. 2Takeaways & Limitations
RepoTrace helps agents focus on relevant code and preserve existing functionality across interactions.
Source: §7 · p. 10Supporting a wider range of harnesses while maintaining effective integration remains an open problem.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Code agents have become increasingly capable of performing software engineering tasks through interaction with software environments. However, agents tend to degrade as the number of interaction rounds increases, making it difficult to maintain existing software during iterative modification. We analyze 2,656 agent trajectories in EvoCode-Bench and identify requirement focus misalignment as a recurring failure pattern, where agents struggle to distinguish the primary modification targets of a requirement from unrelated code. We propose Repotrace a requirement-driven repository understanding harness that combines repository structure with requirement semantics to construct a requirement-code correspondence and injects it into the agent context. This correspondence enables code agents to follow the repository architecture, trace requirements across multi-round interactions, and identify the core code elements corresponding to the current requirement. Experiments show that Repotrace improves the mean per-round reward (MPR) across different models (e.g., +19.99 on GLM-5.2) and agent harnesses (e.g., +10.22 on Claude Code). Across rounds, Repotrace increases the average per-round reward from 0.163 to 0.371 on Terminus-2 with DeepSeek-V4-Pro, while reducing average regression errors by 56.78%. We release the source code and agents' trajectories to support future research.
1 INTRODUCTION
RepoTrace addresses requirement focus misalignment, a recurring failure in multi-round agent work, by linking requirements to relevant repository code. Experiments report improved per-round rewards across models and harnesses, alongside fewer regression errors.
1 INTRODUCTION
- Prior studies report worsening multi-round performance: EvoCode-Bench test pass rates fall from 46.7% in early rounds to 7.7% in later rounds, and 40%–73% of tasks lose previously satisfied behaviors.Source: §1 · pp. 1–2
- Analysis of 2,656 EvoCode-Bench trajectories identifies requirement focus misalignment: agents struggle to distinguish primary modification targets from related code that need not change.Across failure rounds, 55.16% of modifications occur outside the human-oracle target files.Source: §1 · p. 2
- RepoTrace combines repository structure with requirement semantics to construct and inject a requirement-code correspondence that helps agents trace architecture and identify relevant code across rounds.Source: §1 · p. 2
- RepoTrace improves mean per-round reward across models and harnesses, including +19.99 on GLM-5.2 and +10.22 on Claude Code.On Terminus-2 with DeepSeek-V4-Pro, average per-round reward rises from 0.163 to 0.371 while average regression errors fall by 56.78%.Source: §1 · p. 2
2 MOTIVATING EXAMPLES
Real-world examples show agents modifying unrelated code or omitting necessary synchronized updates. These failures motivate helping agents identify the code elements that correspond to a requirement.
2 MOTIVATING EXAMPLES
- In the rollback example, the agent rewrites a shared dispatcher instead of the target logic and omits validate(), breaking validation functionality.Source: §2 · p. 2, Figure 1 · p. 3
- Figure 1 contrasts two failure forms: an unrelated dispatcher rewrite and an output change that omits the code collecting its values.Source: §2 · pp. 2–3, Figure 1 · p. 3
- In the anomaly-evidence example, the agent updates output code but not detector.go, leaving evidence.window values empty.Source: §2 · p. 3
- Across failed rounds after Round 1, 55.16% of source-file modifications fall outside the human-oracle modification target files.Source: §2 · p. 3
3 RELATED WORK
Prior work studies agent performance in multi-round tasks, requirements tracing, and repository-level code understanding. The paper identifies gaps in maintaining requirement-artifact consistency and repository structure across iterative development.
3.1 CODE AGENTS IN MULTI-ROUND INTERACTIONS
- Studies report declining agent performance in multi-round tasks, including roughly 39% lower average success for top-tier models and 20%–27% fewer correct-and-secure outputs across 32 models.SlopCodeBench reports that its best agent passes 14.8% of checkpoints and that 77% of trajectories exhibit structural erosion.Source: §3.1 · p. 3
3.2 REQUIREMENTS TRACING AND REPOSITORY-LEVEL CODE UNDERSTANDING
- Requirements-tracing methods link artifacts or recover trace links, but do not address maintaining requirement-artifact consistency as requirements evolve over multiple rounds.Repository-understanding methods structure code or support graph navigation, but primarily target static repositories or one-shot generation.Source: §3.2 · p. 4
4 REPOTRACE
RepoTrace combines repository structure with requirement semantics to construct requirement-code correspondence for agents in multi-round interactions.
4 REPOTRACE
- RepoTrace combines repository structure with requirement semantics before modification, giving agents structural dependencies and code elements associated with prior requirements.Source: §4 · p. 4
4.1 REPOSITORY STRUCTURE ANALYSIS
- The structural representation links source files through imports and candidate calls, indexes symbols to defining files, and uses PageRank and fan-in to expose structural importance and dependencies.PageRank prioritizes files in prompts and dependent-file examples; fan-in selects multiply referenced files and ranks the existing-file overview, without implying those files need modification.Source: §4.1 · pp. 4–5
4.2 REQUIREMENT SEMANTICS EXTRACTION
- Requirement semantics extraction uses the current requirement, repository structure, structural changes, and prior requirement records to identify candidate target files and symbols, behavior constraints, and potentially affected prior elements.An LLM extracts the record from ranked files, symbol associations, and recent requirement history; a keyword fallback matches paths and identifiers when LLM extraction is unavailable or fails.Source: §4.2 · p. 5
4.3 AGENT CONTEXT AUGMENTATION ACROSS ROUNDS
- The context message combines selected code relations, file overviews, potentially affected prior elements, and verification guidance with the original requirement; RepoTrace retains graph snapshots and requirement records for later rounds.The retained records describe prior analyses, not verified completion states.Source: §4.3 · p. 6
5 EXPERIMENTS
Experiments evaluate RepoTrace on multi-round EvoCode-Bench tasks across models and agent harnesses. The reported results show gains across harnesses, persistent benefits across rounds, and a case where requirement-code correspondence helps preserve existing functionality.
5 EXPERIMENTS
- The experiments assess whether RepoTrace improves code-agent performance on multi-round software tasks using multiple agents.The section introduces benchmarks, baselines, models, and evaluation metrics before presenting results and analysis.Source: §5 · p. 6
5.1 EXPERIMENTAL SETUP
- EvoCode-Bench covers 26 stateful tasks and 227 rounds across six technical domains, with each round adding a requirement and cumulative tests checking current and prior requirements.A round earns reward only when all test cases pass; the experiments use four models with Terminus-2 and Claude Code, using default API configurations.Source: §5.1 · p. 6
5.2 RQ1: TO WHAT EXTENT DOES REPOTRACE IMPROVE PERFORMANCE ACROSS DIFFERENT MODELS?
- RQ1 evaluates four models with and without RepoTrace on EvoCode-Bench, and Table 1 is identified as reporting main results and token usage.The supplied passages give no Table 1 cell values or model-specific outcome for this comparison.Source: §5.2 · p. 7, Table 1 · p. 7
5.3 RQ2: TO WHAT EXTENT DOES REPOTRACE IMPROVE PERFORMANCE ACROSS DIFFERENT HARNESSES?
- On Claude Code, RepoTrace raises MPR from 44.65 to 54.87 (+10.22 points); on Terminus-2, it raises MPR from 14.14 to 36.39 (about 2.57×).Input tokens decrease by 27.61% on Terminus-2 and 19.80% on Claude Code, while output tokens rise only slightly.Source: Table 2 · p. 7, §5.3 · pp. 7–8
5.4 RQ3: HOW DOES REPOTRACE’S IMPROVEMENT VARY ACROSS ROUNDS?
- Across rounds 2–11, RepoTrace raises earlier-round test pass rate from 57.06 to 89.46 (+32.40 points, a 56.78% relative improvement).On Terminus-2, Vanilla reward falls from 0.38 in round 1 to 0.08–0.15 in rounds 2–7, while RepoTrace leads in all 10 rounds.Source: §5.4 · p. 8, Figure 3 · p. 8
5.5 RQ4: HOW DOES REPOTRACE IMPROVE REQUIREMENT TRACING?
- In JobForge, RepoTrace adds a separate critical-command branch while preserving the schedule dispatch, whereas Vanilla breaks the existing schedule functionality.The comparison uses the same requirement chain, model, and execution environment, and both trajectories were independently replayed.Source: Figure 5 · p. 9, §5.5 · p. 9
6 DISCUSSION
RepoTrace can extend to different harnesses through its requirement-code correspondence, which humans and agents can modify. Deeper hook-based integration may offer more control but reduce applicability, leaving effective integration across a wider range of harnesses open.
Applicability
- The correspondence can be modified by humans or agents to support different harnesses without directly changing their execution process.Hooks may offer more control but reduce the range of applicable harnesses; balancing broad applicability with effective integration remains open.Source: §6 · p. 9
Requirements in agentic software engineering
- Requirements provide persistent objectives for agents to retain earlier goals while making new changes as software evolves.The discussion connects this role to agents potentially focusing on the latest requirement and missing earlier ones as interaction horizons grow.Source: §6 · p. 9
7 CONCLUSION AND FUTURE WORK
RepoTrace is presented as a harness for improving requirement tracing and helping agents preserve existing functionality across multi-round interactions; future work will explore structured requirement-change management and RepoTrace’s role in RSI.
7 CONCLUSION AND FUTURE WORK
- Future work will explore structured representations for managing requirement changes and study how RepoTrace can help promote RSI.Source: §7 · p. 10
AI USE STATEMENT
The statement describes how LLMs were used in preparing the manuscript, while the example section illustrates RepoTrace’s three steps using JobForge Round 7.
AI USE STATEMENT
- LLMs assisted with writing and coding to improve clarity, grammar, and phrasing; at least two authors checked all LLM-edited content.The authors state that LLM use did not influence technical contributions or interpretation of experimental findings.Source: AI USE STATEMENT · p. 10
A SPECIFIC EXAMPLES OF REPOTRACE
- JobForge Round 7 illustrates repository structure analysis, requirement semantics extraction, and agent-context augmentation across rounds.The requirement asks for a critical subcommand that reports the longest dependency chain by job count while preserving existing behavior.Source: A SPECIFIC EXAMPLES OF REPOTRACE · p. 14
A.1 REPOSITORY STRUCTURE ANALYSIS
The JobForge example connects a pre-modification repository snapshot to requirement extraction, which identifies candidate targets and behavior constraints for Round 7.
A.1 REPOSITORY STRUCTURE ANALYSIS
- Figure 6 lists 13 source files under cmd/jobforge/ and records schedule.go as added before Round 7, not as a proposed current change.The figure distinguishes the existing schedule.go file from candidate new file critical.go.Source: §A.1 · p. 14, Figure 6 · p. 14
A.2 REQUIREMENT SEMANTICS EXTRACTION
- Requirement extraction uses the current requirement, repository structure, structural changes, and prior records to produce candidate targets, constraints, affected prior elements, and a summary.Figure 7 organizes these inputs and requested fields in a reference prompt.Source: §A.2 · p. 14, Figure 7 · p. 15
- For Round 7, extraction names critical.go and main.go as candidate target files and requires existing subcommands to remain unchanged, while listing no affected prior elements.The preservation requirement appears as a behavior constraint rather than an explicit affected-prior-code entry.Source: §A.2 · p. 14
A.3 AGENT CONTEXT AUGMENTATION ACROSS ROUNDS
The Round 7 correspondence message links candidate targets to the repository and carries earlier requirement context and verification guidance into the agent’s input.
A.3 AGENT CONTEXT AUGMENTATION ACROSS ROUNDS
- The message distinguishes critical.go as a candidate new file from existing main.go, connecting the extracted targets to the repository structure.Figure 8 shows excerpts of the message, including prior requirements, repository structure, target mappings, and verification guidance.Source: §A.3 · p. 15, Figure 8 · p. 16
- Including the Round 6 scheduling requirement and its schedule.go and main.go targets reminds the agent to preserve scheduling behavior when adding critical.Verification guidance asks the agent to check that affected code builds and preserves existing behavior.Source: §A.3 · p. 15
- Prior requirement records preserve earlier analyses but do not certify that a requirement was completed or its tests passed.Source: §A.3 · p. 16
B SETTING FOR REQUIREMENT FOCUS MISALIGNMENT ANALYSIS
The analysis estimates disagreement between agents’ modified source paths and oracle modification scope across later failed trajectories. The supplementary HTML report presents JobForge repository changes by round alongside file and function-call diagrams.
B SETTING FOR REQUIREMENT FOCUS MISALIGNMENT ANALYSIS
- The analysis covers 2,656 trajectories across 26 multi-round tasks and 12 code models; each trajectory is one model–task round’s interaction trace.Source: B SETTING FOR REQUIREMENT FOCUS… · p. 16
- Modified source paths were extracted from agent mutation operations and oracle solution artifacts, with specified languages retained and tests, dependencies, build outputs, caches, temporary files, and common documentation excluded.Source: B SETTING FOR REQUIREMENT FOCUS… · p. 16
- A parsing-quality filter excluded trajectories with unresolved mutation operations or unavailable artifacts, leaving 2,512 trajectories with reliably extracted path sets.The parser resolved paths for 97.52% of detected mutation-related commands.Source: B SETTING FOR REQUIREMENT FOCUS… · p. 17
- 55.16% of agent-modified source-file instances fell outside the oracle modification scope under the reported analysis setting.This corresponds to 2,132 of 3,865 instances.Source: B SETTING FOR REQUIREMENT FOCUS… · p. 17
- The 55.16% figure is a corpus-level static estimate, not replay ground truth, and oracle scope is not assumed to be the only valid implementation.The analysis is path-based and cannot distinguish content-equivalent, whitespace-only, or comment-only writes.Source: B SETTING FOR REQUIREMENT FOCUS… · p. 17
C HTML REPORT OF REPOSITORY CHANGES ACROSS ROUNDS
- The supplementary HTML report organizes JobForge repository snapshots by round and displays candidate targets, extracted symbols, potentially affected prior elements, added files, and diagrams.Source: C HTML REPORT OF REPOSITORY CHANGES… · p. 17
- Across Rounds 6–8, the report shows the file count rising from 12 to 14 as schedule.go and then critical.go appear as added files after being candidate targets.These entries distinguish current-requirement targets from additions in a subsequent snapshot; the complete report covers nine rounds.Source: C HTML REPORT OF REPOSITORY CHANGES… · p. 17
- The function-call diagrams group functions by file, using solid arrows for cross-file calls and dashed arrows for within-file calls; the Round 7 reconstruction’s edge counts do not represent Appendix A.1’s saved file-relation sets.Figures 10 and 11 show local regions whose cross-file arrows may continue beyond the crops; Figure 12 shows the overall Round 7 structure and call graph.Source: Figure 9 · p. 18, C HTML REPORT OF REPOSITORY CHANGES… · p. 19, Figure 10 · p. 19