PaperlayerView exact revision on arXiv

Source-linked AI summary

Chronos Enables Code Agents to Reason over Software Evolution

Xin Yin, Yiang Zhang, Zhiyuan Peng, Chao Ni, Zhe Cui, Xiaohua Xin

arXiv:2610.11578v1cs.SEcs.CL

TL;DR

Relevant repository experience can span related pull requests that emphasize different concerns, making it difficult to retrieve through query similarity alone. Chronos organizes PR history into structured cards connected by typed relations and uses that memory for patch generation and selection. Across the evaluated workflows, it improves resolution and retrieves more useful cards than flat semantic retrieval.

  • Problem

    Relevant experience can span related changes whose descriptions emphasize different concerns, so ranking records only by query similarity leaves their relationships outside the retrieval decision.

    Source: §1 · p. 2

  • Method

    Chronos structures PR history as experience cards connected by typed relations, then uses semantic entry search and graph expansion to guide candidate generation and final patch selection.

    Source: §1 · p. 2

  • Results

    Across six SWE-Bench Verified backbones, Chronos raises mean SWE-Agent resolution from 69.2% to 72.9%, with gains also reported on SWE-Bench Pro and FEA-Bench Lite.

    Source: §7 · p. 18

  • Takeaways & Limitations

    The findings support PR relations as a useful retrieval signal and show that repository memory can support patch generation and selection in the evaluated workflows.

    Source: §7 · p. 18

    Evaluation scope is uneven: six backbones are tested on SWE-Bench Verified, but only MiniMax M2.5 on the transfer benchmarks, and most configurations use single runs.

    Source: §5.4 · p. 16

Abstract

from arXiv · show

Historical pull requests record the design decisions, compatibility constraints, and implementation patterns behind a codebase's current state. Experience relevant to a new task can span related changes whose descriptions emphasize different concerns. We introduce Chronos, a test-time framework that makes this connected history available to large language model (LLM)-based code agents. Chronos distills merged pull requests into structured experience cards and connects them through a typed graph of code-level, developer-intent, and organizational relations. Semantic search identifies entry cards, and weighted multi-hop expansion retrieves connected changes for selective reading. The same memory guides candidate generation and patch selection: a patch-focused change agent and a validation-strategy agent each develop a patch, and an evolution steward consults history to select between them. On SWE-Bench Verified, the full workflow improves SWE-Agent across all six evaluated LLM backbones, raising the mean resolution rate from 69.2% to 72.9% and reaching 79.8% with MiniMax M2.5. With the same backbone, it raises resolution rates from 48.3% to 51.7% on SWE-Bench Pro and from 41.0% to 43.5% on FEA-Bench Lite. Both experience-guided single-agent variants also outperform the base agent. In a human evaluation on 100 tasks with ten cards retrieved per task, graph-grounded retrieval increases the mean number of useful cards from 1.24 to 2.87 over flat semantic retrieval. These results demonstrate the value of PR relations for retrieving useful repository experience and of the evaluated workflows for applying that experience during patch generation and selection.

1 Introduction

Historical PRs preserve the reasoning behind current code, but useful experience can span related changes that semantic similarity alone may not surface. Chronos organizes that history into connected experience cards and uses it to guide patch generation and selection, improving resolution and retrieval results in the reported evaluations.

1 Introduction

  • Historical PRs link code changes to motivating problems, review discussions, and implementation choices, providing reasoning that may otherwise need to be reconstructed from current code.The context can explain compatibility branches, corrected assumptions, and component extensions.Source: §1 · p. 1
  • Relevant experience may span an initial repair, later extension, and backport whose descriptions emphasize different concerns, leaving query-similarity ranking unable to use their relationships.This makes following relations from semantic entry points and prioritizing connected evidence in a compact shortlist a central retrieval challenge.Source: §1 · p. 2
  • Chronos turns PR history into four-field experience cards and connects them with typed relations spanning code continuity, developer intent, and organizational context.Semantic search finds entry cards, weighted graph expansion retrieves related PRs, and agents selectively read the resulting experience.Source: §1 · pp. 2–3, Figure 2 · p. 3
  • A patch-focused change agent and a validation-strategy agent each produce a candidate, while an evolution steward consults memory and source code to select the final patch.The retrieval service integrates with existing agent tools without model fine-tuning.Source: §1 · pp. 2–3
  • 69.2% to 72.9% mean SWE-Agent resolution on SWE-Bench Verified across six backbones, with gains also reported on SWE-Bench Pro and FEA-Bench Lite.The full workflow reaches 79.8% with MiniMax M2.5; graph-grounded retrieval also raises useful cards among ten returned from 1.24 to 2.87 over flat semantic retrieval.Source: Figure 1 · p. 2, §1 · p. 3

2 Approach

Chronos turns merged pull-request history into structured experience cards connected by typed development relations, then retrieves relevant history to guide patch generation and selection. Its workflow combines semantic entry search, bounded graph expansion, and history-informed review of candidate patches.

2 Approach

  • Chronos uses cards describing historical changes and a typed graph of code, intent, and organizational relations to make connected repository experience available at task time.Agents search for entry cards, expand through related pull requests, and selectively read the resulting experience.Source: §2 · p. 4, §2.1 · p. 4, §2.1.2 · p. 5

2.1.1 Software-Evolution Experience Distillation.

  • Each experience card consolidates a pull request’s reasoning into Summary, Signals, Motivation, and Solution fields that support both retrieval and implementation reasoning.Summary anchors embeddings; Signals provide retrieval cues, while Motivation and Solution explain why the change was needed and how to adapt it.Source: §2.1.1 · p. 4

2.1.2 Software-Evolution Graph Construction.

  • The evolution graph links merged pull requests through observable code, developer-intent, and organizational relations, assigning stronger traversal weights to code continuity than to broader organizational associations.The three tiers cover code evolution, explicit intent or workflow continuity, and noisier project-process metadata; the weights are hand-specified priors.Source: §2.1.2 · pp. 5–6

2.2 Retrieving and Applying Software-Evolution Experience

  • Task-time retrieval combines semantic search with typed-relation expansion, returning connected historical changes beyond the initial semantic shortlist for agents to inspect selectively.The two-step graph_search and graph_read workflow exposes lightweight candidates first and full cards when relevant.Source: §2.2 · p. 6, §2.2.1 · p. 7

2.2.1 Agentic Search over Graph-Grounded Experience.

  • Expansion scores reachable pull requests using entry relevance, a path-length penalty, and accumulated relation weights, retaining the best score across explored paths.The shortlist reserves positions for semantic entries and adds up to K graph-expanded pull requests; hop and frontier limits bound exploration.Source: §2.2.1 · pp. 7, 9

2.2.2 Multi-Agent Collaboration over Software Evolution.

  • The patch-focused agent searches around design intent and invariants, while the validation-strategy agent emphasizes behavioral mismatches, APIs, and regression risks.Both assess whether prior experience applies and whether its implementation complexity fits the task.Source: §2.2.2 · p. 9
  • An evolution steward independently consults repository history and current code to select between distinct, non-empty candidate patches based on logical correctness.It compares root-cause treatment, historical consistency, and edge cases, selecting an existing candidate rather than generating another patch.Source: §2.2.2 · pp. 9–10

3 Experimental Setup

The experiments evaluate Chronos with two agent scaffolds across three benchmark settings and assess whether graph-grounded retrieval supplies useful task-specific experience.

3.1 Agents and Retrieval Configuration

  • On SWE-Bench Verified, SWE-Agent runs used six backbones, while transfer benchmarks used MiniMax M2.5; runs allowed 250 steps, a $1.5 budget, and up to ten retrieved cards.The graph server used K=5 semantic entry points and H=3 expansion hops; Chronos mini tested integration with a second scaffold.Source: §3.1 · p. 10

3.2 Benchmarks and Task Coverage

  • SWE-Bench Verified covers 500 human-filtered issue tasks; SWE-Bench Pro’s public setting has 731 tasks from 11 repositories, and FEA-Bench Lite has 200 tasks from 48 repositories.Pro tests longer-horizon changes; FEA-Bench Lite tests feature implementation integrated with existing code.Source: §3.2 · p. 11

3.3 Human Evaluation of Retrieved Experience

  • The human study retrieved ten cards per task for 100 SWE-Bench Verified instances, comparing random sampling, cosine-similarity ranking, and Chronos graph-grounded search.Two experienced annotators judged cards using all four card fields and a task-conditioned rubric for transferable, actionable guidance.Source: §3.3 · p. 11
  • Annotators counted useful cards per task and the fraction of tasks receiving at least one useful card; edge-type ablations measured changes in the former.Useful guidance could include an implementation pattern, failure mode, compatibility constraint, design rationale, or validation strategy.Source: §3.3 · pp. 11–12

4 Evaluation

Chronos improves SWE-Bench Verified results across six evaluated backbones and also improves results on two transfer benchmarks. The evaluation examines repeated-run stability, workflow components, retrieval quality, and the contribution of graph relations.

4.1 Main Results

  • The full Chronos workflow raises mean SWE-Bench Verified resolution from 69.2% to 72.9% (+3.7pp), improving SWE-Agent on all six evaluated backbones.Gains range from +2.8pp to +3.6pp for five backbones, with Qwen3-Max gaining +6.2pp.Source: §4.1 · p. 12, Table 4 · p. 12
  • With MiniMax M2.5, Chronos raises resolution from 48.3% to 51.7% on SWE-Bench Pro and from 41.0% to 43.5% on FEA-Bench Lite.The graph-construction rules and default weight formulas remain unchanged across benchmarks.Source: §4.1 · p. 12, Table 5 · p. 13

4.2 Run-to-Run Stability

  • Across five runs on all 500 SWE-Bench Verified instances, the full configuration averages 80.0% resolution, with a 0.37pp sample standard deviation.The 95% Student’s t interval for the configuration mean is [79.5%, 80.5%]; the lowest run remains 3.2pp above the 76.4% SWE-Agent reference.Source: §4.2 · p. 13

4.3 Ablation

  • Chronos gains +3.4pp over both SWE-Agent and mini-SWE-Agent, reaching 79.8% and 79.2%, respectively.The same memory interface supports both scaffolds while preserving graph-construction and retrieval procedures.Source: §4.3 · p. 13
  • The full workflow reaches 79.8%, exceeding the 78.4% change-agent and 77.8% validation-agent single-agent variants and the 76.4% baseline.The single-agent variants each produce one patch without steward selection; the full workflow’s result is 1.4pp above the change-agent variant and 2.0pp above the validation-agent variant.Source: §4.3 · p. 13, Table 6 · p. 14
  • Graph-grounded retrieval reaches 79.8% resolution versus 79.0% with flat semantic search, a +0.8pp difference.This ablation measures retrieval choice within the full coding workflow.Source: §4.3 · p. 13

4.4 Retrieval Quality and Relation Contributions

  • At a ten-card depth, graph-grounded retrieval raises Useful from 1.24 to 2.87 and Hit from 68% to 89% over Embedding.Random retrieval yields 0.18 Useful and 14% Hit; Useful is the average number of useful cards among the top ten, and Hit is the fraction of tasks with at least one useful card.Source: §4.4 · pp. 13–14, Table 7 · p. 14
  • Removing precedes and extends causes the largest reported Useful drops, 0.60 and 0.44, while organizational relations drop by 0.06–0.18.These are removal losses under the tested configuration, not additive shares; they measure retrieval value in the presence of the remaining graph structure, not relation coverage.Source: §4.4 · pp. 14–15, Figure 3 · p. 15

5 Discussion and Threats to Validity

Chronos separates reusable repository memory from the agents that apply it, using linked PR experience to inform retrieval and maintenance decisions. The discussion also identifies computational costs and evidence-quality and evaluation boundaries.

5.1 Repository Experience as a Reusable Memory Layer

  • Chronos uses the same repository memory interfaces across agent scaffolds and benchmarks, while keeping historical content specific to each target repository.The card schema, graph-construction rules, and retrieval procedure are shared; the reported gains span issue resolution and feature implementation.Source: §5.1 · p. 15

5.2 What Relations Add to Experience Cards

  • Relation-removal results favor code continuity, with developer-intent and organizational links adding retrieval value under a fixed, hand-specified prior.Sensitivity to the numerical weight constants remains unmeasured; agents use card motivations and solutions to assess whether historical reasoning applies to current code.Source: §5.2 · pp. 15–16

5.3 Computational Structure

  • Repository-memory costs span preprocessing, online graph retrieval, shortlist creation, and agent-context use, while the resulting memory can serve multiple tasks.The full workflow runs two candidate generators and may invoke a steward; per-run limits mean total task expenditure includes those runs and retrieval.Source: §5.3 · p. 16

5.4 Evidence Scope and Experience Quality

  • LLM distillation can omit context or introduce inaccuracies, and the human study measures task-conditioned usefulness rather than source-level factual accuracy.The two annotators had κ=0.86 before adjudication, but their shared research context may introduce evaluator bias; agents are prompted to inspect current code before using retrieved experience.Source: §5.4 · p. 16
  • Evaluation covers six backbones on SWE-Bench Verified but only MiniMax M2.5 on the transfer benchmarks, with repeated-run statistics limited to the full MiniMax configuration.Single-agent gains combine graph retrieval with role-specific prompting, while the full workflow’s additional gain reflects removing a strategy, second candidate, and steward selection together; retrieval is bounded by recorded PR timestamps.Source: §5.4 · p. 16

5.5 Implications for Software Maintenance

  • A historical development decision pairs an implementation with its motivating conditions, helping agents assess what transfers to current code and which constraints to preserve.Relations connect decisions to subsequent work on the same files or explicitly linked work, making repository history a record of evolving requirements and implementation choices.Source: §5.5 · pp. 16–17

6 Related Work

Related work spans repository-level software agents and benchmarks, retrieval-augmented systems, code graphs, and methods for reusing software-engineering experience. Chronos differs by retrieving historical PR experience through typed relations between changes.

6.1 Software Engineering Agents and Benchmarks

  • Software-engineering agents support repository exploration, code modification, and varied repair workflows, while benchmarks extend evaluation across harder tasks, languages, feature implementation, migration, and refactoring.Chronos complements these agents with historical PR experience for candidate generation and selection.Source: §6.1 · p. 17

6.2 Knowledge Reuse for Software Engineering Agents

  • Code-graph systems model program structure, whereas Chronos graphs historical PR events to retrieve prior-change experience through task-time tools.Other history-reuse systems draw on agent trajectories, human repairs, blame context, commits, linked issues, or procedural knowledge.Source: §6.2 · pp. 17–18
  • Unlike MemGovern’s embedding-ranked card search, Chronos expands from semantic entry points over typed PR relations supported by files, references, follow-up work, and metadata.Repository Memory retrieves commits and linked issues alongside code navigation; Chronos instead graphs historical PR events, with context available during candidate generation and history-informed selection.Source: §6.2 · p. 18

7 Conclusion

Chronos turns PR history into graph-structured experience for code agents, using semantic entry points and typed relations to retrieve connected changes. Its evaluated workflows improve resolution results across benchmarks and increase the number of useful cards retrieved.

7 Conclusion

  • Chronos uses semantic search and typed PR relations to make connected historical experience available for selective reading.Structured cards supply rationale and implementation logic for candidate generation and patch selection.Source: §7 · p. 18
  • Across six backbones on SWE-Bench Verified, the full workflow raises the mean SWE-Agent resolution rate from 69.2% to 72.9% and reaches 79.8% in the strongest configuration.With MiniMax M2.5, it raises resolution rates from 48.3% to 51.7% on SWE-Bench Pro and from 41.0% to 43.5% on FEA-Bench Lite; single-agent variants also improve on the base agent.Source: §7 · p. 18
  • With ten cards retrieved per task, graph-grounded retrieval raises the average number of human-judged useful cards from 1.24 to 2.87 over flat semantic retrieval.The authors conclude that PR relations provide a useful retrieval signal and that the evaluated workflows apply repository memory during patch generation and selection.Source: §7 · p. 18
Loading 2610.11578v1…