PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-Explore: Benchmarking How Coding Agents Explore Repositories

Shaoqiu Zhang, Yuhang Wang, Jialiang Liang, Yuling Shi, Wenhao Zeng, Maoquan Wang, Shilin He, Ningyuan Xu, Siyu Ye, Kai Cai, Xiaodong Gu

arXiv:2606.07297v1cs.SEcs.CL

TL;DR

Existing coding benchmarks reduce repository-level work to pass/fail outcomes, leaving exploration and line-level evidence retrieval under-measured. SWE-Explore evaluates ranked code-region selection against trajectory-derived ground truth and finds that agentic explorers outperform classical retrieval, while line-level coverage and efficient ranking remain differentiating challenges.

  • Problem

    Existing coding benchmarks reduce repair to pass/fail outcomes and lack precise, line-level evaluation of how agents explore repositories.

  • Method

    SWE-Explore benchmarks ranked code-region selection against ground truth from successful agent trajectories under a fixed line budget.

  • Results

    Agentic explorers form a clear tier above classical retrieval, while file-level localization is strong but line-level coverage and efficient ranking remain differentiating axes.

  • Takeaways & Limitations

    Exploration metrics track downstream repair, with early line-level coverage and compact relevant context especially predictive of successful fixes.

  • Takeaways & Limitations

    The benchmark excludes instances without at least two successful trajectories, limiting supervision to issues with cross-run agreement evidence.

Abstract

from arXiv · show

Repository-level coding benchmarks such as SWE-bench have driven a rapid surge in the capabilities of coding agents. Yet they usually treat coding tasks as a holistic, binary prediction problem (e.g., resolved or unresolved), neglecting fine-grained agent capabilities such as repository understanding, context retrieval, code localization, and bug diagnosis. In this paper, we introduce SWE-Explore, a benchmark that isolates the evaluation of repository exploration, a critical capability of coding agents. Given a repository and an issue, SWE-Explore asks an explorer to return a ranked list of relevant code regions under a fixed line budget. SWE-Explore covers 848 issues across 10 programming languages and 203 open-source repositories. For each instance, we derive line-level ground truth from independent agent trajectories that successfully solved the same issue, distilling the specific code regions their solution paths actually consulted. We evaluate exploration along coverage, ranking, and context-efficiency dimensions, showing that these metrics strongly track downstream repair behavior. Across a broad set of retrieval methods, general coding agents, and specialized localizers, we find that agentic explorers form a clear tier above classical retrieval. While file-level localization is already strong for modern methods, line-level coverage and efficient ranking remain the key axes differentiating state-of-the-art explorers.

1 Introduction

SWE-Explore addresses the limitations of binary repair benchmarks by isolating repository exploration as a ranked, line-level context-selection task. It uses successful agent trajectories for ground truth and validates exploration metrics against downstream repair success.

  • Motivation: Binary pass/fail repair scores obscure whether agents failed during code exploration, bug localization, patch generation, or fix validation.This leaves repository exploration under-measured and prevents precise comparison of retrievers, search agents, and long-context selectors.
  • Benchmark: SWE-Explore formalizes repository exploration as returning a ranked list of code regions for an issue and repository, evaluated at line level.The benchmark is designed to compare methods on the specific capability they aim to improve rather than on holistic repair.
  • Ground truth: Ground truth lines are derived from independent trajectories of agents that successfully solved the same issue, requiring little manual annotation.The evaluation asks how early the ranked output surfaces evidence actually used by successful solution paths.
  • Validation: Coverage, ranking, and budget-efficiency metrics are evaluated with a controlled protocol in which a fixed coding agent receives only each explorer’s output as repository context.The resulting patch is tested against the original test suite to assess whether exploration scores correspond to downstream repair success.

2 Related Work

Prior work evaluates coding agents mainly through executable issue resolution or isolated retrieval and localization tasks. SWE-Explore addresses the missing joint evaluation of trajectory-grounded, line-level exploration quality and its downstream effect on issue resolution.

  • Repository-level benchmarks: Repository-level benchmarks center on executable issue resolution, with SWE-bench variants expanding quality control, contamination control, language coverage, and evaluation settings.SWE-bench couples issue descriptions, repository snapshots, and harness-based verification; Verified and Live tighten evaluation, while later variants broaden multilingual, multi-turn, and rebased settings.
  • Evaluation gap: Existing benchmarks either assess the full issue-to-patch pipeline or isolated intermediate behaviors, leaving trajectory-grounded, line-level exploration and its downstream resolution effect jointly unmeasured.Coarse context labels and final resolve rates can miss whether an explorer reaches the decisive code span, even after finding the correct file.
  • Retrieval and localization: Classical localization ranks repository artifacts from natural-language reports using lightweight TF–IDF and BM25 baselines, while semantic retrieval extends searches to functions and cross-file code completion.nDCG is identified as a standard evaluation measure for retrieval and localization methods.
  • Retrieval and localization: These retrieval evaluations target query–snippet, next-line completion, or bug-file relevance rather than line regions consulted during successful issue resolution.Their methodology is useful but evaluates relevance targets that differ from solution-trajectory exploration.
  • Interactive exploration: Recent LLM-based systems make exploration interactive by combining reasoning, code search, program analysis, localization, code-graph search, or pre-exploration for problem-statement improvement.Examples include AutoCodeRover, LocAgent, OrcaLoca, CoSIL, and CodeScout; general coding agents also connect navigation, context management, tool use, and patch generation.

3 SWE-Explore Benchmark

SWE-Explore isolates repository exploration by requiring a ranked, line-bounded list of relevant code regions for an issue and repository. It evaluates these predictions against trajectory-derived, refined ground truth across a broad benchmark using coverage, ranking, efficiency, and downstream-repair validation.

  • Benchmark formulation: SWE-Explore maps an issue and repository snapshot to a ranked list of file-and-line regions without requiring a patch, ground-truth access, or repository interaction.Each predicted region is represented as a file path and line range.
  • Benchmark composition: 848 retained instances span 10 programming languages and 203 open-source repositories, with each instance averaging 4.3 files, 4.7 regions, and 1,578 visible ground-truth lines.The retained instances come from SWE-bench Verified, SWE-bench-Pro, and SWE-bench Multilingual.
  • Trajectory-grounded supervision: Ground truth is derived from at least two successful issue-resolution trajectories per instance, using intersected line-level reads, selectively promoted load-bearing optional reads, and manual auditing.Ambiguous reads that cannot be mapped to explicit file-interval pairs are discarded rather than heuristically expanded.
  • Evaluation metrics: The benchmark scores predictions at line, file, and region granularity using precision, recall, F1, HITFILE, HITREGION, line-budget NDCG, first useful hit, context efficiency, and noise rate.Line-budget ranking penalizes verbose regions that exhaust the budget without proportional gain.
  • Downstream validation: A one-time restricted-context repair bridge tests whether exploration metrics track downstream repair by hiding repository content outside the explorer’s selected regions.A fixed coding agent produces a patch judged by the original SWE-bench harness; this sanity check is not part of standard evaluation.

4 Experiments

Experiments evaluate exploration with K=5 ranked regions, predictive upstream metrics, and controlled downstream repair tests. Agentic explorers outperform classical retrieval, but line-level recall remains the main bottleneck and missing relevant context harms repair more than redundant context.

  • Experimental setup: K=5 ranked regions are returned by every explorer, matching the refined ground truth average of roughly 4.7 core regions per instance.The fixed budget keeps comparisons fair and aligns each explorer’s output with the supervision target.
  • Downstream validation: Context Efficiency has the highest Pearson correlation with downstream resolve rate at r=0.950, while Rec@100 has the strongest rank correlation at ρ=0.845.These results identify compact relevance and early coverage under a tight line budget as especially predictive of repair.
  • Explorer comparisons: Agentic explorers substantially outperform BM25, TF–IDF, and Potion, which remain close to Random on most exploration metrics.The comparison indicates that multi-step repository interaction reaches a metric range that one-shot lexical and embedding retrieval do not.
  • Explorer comparisons: Most non-oracle explorers achieve Recℓ of only around 0.14–0.19 despite high HitFile and nDCG@500, making line-level recall the usual F1 limitation.AutoCodeRover is highly precise but is likewise recall-limited.
  • Explorer comparisons: CoSIL achieves by far the highest non-oracle Recℓ and F1, whereas other specialized localizers do not uniformly outperform general coding agents.The passage attributes CoSIL’s advantage to iterative code-graph search that broadens high-recall exploration.
  • Robustness: Missing relevant context is the dominant failure mode: resolve rates remain low with partial core context before increasing sharply, whereas random padding represents redundant context.The robustness test sweeps α ∈ {0, 25, 50, 75, 100} on two n=150 subsets with GPT-5.4-mini and GPT-5.4 patchers.

5 Conclusion

SWE-Explore evaluates repository exploration independently from patch generation through ranked, line-level context selection, showing that exploration metrics track downstream repair while current agents remain recall-limited at finding relevant files.

  • 5 Conclusion: SWE-Explore benchmarks repository exploration independently from patch generation using ranked, line-level context selection.It evaluates the evidence surfaced by explorers rather than only final repair outcomes.
  • 5 Conclusion: Trajectory-derived supervision enables comparisons among retrievers, search agents, and long-context selectors.The benchmark compares methods by the evidence they surface.
  • 5 Conclusion: Exploration metrics track downstream repair, while current agents are strong at finding relevant files but remain recall-limited.

A Dataset Details

SWE-Explore retains instances from three public repository-level benchmarks only when multiple trajectories successfully resolve the original task, producing supervision grounded in successful repair behavior. Each record includes line-level supervision and is evaluated on a fixed, canonicalized repository snapshot.

  • Source benchmarks and filtering: 848 instances remain after filtering cases with at least two successful trajectory resolves under the source benchmark’s executable harness.This grounds supervision in successful repair behavior rather than failed exploration attempts.
  • Benchmark composition: The retained set combines Python-centered verified issues, professional software-engineering tasks, and multilingual issue-resolution tasks.This design preserves executable-repair grounding while reducing dependence on one language ecosystem or repository family.
  • Benchmark record schema: Each instance records the issue, repository metadata, trajectory provenance, and line-level supervision through core and optional ground-truth regions.Core regions support scoring, while optional regions support diagnostics and context-efficiency computation.
  • Path and snapshot assumptions: Paths are repository-relative and canonicalized, while line intervals are 1-indexed and closed before scoring.Equivalent spellings such as ./src/foo.py and src/foo.py map to the same repository-relative file.
  • Repository snapshots: Each instance uses one fixed source-benchmark repository snapshot for interval resolution, prediction scoring, and restricted-context downstream validation.Generated, temporary, external-dependency, and out-of-checkout files are invalid repository files.

B Ground-Truth Construction and Refinement Details

SWE-Explore constructs line-level targets from observable repository reads, normalizes them conservatively, and refines the shared evidence across successful trajectories. The default refined-core target balances the conservative pure intersection against the noisy, high-recall full union.

  • Read extraction: The pipeline extracts editor views, resolvable command-line reads, and grep-mapped search hits as repository-relative line regions.Signals that cannot be mapped to a unique file–interval pair are discarded.
  • Normalization: Path and line intervals are normalized, with ambiguous or out-of-repository reads discarded and overlapping or adjacent same-file regions merged.Intervals use 1-indexed closed boundaries; whole-file reads expand to the evaluated checkout’s line count, and out-of-range intervals are clipped.
  • Core and optional context: The raw core is the file-wise, line-level intersection across successful trajectories, while optional context comprises successful reads outside that intersection.The refined core is the main scoring target; optional context is retained for diagnostics and context-efficiency computation.
  • LLM-assisted refinement: LLM refinement evaluates repeatedly visited, adjacent, or modification-proximal optional regions and promotes only precise intervals judged load-bearing.The model receives the issue, candidate region, nearby code, and a trajectory summary; candidates lacking a precise promoted interval are rejected.
  • Human audit and target variants: The refined-core target is the default because it balances the maximally conservative pure intersection against the high-recall but noisy full union.Every promoted region is manually audited against the issue, source trajectories, and final patch, and regions failing any check are removed.

C Metric Definitions and Ideal-Order Implementation

The evaluator scores ranked repository exploration over normalized repository-relative lines, handling duplicates, budgeted prefixes, and rank-aware ordering explicitly. Its nDCG normalization is budget-matched and instance-specific, while first-hit, efficiency, noise, and aggregation rules define additional evaluation behavior.

  • Line universe: Metrics operate on normalized repository-relative line identifiers, counting duplicate predicted lines once for set-based precision and recall while preserving region order for rank-aware metrics.A predicted region contributes all visible lines within its clipped interval.
  • Budgeted prefixes: For B = 500, the evaluator uses the longest prediction prefix whose cumulative visible lines fit the budget, with B ∈{100, 300, 500} additionally supported.Large regions placed early can exhaust the budget before more useful evidence appears.
  • nDCG and ideal ordering: nDCG assigns each predicted region gain equal to newly covered core lines and computes ideal DCG greedily by maximum marginal uncovered-line gain under the same budget.Ties are broken by shorter region length and then repository path, making normalization instance-specific and budget-matched.
  • First useful hit: First Useful Hit measures how early the first predicted region intersecting the line-level core target appears, with higher values indicating earlier useful evidence.If no predicted region intersects the core target, the score is zero.
  • Efficiency and aggregation: Metrics are averaged over instances; empty predictions receive zero for coverage, ranking, first-hit, and efficiency metrics, while invalid paths or empty intervals are discarded.Noise is reported at the region level in the main table.

D Restricted-Context Validation Protocol

The restricted-context protocol tests whether explorer-selected regions support patch generation under a fixed scaffold, using standard harness resolution and diagnostic logging to assess repair failures.

  • Protocol purpose: Restricted-context runs test patch generation from selected regions and supplement, rather than replace, standard upstream scoring.The protocol uses a fixed patching scaffold and is not part of the standard upstream scoring loop.
  • Context materialization: Only predicted line intervals in selected files remain visible; other lines become stable blank placeholders, while unselected files are hidden.Paths are normalized, intervals clipped to file boundaries, and invalid regions removed before materialization.
  • Fixed patch scaffold: The patcher, prompt, tools, and interaction budget are fixed, leaving explorer-produced visible context as the only variable.This controls against resolve-rate differences caused by stronger patch-generation scaffolds rather than better exploration.
  • Evaluation: A run is resolved only when the applied predicted diff passes the benchmark’s standard executable tests.Empty or unparsable patches, failed applications, and test failures count as unresolved.
  • Failure diagnostics: Unresolved runs receive coarse failure diagnostics, but these reasons do not alter the resolved/unresolved label.Logged causes include missing or invalid diffs, application failures, visibility violations, test failures, timeouts, and infrastructure errors.

E Explorer Implementation Details

SWE-Explore evaluates every explorer under a unified ranked-region contract of at most five repository-relative line intervals. Retrieval and agentic outputs are normalized to line-level regions, with invalid predictions removed before scoring.

  • Common output contract: All methods return an ordered list of at most K = 5 repository-relative line regions represented by paths and closed line intervals.Invalid paths, empty intervals, and regions outside the repository checkout are discarded before evaluation.
  • Retrieval baselines: BM25, TF–IDF, and Potion rank repository chunks using the issue statement and convert the top-ranked chunks into line regions.BM25 and TF–IDF use lexical similarity, while Potion provides a lightweight dense retrieval baseline through the same chunk-and-rank interface.
  • Agentic explorers: Agentic explorers retain their original search or localization scaffolds, then normalize file, function, or region outputs into line-level regions.File-level outputs are mapped to the most specific span supported by the method output or associated read trace.
  • Output validation: Predictions are checked for path validity, interval validity, and repository membership before scoring, and valid predictions retain their original order.Invalid predictions and predictions with empty intervals are dropped.

F Case Study: scikit-learn/scikit-learn#10844

This case study examines a numerical-overflow bug whose exploration ranking mirrors the global results. General-purpose agents find both ground-truth files with function-scoped spans, outperforming lexical retrieval and academic localizers.

  • Issue and ground truth: The bug arises when 32-bit integer operands overflow during pk * qk multiplication in fowlkes_mallows_score, corrupting the score and emitting a RuntimeWarning.The fix casts both operands to np.float64 and adds a regression test.
  • Issue and ground truth: The refined ground truth covers two core files and only 26 lines: the modified function and a five-line overflow regression test.The implementation spans sklearn/metrics/cluster/supervised.py lines 850–870, while the test spans test_supervised.py lines 245–249.
  • Explorer behavior: Random, TF–IDF, and Potion reach HitFile = 0, while BM25 partially recovers with HitFile = 0.50 but misses the regression test.The issue’s frequent terms do not identify the implementation, whereas fowlkes_mallows_score helps BM25 locate the file.
  • Explorer behavior: AutoCodeRover, OrcaLoca, LocAgent, and CoSIL locate supervised.py but not test_supervised.py, capping their HitFile at 0.50.AutoCodeRover and OrcaLoca emit only one region, reducing line recall and coverage within the implementation file.
  • Explorer behavior: All five general-purpose agents reach both ground-truth files with HitFile = 1.00 and identify the fowlkes_mallows_score implementation and surrounding test block.Codex uses the tightest fix-line spans, while AweAgent and OpenHands include additional context.

G Reproducibility, Compute, and Limitations

The released artifact supports reproducible evaluation with benchmark records, provenance, schemas, annotations, and metric scripts, while compute costs vary substantially across retrieval and agentic components. The benchmark is limited to agent-solvable issues and trajectory-derived approximations of useful context, with a public-data-only release.

  • Reproducibility: The released artifact includes benchmark records, a common explorer-output schema, metric and restricted-context validation scripts, provenance, repository metadata, and line-level annotations.The evaluation pipeline consumes ranked-region predictions and produces per-instance metrics and aggregate tables.
  • Compute: Agentic explorers and restricted-context validation are the most expensive components because they require LLM calls and executable harness runs.Sparse retrieval runs on CPU workers after indexing, while dense retrieval additionally requires embedding computation but no fine-tuning.
  • Limitations: SWE-Explore covers only instances solved by at least one agent, and its trajectory-derived ground truth empirically approximates useful context rather than proving exhaustiveness.Some valid solution paths may rely on different evidence, so restricted-context results should be interpreted as controlled validation.
  • Responsible release: The release derives from public benchmarks and repository metadata while excluding private repositories, credentials, and user data.Benchmark records preserve source attribution and document the schema, provenance, and intended use.
Loading 2606.07297v1…