PaperlayerView exact revision on arXiv

Source-linked AI summary

Cost-Effective Repository Exploration for Agentic Issue Localization

Mohammad Nour Al Awad, Sergey Ivanov

arXiv:2608.29675v1cs.SEcs.AI

TL;DR

Coding agents must locate relevant repository files before patch generation, but the quality–cost trade-offs of assigning this stage to different models are underexplored. The paper evaluates dedicated explorers under a fixed read-only interface across two task arms and finds that the highest-quality explorer leads localization metrics while cheaper operating points retain useful quality at substantially lower time and token cost. The preferred operating point depends on whether downstream use requires recoverable candidate handoffs or restrictive file gates.

  • Problem

    Repository exploration is a distinct, costly coding-agent stage, and the available quality–cost operating points for dedicated explorers require measurement.

  • Method

    The study evaluates five explorer models through the same read-only interface on two IssueLoc-Bench arms totaling 499 SWE-bench Verified-derived tasks and 500 tasks from additional repositories.

  • Results

    The highest-quality explorer leads localization metrics, while lower-cost explorers retain useful localization quality and reduce agent time and token usage across the evaluated arms.

  • Takeaways & Limitations

    Repository exploration can be treated as a separately measurable and budgetable pipeline stage, with explorer choice guided by the downstream handoff contract.

  • Takeaways & Limitations

    The study does not directly establish how explorer choices affect end-to-end resolved rate or total pipeline cost, which require fixed-repairer evaluation.

Abstract

from arXiv · show

Repository exploration is a distinct and costly stage of coding-agent pipelines: before generating a patch, an agent must identify which repository files are likely to matter. We study whether this stage can be delegated to lower-cost models while retaining useful localization quality. Using our IssueLoc-Bench, we evaluate five explorer models under the same read-only interactive interface on 499 SWE-bench Verified-derived tasks and 500 tasks from 153 additional repositories. We measure early candidate discovery, top-three gold-file coverage, strict file-set recovery, agent time, and token usage, with paired instance-level uncertainty and repository-clustered sensitivity analysis. The highest-quality explorer leads across localization metrics, but substantially cheaper operating points emerge: depending on the model and evaluation arm, lower-cost explorers retain approximately 78-94% of the reference Hit@3 and 73-92% of its F1 while reducing mean agent time by 41-88% and token usage by 84-95%. The preferred operating point depends on how localization is consumed downstream: ranking and coverage metrics characterize recoverable candidate handoffs, whereas F1 and exact match characterize restrictive file gates. These results support treating repository exploration as an independently measurable and budgetable stage of modular coding agents, with explorer selection guided by the downstream handoff contract.

1 Introduction

The paper treats repository exploration as a distinct, measurable allocation stage in coding-agent pipelines and studies its quality–cost trade-offs under a fixed interface. It evaluates dedicated explorers across two arms and distinguishes recoverable ranked handoffs from restrictive file gates.

  • Repository exploration produces a ranked candidate-file artifact whose quality and production cost can be measured independently of patch generation.
  • Poor exploration can waste downstream reasoning budget and misdirect repair by mapping incomplete issue reports onto large repositories.
  • The study compares dedicated explorer models under a fixed repository-interaction interface without directly measuring each explorer’s end-to-end repair effect.
  • The evaluation uses 499 SWE-bench Verified-derived tasks and 500 repository-disjoint tasks from the IssueLoc random-repository arm.
  • Ranking and coverage metrics characterize soft handoffs, whereas F1 and exact match characterize hard gates that constrain downstream file access.
  • The structured file boundary supports independent measurement, budget selection, candidate limits, and escalation without changing the downstream representation.

2 Background and Related Work

Prior work establishes repository-level issue resolution and localization, but end-to-end benchmarks conflate exploration with later repair stages. IssueLoc-Bench isolates file-level explorer choice under a fixed interface while preserving a repair-relevant artifact.

  • Bug-localization research shows that performance depends on combining textual relevance with history, report structure, related reports, and repository organization.
  • Benchmark design, preprocessing, and evaluation choices can change both absolute performance and relative rankings among localization techniques.
  • Ranking metrics measure whether useful files surface early, while F1 and exact recovery measure alignment with the historical resolution footprint.
  • Repository benchmarks such as RepoBench, SWE-bench, and SWE-agent evaluate composed behavior spanning retrieval, interaction, patching, and validation.
  • Unlike escalation and end-to-end exploration studies, this work fixes the exploration interface and varies the explorer model across quality, runtime, and token measures.
  • IssueLoc-Bench isolates file-level localization under repository-grounded pre-fix conditions for selecting a dedicated explorer.
  • Related systems connect localization to repair, while Loc2Repair reports downstream resolved-rate gains from applying localization across repair backbones.

3 Task Definition

The task is pre-fix, repository-grounded file localization: an explorer reads an issue and repository snapshot, then returns an ordered list of existing candidate files without modifying code or validating repairs.

  • The explorer ranks existing repository files that are plausible candidates for downstream repair from an issue report and pre-fix snapshot.
  • The input combines the issue-side task record with a pre-fix repository snapshot containing repository-relative paths.
  • Gold labels represent files modified by the historical resolving commit, defining a reproducible resolution footprint rather than a minimal causal file set.
  • File-level prediction is the operational unit for repository-scale context selection because it determines which files agents read, search, summarize, or pass downstream.
  • Agents interact through read-only repository inspection and must terminate with an ordered list of repository-relative file paths.
  • The evaluation distinguishes soft handoffs using ranking and coverage from hard gates using F1 and exact match.

4 Benchmark Construction and Evaluation Arms

IssueLoc-Bench combines reproducible pre-fix repository snapshots, task and label manifests, and two complementary evaluation arms. The construction uses explicit filtering and split procedures while acknowledging contamination and transfer boundaries.

  • IssueLoc-Bench provides task and label manifests, deterministic pre-fix snapshots, a shared evaluator, and runnable explorer pipelines.
  • The random-repository arm samples exactly linked issue–pull-request pairs with single-commit merged fixes to derive unambiguous file labels.
  • Recency reduces contamination risk but cannot guarantee that public repository instances were absent from model pretraining.
  • Labels join task metadata with filtered repository-relative gold file sets representing historical resolution footprints.
  • The builder materializes each resolving commit’s parent tree, making inputs, labels, and reconstruction inspectable and reproducible.
  • Filtering excludes changes requiring new or removed files, binary-like files, empty gold sets, or entirely markdown/text-like targets.
  • The evaluation uses 499 SWE-bench Verified-derived instances and 500 random-repository instances from 153 repositories, without model-selection splits.
  • The arms are complementary because the SWE-derived arm emphasizes larger repositories, while the random arm has larger historical file footprints.

5 Evaluation Protocol

The evaluation scores ranked candidate-file outputs for early discovery, gold-set coverage, final set recovery, execution validity, and operational cost. Paired and repository-clustered analyses quantify uncertainty while preserving failed runs in quality denominators.

  • Discovery metrics: Hit@1, Hit@3, Hit@5, MRR, and R@3 measure early candidate discovery and gold-file coverage from ranked predictions.Hit@3 records whether any gold file appears in the first three predictions, while R@3 measures the fraction of the gold set covered there.
  • Set-recovery metrics: Macro F1 and exact match evaluate final file-set recovery, with exact match requiring the emitted set to equal the historical resolution footprint.Macro F1 permits partial gold-set recovery while penalizing overprediction.
  • Validity metrics: ErrorRate is the fraction of attempted instances lacking a valid final prediction under benchmark runtime rules, most commonly because trajectories time out.Execution failures remain in the quality denominators and receive zero localization credit when no valid prediction exists.
  • Operational metrics: Agent time includes model/API waiting, repository-tool execution, and timeout penalties, while token averages cover only runs with recorded usage metadata.Reported reductions compare complete explorer operating points rather than intrinsic model speed.
  • Uncertainty analysis: Alternative explorers are aligned instance-by-instance with the highest-quality reference and evaluated using percentile paired bootstrap intervals with 10,000 resamples.Repository-clustered bootstrap resamples repositories while retaining all aligned instances within each sampled repository.

6 Experimental Setup

The experiments compare five explorer checkpoints in two evaluation arms using the same constrained, read-only interactive runner. The setup reconstructs pre-fix repositories, records interaction traces, and scores outputs under a shared evaluation pipeline.

  • Controlled runner: Each instance uses a pre-fix repository snapshot and the same localization task, read-only interaction contract, and per-instance time budget.The runner instructs the agent to localize only and terminate with a structured ranked file prediction.
  • Models: The study evaluates five general-purpose, code-specialized, and compact explorer checkpoints under a controlled model-choice comparison.The checkpoints are GLM-4.7-Flash, Gemma-4-E4B-it, Qwen3-30B-A3B-Instruct, Qwen3-Coder-30B-A3B-Instruct, and Qwen3-4B-Instruct.
  • Interaction protocol: Agents inspect repositories exclusively through shell-based read-only commands such as ls, find, grep, cat, nl, and sed -n.The standard per-instance timeout is 600 seconds.
  • Instrumentation: Isolated workspaces and persistent interaction traces record elapsed time, interaction counts, token metadata, and execution errors for joint quality–cost analysis.The released artifact supports deterministic snapshot materialization, constrained execution, and resumable pipelines.
  • Scoring setup: Outputs are scored against released label manifests for two evaluation arms using the same runner and, unless specified otherwise, a markdown-ignored view focused on code-file localization.IssueLoc uses title and body fields, while transformed SWE-bench instances use linked issue text.

7 Results

Across both evaluation arms, the highest-quality reference explorer leads localization metrics but incurs the greatest time and token cost. Lower-cost explorers occupy distinct operating points, with their practical value depending on whether downstream systems use recoverable rankings or restrictive file gates.

  • Evaluation-arm results: The reference explorer leads every localization metric in both arms but also produces the longest, most token-intensive trajectories.This ordering is reported for both the SWE-bench Verified-derived and random-repository arms.
  • Evaluation-arm results: Qwen3-Coder-30B-A3B is strongest for SWE-derived top-three discovery, while Qwen3-30B-A3B provides stronger alternative F1 and lower token usage.Qwen3-4B and Gemma-4-E4B occupy shorter-time operating points.
  • Evaluation-arm results: Qwen3-30B-A3B has the strongest Hit@3, F1, and exact match on the random-repository arm, while Qwen3-Coder-30B-A3B leads R@3.Gemma-4-E4B and Qwen3-4B again trade quality for shorter mean completion time.
  • Paired model-comparison uncertainty: Every alternative reduces time and tokens, but repository clustering weakens several SWE-derived quality comparisons while leaving operational reductions supported on both arms.The strength of quality ordering therefore depends on whether variation is measured across tasks or repositories.
  • Quality retention and cost reduction: Gemma retains at least 86% of reference Hit@3 and F1 in both arms while cutting mean agent time by more than two thirds.Qwen3-30B offers the strongest alternative F1 with more than 90% observed-token reduction, whereas Qwen3-Coder preserves the strongest SWE-derived top-three discovery but sacrifices stricter set recovery.
  • Answers to the research questions: The preferred operating point depends on the handoff metric: candidate discovery favors Qwen3-Coder on SWE-derived tasks, whereas strict F1 favors Qwen3-30B and Gemma.Ranking and coverage serve soft handoffs; F1 and exact match characterize hard gates.

8 Analysis and Discussion

The analysis finds distinct explorer operating points rather than one universally best lower-cost model. Their value depends on whether downstream use permits recoverable ranked search or imposes a restrictive file gate, while escalation and candidate budgets offer further allocation controls.

  • Explorer Operating Points and Handoff Contracts: Hit@3 records whether a small candidate budget contains any historical target, while R@3 records how much of the target footprint that budget covers.These measures distinguish finding a useful starting point from covering a multi-file issue.
  • Explorer Operating Points and Handoff Contracts: F1 and exact match are better aligned with restrictive context selection, whereas ranking and coverage metrics characterize recoverable candidate handoffs.A later stage may navigate outward from one plausible entry point, but a hard gate cannot recover an omitted file.
  • Cross-Arm Results: The reference configuration provides the highest observed localization quality in both evaluation arms, while every alternative uses less time and fewer tokens.The arms stress different exploration conditions, and repository-clustered analysis tempers some SWE-derived quality contrasts because relatively few repositories can influence aggregate ordering.
  • Escalation-Oriented Explorer Design: A soft-handoff policy could begin with a lower-cost explorer and escalate when candidates are empty, diagnostically implausible, or contradicted by later inspection.For hard gates, escalation must occur earlier because ordinary browsing cannot recover omitted files.
  • Candidate Budgets and Context Transfer: Candidate count can be tuned independently of checkpoint choice because Hit@3 measures a starting point while R@3 measures footprint coverage.Small budgets encourage precise ranking, whereas multi-file issues may require broader coverage.
  • Operational Costs: 34?

9 Threats to Validity

The study’s validity is bounded by its historical file-footprint labels, repository sampling, public-code setting, and deployment-dependent operational measurements. Its isolated localization setup also leaves end-to-end repair effects unresolved.

  • Historical-Footprint Supervision: Historical commit footprints make exact match deliberately strict because touched files are not proven necessary or uniquely required for valid repairs.Ranking and coverage metrics therefore provide complementary evidence, while conclusions concern existing implementation artifacts.
  • Repository and Model Sampling: The two evaluation arms have different repository coverage and linkage criteria, so the paper avoids pooling them and reports repository-clustered sensitivity.Both arms use public repositories, and transfer to private codebases, very large monorepos, other language distributions, or other interfaces remains untested.
  • Operational Measurements: Operational totals combine model behavior, serving, tool interaction, stopping, and timeout policies rather than isolating model throughput.Token comparisons require usage metadata, public-price cost views are contemporaneous, and repeated executions would be needed to measure stochastic trajectory variation.
  • Pipeline Scope: The experiment isolates localization under a fixed interface, so its rankings do not establish effects on resolved rate or total pipeline cost.Those effects require evaluating explorer, repairer, candidate budget, context format, and escalation policy together.

10 Conclusion

Under a fixed read-only interface, the highest-quality explorer leads localization metrics, while alternatives offer lower-time and lower-token operating points. The preferred choice depends on whether downstream systems need recoverable candidate rankings or restrictive file gates, and end-to-end repair effects remain future work.

  • 10 Conclusion: The highest-quality explorer leads localization metrics, while alternatives provide substantially lower-time and lower-token operating points across two complementary arms.Paired comparisons expose task-level trade-offs, with repository-clustered sensitivity tempering several SWE-derived quality contrasts.
  • 10 Conclusion: No alternative dominates simultaneously across ranking quality, strict recovery, time, and tokens.Explorer selection is therefore a multi-axis operating-point decision rather than a single overall ranking.
  • 10 Conclusion: Hit@3, R@3, and MRR characterize a recoverable candidate handoff, whereas F1 and exact match characterize a restrictive file gate.The downstream handoff contract determines which localization behavior matters most.
  • Future Work: Determining whether localization trade-offs translate into end-to-end repair trade-offs requires a fixed-repairer evaluation and remains future work.Future evaluation can vary the explorer, top-k budget, and escalation rule while measuring resolved rate and total pipeline cost.

11 Data Availability Statement

The artifact provides the data, builders, evaluators, runners, outputs, and statistical reporting code needed to reconstruct and rescore the study.

  • 11 Data Availability Statement: The public artifact includes task and label manifests, snapshot construction, shared evaluation, explorer runners, run outputs, and statistical reporting code.Stable identifiers and traces support reconstruction and rescoring.
Loading 2608.29675v1…