Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents

Bowen Qin; Yi Xie

Source-linked AI summary

Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents

Bowen Qin, Yi Xie

arXiv:2607.24882v1Jul 27, 2026cs.IRcs.AIcs.CL

TL;DR

Coding agents need to retrieve the repository files relevant to a task before editing, but this context-acquisition layer is under-evaluated. Agent Retrieval Bench evaluates agentically relevant retrieval across workflow signals and finds that no single retrieval family dominates, supporting hybrid, task-aware retrieval.

Problem
End-to-end coding-agent evaluations undermeasure the preceding question of which repository files should enter the model before editing.
Method
Agent Retrieval Bench evaluates file-level retrieval on 427 evidence-backed samples from frozen repository snapshots, with agentic relevance and realistic distractor controls.
Results
No single retrieval family dominates: Qwen3-4B leads MRR, Qwen3-8B leads Recall@20, and RepoMap leads budgeted context yield, with winners varying by task and budget.
Takeaways & Limitations
Coding-agent retrieval should be hybrid and task-aware because semantic, lexical, structural, and source-test signals solve different benchmark parts.
Takeaways & Limitations
The benchmark is primarily file-level, so full-file credit may not reflect the downstream effort required to localize useful regions within large files.

Abstract

from arXiv · show

Modern coding agents are usually evaluated by whether they eventually produce a correct patch, but patch generation depends on an earlier context-acquisition stage: finding the repository files needed for the task. We introduce Agent Retrieval Bench, a file-level benchmark for this upstream retrieval problem. Samples are built from real coding-workflow signals and evaluated against frozen base-commit repositories, with relevance defined by what an agent needs next rather than direct query-file semantic similarity. The benchmark covers four positive-retrieval tasks: code2test, comment2context, trace2code, and edit2ripple; a fifth subset evaluates selective retrieval using natural evidence-backed no-gold cases and counterfactual wrong-repository controls. Agent Retrieval Bench contains 427 samples across 25 repositories: 345 positive examples, 50 natural no-gold examples, and 32 counterfactual controls. The corpus includes 308 base-commit snapshots, 392,000 files, and 7.9 million chunks. We evaluate lexical retrieval, RepoMap, open-source embeddings, selective abstention, and logged agent context selection. No single retrieval family dominates: Qwen3-Embedding-4B has the best sample-weighted MRR on positive samples, Qwen3-Embedding-8B the best Recall@20, and RepoMap the best budgeted context yield at 8K tokens, with task-level winners differing substantially. Selective thresholds calibrated with counterfactual controls do not improve selective success on natural no-gold cases, revealing a calibration gap. Logged trajectories also miss every gold file on 27-35 percent of samples. A controlled seed-intervention pilot finds that retrieval-derived initial context yields higher file F1 with less post-seed exploration than random non-gold context, while oracle gold context shows substantial remaining headroom.

1. Introduction · 2. Related Work

Agent Retrieval Bench isolates file-level context acquisition as an upstream coding-agent problem, defining relevance by the next useful workflow step rather than semantic similarity or final patch success. It contributes workflow-grounded tasks, frozen-repository evaluation, and analyses showing complementary retrieval methods and limitations in trajectory coverage and causal claims.

1. Introduction: Agentic relevance means retrieving the next useful context, so a workflow query may require tests, constraint files, or root-cause sources beyond its directly mentioned file.This differs from traditional code search, which commonly matches natural-language queries to semantically similar code.
1. Introduction: Logged trajectories still leave every gold file untouched on 27-35 percent of samples despite agents reading or recording multiple files and paths.The reported trajectory track covers 287 code2test, comment2context, and trace2code samples.
1. Introduction: Agent Retrieval Bench evaluates whether agents can find useful repository files before editing, separating context acquisition from patch synthesis.The benchmark uses workflow-derived queries and repositories frozen at their base commits.
1. Introduction: The paper claims component-level effects on acquired context and acquisition cost, not that retrieval dominates patch failure or that File F1 substitutes for test-passing repair.Linking retrieval to patch success requires an executable repair benchmark with aligned outcome labels.
1. Introduction: The four positive tasks are code2test, comment2context, trace2code, and edit2ripple, while selective retrieval tests whether systems should abstain when no repository-local context is appropriate.The tasks respectively target related tests, additional context, root-cause source files, and files affected by an anchored change.
1. Introduction: 427 samples are constructed against frozen base-commit corpora, with gold evidence, schema and corpus checks, and controls for path, patch, commit, and generated-content leakage.The benchmark includes positive tasks plus natural and counterfactual no-gold retrieval cases.
1. Introduction: No retrieval family dominates across workflow signals or context budgets: structural and semantic methods are complementary, and Qwen3-4B and Qwen3-8B reverse order across edit2ripple budgets.A controlled-input pilot also finds retrieval-derived seeds more context-efficient than random context in a single run.
2. Related Work: Related work spans semantic code search, repository-level completion, end-to-end agents, localization, agentic exploration, traceability, fault localization, and repository maps; this benchmark targets file-level context finding before editing.It is positioned as complementary to nearby settings rather than as a replacement for them.

3. Problem Definition

Agent Retrieval Bench defines file-level retrieval from base-commit repositories using workflow-derived queries and agentically relevant gold files. It evaluates ranked-file recovery, budgeted context exposure, and complementarity with logged agent trajectories.

Sample and corpus: Each sample pairs a repository base commit and query with gold files, while retrievers rank candidate files and metrics operate on unique file paths.Selective samples additionally specify why an empty gold set is expected.
Query construction: Queries derive from real coding-workflow signals and remove final patches, fix diffs, hashes, generated artifacts, and exact gold paths.Signals include PR or implementation summaries, review comments, failure traces, anchored edits, and issues.
Gold and relevance: Gold files are task-specific files an agent needs next, including related tests, context files, root-cause sources, or additional affected files.Reviewed and anchor files may be given context rather than counted as main gold, and selective samples use empty gold sets.
Metrics: The benchmark reports Recall@5, Recall@10, Recall@20, MRR, and BCY@B, where Recall@k measures gold-file recovery within the top k unique files.BCY measures labeled gold context packed within a fixed token budget after ranking and uses canonical deterministic file rendering and tokenization.
Logged trajectories: Logged trajectories report a 2-by-2 retriever-agent complementarity matrix and PES@k, an upper-bound proxy for avoidable gold-localization delay rather than a causal savings estimate.The matrix distinguishes both hit, retriever-only, agent-only, and both miss outcomes.

4. Benchmark Tasks · 5. Dataset Construction

Agent Retrieval Bench defines four evidence-backed positive retrieval tasks and two distinct no-gold strata, then constructs them from real workflow artifacts against frozen repository snapshots. Its corpus combines 427 samples with large, multilingual repository snapshots and realistic workflow distractors.

4.1 code2test: code2test retrieves relevant tests from implementation-change signals while excluding exact test paths and final-patch evidence.Gold files are tests linked by released pull-request or change evidence.
4.2 comment2context: comment2context retrieves additional files needed to understand or satisfy a review comment, treating the reviewed file as given rather than gold.The query includes the review comment and reviewed file.
4.3 trace2code: trace2code identifies root-cause source files from a reproduced failure, while tests and failure frames count only as supporting context unless root-cause evidence supports them.The query contains the command and failure excerpt.
4.4 edit2ripple: edit2ripple finds source or test files affected by a localized edit, using changed-file evidence and recorded ripple relations as gold support.The anchor file is given context rather than gold, alongside its diff and change intent.
4.5 Selective No-Gold Retrieval: 50 natural no-gold cases and 32 counterfactual controls have empty gold sets but separately test upstream or user-caused resolution versus query–repository mismatch.The natural cases reflect maintainer-attributed resolutions involving an upstream dependency, external service, or user error.
5. Dataset Construction: Each sample pairs a repository workflow artifact with task-specific relevance evidence and is evaluated at a frozen base commit across four positive tasks and two no-gold strata.Queries come from workflow evidence rather than model-written task descriptions, and construction removes exact target paths and raw resolution artifacts.
5.1 Provenance and Construction Pipeline: 427 samples span 25 repositories, while the reusable corpus manifest contains 308 repo/base rows, 391,932 files, and 7,922,369 chunks across 29 repositories.Sample-bearing repositories cover six primary languages, and the corpus is intentionally much larger than the sample count to provide a realistic candidate universe.
5.1 Provenance and Construction Pipeline: The benchmark preserves realistic distractors, including exposed implementation files, reviewed files, stack frames, and test paths that are visible but not necessarily gold.These distractors mirror common agent workflows and require retrieval beyond the immediately exposed context.

6. Quality and Leakage Controls · 7. Baselines · 8. Main Results

Agent Retrieval Bench controls direct-answer leakage and evaluates retrieval across diverse baselines, tasks, context budgets, trajectories, and selective settings. Results show complementary method strengths rather than a universal winner, with hybrid and retrieval-derived contexts improving coverage while natural no-gold abstention remains difficult.

6. Quality and Leakage Controls: All 345 positive samples have released workflow-to-corpus evidence, while fatal leakage checks for exact paths, patches, commits, and generated artifacts are zero.The controls establish traceability and exclude direct answer leakage, but do not prove that every useful file is labeled.
7. Baselines: The benchmark compares path-aware lexical retrieval, RepoMap structure signals, and prompt-free zero-shot open-source embeddings under fixed all-file positive-task evaluation.Embedding rankings use separately encoded, L2-normalized query and corpus vectors with file scores determined by the maximum-scoring chunk.
8.1 Complete Positive Leaderboard: 0.2379 weighted MRR goes to Qwen3-4B, 0.7029 weighted Recall@20 to Qwen3-8B, and 0.3788 weighted BCY@8k to RepoMap.Under repository-equal weighting, Qwen3-4B remains the MRR leader and becomes the Recall@20 leader, exposing task-frequency effects.
8.2 Task Winners / 8.3 Agentic Relevance Relations: Task winners differ substantially, and 272 (94.8%) of 287 taxonomy-covered samples have at least one secondary agentic-relevance relation.The primary projection partitions the 287 samples, while causal-indirect cases favor RepoMap for MRR and RRF-Q8+RM for Recall@20.
8.4 Simple Hybrid Rank Fusion: 0.2296 to 0.2713 overall MRR and 0.7070 to 0.7331 overall Recall@20 are the gains from fusing Qwen3-8B with RepoMap using RRF.Trace2code Recall@20 reaches 0.8795, above RepoMap at 0.8366 and Qwen3-8B at 0.7970, but code2test MRR remains slightly below Qwen3-4B.
8.7 Trajectory Cost and Seed Intervention: 35.2% of OpenAI strict-context samples and 27.2–29.3% of Codex samples contain no gold file in logged context events, while retrieval-derived seeds improve File F1 over random non-gold context.The 45-sample controlled pilot associates lexical and RRF seeds with about 0.075 additional File F1, earlier gold contact, and fewer post-seed read tokens; oracle context indicates remaining headroom.
8.8 Selective Retrieval with Abstention: No ranker improves selective success after counterfactual controls are removed and thresholds are recalibrated on natural-only folds.Jina correctly abstains on 94.0% of natural no-gold cases but passes only 37.7% of positives, showing that wrong-repository separation does not calibrate natural abstention.
8.9 Budget-Dependent Retrieval on edit2ripple / 8.10 trace2code Reverses the Aggregate Picture: Edit2ripple reverses the verified Qwen ordering between fully packed 4k and 8k budgets, while trace2code is led by RepoMap for both MRR and Recall@20.Qwen3-4B has edit2ripple Recall@20 of 0.7112, whereas its trace2code MRR is only 0.0827 despite leading aggregate MRR.

9. Rank and Error Analysis

First-gold depth reveals a tradeoff beyond aggregate MRR: Qwen3-4B ranks successful hits earlier, while Qwen3-8B covers more samples by depth 20. Error analyses show structural and semantic retrieval signals are complementary, with failures arising from given-file traps and project-specific test organization.

First-gold depth: 0.7944 vs. 0.6864: Qwen3-8B has broader top-20 gold-file coverage than Qwen3-4B, despite Qwen3-4B having the highest overall MRR.The 4B model places successful hits earlier, while the 8B model retrieves at least one gold file for more samples by depth 20.
First-gold depth: 22 of 287 samples are missed by every reported baseline at top 20, indicating complementary retrieval behavior rather than uniformly impossible examples.These samples come from code2test, comment2context, and trace2code.
Trace2code errors: RepoMap recovers Gin trace2code gold files that both Qwen3 models miss, using structural proximity between tests and source files.The gold files are response_writer.go and errors.go; failure excerpts mention tests and compile failures, which can bias semantic models toward visible tests or repository metadata.
Trace2code errors: pplx-embed-v1-4b ranks Caddy’s gold caddyauth.go first while RepoMap misses it at depth 20, showing that semantic and structural signals are complementary.This counterexample rules out a simplistic conclusion that RepoMap consistently beats embeddings.
Comment2context errors: RepoMap falls into the given-file trap in a Hypothesis sample, ranking the reviewed test first while missing gold files in core.py and the conjecture engine at depth 20.The reviewed file is already available to the agent, so returning it is insufficient.
Code2test errors: All seven baselines miss etcd’s gold end-to-end control-plane tests and cluster framework file at depth 20, despite nine implementation and two existing test files in the query summary.The failure demonstrates that source-to-test retrieval requires mapping implementation changes to project-specific test organization, not only semantic similarity.

10. Candidate Filter Ablation

A diagnostic ablation shows that candidate filtering changes vectorless-baseline performance in task-specific ways, while the official leaderboard retains the all_files setting.

Candidate filtering: Restricting candidates to tests_only sharply improves code2test but reduces trace2code to zero by removing source-file golds.The ablation is diagnostic and does not replace the official all_files leaderboard.
Candidate filtering: The code_only filter improves vectorless overall MRR, especially by reducing distracting non-source candidates.These results motivate interpreting candidate-filter effects diagnostically rather than as the official evaluation.

11. Limitations

The benchmark is a diagnostic, file-level evaluation on a fixed base-commit corpus, with limited coverage and substantial uncertainty in how labels and relevance annotations represent downstream needs. Its scores may overstate localization because gold files can be incomplete and full-file credit can hide within-file search.

Scope and coverage: 427 samples across 25 repositories make the benchmark diagnostic rather than a large statistical evaluation of languages and ecosystems.The sample distribution is uneven: Gin contributes 25.5% of positive samples, and the four largest repositories contribute 58.8%.
File-level evaluation: 51.8% of sample–gold-file occurrences exceed 500 lines, while median labeled span evidence covers only 4.7% of its file.File-level credit can therefore leave downstream agents substantial within-file search, despite identifying the correct file.
Static evaluation: The corpus is fixed at the base commit, and the main static leaderboard omits dynamic tool use, iterative retrieval, and multi-turn agent exploration.Logged trajectories and an additional closed-tool diagnostic are reported separately rather than incorporated into the main static-retrieval leaderboard.
Label uncertainty: Gold labels approximate files supported by construction evidence, but alternative useful files may be omitted from the gold set.This makes gold-file relevance an approximation of what an agent needs to read, rather than an exhaustive specification.
Annotation and taxonomy: Agentic-relevance relations are deterministic diagnostic annotations, while multi-label categories and fixed task-specific thresholds are benchmark conventions rather than independently validated latent classes.The primary projection thresholds do not enter the benchmark’s main retrieval scores.

12. Future Work

Future work centers on hybrid retrieval, broader and more robust benchmark coverage, and finer-grained evaluation tied to downstream coding outcomes. Proposed extensions combine complementary retrieval signals, expand tasks and languages, test query rewriting, add span-level annotations, and use controlled interventions on executable issue-resolution instances.

Hybrid retrieval: Hybrid retrieval should combine semantic vectors, lexical and path signals, repository graphs, source-test relations, and task-aware reranking.The motivation is that embeddings, lexical matching, and RepoMap have complementary strengths.
Benchmark expansion: A larger release should add more languages, repositories, and workflow signals, alongside issue-to-code, bug-report-to-root-cause, migration-to-affected-files, and API-usage-to-implementation tasks.These additions would broaden the benchmark’s repository and workflow coverage.
Query rewriting: Paired developer-facing query rewrites could test phrasing robustness and whether normalized coding-assistant queries change the value of semantic, lexical, and structural signals.The rewrites would preserve the same underlying workflow-derived samples, including PR summaries, review comments, and failure excerpts.
Granularity and downstream evaluation: Future releases should add span annotations to edit2ripple, require line-resolved intervals, evaluate span yield under BCY’s canonical token packing, and run paired interventions on executable issue-resolution instances.The intervention would hold the model, scaffold, tool and token budgets, and sampling configuration fixed while varying only the initial context.

13. Conclusion · A. Repository and Language Coverage

Agent Retrieval Bench shows that coding-agent retrieval remains an upstream, task-dependent problem requiring hybrid methods rather than a single dominant retrieval family. Its controlled benchmark spans diverse repositories and languages, enabling these differences to be measured before end-to-end patch generation obscures them.

13. Conclusion: The benchmark evaluates whether agents find needed repository files before patch generation, including when no file should be returned.This targets an upstream failure mode rather than only final patch correctness.
13. Conclusion: No single retrieval family dominates across sample-weighted MRR, Recall@20, aggregate BCY@8k, and task-specific performance.Qwen3-4B leads sample-weighted MRR, Qwen3-8B leads sample-weighted Recall@20, and RepoMap leads aggregate BCY@8k and trace2code.
13. Conclusion: Qwen3-4B and Qwen3-8B reverse order across verified edit2ripple context budgets.This further shows that retrieval rankings depend on task and context budget.
13. Conclusion: Coding-agent retrieval should combine semantic similarity, lexical/path matching, repository structure, and source-test relations.The practical recommendation is hybrid and task-aware because these signals solve different parts of the benchmark.
13. Conclusion: The benchmark provides a controlled way to measure retrieval differences before they are hidden inside end-to-end patch generation.Its focus is the context-acquisition stage preceding patch generation.
A. Repository and Language Coverage: The corpus covers 13 Python repositories, three each for Go, Rust, and TypeScript, two Java repositories, and one JavaScript repository.Each repository receives one primary language label, including multilingual repositories; counterfactual no-gold rows count under the searched corpus repository.
A. Repository and Language Coverage: Table 22 reports repository, primary language, and per-task counts for code2test, comment2context, trace2code, edit2ripple, natural no-gold, and counterfactual wrong-repository samples.The table abbreviates these categories as C2T, C2C, T2C, E2R, Nat., and Ctr.

B. Released Sample Schema and Examples

The released benchmark uses one JSON object per line with shared repository, commit, query, gold, and evidence structure, while task-specific examples define distinct retrieval targets and abstention cases. Examples span four positive tasks plus natural no-gold abstention, where evidence can establish that no repository-local file should be returned.

Released JSONL schema: Each JSONL record identifies its task, repository, base commit, query, gold target, and optional file or span-level evidence fields.Optional span fields refine supporting regions without changing the file-level gold set.
Positive-task examples: code2test retrieves existing tests from an implementation change, while comment2context retrieves implementation context for a reviewed test but excludes the reviewed test from main gold.The code2test example has two hidden test paths, whereas comment2context requires tokio/src/sync/mpsc/chan.rs as context for a reviewed tokio/tests/sync_mpsc.rs change.
Positive-task examples: trace2code maps a reproduced test failure to its root-cause implementation file, exemplified by debug_test.go exposing debug.go after a compile failure.The example records debug.go as the root-cause file and debug_test.go as the related test.
Positive-task examples: edit2ripple uses an anchored edit to identify additional affected files, with an API declaration and tests serving as gold.The example anchors on clap_builder/src/parser/parser.rs and identifies clap_builder/src/builder/arg.rs plus tests/builder/default_vals.rs.
Abstention example: Natural no-gold records represent plausible local queries whose evidence shows an external resolution, using an empty file list, no_gold=true, and reason=upstream_dependency.The pytest example is genuinely associated with pytest but points to an upstream CPython defect; maintainer evidence classifies it as requiring no repository-local fix.

C. Embedding Baseline Implementation

The embedding baselines use a common prompt-free, L2-normalized SentenceTransformers interface with right truncation and max chunk-to-file aggregation. This standardization improves protocol uniformity but does not represent each checkpoint’s best task-specific configuration, and the pplx-embed-v1-4b result remains provisional pending tokenizer correction.

Interface and aggregation: The common prompt-free interface improves protocol uniformity but should not be interpreted as each model’s best task-specific configuration.Recommended retrieval prompts are not selected automatically because checkpoints’ default_prompt_name is unset.
Interface and aggregation: All embedding rows use no added prompt, L2-normalized output, right truncation, and max chunk-to-file aggregation.Queries use the benchmark evaluation text without an added instruction; file scores are formed from chunk cosine similarities using maximum aggregation.
Interface and aggregation: Queries use benchmark evaluation text without an added instruction, while tokenizers use right padding and right truncation with the effective SentenceTransformers sequence cap.The cap reconciles checkpoint model and tokenizer settings.
Ranking details: Ties are resolved deterministically by path and chunk identifier before retaining the first occurrence of each file.This makes file-level ranking behavior reproducible when chunk scores tie.
Implementation caveat: The pplx-embed-v1-4b row is provisional because Transformers 4.57.6 warned about fix_mistral_regex=true, but the evaluator did not forward that tokenizer keyword.The reported row corresponds to the tokenizer loaded in the recorded run, and the regex fix was not applied.

D. Full edit2ripple Results

The edit2ripple results report retrieval performance on 58 evidence-backed samples using BCY’s canonical token-based file-packing protocol and stored top-20 rankings. Results at 16k and 32k are lower bounds when prefixes underfill budgets, while the pplx result remains provisional.

BCY applies canonical token-based file packing at each budget using stored top-20 file rankings.
4k and 8k budgets are fully packed in recorded runs, whereas 16k and 32k values are lower bounds when stored prefixes underfill budgets.
The pplx run is provisional pending a corrected-tokenizer rerun.
58 evidence-backed samples comprise the edit2ripple retrieval evaluation.

E. Closed-Tool Context Acquisition Diagnostic

The closed-tool diagnostic extends Agent Retrieval Bench from static file ranking to evaluator-mediated interactive context acquisition using Docker-isolated policies. It supports trajectory-based evaluation while revealing substantial task heterogeneity and limited interpretive scope.

Protocol: The Docker-isolated policy accesses only the prompt, action schema, and prior observations, with the benchmark corpus unavailable inside its container.The protocol requires iterative repository-information requests through evaluator-mediated tools.
Protocol: 287 samples span code2test, comment2context, and trace2code in the closed-tool diagnostic.Both scripted grep and Codex use evaluator-mediated repository access.
Task variation: Trace2code recovers substantially more context under interactive tool use, while comment2context remains difficult even with adaptation.The task breakdown is diagnostic rather than a replacement for the static leaderboard.
Trajectory analysis: The main prefix gain occurs between two and four tool calls, after which acquired-context scores saturate.This post-hoc diagnostic measures how quickly completed trajectories acquired useful files rather than rerunning them under smaller budgets.
Limitations: The diagnostic validates interactive retrieval evaluation but does not establish deterministic grep as a meaningful agent baseline or represent all production-agent behavior.The scripted-grep row is a non-adaptive lower bound and tool-path smoke test.