PaperlayerView exact revision on arXiv

Source-linked AI summary

ContextBench: A Benchmark for Context Retrieval in Coding Agents

Han Li, Letian Zhu, Bohan Zhang, Rili Feng, Jiaming Wang, Yue Pan, Earl T. Barr, Federica Sarro, Zhaoyang Chu, He Ye

arXiv:2602.05892v3cs.LG

TL;DR

Existing issue-resolution benchmarks mainly measure whether coding agents succeed, leaving their intermediate context retrieval insufficiently evaluated. CONTEXTBENCH addresses this with human-annotated gold contexts and automated trajectory-based retrieval metrics, finding recall-heavy behavior, limited benefits from complex scaffolding, and gaps between retrieved and utilized context.

  • Problem

    Existing benchmarks emphasize final task success and provide limited insight into the code context agents retrieve during problem solving.

  • Method

    CONTEXTBENCH combines human-annotated gold contexts with automated trajectory tracking to measure context recall, precision, F1, and process dynamics.

  • Results

    Recall-heavy retrieval, limited gains from sophisticated scaffolding, and gaps between retrieved and utilized context are consistent findings across evaluated LLMs and coding agents.

  • Takeaways & Limitations

    The benchmark supplies intermediate context signals for more transparent, process-oriented evaluation of coding-agent reasoning and context utilization.

  • Takeaways & Limitations

    Gold contexts are derived from human annotations based on a gold patch, but multiple semantically equivalent patches may yield different valid contexts.

Abstract

from arXiv · show

LLM-based coding agents have shown strong performance on automated issue resolution benchmarks, yet existing evaluations largely focus on final task success, providing limited insight into how agents retrieve and use code context during problem solving. We introduce ContextBench, a process-oriented evaluation of context retrieval in coding agents. ContextBench consists of 1,136 issue-resolution tasks from 66 repositories across eight programming languages, each augmented with human-annotated gold contexts. We further implement an automated evaluation framework that tracks agent trajectories and measures context recall, precision, and efficiency throughout issue resolution. Using ContextBench, we evaluate four frontier LLMs and five coding agents. Our results show that sophisticated agent scaffolding yields only marginal gains in context retrieval ("The Bitter Lesson" of coding agents), LLMs consistently favor recall over precision, and substantial gaps exist between explored and utilized context. ContextBench augments existing end-to-end benchmarks with intermediate gold-context metrics that unbox the issue-resolution process. These contexts offer valuable intermediate signals for guiding LLM reasoning in software tasks.

1 Introduction

Existing coding-agent benchmarks emphasize final issue-resolution success while overlooking the intermediate code context agents retrieve. CONTEXTBENCH addresses this gap with expert gold contexts and process-level retrieval metrics, revealing recall-heavy behavior and limited benefits from complex scaffolding.

  • Motivation: Existing benchmarks prioritize end-to-end success rates and do not evaluate the code context retrieved during issue resolution.This limits visibility into how agents arrive at solutions and may permit trial-and-error or test-case overfitting.
  • Benchmark: CONTEXTBENCH contains 1,136 issue-resolution tasks from 66 repositories across 8 programming languages, each annotated with verified gold context.The benchmark provides an intermediate basis for evaluating context retrieval beyond final task success.
  • Evaluation: The evaluation framework records agent trajectories and compares retrieved and gold contexts using recall, precision, and F1 at file, block, and line levels.The shared coordinate system aligns repository file paths, AST blocks, and line ranges for interval-overlap measurement.
  • Findings: Across evaluated LLMs and coding agents, retrieval consistently favors broad recall over precision, introducing noise and yielding limited precision and F1 gains.The benchmark evaluates four frontier LLMs and five coding agents.
  • Findings: More complex retrieval scaffolds do not consistently outperform a simple baseline, while substantial gaps remain between retrieved and utilized context.These findings motivate process-oriented signals for understanding and improving coding-agent behavior.

2 CONTEXTBENCH: A Benchmark to Assess Context Retrieval Ability for Coding Agents

CONTEXTBENCH is constructed by deduplicating issue tasks, selecting challenging cases, and building compact human-verified gold contexts. The resulting repository-level benchmark exposes intermediate retrieval signals for evaluating coding agents.

  • 2.1 Step 1: Task Deduplication: CONTEXTBENCH pools issue-resolution tasks from multiple benchmarks and removes exact and near duplicates using rule-based and embedding-based detection.The source pool contains 4,497 tasks before deduplication.
  • 2.2 Step 2: Task Selection: Task selection ranks candidates by agent solvability, edit scope, and edit dispersion to emphasize difficult, context-intensive issues.These metrics target tasks with few successful agents, larger patches, and edits spread across distant repository regions.
  • 2.2 Step 2: Task Selection: From 1,500 candidates, manual review removes semantically trivial cases and produces a final set of 1,136 tasks.Excluded examples include broad variable renaming and formatting changes that do not affect program logic.
  • 2.3 Step 3: Expert Annotation: Expert annotators trace dependencies from ground-truth patch locations through invocations, inheritance, control flow, data flow, and related surrounding code.Guidelines and examples standardize dependency tracing and artifact recording.
  • 2.3 Step 3: Expert Annotation: Annotated contexts are validated through LLM-generated patches, compactness review, inter-annotator agreement, and iterative reannotation when verification fails.A context is considered sufficient when at least one generated patch passes the official test suite.
  • 2.4 CONTEXTBENCH Statistics: The benchmark spans 66 repositories and 8 programming languages and introduces human-verified gold contexts as intermediate retrieval signals.Its repository-level design complements final task-resolution evaluation.

3 Can Coding Agents Retrieve Effective Contexts to Resolve Issues?

ContextBench evaluates coding-agent context retrieval as an intermediate process rather than relying only on final issue-resolution success. Results show that sophisticated scaffolding offers no consistent advantage, LLMs trade precision for recall, retrieval strategies affect performance and cost, and retrieved context is often not used.

  • RQ1: Benchmarking Coding Agents: Sophisticated agent scaffolding does not necessarily improve context retrieval, as more complex agents often perform no better than the simple mini-SWE-agent baseline.The authors attribute this pattern to potential over-engineering in current retrieval designs.
  • Benchmark setup: ContextBench compares five coding agents and four LLMs using retrieval metrics alongside Pass@1 issue-resolution performance.The benchmark reports recall, precision, and F1 across file, block, and line levels, plus retrieval-process measures.
  • RQ2: Benchmarking Large Language Models: Block-level F1 remains below 0.45 and line-level F1 below 0.35, indicating that frontier LLMs still struggle to retrieve effective code context.LLMs frequently retrieve relevant information while introducing substantial irrelevant content.
  • RQ2: Benchmarking Large Language Models: GPT-5 achieves higher block- and line-level recall but lower precision and overall F1 than Claude Sonnet 4.5.This result illustrates the trade-off between broad retrieval and accurate evidence selection.
  • RQ3: Analysis of Context Retrieval Patterns: LLMs balance retrieval rounds against context granularity differently, and more balanced strategies tend to produce better line-level retrieval and issue-resolution performance.GPT-5 uses 5.87 rounds and 119.29 lines per step, whereas Devstral 2 uses 22.16 rounds and 11.98 lines per step; Claude Sonnet 4.5 is described as more balanced.
  • RQ4: Analysis of Context Retrieval Dynamics: Retrieved context is substantially underused during final patch generation, so successful intermediate retrieval does not always translate into successful issue resolution.The largest context loss is observed for Gemini 2.5 Pro and Devstral 2, while retrieval dynamics also expose efficiency–redundancy trade-offs.
  • RQ5: Analysis of Gold Context Robustness: Gold contexts are derived from a gold patch, but their reliability is tested against semantically equivalent patches because one issue may have multiple valid implementations.A case study of 82 tasks found high consistency across patch-conditioned gold contexts.

4 Related Work

Related work establishes repository-level benchmarks for end-to-end patch success, develops diverse coding-agent architectures, and analyzes trajectories for failure diagnosis. Context retrieval remains a shared concern across these lines of research.

  • Repository-Level Code Evaluation Benchmarks: Repository-level benchmarks evaluate whether systems produce test-passing patches for real-world issues in executable codebases.SWE-bench and its variants form the de facto standard for end-to-end issue-resolution evaluation.
  • Coding Agents: Coding agents use either fixed pipeline stages with controlled context selection or open-ended tool interactions that dynamically search, inspect, and modify repositories.Despite architectural differences, context retrieval remains a shared bottleneck.
  • Trajectory Analysis and Failure Diagnosis: Trajectory-analysis studies diagnose agent failures through manual inspection, error taxonomies, and statistics over execution logs.These studies improve understanding of failures such as reasoning deadlocks, tool misuse, runtime exceptions, and context-related errors.

5 Conclusion

ContextBench introduces process-level evaluation of context retrieval in coding agents, extending issue-resolution benchmarks with gold contexts and retrieval-centric metrics. Its evaluation reveals limited scaffolding benefits, recall-over-precision behavior, and gaps between retrieved and utilized context.

  • ContextBench augments end-to-end issue-resolution benchmarks with human-annotated gold contexts and retrieval-centric metrics.The benchmark enables analysis beyond final success rates.
  • Evaluations across four frontier LLMs and five coding agents reveal limited benefits from complex agent scaffolding.
  • LLM coding agents consistently favor recall over precision, while retrieved context substantially exceeds utilized context.
  • Intermediate context signals are presented as a promising direction for improving and evaluating LLM-based software engineering systems.The benchmark supports transparent, reproducible, and fine-grained assessment of codebase interaction.

A Details on Data Filtering

The data-filtering pipeline progressively removes redundancy, low-information tasks, difficult-to-assess cases, and annotation artifacts. It combines automated deduplication and difficulty filtering with manual quality review and standardized context-extraction procedures.

  • Task filtering: Approximately 4.4k source tasks are aggregated from established software-engineering benchmarks as an intentionally broad initial pool.
  • Task filtering: ID-based deduplication removes identical or invalid issue/PR identifiers, leaving 3.9k tasks.
  • Task filtering: Embedding-based semantic deduplication filters task pairs whose cosine similarity exceeds 0.90.
  • Task filtering: Difficulty filtering excludes trivially solvable, low-dispersion, or limited-scope tasks to preserve evaluative value.
  • Task filtering: Manual annotation review retains tasks with complete, precise, and suitable gold contexts, producing a final benchmark of 1.2k tasks.
  • Context extraction: The context-extraction framework uses structured prompts, verification functions, and regular expressions to produce consistent machine-readable context records.

C.1.4 Cross-Model Evaluation Results

The cross-model evaluation applies a standardized context-extraction protocol across four models and multiple agent frameworks. Models generally produced structured, machine-verifiable contexts, although DevStral2 occasionally deviated from the output protocol.

  • Cross-model evaluation: Four models—Claude Sonnet 4.5, GPT-5, Gemini 2.5 Pro, and DevStral2—were evaluated under a standardized extraction protocol.
  • Observed limitation: DevStral2 occasionally deviated from the standardized output protocol during evaluation.The corresponding anomalies are discussed with raw interaction traces.
  • Context extraction protocol: The protocol requires agents to provide inspected source-file paths and line ranges before final submission.
  • Cross-agent adaptation: Prompt templates and verification routines preserve a uniform, machine-readable <PATCH CONTEXT> structure across heterogeneous agents.

C.2.2 SWE-Agent Implementation

The SWE-Agent implementation enforces structured context reporting before submission and adapts extraction across agent workflows. Agentless enhancements additionally make repair prompts repository- and language-adaptive through dynamic extension detection and universal examples.

  • SWE-Agent implementation: SWE-Agent requires a mandatory <PATCH CONTEXT> block listing every examined source file and exact line range before invoking submission.
  • Context format: The extraction protocol permits only absolute file paths and line ranges, excluding explanations, debugging output, tests, and other non-code content.
  • Cross-agent adaptation: OpenHands records source exploration with machine-parseable file-range blocks only when commands print source-code content.
  • Agentless enhancements: Agentless detects the repository’s primary programming language from file-extension frequency and injects the corresponding extension into prompt examples.
  • Agentless enhancements: These adaptations aim to eliminate language-specific fault-localization bias and support cross-language applicability without manual configuration.
  • Agentless enhancements: Language-agnostic DIFF examples replace domain-specific examples with universal Hello World modification patterns adapted to the detected language.

C.2.5 Framework Generalization Results

The standardized context-extraction framework produced structured patch-context outputs across five heterogeneous coding agents, supporting reproducible downstream evaluation. Its block-level alignment uses definition-level AST symbols, while gold contexts are verified for sufficiency rather than repair performance.

  • Framework Generalization: All five evaluated agents produced the corresponding <PATCH CONTEXT> outputs under the standardized extraction framework.The framework is described as robust across heterogeneous agent architectures and constrains outputs to a structured, machine-verifiable format.
  • Framework Generalization: Consistent patch-context extraction supports reproducibility, auditing, and downstream evaluation across differing prompts, workflows, and system architectures.The reported adaptation preserves the fidelity and completeness of extracted context across these agent differences.
  • Gold Context Scope: Gold contexts are compact, human-annotated references verified to be sufficient for issue resolution, but they are not claimed to be globally minimal.Precision and F1 therefore measure agreement with a compact, verified reference rather than penalizing every additional context region that may help.
  • Block-Level (AST) Alignment: Block-level evaluation standardizes AST units as definition-level functions, methods, classes, interfaces, or traits mapped to canonical file-and-line spans.This design avoids treating arbitrary low-level AST nodes as evaluation units and improves comparability across languages.
  • Context Verification: Context sufficiency is established when at least one of five independently generated patches passes the complete official test suite.This existence-based criterion checks whether a correct solution is possible from the context, rather than measuring consistent repair performance.
  • Context Verification: Verification is a feasibility-oriented validation of annotated contexts, not a benchmark of language-model repair performance.Correctness is determined exclusively by the official test suite, independently of model confidence or reasoning traces.

H Details on Evaluation Metrics

The evaluation measures how agents retrieve gold code context both in final outputs and throughout execution, using multiple granularities and process-oriented metrics. It also examines retrieval speed, redundancy, evidence retention, and robustness across alternative patches.

  • Final Context Metrics: Context recall measures the proportion of gold context retrieved, while context precision measures the retrieved context overlapping with gold context.F1 balances these complementary dimensions as their harmonic mean.
  • Process Metrics: AUC-Cov summarizes how quickly agents reach high cumulative gold coverage during execution.Higher values indicate earlier retrieval of critical gold evidence with fewer observation steps.
  • Process Metrics: Redundancy measures the fraction of newly retrieved context that overlaps with previously observed elements across execution steps.Higher redundancy indicates repeated inspection, whereas lower redundancy indicates more novel context acquisition.
  • Process Metrics: Evidence Drop measures gold evidence observed during exploration but not retained in the final aggregated patch context.Lower Drop indicates better consolidation and preservation of discovered evidence.
  • Robustness Analysis: Gold-context robustness is assessed by averaging pairwise Jaccard similarity across contexts derived from multiple semantically equivalent patches.The case study covers 82 instances with two or three test-passing patches per instance.
  • Evaluation Setup: Metrics are averaged across evaluated tasks and computed at file, block, and line granularities to provide fine-grained behavioral analysis.The pipeline also records tool calls, retrieved contexts, and edit trajectories for manual inspection.

I.1 Case 1: Prometheus Agent – Incomplete Class Semantics Retrieval

The case studies show that incomplete context retrieval can produce semantically incorrect patches or derail all later localization stages. Failures arise from missing class semantics, incorrect initial file localization, and search strategies that tunnel into one implementation.

  • Prometheus Agent – Incomplete Class Semantics Retrieval: The agent retrieved operational methods but missed CaseInsensitiveDict’s init and update methods, producing an incomplete class context.The gold fix instead filters merged_setting after merging while preserving proper initialization.
  • Prometheus Agent – Incomplete Class Semantics Retrieval: Missing CaseInsensitiveDict constructor semantics led the agent to pass a generator where initialization required a mapping with .items().The resulting issue caused 26 test failures involving headers, authentication, and cookie persistence.
  • Localization Cascade: For django-11630, failure to retrieve model checks.py caused subsequent element and edit localization to operate on the wrong files.The gold patch modifies model checks.py, not options.py, to conditionally emit warnings when database routers are configured.
  • Search-Induced Context Tunneling: For django-11138, grep results for a MySQL-specific SQL function anchored search on MySQL and prevented horizontal exploration of SQLite and Oracle modules.The case identifies this as search-induced context tunneling in modular architectures.

J Cross-Language Radar Distributions for File/Block/Line Metrics

Figure 6 presents localization quality across eight programming languages, separating file-, block-, and line-level granularity and reporting precision, recall, and F1.

  • Figure 6: The radar figure uses one axis per programming language to show localization-quality values under a common evaluation protocol.Columns encode File, Block, and Line granularity, while rows encode Precision, Recall, and F1.

K Potential Data Contamination and Protocol Misalignment in Devstral2

The Devstral2 traces exhibit behavior that may reflect data contamination or protocol misalignment, including precomputed plans, hallucinated states, and premature submission. These artifacts can distort benchmark evaluation and motivate contamination detection and protocol-compliance checks.

  • Observed Behaviors: Devstral2 often produced complete structured solution plans immediately after reading the issue, without waiting for framework feedback.The model also hallucinated intermediate system states during execution.
  • Potential Explanations: Possible explanations include exposure to SWE-bench artifacts during training and mismatch between interactive training protocols and mini-SWE-agent execution semantics.The text presents these as plausible explanations rather than established causes.
  • Evaluation Consequences: Benchmark contamination can artificially inflate performance, while hallucinated states make genuine reasoning difficult to distinguish from spurious pattern completion.Premature submission also disrupts iterative correction and reduces the reliability of automated correctness assessment.
  • Mitigations: The paper recommends strict data hygiene, protocol-aligned training, contamination detection, and protocol-compliance checks for fair and reproducible evaluations.These recommendations are presented as future safeguards for public software-engineering benchmarks.
Loading 2602.05892v3…