PaperlayerView exact revision on arXiv

Source-linked AI summary

The Recall Trap: A Recall-Maximizing Retriever Configuration Reduces Issue Resolution in Fixed-Budget Code Context

Alexander Adkins, Teimuraz Trapaidze

arXiv:2608.14838v1cs.SEcs.CLcs.IR

TL;DR

Retrieval systems often optimize recall under fixed context budgets, but evidence that higher retrieval recall improves code-repair success remains limited. This paper evaluates a file-deduplication flag in controlled, execution-graded code repair and finds that disabling it improves single-shot issue resolution, with the effect reversing for BM25 retrieval.

  • Problem

    Under fixed context budgets, retrieval systems favor file recall, but whether higher recall improves downstream code-repair success remains insufficiently established.

  • Method

    The study compares otherwise identical pack-style retrieval configurations with and without one-chunk-per-file deduplication in controlled, execution-graded code repair.

  • Results

    Disabling file deduplication improves single-shot issue resolution on two confirmatory model arms, but the effect reverses for lexical BM25 retrieval.

  • Takeaways & Limitations

    At tight fixed context budgets, packing policies should be evaluated against code-repair success rather than retrieval recall alone.

  • Takeaways & Limitations

    The multi-file case is an underpowered null, so cross-file workloads are required to assess that boundary.

Abstract

from arXiv · show

Retrieval components for code assistants are tuned against retrieval metrics: a configuration that raises recall@k is adopted, and downstream task success is assumed to follow. We report a controlled case study in code repair, not a new phenomenon but a deployed-flag, execution-graded instance of the known relevance-diversity and objective-mismatch tradeoff (Levy et al., 2025). On SWE-bench Verified we inject a retriever's hits as a fixed 12-slot context pack with no search tools and toggle one flag (one-chunk-per-file deduplication) on an otherwise identical stack. The flag is the higher-recall configuration (gold file present in 0.878 of served packs against 0.806 disabled), yet disabling it, trading file breadth for within-file depth, raises the single-shot resolve rate: gpt-5.6-sol +7.6pp (39.2% to 46.8%, n=500, McNemar exact p=0.0003), and a pre-registered open-weights replication any reviewer can re-run (Qwen3.6-27B, +3.6pp, n=499, p=0.0133); both survive repository-clustered inference. The gain tracks within-file anchor dose, and a random-chunk control refutes an argmax-selection artifact. We map where it holds: it reverses on a lexical BM25 retriever (-3.2pp, significant cross-paradigm interaction), is not detected under unrestricted-Read agents (a powered null), and across four languages (SWE-PolyBench, N=617) is positive but not significant (+2.6pp, p=0.056), a mapped boundary rather than a confirmed extension. Operationally, at a tight fixed budget: do not hard-deduplicate by file, and A/B packing policies against the task, not the metric the flag was tuned to.

1. Introduction

In a fixed 12-slot, single-shot code-repair setting, a standard one-chunk-per-file deduplication flag improves file recall but reduces execution-graded issue resolution by favoring breadth over within-file depth. The paper presents this as a controlled, mechanism-linked map of where that recall–resolution inversion holds and where it does not.

  • Mechanism: The recall-maximizing configuration reduces resolution because zero-sum packing favors shallow breadth—about 12 files with one chunk each—over deep coverage of about 5 files.Every slot assigned to one file is unavailable to another, making retrieval allocation zero-sum under the fixed budget.
  • Primary finding: +3.6pp: a pre-registered Qwen3.6-27B replication improves from dedup-ON 9.2% to dedup-OFF 12.8%, with McNemar exact p=0.0133.The replication is open weights, independently re-runnable, and its result survives conditional inference.
  • Boundary map: −3.2pp: the effect reverses with a lexical BM25 retriever, producing a significant cross-paradigm interaction rather than a universal document-count law.The effect tracks within-file anchor dose, with BM25’s depth arm at +0.029 versus fusion’s +0.089.
  • Boundary map: +2.6pp, p=0.056: the pre-registered SWE-PolyBench test across four languages is directionally positive but not significant.The authors treat this as a mapped boundary, not a confirmed multilingual extension.
  • Mechanism: 3.0% against 6.6%, against depth 12.4%: random reselection of non-argmax chunks performs worse than the retained argmax and refutes an argmax-selection artifact.The control was designed before generation but was not pre-registered.

2. Background and Problem Setting

The study fixes retrieval context at 12 ranked code chunks and requires a single-shot patch from that pack, isolating retrieval configuration as the manipulated factor. File deduplication reallocates slots toward distinct files and maximizes file-level recall, but changes several aspects of the served ranking beyond within-file depth.

  • Fixed-budget pack regime: The solver receives the top-12 ranked code chunks and must emit the full patch in one completion from that pack alone.This fixed-budget, single-shot regimen isolates retrieval configuration as the only manipulated factor.
  • The dedup flag and what it confounds: With deduplication ON, the pack contains about 12 distinct files with one chunk each; OFF typically serves about 5 files with multiple chunks per file.The candidate pool and scoring function are identical; the flag reallocates slots.
  • The dedup flag and what it confounds: The deduplication comparison is a compound treatment that changes file count, chunks per file, served rank and score distribution, and which within-file chunk is kept.The study does not claim to isolate within-file depth from the accompanying change in file count.
  • Why dedup is the recall-maximizing configuration: 0.878 versus 0.806: gold-file presence is higher in dedup-ON than dedup-OFF served packs (n=500), confirming the recall-maximizing premise on model-seen packs.The corresponding retrieval-index evaluation changes from 0.666 to 0.817.

3. Related Work

This work is a controlled causal code-repair case study of established relevance–diversity and retrieval–generation objective mismatches, adding a deployed fixed-budget packing toggle. It connects prior depth–breadth, fault-localization, and retrieval-granularity findings while locating a budget- and retriever-dependent boundary.

  • 3.1 Diversity objectives and the QA/IR prior art we build on: The paper contributes a controlled causal code-repair instance of retrieval-metric objective mismatch by toggling a deployed default under execution grading, not by claiming a new phenomenon.Its embedding-retrieval boundary is not predicted by a universal document-count interpretation.
  • 3.1 Diversity objectives and the QA/IR prior art we build on: File deduplication instantiates diversity-aware retrieval, trading per-item relevance and within-file depth for coverage of distinct file-level aspects.This extends maximal marginal relevance, α-nDCG, novelty/diversity evaluation, and xQuAD, but the depth-versus-breadth tradeoff itself is established prior art.
  • Fault localization and retrieval granularity: Fault-localization literature already identifies right-file-wrong-lines failures; this paper’s contribution is linking the packing flag causally to served-pack line localization and resolution.Anchor dose is positioned as a served-pack instance of known line-level localization recall, not as a new metric.
  • Budget-dependent regime boundary: 70.4% to 78.6% resolution in prior work is compatible with this paper because that study uses tens to hundreds of thousands of tokens, whereas this study uses 12 slots.Prior data show micro-precision flat at about 10% despite 84 to 92% file-recall, retrieving roughly ten times the gold set.
  • Budget-dependent regime boundary: Within the tested deployable slot-budget regime, depth advantage grows with K, while any breadth crossover lies outside the tested tens-of-thousands-token regime.Reported weak-open A3B gains are +2.0/+4.2/+6.2pp at K=4/12/40.

4. Method

The method isolates one file-level deduplication flag in an otherwise identical multi-signal retriever, then evaluates fixed 12-slot packs in single-shot, tool-free code repair. It supplements paired per-instance testing with repository-clustered inference and explicitly treats unrestricted-read agents as a separate boundary regime.

  • Retriever configuration: The retriever fuses dense, lexical, and graph signals, while RAGD_QUERY_DEDUP_FILE alone controls whether the top-12 ranking is limited to one chunk per file.Both arms use the same code path, index, embedder, ranking weights, chunking, and slot budget.
  • Pack construction: Dedup-ON produces about 12 distinct files with one chunk each, whereas dedup-OFF produces about 5 files with multiple chunks per file.Dedup-ON is also the file-level recall@k maximizer: 0.666 to 0.817 in index evaluation and 0.878 against 0.806 gold-file presence in served packs.
  • Evaluation harness: The primary runner makes one API call per instance, supplies the issue and 12-slot pack, and requires a single SEARCH/REPLACE completion without tools, turns, or reading beyond the pack.Edits are applied to pristine per-arm isolated checkouts, separating pre-fetched retrieval from generation.
  • Boundary condition: The boundary analysis uses a claude CLI with pre-injected packs, Read/Edit/Write, no search tools, at most 10 turns, and unrestricted Read.Because iterative Read is available, the 12-slot budget no longer binds; this is a separate no-search agentic regime whose absolute rates are not comparable to the single-shot results.
  • Statistical analysis: McNemar’s exact test compares paired per-instance resolve outcomes, with repository-cluster bootstrap, leave-one-repository-out deltas, and repository-level discordant counts reported additionally.The primary gpt result survives clustering, while the DENSE-1 dense control weakens.

5. Results … 5.3 Retriever scope: two embedding retrievers, and a lexical reversal

Under a fixed single-shot context budget, deduplication-off improves resolution for the primary embedding-retriever evaluations, chiefly through answer quality, but the effect reverses for BM25. The evidence supports a bounded retriever-scope claim rather than a universal document-count law.

  • 5.1 Primary: single-shot cross-model table, with clustered inference: +6.3pp is the Qwen3.6-27B complete-case effect (n=269, p=0.0095), while Manski bounds remain +3.0 to +3.6pp under alternative treatment of truncated empties.Symmetric truncation rates and discordant wins indicate the advantage is associated with within-file depth rather than differential censoring.
  • 5.2 Attempt-rate decomposition (corrected): 88 to 94% of the gain for gpt-5.6-sol and DENSE-1 comes through correct-given-attempt quality, whereas DeepSeek’s producibility channel contributes about 24%.For DeepSeek, 20 of 38 logged failures produced text but no edit because the SEARCH/REPLACE anchor did not match served context.
  • 5.3 Retriever scope: two embedding retrievers, and a lexical reversal: +5.7pp favors dedup-OFF over dedup-ON for the dense-only DENSE-1 control (41.5% to 47.2%, n=494, raw p=0.0097).Its clustered 95% CI is [−0.9, +8.6]pp and includes zero, so it is directional replication evidence rather than confirmatory evidence.
  • 5.3 Retriever scope: two embedding retrievers, and a lexical reversal: −2pp favors dedup-ON over dedup-OFF for the exploratory BM25 pilot (76% to 74%, n=100, p=0.69), indicating the reversal direction.The powered BM25 analysis was designed to test whether the embedding-retriever effect generalized across retrieval paradigms.
  • 5.3 Retriever scope: two embedding retrievers, and a lexical reversal: +10.8pp for fusion minus BM25 and +8.9pp for dense minus BM25 both have confidence intervals excluding zero, supporting a significant cross-retriever interaction.The equivalence test at a 5pp margin does not support equivalence, but the interaction is the primary evidence for a difference.
  • 5.3 Retriever scope: two embedding retrievers, and a lexical reversal: 0.69 versus 0.61 is BM25’s served-pack gold-file recall for dedup-ON versus dedup-OFF, yet BM25 resolves at 34 to 37%, comparable to fusion.The reversal therefore is not explained by a recall-floor artifact; BM25 remains a functional retriever.
  • 5.3 Retriever scope: two embedding retrievers, and a lexical reversal: +0.029 is BM25’s within-file gold-line coverage gain (0.059 to 0.088), versus fusion’s +0.089, so the anchor-dose increase barely materializes under BM25.The paper therefore limits the positive effect to the two embedding retrievers tested and does not claim a clean universal embedding-versus-lexical boundary.

5.4 Mechanism: anchor dose (an oracle diagnostic and partial mediator) and a selection … 5.7 Failure taxonomy and the parse-apply-test funnel

The effect is consistent with greater within-file anchor dose, which explains only part of the advantage and is not isolated from file-count changes. It disappears under unrestricted Read, survives failure-funnel checks, and the withdrawn concentration sweep supports no inverted-U claim.

  • 5.4 Mechanism: anchor dose (an oracle diagnostic and partial mediator) and a selection: Dedup-OFF beats dedup-ON by +7 to +13.6pp among anchor-visible instances, making anchor dose a partial mediator, descriptively about 40% of the effect.The residual remains unexplained, and the analysis stops short of a full causal claim.
  • 5.4 Mechanism: anchor dose (an oracle diagnostic and partial mediator) and a selection: 82% of dedup-OFF wins occur when both packs contain the gold file, with resolution 44.5% versus 52.0% (+7.5pp), supporting a within-file-depth reading.This stratification is observational and supportive rather than an identification.
  • 5.5 Boundary condition: the effect is not detected under unrestricted Read: 65.9% versus 64.5% (Δ −1.4pp, p=0.45) under unrestricted Read is a powered null, so the fixed-pack effect is not detected in free-reading agents.The result is bounded below the single-shot effect rather than establishing a reversal.
  • 5.6 Concentration sweep (withdrawn, confounded): The earlier inverted-U concentration sweep was withdrawn because it mixed confounded models; within a single model, served points were flat.No numeric optimum or inverted-U claim is made.
  • 5.7 Failure taxonomy and the parse-apply-test funnel: Resolve rate rises monotonically from 15.0% to 37.3% to 67.9% across failure classes, while Class B’s 74.8% share tracks its 72.2% prevalence.This argues against describing failures as dominated by an allocation problem.
  • 5.7 Failure taxonomy and the parse-apply-test funnel: Among patches that applied, depth still wins test-pass: DeepSeek 23.9% against 31.9% (+8.0pp), DENSE-1 48.4% against 55.1% (+6.7pp), and gpt-5.6-sol 47.9% against 53.1% (+5.2pp).Thus the effect is not merely that depth produces more anchorable patches.

5.8 Contamination-dosage stratification · 5.9 Budget×dedup dose-response: the trap grows with the slot budget · 5.10 Multilingual generality: a mapped boundary (SWE-PolyBench)

The deduplication trap is not explained by repository popularity and grows with the clean slot budget, while multilingual results remain directionally positive but below the preregistered detection threshold. Together, these analyses define a scoped effect with mapped limits rather than a confirmed cross-language generalization.

  • 5.8 Contamination-dosage stratification: ρ≈+0.12: repository popularity is essentially uncorrelated with the deduplication effect, whose largest values span roughly 4k to 60k stars.Astropy shows about +9pp at about 4k stars, while scikit-learn shows about +10pp at about 60k; some high-star repositories are null.
  • 5.8 Contamination-dosage stratification: +3.4 to +6.4pp: the effect remains substantial and mostly significant in the low-popularity stratum, while the median high-minus-low tilt is only about +2.7pp.The popularity proxy has restricted range because all repositories are popular public Python projects, so the analysis cannot exclude shared contamination.
  • 5.9 Budget×dedup dose-response: the trap grows with the slot budget: Δ +4.80pp at K=4 and Δ +9.20pp at K=40: dedup-OFF’s depth advantage roughly doubles as the clean slot budget increases.The clean Qwen3.6-27B runs resolve 43/499 versus 67/499 at K=4 and 44 versus 90 at K=40; the within-model difference-in-differences is ΔΔ = +4.40pp, p=0.0003.
  • 5.9 Budget×dedup dose-response: the trap grows with the slot budget: 6.8 to 7.4% versus 8.8 to 13.0%: in an exploratory weak-open sweep, dedup-ON stays flat across budgets while dedup-OFF rises.The analysis supports a monotone Δ-trend, but its absolute levels are truncation-contaminated and the K=4 null is inconclusive.
  • 5.10 Multilingual generality: a mapped boundary (SWE-PolyBench): Δ = +2.59pp, exact McNemar p = 0.056: pooled across two models and four languages on SWE-PolyBench, the trap direction is positive but nonsignificant.The realized pool was N=617 model×instance pairs, below the planned N≈1000 because of the gold-validity gate.
  • 5.10 Multilingual generality: a mapped boundary (SWE-PolyBench): 95% CI [0.0, +5.3]pp: instance-clustered bootstrap leaves the multilingual verdict unchanged, with an interval reaching zero.Neither nominally significant cell survives Holm correction; the authors therefore report multilingual behavior as a mapped boundary, not confirmed generalization.
  • 5.10 Multilingual generality: a mapped boundary (SWE-PolyBench): −1.3pp, p=1.00: gpt-5.6-sol is flat on SWE-PolyBench Python despite +7.6pp, p=0.0003, on SWE-bench Verified Python.This contrast indicates that language alone does not explain the difference; pack composition and the instance pool are implicated instead.

6. Discussion

The discussion presents the depth advantage as a registered, clustering-robust finding while delimiting its interpretation through token-size, construct, locality, and generality analyses. Operationally, it recommends task-level A/B testing rather than hard file deduplication, but scopes that advice to fixed-slot, embedding-retrieval settings tested here.

  • Evidence status: The surviving claim rests on two confirmatory single-shot arms that survive repository clustering and does not depend on any single arm.These are gpt-5.6-sol and the pre-registered Qwen3.6-27B replication; DeepSeek is supportive, while DENSE-1 is directional with an interval including zero.
  • Mechanism and confounds: 68 to 94% of every arm’s gain comes through the quality channel, while only DeepSeek has a material producibility share.The parse-apply-test funnel shows depth wins on test-pass conditional on applying, including a logged gpt replication of +5.2pp.
  • Mechanism and confounds: The effect survives the token-size objection: among 179 instances where dedup-OFF is smaller, gpt discordance is 23/9, p=0.020, and Qwen discordance is 18/4, p=0.0043.Native-tokenizer recounts reproduce the gap, and discordance direction is uncorrelated with token delta (MWU p≈0.99).
  • Boundary conditions: +2.6pp, p=0.056 is the pooled multilingual result, so SWE-PolyBench maps a boundary rather than confirming generality.The core evidence remains scoped to SWE-bench Verified, a 12-slot budget, the reported chunker, and tested systems.
  • Operational implications: Practitioners should not hard-deduplicate by file at a tight slot budget and should A/B packing policies against task success rather than the tuning metric.The recommendation applies to the tested embedding retrievers and single-file-dominated Python, single-shot workloads; BM25, tool-using agents, and regimes far beyond K=40 are outside its support.

7. Conclusion

At a tight fixed context budget, file deduplication increases file recall but reduces single-shot issue resolution across confirmatory model arms. The effect is mechanism-linked and scoped: it reverses under BM25, is absent with unrestricted reading, and supports task-based packing validation.

  • Core result: +7.6pp for gpt-5.6-sol and +3.6pp for Qwen3.6-27B show lower resolution with file deduplication across two confirmatory arms.Both effects survive conditional-estimand and repository-clustered inference.
  • Core result: +4.8pp at K=4 to +9.2pp at K=40 shows the effect grows with slot budget on Qwen3.6-27B.The within-model difference-in-differences is p=0.0003.
  • Interpretation: A random-chunk control refutes the argmax chunk-selection alternative, while the study frames the result as an execution-graded code-repair instance of a known tradeoff.The control does not by itself separate within-file depth from the file-count change.
  • Boundary conditions: The effect reverses under lexical BM25, is not detected under unrestricted-Read agents, and is positive but not significant across four languages on SWE-PolyBench.The pooled cross-language result is +2.6pp, p=0.056; the BM25 interaction is significant.
  • Mechanism: The effect tracks within-file anchor dose, with the quality channel accounting for 68 to 94% of every arm's gain, and concentrates on single-file fixes.The mediator is supported but partial, and small under BM25.
  • Operational implication: For fixed-budget packed-context consumers, do not hard-deduplicate by file; validate packing policies against the task rather than the tuning metric.The BM25 reversal indicates this is not a universal document-count law, though one configuration cannot establish a clean retrieval-paradigm boundary.

Appendix A: Reproducibility handles

The released artifact provides the execution harnesses, served inputs, outputs, grading materials, and analysis scripts needed to audit the reported results. It is archived on Zenodo with per-instance manifests linking table rows to their inputs and recorded model and decoding configurations.

  • Artifact contents: The artifact includes the single-shot runner, agentic harness, served packs, predictions, raw responses, grading reports, inclusion tables, and analysis scripts.The runner applies SEARCH/REPLACE in isolated per-arm checkouts and persists raw responses where logged.
  • Configuration: Model slugs, providers, decoding parameters, embedding identity, chunking settings, and the exact ragd commit are recorded in the artifact and preregistration materials.The embedding is identified as Qwen3-Embedding-8B-Q8_0 via llama.cpp, with chunker size, overlap, and boundary policy specified.
  • Auditability: Every table row maps to its inputs through a per-instance manifest.This provides a direct audit trail from reported rows to their underlying materials.
  • Archive: The archive is available at Zenodo under DOI 10.5281/zenodo.21879550.The archive includes the file inventory and SHA-256 hashes for verifiable-core files, including the paper, preregistration documents, ledger, harnesses, and analysis scripts.

Appendix B: Pre-registration records

The appendix distinguishes genuinely pre-registered Class-1 arms from Class-2 working notes and same-day design-committed analyses. It also limits claims for later replications and controls when their specifications followed the primary result.

  • Provenance classes: Class 1 requires a verified pre-data design commit; Class 2 notes were authored before release but not committed before their runs and are not called pre-registered.Class-2 examples include the random-chunk decision rule and BM25-agnostic TOST commitment.
  • Provenance classes: The GPT logged replication is Class 1 for its replication run and funnel, but it does not upgrade the original headline estimate.Its reporting rule, endpoint hierarchy, and failure branch were fixed pre-data.
  • Timing disclosures: Same-day ordering supported by run logs makes the gpt and DeepSeek arms design-committed confirmatory arms rather than pre-registered ones.The E1–E2 allocation design was committed on 2026-07-14, the day the primary gpt-5.6-sol result was first unblinded.
  • Timing disclosures: DENSE-1 was specified before its own data but after the primary effect, so it is reported as a replication and generality control rather than a pre-registered primary-result control.Its design documents were committed about a day after the primary gpt-5.6-sol result was first unblinded.
  • Timing disclosures: The budget×dedup dose-response has no pre-registration document, and the earlier reference to a budget-sweep pre-registration was withdrawn.The paper uses “pre-registered” only for Class-1 arms and “design-committed” for same-day log-supported ordering.

Appendix C: Reasoning-model incompatibility log

GLM-5.2 and Kimi-k2.7-code were incompatible with the single-shot no-tools regime because of unbounded or mandatory reasoning under a bounded completion budget. Raw-response evidence supported this classification, so they were counted as neither replications nor non-replications.

  • Compatibility classification: GLM-5.2 and Kimi-k2.7-code were incompatible with the single-shot no-tools regime.The incompatibility was documented by the provider.
  • Compatibility classification: Unbounded or mandatory reasoning under a bounded completion budget caused the incompatibility.This applied to the single-shot no-tools regime.
  • Reporting treatment: Raw-response evidence supported counting these models as neither replications nor non-replications.Their results were therefore excluded from both replication categories.

Appendix D: Hypothesis registry

The fifteen-entry registry distinguishes outcomes such as supported, refuted, inconclusive, not-novel, and mapped-boundary findings. Its key dispositions include a BM25 reversal, confirmed replication under the primary setup, and multilingual evidence classified as a mapped edge rather than a confirmation.

  • Registry dispositions: +8.7pp@92 collapses to +1.6pp p=0.45@499, making the result inconclusive at scale and not novel.The registry associates this entry with an oracle-ceiling result.
  • Replication: DENSE-1 reproduces the direction with instance-level p=0.0097, but its clustered confidence interval includes zero.The registry records this branch as withdrawn in direction.
  • Scope controls: −3.2pp reverses the effect for BM25, with a significant interaction limiting scope to the two embedding retrievers tested.This entry refutes the retriever-agnostic hypothesis.
  • Replication: +6.4pp replicates the primary direction with McNemar p=0.002 and clustered CI [+4.9,+11.3], while the funnel adds +5.2pp for dedup-OFF>ON on test-pass|applied.The downgrade branch did not fire, and the funnel result favors depth.
  • Registry dispositions: The registry contains fifteen entries spanning distinct dispositions, including not-novel, refuted, inconclusive, supported, and mapped-boundary outcomes.Underpowered nulls are labeled inconclusive rather than refuted.
  • Boundary findings: Multilingual generality is directionally positive but below the pre-registered significance threshold, so it is reported as a mapped edge rather than a confirmation.This is the registry’s G-4 honest-boundary branch.
Loading 2608.14838v1…