PaperlayerView exact revision on arXiv

Source-linked AI summary

Do not copy and paste! Rewriting strategies for code retrieval

Andrea Gurioli, Federico Pennino, Maurizio Gabbrielli

arXiv:2605.08299v1cs.SEcs.AI

TL;DR

Code retrieval encoders can overemphasize surface syntax, leaving open how much representational shift helps and when online query rewriting is worth its cost. The paper compares three rewriting representations and two augmentation regimes across broad evaluations, finding that NL+QC delivers the largest gains while ΔH predicts QC benefit. It frames rewriting as a cost–benefit decision, especially for lightweight encoders and code-dominant queries.

  • Problem

    Prior code-retrieval rewriting fixes a code-level representation and requires an LLM call per query, leaving the useful representational shift and online cost-benefit boundary unclear.

  • Method

    The paper compares stylistic rephrasing, NL-enriched PseudoCode, and full natural-language transcription under joint query–corpus and corpus-only regimes, using five encoders, three rewriters, and two diagnostics.

  • Results

    +0.51 absolute NDCG@10 is achieved by NL+QC for MoSE-18 on CT-Contest, while corpus-only rewriting degrades retrieval in 56/90 configurations and ΔH predicts QC gains.

  • Takeaways & Limitations

    Rewriting is most effective as a remediation layer for lightweight encoders on code-heavy retrieval, with diminishing returns for stronger encoders or NL-heavy queries.

  • Takeaways & Limitations

    The study excludes closed-source rewriters, focuses on Python-heavy benchmarks, and uses corpus-level diagnostics that do not predict per-query gains.

Abstract

from arXiv · show

Embedding-based code retrieval often suffers when encoders overfit to surface syntax. Prior work mitigates this by using LLMs to rephrase queries and corpora into a normalized style, but leaves two questions open: how much representational shift helps, and when is the per-query LLM call justified? We study a hierarchy of three rewriting strategies: stylistic rephrasing, NL-enriched PseudoCode, and full Natural-Language transcription, under joint query-corpus (QC, online) and corpus-only (C, offline) augmentation, across six CoIR benchmarks, five encoders, and three rewriters spanning independent model families (Qwen, DeepSeek, Mistral). We are the first to evaluate NL-enriched PseudoCode and snippet-level Natural Language as direct retrieval representations, rather than as transient intermediates. Full NL rewriting with QC yields the largest gains (+0.51 absolute NDCG@10 on CT-Contest for MoSE-18), while corpus-only rewriting degrades retrieval in 56 of 90 configurations, about 62%. We introduce two diagnostics, Delta H, token entropy, and Delta s, embedding cosine, and show that Delta H predicts retrieval gain under QC across all three rewriter families: pooled Spearman rho = +0.436, p < 0.001 on DeepSeek+Codestral; rho = +0.593 on Codestral alone; rho = +0.356 on Qwen. This establishes Delta H as a cheap, rewriter-agnostic proxy for deciding when rewriting pays off before running retrieval. Our analysis reframes LLM rewriting as a cost-benefit decision: it is most effective as a remediation layer for lightweight encoders on code-dominant queries, with diminishing returns for strong encoders or NL-heavy queries.

1 Introduction

The paper asks how much representational shift improves code retrieval and when online query rewriting justifies its cost. It studies three rewriting levels and two augmentation regimes, finding that NL+QC is strongest for code-heavy retrieval while ΔH predicts QC gains.

  • Surface-syntax overfitting motivates testing how much representational shift helps and when per-query LLM rewriting is worth its cost.
  • The study varies stylistic rephrasing, NL-enriched PseudoCode, and full natural-language transcription under joint query–corpus and corpus-only augmentation.
  • +0.51 absolute NDCG@10 lifts MoSE-18 on CT-Contest from 0.23 to 0.74 with NL+QC.
  • Corpus-only rewriting degrades retrieval in 56/90 configurations, whereas ΔH predicts retrieval gain under QC across rewriter families.

2 Background and Related Work

The paper situates rewriting within code information retrieval and identifies gaps in prior work’s retrieval target, cost, rewriter coverage, and diagnostics. It addresses these gaps by directly evaluating new representations in a controlled hierarchy.

  • Code Information Retrieval retrieves software artifacts across code-to-code, text-to-code, and hybrid query–corpus modalities.
  • Prior rewriting methods normalize surface form or use PseudoCode transiently, but ultimately retrieve over code.
  • The paper identifies four gaps: fixed abstraction level, online per-query cost, limited rewriter-family coverage, and absent diagnostics for when rewriting is worthwhile.
  • NL-enriched PseudoCode and snippet-level full NL are introduced as direct retrieval representations rather than intermediate forms.
  • The comparison unifies three abstraction levels, adds corpus-only rewriting, evaluates three independent rewriter families, and supplies a predictive representation-level diagnostic.

3 The Paraphrasing Strategy

The paraphrasing strategy applies the same target representation to indexed documents and queries, comparing online joint rewriting with offline corpus-only rewriting. Representational analysis uses token entropy and embedding cosine to characterize rewriting effects.

  • The Paraphrasing Strategy: The study asks whether corpus-only rewriting can replace per-query online rewriting while preserving retrieval quality.
  • The Paraphrasing Strategy: NL-enriched PseudoCode and snippet-level Natural Language become final representations passed to a frozen encoder, unlike prior transient PseudoCode.
  • The Paraphrasing Strategy: The rewriter comprehends each snippet, generates the target form, and applies that form consistently to corpus documents and queries.
  • The Paraphrasing Strategy: Evaluation spans six CoIR test sets covering code-to-code, text-to-code, and hybrid tasks, with NDCG@10 across a tractable 5 × 3 × 6 design.
  • Representational Analysis: ΔH measures rewritten-minus-baseline token entropy, capturing lexical diversity received by the encoder.
  • Representational Analysis: Δs measures rewritten-minus-baseline mean pairwise embedding cosine, complementing ΔH with embedding-level geometric change.

4 Main Evaluation

Across six CoIR tasks, rewriting helps most for code-to-code retrieval with smaller encoders, while corpus-only augmentation and gains on natural-language queries are limited.

  • Code-to-code: QC-NL is best for every encoder on CT-Contest and four of five on CT-DL.MoSE-18 gains +0.51 NDCG@10 on CT-Contest and +0.16 on CT-DL; E5-base-v2 gains +0.24 and +0.10.
  • Code-to-code: +0.51 NDCG@10 lifts MoSE-18 from 0.23 to 0.74 on CT-Contest under QC-NL.
  • Text-to-code: On text-to-code tasks, QC-Rephrasing is best on Apps, whereas no QC configuration improves over the strongest baselines on CosQA.On Apps, CodeXEmbed gains +0.14; on CosQA, the strongest baselines are Qwen3-Emb at 0.38 and CodeXEmbed at 0.34.
  • Hybrid: Under QC, PseudoCode and NL tie for best on hybrid retrieval, while C-NL is the only setting below baseline on average.The hybrid comparison aggregates NDCG@10 across five encoders.
  • Rewriter size: Larger rewriters generally improve retrieval quality on average, but the trend is not monotonic for every encoder–strategy pair.The study also reports hardware and latency constraints for practitioners.

5 Representational Analysis

The analysis links rewriting’s representational shifts to retrieval outcomes, showing that entropy increases with abstraction and that query–corpus co-transformation is critical for exploiting those shifts.

  • Diagnostic analysis covers four pure code-to-code and text-to-code benchmarks, excluding hybrid corpora because their intrinsic NL/code mixture confounds rewriting-induced shifts.
  • Token-level effects: Token entropy increases monotonically with abstraction for four of five encoders, with Rephrase < Pseudo < NL.Qwen3-Emb is the exception because its large vocabulary absorbs NL diversity into subword merges.
  • Token-level effects: NL rewriting produces the richest lexical tail, reaching 47.5% Hapax% for Qwen3-Emb and 45.4% for MoSE-18 versus baselines of 36.6% and 33.3%.NL requires ∼1.9× more distinct words than code to cover 80% of the text and achieves 43.5% overall Hapax.
  • Embedding geometry: NL reduces mean pairwise cosine for all five encoders, with the largest reductions for UniXcoder (−0.15) and Qwen3-Emb (−0.131).PseudoCode is inconsistent, increasing cosine for UniXcoder and E5-base-v2.
  • Retrieval landscape: Under corpus-only rewriting, large representational shifts enter a retrieval-loss region, whereas query–corpus rewriting moves the same configurations into a gain region.The landscape plots ∆NDCG@10 against ∆H and ∆¯s, with the query–corpus setting recovering and often exceeding baseline performance.
  • Predictive diagnostics: Under QC, ∆H is the sole significant predictor of retrieval gain, while ∆¯s is not significant and neither diagnostic reaches significance under C.The diagnostics are largely independent, with ρ=+0.229, p=0.078.
  • Efficiency: QC rewriting adds ∼725 ms of decoding latency per query, while rewriting ∼38K snippets costs ∼16.5 GPU-hours for NL as a one-time offline operation.These measurements support a deployment trade-off between online latency and offline corpus-processing cost.

6 Cross-Rewriter Robustness

Cross-rewriter experiments show that NL rewriting generalizes on code-heavy retrieval, but strategy rankings vary by rewriter, while ∆H consistently tracks retrieval gain under QC.

  • Retrieval robustness: NL is best for Qwen and DeepSeek on CT-Contest and competitive for Codestral, but the best strategy depends on the rewriter.Codestral-Rephrase reaches 0.74, and DeepSeek-Pseudo underperforms DeepSeek-Rephrase.
  • Diagnostic replication: The ∆H–∆NDCG@10 correlation replicates across rewriters, reaching ρ=+0.593 for Codestral and ρ=+0.436, p<0.001 when DeepSeek and Codestral are pooled.The correlation preserves its positive sign for DeepSeek at ρ=+0.274.
  • Diagnostic replication: Because ∆H correlates with gain within each rewriter, it can identify the best strategy without requiring full retrieval evaluation.On NL-heavy CosQA, negative or small ∆H values correspond to weak or negative gains, unlike the positive shifts on CT-Contest.
  • Retrieval robustness: On NL-heavy CosQA, no rewriting strategy beats the baseline for any rewriter.This indicates that rewriting failure on this task is not specific to one rewriter family.

7 Conclusion

The paper evaluates three rewriting levels as direct retrieval representations and frames rewriting as a cost–benefit decision shaped by encoder strength, query modality, and representational shift.

  • The study introduces NL-enriched PseudoCode and snippet-level Natural Language as direct retrieval representations alongside stylistic rephrasing.
  • NL+QC provides the largest reported gain, up to +0.51 NDCG@10 on CT-Contest for MoSE-18, especially for lightweight encoders.
  • Corpus-only rewriting degrades retrieval in ∼62% of configurations, while QC outperforms C in 78/90 paired comparisons.
  • ∆H is a significant cross-rewriter predictor of retrieval gain under QC, with pooled non-Qwen ρ=+0.436, p<0.001.
  • The conclusion recommends QC rewriting for lightweight encoders on code-dominant queries and skipping it for strong encoders or NL-rich queries.

8 Limitations and Broader Impact

The study’s evidence is bounded by rewriter, language, diagnostic, and deployment coverage, while semantic drift and hallucination remain risks for downstream retrieval systems.

  • Limitations: The study excludes closed-source rewriters and remains Python-heavy, leaving frontier proprietary models and low-resource languages unevaluated.
  • Limitations: ∆H and ∆¯s are corpus-level aggregates and do not predict per-query gains.Extending them to per-query confidence estimation remains open.
  • Limitations: Latency measurements assume a single H100 without production batching, caching, or query-side pre-computation.Such optimizations could shift the QC-versus-C trade-off toward QC.
  • Broader impact: Rewriter bias and hallucination can silently change semantics, misleading retrieval and downstream systems such as code completion, security audit, or program repair.The paper recommends auditing rewrites and retaining pointers to original entries.

A Technical Appendices and Supplementary Material

The supplementary results detail benchmark layouts, tokenisation statistics, rewriter-capacity effects, cross-family comparisons, and rewriting failures on NL-heavy queries.

  • Evaluation tables: Tables 7 and 8 report NDCG@10 across encoder, technique, and augmentation combinations for code-to-code, text-to-code, and hybrid CoIR tasks.QC rewrites queries and corpora jointly, whereas C rewrites only the corpus; baselines remain unmodified.
  • Tokenisation statistics: Table 9 summarizes vocabulary size, distinct-token counts, unigram entropy, type–token ratio, distributional skew, and hapax rates across encoder–strategy combinations.The statistics are averaged across tasks, with n=4 tasks per cell.
  • Rewriter capacity: Larger Qwen2.5-Coder-Instruct rewriters consistently improve retrieval quality across all three strategies and five encoders.The comparison spans rewriter sizes from 1.5B to 14B on codetrans-contest using NDCG@10.
  • Cross-rewriter robustness: NL rewriting is the best or a competitive strategy for every encoder under at least two of three independent rewriter families on codetrans-contest.The cross-family comparison uses Qwen3-Coder-30B, DeepSeek-Coder-V2-Lite-Instruct, and Codestral-22B with QC-MANIPULATION.
  • NL-heavy queries: On the NL-heavy cosqa benchmark, no rewriting strategy improves over the unmodified baseline under any of the three rewriters.The accompanying Delta H diagnostic is near-zero or negative for cosqa across all three rewriters.
Loading 2605.08299v1…