PaperlayerView exact revision on arXiv

Source-linked AI summary

Better Call Grep: Evaluating and Improving Grep-Like Lexical Retrieval for Repository-Level Code Completion

Baoyi Wang, Xingliang Wang, Guochang Li, Chen Zhi, Junxiao Han, Xinkui Zhao, Nan Wang, Shuiguang Deng, Jianwei Yin

arXiv:2601.23254v3cs.SE

TL;DR

Repository-level code completion requires cross-file context, but limited context windows constrain what LLMs can use, and existing retrieval can entail substantial preprocessing costs. The paper evaluates LLM-generated ripgrep retrieval, then adds identifier-weighted re-ranking and structure-aware deduplication. GrepRAG consistently improves over strong baselines across CrossCodeEval and RepoEval_Updated and establishes SOTA performance.

  • Problem

    Repository-level completion needs cross-file dependencies beyond the edited file, while LLM context windows limit the repository context that can be supplied and retrieval methods can incur preprocessing costs.

    Source: §1 · p. 2

  • Method

    The paper evaluates Naive GrepRAG, which uses LLM-generated ripgrep queries, then proposes GrepRAG with identifier-weighted re-ranking and structure-aware deduplication.

    Source: §1 · pp. 3–4

  • Results

    GrepRAG consistently improves over strong baselines across CrossCodeEval and RepoEval_Updated, establishing SOTA performance.

    Source: §8 · p. 20

  • Takeaways & Limitations

    Naive GrepRAG can achieve competitive performance by capturing explicit lexical dependencies, while optimized lexical retrieval improves completion across the evaluated benchmarks.

    Source: §8 · p. 20

    High-frequency generic identifiers can introduce noisy matches, and overlapping retrieved chunks can waste context and fragment code flow.

    Source: §4.3.3 · p. 12

Abstract

from arXiv · show

Repository-level code completion remains challenging for large language models (LLMs) due to cross-file dependencies and limited context windows. Prior work addresses this challenge using Retrieval-Augmented Generation (RAG) frameworks based on semantic indexing or structure-aware graph analysis, but these approaches incur substantial computational overhead for index construction and maintenance. Motivated by common developer workflows that rely on lightweight search utilities (e.g., ripgrep), we revisit a fundamental yet underexplored question: how far can simple, index-free lexical retrieval support repository-level code completion before more complex retrieval mechanisms become necessary? To answer this question, we systematically investigate lightweight, index-free, intent-aware lexical retrieval through extensive empirical analysis. We first introduce Naive GrepRAG, a baseline framework in which LLMs autonomously generate ripgrep commands to retrieve relevant context. Despite its simplicity, Naive GrepRAG achieves performance comparable to sophisticated graph-based baselines. Further analysis shows that its effectiveness stems from retrieving lexically precise code fragments that are spatially closer to the completion site. We also identify key limitations of lexical retrieval, including sensitivity to noisy matches from high-frequency ambiguous keywords and context fragmentation caused by rigid truncation boundaries. To address these issues, we propose GrepRAG, which augments lexical retrieval with a lightweight post-processing pipeline featuring identifier-weighted re-ranking and structure-aware deduplication. Extensive evaluation on CrossCodeEval and RepoEval-Updated demonstrates that GrepRAG consistently outperforms state-of-the-art (SOTA) methods, achieving 7.04-15.58 percent relative improvement in code exact match (EM) over the best baseline on CrossCodeEval.

1 Introduction

Repository-level completion requires cross-file context that limited context windows cannot provide from the edited file alone, while complex retrieval can impose high preprocessing and maintenance costs. The paper evaluates lightweight lexical retrieval, explains its strengths and limits, and proposes an optimized variant.

1 Introduction

  • Repository-level completion needs cross-file dependencies that may lie outside the edited file, while limited context windows make supplying an entire repository infeasible.Relevant dependencies include class definitions, utility interfaces, and global constants.Source: §1 · p. 2
  • Existing RAG approaches can require costly preprocessing and index construction, and repository changes can make static graphs and vector indices stale.GraphCoder takes about 91 seconds to build a graph index and 7 seconds to retrieve on a roughly 100K-line repository.Source: §1 · p. 2
  • The study asks whether lightweight lexical search, common in developer workflows, can support repository-level completion before more complex retrieval is needed.The paper notes that systematic evaluation of lexical matching for code completion remains unexplored.Source: §1 · p. 2
  • Naive GrepRAG uses LLM-generated ripgrep commands and performs competitively, with success linked to precise lexical queries retrieving fragments closer to the completion site.The paper identifies class, method, and variable names among the effective keyword types.Source: §1 · p. 3
  • GrepRAG combines identifier-weighted re-ranking and structure-aware deduplication, addressing Naive GrepRAG’s limitations and achieving SOTA performance across multiple benchmarks.The paper identifies keyword ambiguity, context redundancy, and fragmentation as the issues the pipeline targets.Source: §1 · pp. 3–4

2 Motivation

The motivation section contrasts the latency of indexed retrieval with index-free grep and illustrates how lexical search can locate relevant cross-file definitions and usages.

2.1 Quantitative Analysis of Retrieval Latency and Scalability

  • On large repositories, a single VanillaRAG or GraphCoder retrieval can exceed 40 seconds, and Table 1 excludes index construction and static graph modeling.The table caption notes that latencies above 2 seconds are considered unacceptable for real-time completion.Source: §2.1 · p. 4, Table 1 · p. 4
  • For the 754K-LOC FloatingPoint repository, grep retrieval takes 1.45 seconds, about 1/35th of the over-50-second overhead of existing methods.The latency comparison concerns retrieval; Table 1 reports average latency for Line and API completion tasks on RepoEval_Updated.Source: §2.1 · p. 4, Table 1 · p. 4

2.2 Qualitative Analysis of Retrieval Effectiveness

  • In the Deck method-invocation example, GraphCoder ranks irrelevant logging code first and the truncated class definition ninth, while grep retrieves definitions and usages.Grep targets the explicit identifier Deck and locates both class definitions and object-instantiation code.Source: §2.2 · p. 5, Figure 2 · p. 5

3 Experimental Setup

The experiments evaluate grep-like retrieval for repository-level completion using two datasets, defined quality and latency metrics, Naive GrepRAG, representative baselines, and standardized implementation settings.

3 Experimental Setup

  • The setup systematically evaluates the effectiveness and efficiency of grep-like lexical retrieval for repository-level completion.It introduces datasets, evaluation metrics, Naive GrepRAG, baselines, and implementation details.Source: §3 · p. 6

3.1 Datasets

  • CrossCodeEval uses Python and Java cases filtered to require external context, while RepoEval_Updated provides 8,000 tasks and large repositories for latency and scalability testing.Some RepoEval_Updated projects exceed 500K lines of code.Source: §3.1 · p. 6

3.2 Evaluation Metrics

  • Generation quality is measured with code- and identifier-match metrics, and retrieval efficiency with average CPU time per query.The metrics include EM, Edit Similarity, Recall, and F1; identifier matching treats identifiers as a set.Source: §3.2 · p. 6

3.3 Naive GrepRAG

  • Naive GrepRAG has the LLM generate ripgrep queries from local context, executes them across the repository, then ranks and concatenates candidate snippets.The candidates are ranked by Jaccard similarity to the local context before the top-K fragments form the prompt context.Source: §3.3 · pp. 6–7, Figure 3 · p. 7

3.4 Baselines

  • The baselines span no retrieval, BM25-based retrieval, graph-based retrieval, analogy and import-dependency retrieval, and reinforcement-learning retrieval.The named systems are NoRAG, VanillaRAG, GraphCoder, RepoFuse, and RLCoder.Source: §3.4 · p. 7

3.5 Implementation Details

  • Experiments use DeepSeek-V3.2-EXP and Qwen3-Coder-Plus at temperature zero, with a 10-snippet retrieval budget and a 4,096-token context limit for RAG methods.The prompt specifies 10 ripgrep queries; GPU-based methods run on one NVIDIA A6000, and latency is measured on the same CPU environment.Source: §3.5 · p. 7

4 Evaluating the Potential of Naive GrepRAG

The empirical study evaluates Naive GrepRAG against sophisticated baselines, analyzes why it succeeds, and diagnoses its limitations to inform optimization.

4 Evaluating the Potential of Naive GrepRAG

  • The study first evaluates Naive GrepRAG’s end-to-end completion performance against sophisticated baselines.This is research question RQ1.Source: §4 · p. 8
  • It then examines the mechanisms behind Naive GrepRAG’s success.This is research question RQ2.Source: §4 · p. 8
  • Finally, it diagnoses Naive GrepRAG’s limitations to inform optimization directions.This is research question RQ3.Source: §4 · p. 8

4.1 RQ1: Evaluating the Performance of Naive GrepRAG

RQ1 evaluates whether LLM-generated ripgrep commands can improve repository-level completion, comparing Naive GrepRAG with retrieval baselines across CrossCodeEval’s Python and Java subsets.

4.1 RQ1: Evaluating the Performance of Naive GrepRAG

  • The research question tests whether LLM-generated grep commands can retrieve useful context and improve end-to-end completion on CrossCodeEval.Source: §4.1.1 · p. 8

4.1.1 Motivation.

  • The evaluation compares Naive GrepRAG with five baselines across Python and Java, using DeepSeek-V3.2-EXP and Qwen3-Coder-Plus to reduce model-specific bias.Source: §4.1.2 · p. 8

4.1.2 Analysis of End-to-End Performance.

  • Less than 0.02s average retrieval latency makes Naive GrepRAG faster than GraphCoder and RepoFuse while reducing retrieval computational cost.Source: §4.1.2 · p. 8
  • 38.61% EM on Python with DeepSeek-V3.2-EXP exceeds Vanilla RAG (24.99%) and RLCoder (36.59%); Naive GrepRAG also reaches 41.70% EM on Java.The comparison uses two backbone models across Python and Java; the paper also reports the highest identifier EM and F1 for Naive GrepRAG.Source: Table 3 · p. 8, §4.1.2 · p. 8

4.1.3 Analysis of Solved Instance Overlap.

  • 161 Python instances were solved uniquely by GrepRAG, versus 24 by RepoFuse; across both languages, only 20%–30% of cases were solved by all models.GrepRAG’s unique solutions account for 11.6% of Python cases, and the distinct solution sets suggest lexical matching complements other retrieval strategies.Source: §4.1.3 · pp. 8–9, Figure 4 · p. 9

4.2 RQ2: Analyzing the Success of Naive GrepRAG

RQ2 examines Naive GrepRAG’s retrieval patterns and why it succeeds where RAG baselines fail, relating its precise, local identifier retrieval to baseline retrieval and re-ranking errors.

4.2.1 Motivation.

  • The retrieval-pattern analysis samples 381 of 45,615 ripgrep commands, stratified by language: 212 Python and 169 Java commands.Source: §4.2.2 · p. 9

4.2.2 Analysis of Retrieval Patterns.

  • Across 381 sampled commands, method-name queries account for 41.47%, class-name queries 35.96%, variable-name queries 18.37%, and other keywords 4.20%.Source: §4.2.2 · pp. 9–10
  • 23.5% of sampled commands use wildcards, and multi-query sets combine class-, method-, and variable-name searches to provide overlapping lexical views.Wildcards retrieve similarly named definitions; combining queries can surface signatures, callsites, implementations, and variable usages.Source: §4.2.2 · p. 10

4.2.3 Failure Analysis of RAG-based Baselines.

  • Failure analysis distinguishes coarse-retrieval misses from re-ranking failures, using 80% coverage of GrepRAG’s golden-context lines as the threshold.A failed completion below the threshold is classified as coarse retrieval failure; one at or above it is classified as re-ranking failure.Source: §4.2.3 · pp. 10–11
  • Naive GrepRAG succeeds with precise queries and code fragments closer to the completion site, while baselines fail through coarse retrieval and re-ranking errors.Baselines often miss relevant context during initial retrieval or fail to prioritize precise identifier matches afterward.Source: §4.2.3 · p. 11

4.3 RQ3: Limitations of Naive GrepRAG

Naive GrepRAG’s failures arise both from noisy keyword matches that distort ranking and from lexical retrieval’s difficulty finding implicit dependencies. Even when relevant context is retrieved, redundancy and fragmentation can make it harder for the model to use.

4.3.1 Motivation.

  • The failure set includes cases where Naive GrepRAG predicts incorrectly but at least one other RAG baseline succeeds.The study examines 362 Python and 281 Java cases meeting this criterion and distinguishes failure to retrieve a successful baseline’s context from failure after retrieving it.Source: §4.3.2 · p. 12, §4.3.1 · p. 12

4.3.2 Methodology.

  • Re-ranking failures account for 71.5% of Python and 75.1% of Java cases among 643 failures; recall failures make up the remainder.The analysis groups failures into two re-ranking patterns and one recall-failure class.Source: §4.3.3 · p. 12
  • High-frequency identifiers such as init, config, and run trigger irrelevant matches that can receive inflated Jaccard scores and displace relevant blocks from Top-K.Jaccard overlap does not distinguish frequent generic terms from task-specific identifiers.Source: §4.3.3 · p. 12
  • Independent queries can retrieve overlapping or adjacent snippets, wasting context tokens and sometimes placing a variable’s usage before its definition.This fragmentation and chronological disorder can disrupt code’s logical flow for the LLM.Source: §4.3.3 · p. 12, Figure 5 · p. 13
  • Lexical retrieval has a fundamental recall limit when critical context depends on implicit structural relationships, such as inheritance, rather than explicit lexical clues.In these cases, ripgrep commands may not locate the needed context.Source: §4.3.3 · p. 12

5 Improving Naive GrepRAG via Post-Processing

GrepRAG retains ripgrep retrieval and adds identifier-weighted re-ranking and structure-aware deduplication. It improves results across the reported benchmarks, with deduplication contributing strongly and performance remaining robust across tested generator models and candidate-pool sizes.

5.1 Overview

  • The post-processing targets Naive GrepRAG’s high-recall but weak-ranking and fragmented-context limitations.The stated goals are to address keyword ambiguity, redundancy, and fragmentation.Source: §5.1 · p. 13

5.2 Approach

  • GrepRAG keeps ripgrep retrieval, replaces Jaccard re-ranking with BM25, and merges overlapping or adjacent snippets by line intervals before selecting Top-K blocks.Deduplication processes the Top-N% of ranked candidates; the final context is capped at 4,096 tokens.Source: §5.2 · pp. 13–14

5.3 Main Results

  • GrepRAG improves CrossCodeEval code EM by 7.04%–15.58% relative to the best baselines; on RepoEval_Updated, API-task code EM rises 13.8% for Python and 13.4% for Java.The RepoEval_Updated changes are 35.75 → 40.70 for Python and 40.27 → 45.67 for Java.Source: §5.3 · p. 14, Table 5 · p. 15

5.4 Ablation Study: Dissecting the Improvement

  • The ablation reports 42.29% EM for full GrepRAG, 41.93% without BM25, and 39.12% without deduplication on Python DeepSeek-V3.2-EXP.The no-dedup variant improves only 0.51% over Naive GrepRAG, while adding deduplication without BM25 yields a 3.32% increase.Source: §5.4 · pp. 14–15, Table 6 · p. 16

5.5 Generalization of Ripgrep Command Generation

  • Changing the ripgrep instruction generator between DeepSeek-V3.2-EXP and Qwen3-Coder-Plus has little effect on downstream completion performance.The authors report that command generation’s downstream performance is not unique to one instruction generator.Source: Table 7 · p. 16, §5.5 · p. 16

5.6 Hyperparameter Sensitivity Analysis

  • Across both datasets, all tested N values from 10% to 90% outperform the strongest baseline, with performance generally peaking around N = 50%.Below 50%, deduplication may leave only one or two blocks; above 50%, performance plateaus or slightly degrades as low-relevance blocks add noise.Source: §5.6 · pp. 16–17, Figure 7 · p. 17

6 Discussion

The discussion reports that GrepRAG performs better than Claude Code under the same backbone and that a fine-tuned 0.6B model exceeds larger general-purpose models on specified RepoEval_Updated tasks. It also examines command-generation cost, pipeline timing, and potential benchmark contamination.

Mitigating LLM Overhead for Retrieval Command Generation

  • Because ripgrep command generation adds LLM inference latency, the authors constrain generation to keywords with a static command template and distill from a capable teacher model.The template reduces generated tokens; teacher quality is described as determining the supervised fine-tuning performance upper bound.Source: §6 · p. 17
  • On RepoEval_Updated line-level Python and Java tasks, the fine-tuned 0.6B model surpasses substantially larger general-purpose models in completion quality and computational cost.Results across all evaluation datasets are reported in Table 5.Source: §6 · pp. 17–18

RAG Pipeline Time

  • GrepRAG's reported pipeline time combines command generation and retrieval, while baseline time includes indexing and retrieval; command generation is O(1), versus O(N) or higher for graph-based approaches.The authors state this supports scalability and deployment practicality for large, frequently evolving repositories where index maintenance is a bottleneck.Source: §6 · p. 18

Potential Data Contamination

  • The authors cannot rule out evaluation-benchmark overlap in LLM pretraining, but base their conclusions on relative gains among methods using identical backbone models.They attribute GrepRAG's improvements over baselines to more precise context retrieval rather than model-parameter leakage.Source: §6 · p. 18

Comparison with Coding Agents

  • GrepRAG achieves higher code-level and identifier-level scores than Claude Code across RepoEval_Updated and CrossCodeEval, while Claude Code averages 97.97 s per instance.The comparison uses randomly sampled evaluation sets and the same DeepSeek-V3.2 backbone.Source: §6 · p. 18

7 Related Work

Related work covers language models for code and retrieval strategies for repository-level completion. It describes model limitations in cross-file settings and traces retrieval research from similarity and graph methods to strategy optimization and alignment.

7.1 Large Language Models for Code

  • Code models include proprietary systems such as GPT-5.2, Gemini 3, and Claude Opus 4.5 [1, 46], alongside open-source general-purpose and programming-focused models.Examples include DeepSeek-V3 [10, 25, 26], Qwen 3 [2, 14, 60, 61], Code Llama [40], and StarCoder [21, 30].Source: §7.1 · pp. 18–19

7.2 Repository-Level Code Completion

  • Repository-level completion remains difficult because cross-file definitions and constants challenge limited context windows, motivating RAG to retrieve codebase context [8] [42] [43] [45] [57].The passage says models that handle intra-file logic can struggle to capture global semantics in modular codebases [27] [33] [41].Source: §7.2 · p. 19
  • Retrieval work spans similarity-based methods [12] [22] [31], structure-aware graph methods [3] [9], and strategy optimization or alignment [53] [16].Examples include RepoCoder [67], GraphCoder [29], RepoHyper, Cocomic [8], RepoFuse [24], and AlignCoder [16]; graph construction and traversal add computational latency.Source: §7.2 · p. 19

8 Conclusion

The conclusion presents lightweight lexical retrieval as effective for code completion while identifying noisy and fragmented contexts as limitations. GrepRAG combines re-ranking and deduplication, and the paper reports consistent improvements over strong baselines.

8 Conclusion

  • Experiments on CrossCodeEval and RepoEval_Updated show GrepRAG consistently improves over strong baselines and establishes a new state of the art.The conclusion identifies future investigation of adaptive routing for implicit dependency scenarios.Source: §8 · p. 20
  • Naive GrepRAG captures explicit lexical dependencies but is challenged by noisy and fragmented contexts; GrepRAG combines identifier-weighted re-ranking with structural deduplication.Source: §8 · p. 20
  • The authors identify adaptive routing for implicit dependency scenarios as future work.Source: §8 · p. 20
Loading 2601.23254v3…