PaperlayerView exact revision on arXiv

Source-linked AI summary

MAP4CS: A Multi-dimensional Data Pruning Framework for Efficient Code Retriever Fine-tuning

Yuxuan Chen, Mingwei Liu, Guangsheng Ou, Zekai Zhang, Zike Li, Yanlin Wang, Pelin Zheng

arXiv:2610.11727v1cs.SEcs.AI

TL;DR

Full-corpus retriever fine-tuning can be costly and affected by noisy, redundant data, while random sampling may not preserve representative examples. MAP4CS combines structural, semantic, and distributional assessment with rule-based filtering to select a small training subset. Across in-domain and out-of-domain benchmarks, using about 5% of the data, it outperforms random sampling and matches or exceeds full-data fine-tuning.

  • Problem

    Full-corpus fine-tuning is computationally expensive and can suffer negative transfer from low-quality samples, while random sampling does not ensure a representative subset.

    Source: §1 · p. 2

  • Method

    MAP4CS selects a core subset by assessing syntactic structure, semantic consistency, and distributional representation, then applies rule-based filtering.

    Source: §1 · p. 3, §7 · p. 20

  • Results

    Using about 5% of training data, MAP4CS outperforms random sampling across in-domain and out-of-domain benchmarks and matches or exceeds full-dataset fine-tuning.

    Source: §7 · p. 20

  • Takeaways & Limitations

    MAP4CS acts as a de-duplicator for redundant corpora and a denoiser for chaotic ones, producing a lexically diverse, information-dense training corpus.

    Source: §7 · p. 20

    Validation on anonymized industrial search logs remains future work because confidential enterprise logs limit replication of real-world deployment conditions.

    Source: §5 · p. 18

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) has become a cornerstone in software engineering for enhancing Large Language Models (LLMs) with domain-specific knowledge. However, adapting retrievers to evolving code repositories remains challenging due to the noise and redundancy inherent in massive code corpora. Standard fine-tuning on the full corpus is computationally expensive and often leads to sub-optimal performance due to negative transfer from low-quality samples. Conversely, simple random sampling fails to guarantee data representativeness. To address these challenges, we propose MAP4CS (Multi-dimensional Awareness Pruning for Code Search), an adaptive data pruning framework. MAP4CS identifies a small, high-quality core subset by integrating syntactic structure, semantic diversity, and distributional representation, followed by a rigorous rule-based filtering pipeline. Extensive experiments on two large-scale datasets demonstrate that MAP4CS consistently outperforms random sampling baselines using only 5% of the training data. Remarkably, it achieves performance comparable to, or even superior to, fine-tuning on the full dataset, validating the ''less is more'' hypothesis in data-centric AI. Furthermore, linguistic analysis reveals an adaptive optimization mechanism: MAP4CS automatically functions as a de-duplicator for redundant corpora and a denoiser for chaotic ones, constructing a training corpus that is both lexically diverse and information-dense.

1 Introduction

Full-corpus fine-tuning is costly and vulnerable to noisy data, while random sampling can discard rare knowledge. MAP4CS selects and filters a small, multidimensionally assessed subset; experiments report strong retrieval performance with only 5% of the data.

1 Introduction

  • Full-corpus fine-tuning is costly and can transfer noise, while random sampling may lose long-tail APIs and retain confusing high-entropy pairs.The paper frames identifying a small, high-quality subset that can outperform the full corpus as its central research question.Source: §1 · p. 2
  • MAP4CS selects code-search training data using syntactic structure, semantic alignment, and distributional representation, followed by a cascade filtering and sampling pipeline.It first clusters code into local semantic contexts, then scores samples and selects a core subset.Source: §1 · p. 3
  • Using 5% of training data, MAP4CS outperforms random sampling across dense encoders and matches or exceeds full-dataset fine-tuning.On CSN-Java, it improves retrieval MRR over random sampling by 31.9% with UniXcoder and 45.5% with Qwen3-Embedding.Source: §1 · p. 3
  • The framework’s adaptive selection mechanism filters high-entropy noise while maximizing sample diversity to address the data scale-quality trade-off.The paper describes the resulting training-cost reduction as over 90%, with performance comparable to or better than full-data fine-tuning.Source: §1 · p. 3

2 Background and Motivation

The section contrasts data-pruning approaches that overlook code structure with cleaning methods that remove obvious noise but do not actively value samples. Its examples motivate pruning that accounts for semantic alignment and structural diversity.

2.1 The Landscape of Data Pruning

  • General pruning can treat code as flat text and miss syntax or execution logic, while conventional cleaning removes obvious noise without identifying samples that bridge query intent and code.The paper frames these limitations as a compromise between methods that misunderstand code structure and methods that lack selection intelligence.Source: §2.1 · p. 4

2.2 Motivation

  • A query about an internal buffer capacity hint can match a distractor’s explicit keyword even though the ground-truth method handles the logic implicitly.The example motivates selecting hard positives to encourage semantic alignment beyond surface lexical overlap.Source: §2.2 · pp. 4–5
  • A query requiring a specific action sequence can be confused with boilerplate sharing domain terms, motivating AST-based coverage of diverse control flows.The paper’s example describes the required sequence as “Signal value AND onComplete.”Source: §2.2 · pp. 4–5, Figure 2 · p. 5

3 Approach

MAP4CS is presented as a coarse-to-fine data-pruning pipeline for code-search training. It partitions a streaming corpus into local contexts, scores samples across multiple dimensions, then filters and samples under a data budget.

3 Approach

  • Stream-Based Semantic Clustering partitions the streaming corpus into local contexts.This is the first stage of the framework’s coarse-to-fine pipeline.Source: §3 · p. 5
  • Multi-Metric Sample Scoring evaluates samples based on structure, semantics, and distribution.This stage follows clustering in the pipeline.Source: §3 · p. 5
  • Data Filtering and Sampling performs final data distillation under a strict budget.This is the final stage of the pipeline.Source: §3 · p. 5

3.1 Problem Formulation

The paper formulates pruning as selecting a budget-limited subset of query–code training pairs that maximizes retrieval performance after fine-tuning. Performance is evaluated on a validation set using a metric such as MRR.

3.1 Problem Formulation

  • The corpus contains N training pairs, with each natural-language query q_i corresponding to a code snippet c_i.Source: §3.1 · p. 5
  • The objective is to choose a subset S of the corpus, with |S| ≤ B, that maximizes the evaluation metric for a model fine-tuned on S.The stated example sets B = 0.05N.Source: §3.1 · p. 5
  • The objective's evaluation metric M is measured on a validation set, with MRR given as an example.Source: §3.1 · p. 5

3.2 Stream-Based Semantic Clustering

MAP4CS structures long-tailed code corpora into local semantic contexts using streaming clustering. It uses dynamic clustering to accommodate evolving data and a similarity threshold to determine cluster assignment.

3.2 Stream-Based Semantic Clustering

  • Long-tailed code distributions can let mainstream functions obscure domain-specific logic, motivating organization into local semantic contexts.Source: §3.2 · p. 6

3.2.1 Clustering Algorithm Selection.

  • Single-Pass Clustering addresses fixed-K, noise-sensitivity, and efficiency challenges with streaming, O(N)-complexity clustering that dynamically discovers clusters.Its dynamic centroid updates adapt to concept drift.Source: §3.2.1 · p. 6

3.2.2 Clustering Process.

  • Code-query pairs are embedded into dense semantic vectors, then assigned to the most similar centroid when similarity exceeds τ; otherwise, a new cluster is initialized.Centroids are updated by a weighted moving average, and τ is set to 0.55 to balance cohesion and coverage.Source: §3.2.2 · pp. 6–7

3.3 Multi-Metric Sample Scoring

MAP4CS scores samples across syntactic structure, semantic alignment, and distributional representation. The measures favor structurally distinctive examples, a range of query–code alignment levels, and both dense and diverse regions of semantic space.

3.3.1 Structural Diversity

  • Structural diversity is measured from linearized AST sequences as the complement of average sequence similarity within a cluster, with higher scores indicating more unique syntax.The representation abstracts away variable names in favor of identifiers and syntactic tags.Source: §3.3 · pp. 7–8

3.3.2 Semantic Diversity

  • MAP4CS samples across high, medium, and low code–query cosine-similarity tiers rather than retaining only highly aligned examples.This is intended to retain standard positives, hard positives, and edge cases.Source: §3.3 · p. 8

3.3.3 Distributional Representation

  • The distributional score combines density and diversity over semantic vectors to address the trade-off between selecting cluster centers and selecting outliers.Density identifies mainstream cluster patterns, while diversity selects samples distant from their nearest neighbors.Source: §3.3 · p. 9

3.4 Data Filtering and Sampling

MAP4CS filters invalid or redundant samples before ranking the survivors by a weighted combination of normalized quality scores. It then selects the top K samples to form the pruned dataset.

3.4 Data Filtering and Sampling

  • The Filter-then-Rank pipeline combines deterministic hard rules with adaptive scoring to reduce redundancy and engineering incompatibilities before final selection.Source: §3.4 · p. 9

3.4.1 Step 1: Rule-Based Filtering.

  • Rule-based filtering removes duplicate normalized code, snippets exceeding 1,024 tokens, and queries with noisy content or insufficient length.The query rules target excessive non-ASCII characters, URLs, and specific metadata tags.Source: §3.4.1 · p. 9

3.4.2 Step 2: Weighted Ranking and Sampling.

  • After min-max normalization, the ranking score weights syntactic and semantic scores at 1/3 each and density and diversity at 1/6 each, then selects the top K samples.The paper motivates the larger structural and semantic weights as prioritizing foundational data correctness.Source: §3.4.2 · pp. 9–10

4 Evaluation

The evaluation compares MAP4CS with random sampling and full-data fine-tuning, then examines retrieval outcomes and the semantic and linguistic properties of the pruned data. Across the reported settings, the 5% subset consistently outperforms random sampling and often matches or exceeds full-data training.

4.1 Experimental Setup

  • The evaluation covers CodeSearchNet and CoSQA in-domain, plus APPS and Synthetic Text2SQL for OOD retrieval, using three dense encoders and BM25.RQ1 fine-tunes UniXcoder, GTEbase, and Qwen3-Embedding on CSN-Java and Query4Code; MAP4CS and random sampling use the same 5% budget, with MRR as the retrieval metric.Source: §4.1.1 · p. 10, §4.1.2 · p. 10, §4.2.1 · p. 11

4.1.3 Implementation.

  • Experiments use six epochs, batch size 4, AdamW with a 2e-5 initial learning rate, and a maximum sequence length of 512 tokens.The authors state that settings combine prior protocols with validation-set tuning and fix the random seed at 42.Source: §4.1.3 · p. 11

4.2.2 Results.

  • With 5% of training data, MAP4CS consistently beats random sampling and achieves comparable or better retrieval results than full-data fine-tuning across in-domain and OOD tasks.Table 1 reports downstream-task performance across fine-tuning strategies.Source: §4.2.2 · pp. 12–14, Table 1 · p. 12
  • On CSN-Java, UniXcoder’s average score rises from 0.166 (Random 5%) to 0.219, a 31.9% relative improvement.The passage also reports gains on CSN-Java with GTEbase and Qwen3-Embedding, and on Query4Code with Qwen3-Embedding.Source: §4.2.2 · p. 13
  • Full-data fine-tuning scores 0.053 for GTEbase on CSN-Java, while MAP4CS scores 0.178; on Query4Code, UniXcoder scores 0.200 versus 0.192.The authors relate larger gains on noisier CSN-Java to its mismatches, duplicates, and boilerplate, while Query4Code has higher semantic consistency.Source: §4.2.2 · p. 13
  • MAP4CS also improves OOD retrieval; on APPS, GTEbase trained on Query4Code scores 0.027 versus 0.012 with random sampling.The passage reports a 125% improvement for this comparison and says Qwen3-Embedding improves on every downstream task over random and full-data fine-tuning.Source: §4.2.2 · p. 13

4.2.3 Visualization of Semantic Distribution.

  • In the Code-Query Semantic Space, MAP4CS has an overlap coefficient of 0.5549 and a JS distance of 0.4895, with a more concentrated distribution than random sampling.The authors interpret the distributional difference as refinement away from noisy, mismatched regions toward high-consistency clusters.Source: §4.2.3 · pp. 14–15, Figure 4 · p. 14

4.2.4 Reason Analysis.

  • MAP4CS raises TTR from 0.0402 under Random Sampling to 0.0579, while Shannon entropy decreases from 6.269 to 6.160, a drop of approximately 1.7%, consistent with filtering repetitive boilerplate and noise.Source: §4.2.4 · p. 15

4.3 RQ2: Ablation Study

The ablation separates clustering-based sampling from MAP4CS’s additional scoring and filtering, while RQ3 examines performance and linguistic properties across sampling ratios.

4.3.1 Design.

  • The ablation used GTEbase on Query4Code and compared MAP4CS and cluster-only random selection of 5% of the data against random-sampling fine-tuning.Clustering Only samples within clusters generated by Single-pass Clustering.Source: §4.3.1 · p. 16

4.3.2 Results.

  • MAP4CS achieved an average score of 0.161, versus 0.152 for Clustering Only and 0.144 for Random Sampling.The authors attribute the additional gain over clustering to filtering low-quality samples within clusters.Source: §4.3.2 · p. 16
  • On APPS, MAP4CS scored 0.027, versus 0.018 for Clustering Only and 0.012 for Random Sampling.The passage presents this as evidence of an advantage on a challenging task.Source: §4.3.2 · p. 16

4.4.1 Design.

  • RQ3 varied the Query4Code sampling ratio across 3%, 5%, 10%, 20%, 50%, 70%, and 100% to compare MAP4CS with random sampling.The evaluation used both UniXcoder and GTEbase.Source: §4.4.1 · p. 17

4.4.2 Results.

  • With 3% of the training data, MAP4CS consistently outperformed random sampling; for UniXcoder, 3% pruned data surpassed 50% randomly sampled data.The sensitivity analysis varied the ratio from 3% to 100% on Query4Code for UniXcoder and GTEbase.Source: §4.4.2 · p. 17, §4.4.1 · p. 17, Figure 6 · p. 17
  • Random sampling showed performance volatility and sometimes degraded as its data ratio increased, while MAP4CS maintained a stable, superior trajectory.The authors suggest that adding more randomly sampled data can introduce detrimental noise.Source: §4.4.2 · p. 17

4.4.3 Reason Analysis.

  • Below a 10% ratio, MAP4CS maintained higher TTR than random sampling, while random sampling showed slightly higher entropy.The authors interpret the TTR gap as prioritizing non-redundant samples and the entropy pattern as consistent with filtering high-entropy noise; Figures 7 and 8 track these trends.Source: §4.4.3 · p. 17, Figure 7 · p. 17, Figure 8 · p. 17

5 Threats to Validity

The authors address internal validity by holding fine-tuning hyperparameters constant and testing out-of-distribution benchmarks against possible pretraining contamination. External validity remains bounded by selected models, datasets, and the gap between academic benchmarks and industrial search.

Internal Validity

  • UniXcoder’s possible prior exposure to CodeSearchNet could inflate in-domain results, so APPS and Synthetic Text2SQL are used as out-of-distribution benchmarks.The authors also keep fine-tuning hyperparameters consistent across experiments to support fair comparisons.Source: §5 · p. 18

External Validity

  • Findings may be limited by the selected retrieval models and datasets, although the evaluation spans varied architectures, data distributions, and seven programming languages.The authors include CSN-Java, Query4Code, CodeSearchNet, and Synthetic Text2SQL in the evaluation.Source: §5 · p. 18
  • Academic benchmarks do not fully replicate industrial code search, and validation on anonymized industrial search logs remains future work.CoSQA’s real user queries from Microsoft Bing search logs partially bridge this gap.Source: §5 · p. 18

6 Related Work

Related work spans neural code retrievers, data-selection methods, and retrieval-augmented code generation. MAP4CS differs by focusing on data-centric fine-tuning and multi-dimensional pruning for code retrieval.

6.1 Neural Code Retrieval

  • Neural code retrieval progressed from lexical matching to shared-vector-space bi-encoders, while prior work largely emphasizes model architectures and pretraining on massive, unfiltered corpora.MAP4CS instead studies fine-tuning strong backbones with a small representative data subset.Source: §6.1 · pp. 18–19

6.2 Data Pruning and Selection

  • Data selection spans NLP quality and diversity metrics, retrieval negative mining, and code-specific cleaning, but existing pruning often evaluates only one dimension.MAP4CS combines semantic consistency, AST structure, and distributional density for retrieval-augmented generation tasks.Source: §6.2 · p. 19

6.3 Retrieval-Augmented Code Generation (RACG)

  • RACG research covers diverse applications, but retriever adaptation efficiency is under-explored despite the need to update retrievers for evolving codebases.MAP4CS addresses the training-efficiency bottleneck by demonstrating high-performance retriever training with less data.Source: §6.3 · p. 20

7 Conclusion

MAP4CS selects a high-quality code-training subset through multi-dimensional assessment and rule-based filtering. Experiments report strong results with about 5% of the data, while linguistic analysis describes different pruning behavior for redundant and chaotic corpora.

7 Conclusion

  • MAP4CS integrates syntactic structure, semantic consistency, and distributional representation, then applies rule-based filtering to select a unique, engineering-quality core subset.The framework is designed for efficient and robust retriever fine-tuning for RAG in software engineering.Source: §7 · p. 20
  • ∼5% of training data lets MAP4CS outperform random sampling across in-domain and out-of-domain benchmarks while matching or exceeding full-data fine-tuning.The authors present this as empirical support for the “less is more” hypothesis in data-centric AI.Source: §7 · p. 20
  • Linguistic analysis finds that MAP4CS acts as a de-duplicator on redundant corpora and a denoiser on chaotic ones, producing lexically diverse, information-dense training data.The authors identify larger decoder-only LLMs and iterative pruning for continuous learning as future work.Source: §7 · p. 20

Related papers

Loading 2610.11727v1…