PaperlayerView exact revision on arXiv

Source-linked AI summary

Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruning

Yun-Da Tsai, Mingjie Liu, Haoxing Ren

arXiv:2407.05040v1cs.SEcs.LG

TL;DR

Synthetic code datasets can be costly to expand and contain redundancy and noise, motivating more efficient code-LLM fine-tuning. The paper combines dimensionality reduction, clustering, and pruning metrics to select smaller training subsets. It reports that 10% of the data retains most benchmark performance, while moderate pruning can improve results.

  • Problem

    Synthetic code datasets can be costly to scale and contain redundant or noisy instruction-code pairs, motivating more efficient pruning for code LLM fine-tuning.

    Source: §1 · pp. 1–2

  • Method

    The method embeds instruction-code pairs, reduces their dimensionality, clusters similar samples, and selects representative subsets using pruning metrics.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Training on 10% of the data retains most benchmark performance, with 3.9% degradation on HumanEval and 1.5% on MBPP; moderate pruning improves results by up to 2.7% on HumanEval and 3.5% on MBPP.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The pruning strategy reduces computational costs while maintaining performance, and the authors suggest future work could generate more informative data from clusters with low pruning metrics.

    Source: §6 · p. 8

    The study focuses exclusively on English prompts for Python code generation, so prompts in other languages might not produce accurate or functional code.

    Source: Potential Risks · p. 9

Abstract

from arXiv · show

Recent work targeting large language models (LLMs) for code generation demonstrated that increasing the amount of training data through synthetic code generation often leads to exceptional performance. In this paper we explore data pruning methods aimed at enhancing the efficiency of model training specifically for code LLMs. We present techniques that integrate various clustering and pruning metrics to selectively reduce training data without compromising the accuracy and functionality of the generated code. We observe significant redundancies in synthetic training data generation, where our experiments demonstrate that benchmark performance can be largely preserved by training on only 10% of the data. Moreover, we observe consistent improvements in benchmark results through moderate pruning of the training data. Our experiments show that these pruning strategies not only reduce the computational resources needed but also enhance the overall quality code generation.

1 Introduction

Synthetic code data can improve code LLMs but increases training costs and contains redundancy and noise. This work prunes such data, reporting that 10% retains most benchmark performance and moderate pruning can improve results.

1 Introduction

  • Synthetic code fine-tuning datasets can contain redundant, noisy instruction-code pairs, partly because verifying every program's functional correctness is impractical.Expanding synthetic datasets also raises training costs and computational demands.Source: §1 · pp. 1–2
  • The method embeds instruction-code pairs, reduces feature dimensionality, clusters similar samples, and selects representative subsets using pruning metrics.The authors describe this as a scalable pruning approach based on unsupervised learning.Source: §1 · p. 2
  • Training on 10% of synthetic data retains most benchmark performance, with 3.9% degradation on HumanEval and 1.5% on MBPP versus using all data.Moderate pruning also improves results by up to 2.7% on HumanEval and 3.5% on MBPP compared with using all data.Source: §1 · p. 2
  • Ablations identify the clustering algorithm as critical, while pruning metrics are less important.Source: §1 · p. 2

2 Related Work

Related work covers code LLMs and synthetic instruction data, instruction fine-tuning, and methods for selecting or pruning training examples. It describes both clustering-based approaches and metrics that track sample difficulty, influence, or quality.

2.1 Large Language Models for Code Generation

  • Code LLMs including CodeAlpaca, CodeLlama, WizardCoder, and Magicoder build on synthetic instruction data generated or evolved through different techniques.Source: §2.1 · p. 2

2.2 Instructional Fine-tuning

  • Instruction fine-tuning has improved zero-shot performance, while research has explored scaling tasks and models and filtering instruction data for quality.The Superficial Alignment Hypothesis suggests that minimal fine-tuning data may suffice for alignment, and one study reports improvement with 1,000 high-quality examples.Source: §2.2 · pp. 2–3

2.3 Data Pruning for Efficient Training

  • Clustering-based pruning groups similar data and selects samples, with prior work applying it to vision-text matching and data-efficient fine-tuning.Source: §2.3 · p. 3
  • Pruning metrics target sample difficulty, influence, or quality, but gradient-based methods can be computationally costly and external oracles may be infeasible due to cost.Source: §2.3 · p. 3

3 Methodology

The methodology seeks representative subsets whose training performance is comparable to or better than training on the full dataset. It uses embeddings, dimensionality reduction, clustering, and pruning metrics, with syntactical analysis favored to reduce computational cost.

3 Methodology

  • Instruction-code pairs are embedded, reduced in dimensionality, clustered by similarity, and further reduced using pruning metrics.The process aims to select high-quality, representative subsets for fine-tuning.Source: §3 · p. 3, Figure 1 · p. 4
  • The method focuses on syntactical differences because detecting semantic differences typically requires costly fuzzing and program execution.Source: §3 · p. 3

3.1 Dimension Reduction

  • PCA reduces high-dimensional LLM embeddings to lower the computational complexity of clustering and pruning metrics, including kernel-based metrics.The paper notes that LLM representations often exceed a thousand dimensions and that kernel methods can be hindered by the curse of dimensionality.Source: §3.1 · pp. 3–4

3.2 Clustering

Clustering groups similar instruction-code pairs so the method can select diverse, representative samples from each cluster. The section describes three clustering approaches and several metrics or strategies for choosing samples.

3.2 Clustering

  • Clustering groups similar instruction-code pairs, then samples from each cluster to form a representative subset.The sampling strategy depends on the pruning metric.Source: §3.2 · p. 4

3.2.1 KMeans

  • KMeans partitions data into k compact clusters by minimizing within-cluster sum-of-squares, and scales efficiently to large datasets.Source: §3.2.1 · p. 4

3.2.2 Agglomerative Clustering

  • Agglomerative clustering builds nested clusters using linkage criteria and does not require specifying the number of clusters in advance.The authors describe that flexibility as useful for selecting representative samples while maintaining dataset quality.Source: §3.2.2 · p. 4

3.2.3 HDBSCAN

  • HDBSCAN clusters around high-density core samples and removes unclustered samples as outliers.The authors say this aligns with their aim of finding syntactically representative samples; reported results use HDBSCAN-diversity and average three runs.Source: §3.2.3 · p. 4, Table 1 · p. 5

3.3 Pruning Metrics

  • Pruning metrics are chosen to detect syntactic differences and identify representative samples.Source: §3.3 · p. 4

3.3.1 Diversity Metric

  • The diversity metric scores a sample by its minimum distance from the selected query set within its cluster.The distance function is the dot product because embeddings are normalized before pruning.Source: §3.3.1 · p. 4

3.3.2 Density Metric

  • Kernel density estimation measures sample density in feature space, with high scores indicating crowded regions containing similar instances.Such samples are suggested to be less critical for maintaining diversity.Source: §3.3.2 · p. 4

3.3.3 Random

  • Random selection samples data from a chosen cluster or from the entire training dataset without clustering.Source: §3.3.3 · p. 5

4 Experiments

The experiments compare HDBSCAN-diversity pruning with baselines across code-generation benchmarks and report results from four datasets. The main findings are that moderate pruning can improve performance and substantial pruning can preserve much of it.

4.1 Setup

  • Experiments use DeepSeek-Coder-Base 6.7B, PCA-reduced 10-dimensional representations, and three runs to report averages and standard deviations.The model is identified as the base model; PCA and repeated runs address the experimental setup.Source: §4.1 · p. 5

4.2 Training

  • Training combines and shuffles two synthetic datasets totaling 185k entries, then fine-tunes for two epochs with full parameter updates.The datasets are Magicoder-OSS-Instruct-75K and Magicoder-Evol-Instruct-110K; batch size varies with pruning level.Source: §4.2 · p. 5

4.5 Main Results

  • HDBSCAN-diversity consistently outperforms nocluster-random across four datasets, typically peaking at 10–20% compression and remaining above full-data pass@1 even at 90% compression.Figure 2 compares MBPP, MBPP+, HumanEval, and HumanEval+ across compression ratios.Source: Figure 2 · p. 6, §4.5 · p. 7

4.3 Evaluation

  • Evaluation compares pass@1 on HumanEval and MBPP and their plus variants, using EvalPlus inference scripts with greedy decoding.HumanEval+ and MBPP+ provide 80× and 35× more tests, respectively.Source: §4.3 · p. 6

4.4 Implementation Details

  • Implementation embeds data with text-embedding-ada-002, applies PCA before clustering, and uses Scott’s Rule for kernel-density bandwidth selection.The diversity metric uses a random 10% of the dataset as its query set.Source: §4.4 · p. 6

4.5 Main Results

  • 3.9% on HumanEval and 1.5% on MBPP are the reported degradations at 10% data versus full training, while slight pruning improves results by up to 2.7% and 3.5%, respectively.The improvements are reported for HumanEval and MBPP, respectively.Source: §4.5 · p. 6
  • At 1% of the data, MBPP performance is nearly equivalent to full-data training and improves 4.1% over the base model; HumanEval degrades versus full-data training but improves 17.0% over the base model.Figure 3 reports these extreme-pruning comparisons.Source: Figure 3 · p. 7, §4.5 · p. 7

5 Ablation Studies

The ablations focus mainly on MBPP because it provides more stable and consistent results. They find clustering choice matters more than pruning metric choice, while also examining PCA and embedding inputs.

5.1 Compare Clustering Algorithm

  • Clustering algorithms generally improve MBPP performance over no clustering, with HDBSCAN maintaining higher pass@1 and KMeans and Agglomerative Clustering showing greater variability.Figure 4 compares clustering algorithms across compression ratios; HDBSCAN remains above full-data performance at 90% compression.Source: Figure 4 · p. 7, §5.1 · p. 7

5.2 Compare Pruning Metrics

  • With HDBSCAN, diversity pruning offers a slight advantage only at 10–40% compression, and its benefit is limited and dependent on the clustering algorithm.Figure 5 compares pruning metrics across compression ratios.Source: Figure 5 · p. 8, §5.2 · p. 8

5.3 Effect of PCA

  • At 50% compression with KMeans and density pruning, PCA reduces pass@1 by less than 0.6% on MBPP and 4.3% on HumanEval, while reducing dimensions from 1536 to 10 speeds KMeans by about 12×.HDBSCAN clustering without PCA does not complete within four hours; Table 2 compares scores, dimensions, and pruning runtime.Source: §5.3 · p. 8, Table 2 · p. 8

5.4 Embeddings for Instruction or Code

  • At 50% compression with KMeans, embedding both instructions and code performs better than using either alone, while instruction-only and code-only results do not differ significantly.Table 3 compares the three embedding-input choices.Source: Table 3 · p. 8, §5.4 · p. 8

6 Conclusion

The study presents an efficient pruning strategy for fine-tuning LLMs on coding datasets. It reports improved data efficiency while reducing computational costs and maintaining performance.

6 Conclusion

  • The study presents an efficient data-pruning strategy for fine-tuning LLMs on coding datasets.Source: §6 · p. 8
  • The reported clustering and pruning techniques reduce computational costs while maintaining performance.Source: §6 · p. 8
  • Future work could generate more informative data from clusters with low pruning metrics.Source: §6 · p. 8

Limitations

The study's results are constrained by clustering and training randomness and a limited number of experimental runs. Different hyperparameter settings may also affect outcomes, and optimal settings for different data sizes remain future work.

Limitations

  • The clustering algorithms and training framework introduce inherent randomness into the results.Source: Limitations · p. 9
  • Only three runs were averaged per experiment, which may not capture variability or support accurate conclusions.The authors say more runs are needed for greater confidence.Source: Limitations · p. 9
  • For pruned datasets below 10% of the original size, training used batch size 32 rather than 512; optimal settings for different data sizes remain undetermined.The authors note that hyperparameter settings could affect outcomes.Source: Limitations · p. 9

Potential Risks

The study covers English prompts for Python code generation and identifies safety and output-quality risks associated with code generation using LLMs.

Potential Risks

  • The study focuses exclusively on English prompts for Python code generation; prompts in other languages might not produce accurate or functional code.Source: Potential Risks · p. 9
  • Without safety alignment, generated outputs risk malicious code or harmful language that could lead to security vulnerabilities or unintended consequences.Source: Potential Risks · p. 9
  • LLM-generated code may be incorrect or suboptimal, fail to follow best practices, or introduce security flaws, while training-data biases may propagate into outputs.Source: Potential Risks · p. 9

Use of AI Assistants

The authors report using ChatGPT to refine the paper's writing and generate code templates for figure drawing, and say they checked the generated content for accuracy and alignment with their intentions.

Use of AI Assistants

A Code Samples from Data Pruning

This section presents examples of data selected by the pruning strategy and an example of a removed outlier. The selected samples align with downstream coding tasks from English problem descriptions to code generation.

A Code Samples from Data Pruning

Related papers

Loading 2407.05040v1…