Source-linked AI summary
Uncertainty-Aware Gradient Signal-to-Noise Data Selection for Instruction Tuning
Zhihang Yuan, Chengyu Yue, Long Huang, Litu Ou, Lei Shi
TL;DR
Large, noisy, and redundant instruction datasets make full-data fine-tuning costly, while existing selection methods often rely on expensive gradient datastores or static proxy scores. GRADFILTERING uses a GPT-2 proxy with a LoRA ensemble to score examples by G-SNR, and its selected subsets match or exceed strong baselines while converging faster.
Problem
Large instruction datasets contain redundant or noisy examples, creating a need for smaller subsets that retain or improve fine-tuning performance at lower cost.
Method
GRADFILTERING combines a LoRA ensemble on a frozen backbone with a small GPT-2 proxy and aggregates per-example gradients into objective-agnostic G-SNR utilities.
Results
GRADFILTERING-selected 5–15% subsets match or outperform Random and Superfiltering in most LLM-as-a-judge settings, with aligned human preferences and faster convergence.
Takeaways & Limitations
Uncertainty-aware gradient statistics provide an effective signal for curating large instruction-tuning corpora under constrained compute budgets.
Takeaways & Limitations
G-SNR ignores gradient direction, depends on an early/late snapshot scheme and modest ensemble size, still requires proxy backpropagation, and assumes early usefulness predicts later value.
Abstract
from arXiv · showhide
Instruction tuning is a standard paradigm for adapting large language models (LLMs), but modern instruction datasets are large, noisy, and redundant, making full-data fine-tuning costly and often unnecessary. Existing data selection methods either build expensive gradient datastores or assign static scores from a weak proxy, largely ignoring evolving uncertainty, and thus missing a key source of LLM interpretability. We propose GRADFILTERING, an objective-agnostic, uncertainty-aware data selection framework that utilizes a small GPT-2 proxy with a LoRA ensemble and aggregates per-example gradients into a Gradient Signal-to-Noise Ratio (G-SNR) utility. Our method matches or surpasses random subsets and strong baselines in most LLM-as-a-judge evaluations as well as in human assessment. Moreover, GRADFILTERING-selected subsets converge faster than competitive filters under the same compute budget, reflecting the benefit of uncertainty-aware scoring.
1 Introduction
Instruction-tuning datasets are large, noisy, and redundant, motivating efficient selection of small subsets that preserve or improve fine-tuning performance. GRADFILTERING addresses this with objective-agnostic, uncertainty-aware gradient scoring and shows strong evaluation results and faster convergence.
- Large instruction datasets make full-data fine-tuning expensive and often unnecessary because many examples are redundant or noisy.
- GRADFILTERING uses a LoRA ensemble on a frozen backbone with a small GPT-2 proxy to aggregate per-example gradients into G-SNR utility scores.
- 19/24 LLM-as-a-judge evaluation cases favor GRADFILTERING-selected 5–15% subsets over Random and Superfiltering across Alpaca, Alpaca-GPT4, and LLaMA-2-7B/13B.
- A small human study confirms that the reported model preferences align with human judgments.
- GRADFILTERING-selected subsets converge faster than competitive filtering baselines without degrading final instruction-following quality.
2 Related Work
Prior data-valuation methods are theoretically grounded but often expensive or difficult to scale, while instruction-tuning filters use strong-model gradient datastores or static proxy scores. GRADFILTERING instead combines in-situ gradient dynamics with a lightweight LoRA ensemble and GPT-2 proxy.
- Data Valuation and Selection: Influence functions and TracIn estimate training-example effects using second- or first-order approximations, but are expensive and brittle for deep networks.
- Instruction-Tuning Selection: LESS builds a low-rank gradient datastore from strong-model per-example gradients and tailors selection to a particular downstream task.
- Instruction-Tuning Selection: Superfiltering uses GPT-2 proxy perplexity and IFD rankings to select a top-k subset, reducing filtering cost while assigning static scores.
- Epistemic Uncertainty: Epistemic uncertainty can identify noisy or spurious instruction examples, but full deep ensembles are computationally expensive.
- Epistemic Uncertainty: LoRA-Ensemble approximates deep-ensemble benefits by training independent adapters on a shared backbone at lower parameter and compute cost.
- Positioning Our Approach: GRADFILTERING combines GPT-2 with a LoRA ensemble in an in-situ, gradient-based selection method.
3 LoRA-Ensemble Approximation for Gradient-Based Data Valuation
The method trains independent LoRA adapters sharing a frozen backbone, then uses their per-example gradient trajectories to estimate learning dynamics and epistemic uncertainty. Gradient-profile visualizations show divergent early trajectories and reduced later disagreement.
- LoRA-Ensemble Approximation: Independent LoRA members share a frozen pretrained backbone but maintain separately initialized low-rank adapters and distinct training trajectories.
- Gradient-Based Data Valuation: Per-example LoRA gradients summarize how each instruction-response pair drives ensemble parameter updates during training.
- LoRA-Ensemble Approximation: The ensemble minimizes average per-member loss while backpropagating separately through each adapter, preserving member-specific behavior.
- Gradient-Based Data Valuation: Gradient norms measure how strongly an example drives parameter updates, while ensemble moments capture expected update magnitude and epistemic uncertainty.
- Empirical Geometry of LoRA Training Dynamics: Gradient-profile trajectories embed relative per-example gradient drops across epochs, with each polyline representing one LoRA member.
- Empirical Geometry of LoRA Training Dynamics: Ensemble trajectories quickly diverge into stable regions, while later-epoch points cluster more tightly, indicating reduced disagreement during training.
- Implementation: The implementation uses M = 5 ensemble members and sets T = 2 because trajectories change most between epochs 1 and 2 before stabilizing.
4 GRADFILTERING : Data Selection with G-SNR
GRADFILTERING scores instruction examples from evolving per-example LoRA gradients, combining learning progress with ensemble disagreement to select a targeted subset. Its G-SNR utility favors large, consistent gradient drops while down-weighting uncertain signals.
- Method: GRADFILTERING turns LoRA-ensemble training dynamics into per-example data-selection scores.The framework uses a frozen backbone and tracks gradients across ensemble members and epochs.
- Per-Example Gradient Statistics: Each example is characterized by initial difficulty, learning progress, and stability across ensemble members using per-example LoRA gradients.Gradient norms quantify instantaneous training signal, while ensemble variation indicates consistency of the response.
- Uncertainty-Aware Utility Scoring: The disagreement term measures late-stage variance in gradient norms across ensemble members, with high variance indicating ambiguity or noise.Low variance indicates agreement on update magnitude, whereas persistent disagreement is treated as an unstable signal.
- G-SNR Utility: G-SNR combines normalized gradient drop with uncertainty normalization, preferring consistent signal and suppressing high-disagreement examples.The relative gradient drop removes scale effects, and the variance factor down-weights unreliable signals.
- Gradient-Drop Signal: The information-gain term measures gradient-norm reduction from an early epoch s to a later epoch t.Examples with clearer gradient reductions are treated as showing more learning progress during training.
- Selection Protocol: Selecting the top-α fraction using G-SNR yields a subset under the same procedure across datasets and base models without task-specific reward signals or external proxy scores.The selection protocol assumes access to per-example gradients under a parametric loss.
5 Experiments
Experiments evaluate GRADFILTERING across subset sizes, datasets, LLaMA-2 backbones, adaptation regimes, preference judgments, ablations, and convergence. Selected subsets generally match or outperform baselines and converge faster under identical compute.
- Main Results: GRADFILTERING consistently outperforms random subsets and is better or comparable with Superfiltering across Alpaca, Alpaca-GPT4, backbones, subset ratios, and adaptation regimes.Table 1 reports Pairwise Winning Scores across 5%, 10%, 15%, and 100% settings.
- Main Results: With only 5%–15% of the data, GRADFILTERING often matches or exceeds the full-data counterpart, whereas random subsets are frequently below the full-data baseline.This pattern indicates that selected examples preserve high training signal.
- Main Results: Human evaluation produced win/tie/lose counts of 44/19/37 on Alpaca and 49/8/43 on Alpaca-GPT4 for LLaMA-2-13B full fine-tuning with 10% data.The results are consistent with the LLM-judge trend across Helpfulness, Relevance, Accuracy, and Level of Detail.
- Ablation Study: All three simpler utility alternatives typically yield negative pairwise-winning-score deltas relative to G-SNR across datasets, model sizes, and adaptation regimes.Raw gradient drop performs worst overall; normalization or variance adjustment mitigates instability but still underperforms full G-SNR.
- Convergence Analysis: GRADFILTERING converges faster and reaches lower loss earlier than Superfiltering in both full fine-tuning and LoRA under identical LLaMA-2-13B, 10% Alpaca settings.Figure 3 compares training-loss trajectories and final loss for the two selections.
6 Further Discussion
G-SNR is presented as a proxy for instruction-tuning data utility that combines gradient dynamics with ensemble uncertainty. The framework is objective- and largely model-agnostic in principle, but experiments are limited to supervised instruction tuning.
- G-SNR Interpretation: Examples with large, consistent gradient drops under the LoRA ensemble tend to yield stronger downstream instruction-following performance than examples with small or noisy gradient changes.G-SNR summarizes how strongly an example pulls the model during training.
- Scope: G-SNR is objective-agnostic because it uses per-example gradients of a parametric loss without task-specific reward models, preference labels, or handcrafted difficulty scores.The framework requires a differentiable backbone and parameter-efficient adaptation mechanism.
- Limitations: Experiments instantiate and evaluate GRADFILTERING only on supervised instruction tuning, while extension to objectives such as multi-task mixtures remains future work.This boundary reflects computational constraints and page limits.
7 Conclusion
GRADFILTERING selects instruction-tuning data by combining LoRA-ensemble gradient dynamics into an objective-agnostic G-SNR utility. Its selected small subsets generally preserve or improve evaluation preferences while converging faster than competitive filters.
- GRADFILTERING fine-tunes a small GPT-2 proxy with a LoRA ensemble and aggregates per-example adapter gradients into a G-SNR utility.G-SNR combines relative gradient drop with late-epoch gradient variance.
- 5–15% subsets match or outperform random splits and Superfiltering in most LLM-as-a-judge settings across Alpaca, Alpaca-GPT4, and LLaMA-2-7B/13B.A small human study confirms the same preference trends.
- GRADFILTERING-selected subsets converge faster and reach lower training loss than competitive filters under the same compute budget.This pattern holds under both LoRA and full-parameter fine-tuning.
Limitations
The method has limitations involving its gradient representation, snapshot and ensemble choices, proxy-training cost, and assumptions about when useful examples become identifiable.
- G-SNR ignores gradient direction, so examples important for rare or long-horizon behaviors may be under-valued.The method operates on gradient norms and their variance.
- G-SNR depends on an early/late snapshot scheme and a modest ensemble size of M = 5, which different schedules or proxies could change.The authors note that t-SNE analyses indicate these settings already induce diverse trajectories.
- The GPT-2 proxy still requires backpropagation, making its cost lower than strong-teacher or full-gradient-datastore approaches but not free.
- Training-dynamics selection assumes useful examples look useful early, an assumption that delayed credit assignment or strong curriculum effects may violate.
A.1 Prompt for Evaluation
The evaluation compares selected-data models with full-data models using a fixed symmetric LLM-as-a-judge protocol. GPT-5.1 and Qwen3-235B-Instruct provide the judge evaluations.
- The protocol uses a fixed, symmetric prompt to compare two candidate responses for the same instruction.
- GPT-5.1 and Qwen3-235B-Instruct serve as proprietary and open-source judge models, respectively.
- The prompt template is adopted from Vicuna with only minimal renaming of model identifiers.
Prompt for Performance Evaluation
The performance-evaluation prompt presents an instruction and two anonymized candidate responses to a judge. The judge outputs a preference label and brief explanation, with response order shuffled to reduce ordering bias.
- The judge is asked to check the quality of the answer.
- The prompt shows the instruction alongside two anonymized candidate responses labeled Model A and Model B.
- The judge outputs A, B, or Tie together with a brief explanation, while shuffling candidate order to avoid ordering bias.