Uncertainty-Aware Gradient Signal-to-Noise Data Selection for Instruction Tuning

• Large instruction datasets make full-data fine-tuning expensive and often unnecessary because many examples are redundant or noisy.Source: §1 · p. 1
• GRADFILTERING uses a LoRA ensemble on a frozen backbone with a small GPT-2 proxy to aggregate per-example gradients into G-SNR utility scores.Source: §1 · p. 2
• 19/24 LLM-as-a-judge evaluation cases favor GRADFILTERING-selected 5–15% subsets over Random and Superfiltering across Alpaca, Alpaca-GPT4, and LLaMA-2-7B/13B.Source: §1 · p. 2
• A small human study confirms that the reported model preferences align with human judgments.Source: §1 · p. 2
• GRADFILTERING-selected subsets converge faster than competitive filtering baselines without degrading final instruction-following quality.Source: §1 · p. 2

Data Valuation and Selection

• Influence functions and TracIn estimate training-example effects using second- or first-order approximations, but are expensive and brittle for deep networks.Source: §2.1 · p. 2

Instruction-Tuning Selection

• LESS builds a low-rank gradient datastore from strong-model per-example gradients and tailors selection to a particular downstream task.Source: §2.1 · p. 2
• Superfiltering uses GPT-2 proxy perplexity and IFD rankings to select a top-k subset, reducing filtering cost while assigning static scores.Source: §2.1 · p. 2

Epistemic Uncertainty

• Epistemic uncertainty can identify noisy or spurious instruction examples, but full deep ensembles are computationally expensive.Source: §2.2 · p. 2
• LoRA-Ensemble approximates deep-ensemble benefits by training independent adapters on a shared backbone at lower parameter and compute cost.Source: §2.2 · pp. 2–3

Positioning Our Approach

• GRADFILTERING combines GPT-2 with a LoRA ensemble in an in-situ, gradient-based selection method.Source: §2.2 · p. 3

LoRA-Ensemble Approximation

• Independent LoRA members share a frozen pretrained backbone but maintain separately initialized low-rank adapters and distinct training trajectories.Source: §3.1 · p. 3

Gradient-Based Data Valuation

• Per-example LoRA gradients summarize how each instruction-response pair drives ensemble parameter updates during training.Source: §3.1 · p. 3

LoRA-Ensemble Approximation

• The ensemble minimizes average per-member loss while backpropagating separately through each adapter, preserving member-specific behavior.Source: §3.1 · p. 3

Gradient-Based Data Valuation

• Gradient norms measure how strongly an example drives parameter updates, while ensemble moments capture expected update magnitude and epistemic uncertainty.Source: §3.1 · p. 3

Empirical Geometry of LoRA Training Dynamics

• Gradient-profile trajectories embed relative per-example gradient drops across epochs, with each polyline representing one LoRA member.Source: §3.1 · pp. 3–4, Figure 1 · p. 4
• Ensemble trajectories quickly diverge into stable regions, while later-epoch points cluster more tightly, indicating reduced disagreement during training.Source: §3.1 · p. 4

Implementation

• The implementation uses M = 5 ensemble members and sets T = 2 because trajectories change most between epochs 1 and 2 before stabilizing.Source: §3.2 · p. 4

Method

• GRADFILTERING turns LoRA-ensemble training dynamics into per-example data-selection scores.The framework uses a frozen backbone and tracks gradients across ensemble members and epochs.Source: §4 · p. 4, §4.2 · p. 4

Per-Example Gradient Statistics

• Each example is characterized by initial difficulty, learning progress, and stability across ensemble members using per-example LoRA gradients.Gradient norms quantify instantaneous training signal, while ensemble variation indicates consistency of the response.Source: §4.2 · pp. 4–5

Uncertainty-Aware Utility Scoring

• The disagreement term measures late-stage variance in gradient norms across ensemble members, with high variance indicating ambiguity or noise.Low variance indicates agreement on update magnitude, whereas persistent disagreement is treated as an unstable signal.Source: §4.2 · p. 5

G-SNR Utility

• G-SNR combines normalized gradient drop with uncertainty normalization, preferring consistent signal and suppressing high-disagreement examples.The relative gradient drop removes scale effects, and the variance factor down-weights unreliable signals.Source: §4.3 · pp. 5–6

Gradient-Drop Signal

• The information-gain term measures gradient-norm reduction from an early epoch s to a later epoch t.Examples with clearer gradient reductions are treated as showing more learning progress during training.Source: §4.3 · p. 5

Selection Protocol

• Selecting the top-α fraction using G-SNR yields a subset under the same procedure across datasets and base models without task-specific reward signals or external proxy scores.The selection protocol assumes access to per-example gradients under a parametric loss.Source: §4.3 · p. 6

Main Results

• GRADFILTERING consistently outperforms random subsets and is better or comparable with Superfiltering across Alpaca, Alpaca-GPT4, backbones, subset ratios, and adaptation regimes.Table 1 reports Pairwise Winning Scores across 5%, 10%, 15%, and 100% settings.Source: §5.2 · p. 6
• With only 5%–15% of the data, GRADFILTERING often matches or exceeds the full-data counterpart, whereas random subsets are frequently below the full-data baseline.This pattern indicates that selected examples preserve high training signal.Source: §5.2 · p. 6
• Human evaluation produced win/tie/lose counts of 44/19/37 on Alpaca and 49/8/43 on Alpaca-GPT4 for LLaMA-2-13B full fine-tuning with 10% data.The results are consistent with the LLM-judge trend across Helpfulness, Relevance, Accuracy, and Level of Detail.Source: §5.3 · p. 7

Ablation Study

• All three simpler utility alternatives typically yield negative pairwise-winning-score deltas relative to G-SNR across datasets, model sizes, and adaptation regimes.Raw gradient drop performs worst overall; normalization or variance adjustment mitigates instability but still underperforms full G-SNR.Source: §5.4 · p. 7, Table 2 · p. 8

Convergence Analysis

• GRADFILTERING converges faster and reaches lower loss earlier than Superfiltering in both full fine-tuning and LoRA under identical LLaMA-2-13B, 10% Alpaca settings.Figure 3 compares training-loss trajectories and final loss for the two selections.Source: §5.5 · pp. 7–8, Figure 3 · p. 7

G-SNR Interpretation

• Examples with large, consistent gradient drops under the LoRA ensemble tend to yield stronger downstream instruction-following performance than examples with small or noisy gradient changes.G-SNR summarizes how strongly an example pulls the model during training.Source: §6 · p. 8

Scope

• G-SNR is objective-agnostic because it uses per-example gradients of a parametric loss without task-specific reward models, preference labels, or handcrafted difficulty scores.The framework requires a differentiable backbone and parameter-efficient adaptation mechanism.Source: §6 · p. 8

Limitations

• Experiments instantiate and evaluate GRADFILTERING only on supervised instruction tuning, while extension to objectives such as multi-task mixtures remains future work.This boundary reflects computational constraints and page limits.Source: §6 · p. 8

• GRADFILTERING fine-tunes a small GPT-2 proxy with a LoRA ensemble and aggregates per-example adapter gradients into a G-SNR utility.G-SNR combines relative gradient drop with late-epoch gradient variance.Source: §7 · p. 8
• 5–15% subsets match or outperform random splits and Superfiltering in most LLM-as-a-judge settings across Alpaca, Alpaca-GPT4, and LLaMA-2-7B/13B.A small human study confirms the same preference trends.Source: §7 · p. 8
• GRADFILTERING-selected subsets converge faster and reach lower training loss than competitive filters under the same compute budget.This pattern holds under both LoRA and full-parameter fine-tuning.Source: §7 · p. 8

• G-SNR ignores gradient direction, so examples important for rare or long-horizon behaviors may be under-valued.The method operates on gradient norms and their variance.Source: Limitations · p. 9
• G-SNR depends on an early/late snapshot scheme and a modest ensemble size of M = 5, which different schedules or proxies could change.The authors note that t-SNE analyses indicate these settings already induce diverse trajectories.Source: Limitations · p. 9
• The GPT-2 proxy still requires backpropagation, making its cost lower than strong-teacher or full-gradient-datastore approaches but not free.Source: Limitations · p. 9
• Training-dynamics selection assumes useful examples look useful early, an assumption that delayed credit assignment or strong curriculum effects may violate.Source: Limitations · p. 9

• The protocol uses a fixed, symmetric prompt to compare two candidate responses for the same instruction.Source: §A.1 · p. 11
• GPT-5.1 and Qwen3-235B-Instruct serve as proprietary and open-source judge models, respectively.Source: §A.1 · p. 11
• The prompt template is adopted from Vicuna with only minimal renaming of model identifiers.Source: §A.1 · p. 11

• The judge is asked to check the quality of the answer.Source: Prompt for Performance Evaluation · p. 11
• The prompt shows the instruction alongside two anonymized candidate responses labeled Model A and Model B.Source: Table 3 · p. 11
• The judge outputs A, B, or Tie together with a brief explanation, while shuffling candidate order to avoid ordering bias.Source: Table 3 · p. 11