Robust LLM Data Selection via Content-Invariant Gradient Attribution

1 Introduction

• Carefully selected subsets can match or outperform substantially larger training datasets, motivating target-aware selection from corpora that may contain redundant, noisy, or weakly informative examples.The paper frames data selection as a key component of efficient LLM post-training.Source: §1 · p. 1
• Minor wording or formatting changes can produce different gradient representations and low-overlap selected subsets, a failure mode the paper calls surface-form bias.The changes preserve the validation example’s content, yet can alter the selected training subset.Source: Figure 1 · p. 1, §1 · p. 1
• CIGA selects a layer with strong updates that remain stable across surface variants, then uses task-adaptive semantic similarity to calibrate its gradient ranking.This combines optimization-level relevance with content-level information rather than replacing gradient attribution with semantic similarity.Source: §1 · pp. 1–2
• Across five benchmarks, CIGA improves selection consistency while maintaining downstream capabilities and improving average performance, at lower attribution cost than multi-checkpoint gradient selection.The paper reports this as its robust and effective data-selection contribution.Source: §1 · p. 2

2.1 Data Attribution

• Attribution spans Hessian-based Influence Functions, first-order gradient similarity, Data Shapley, and downstream-utility evaluation alongside costly LOO retraining.First-order methods such as TracIn and LESS use gradient similarity without the Hessian.Source: §2.1 · p. 2

2.2 Data Selection

• Data-selection work pursues scalability and uses gradient, semantic, internal-signal, or information-theoretic criteria, but global-gradient methods remain sensitive to expression-form variation.The paper characterizes gradient methods as capturing update utility and semantic or internal-signal methods as capturing content with less connection to gradient dynamics.Source: §2.2 · p. 2
• Figure 2 depicts CIGA’s sequence of warm-up fine-tuning, consistent influential-layer selection, task-adaptive fusion, and Top-k% selection with retraining.The overview presents the proposed pipeline in the context of data selection.Source: Figure 2 · p. 3

3.1 Warm-up Fine-tuning

• A warm-up fine-tune yields one checkpoint and optimizer state used for Adam-preconditioned update features, layer selection, and attribution.Source: §3.1 · p. 3

3.2 Consistent Influential Layer Selection

• CIGA selects the layer with the highest score balancing average update magnitude against cross-variant dispersion in update magnitude and direction.It constructs validation-example variants through semantics-preserving wording or formatting changes, then Z-scores the layer statistics before scoring.Source: §3.2 · p. 3

3.3 Dynamic Feature Fusion

• CIGA ranks candidates using cosine similarities between selected-layer gradient features and validation features, and between text embeddings, then fuses the rankings with reciprocal-rank fusion.The fusion weight β controls reliance on semantic features.Source: §3.3 · pp. 3–4
• Task-specific β is derived from validation-set diagnostics: high gradient SNR shrinks it, while a concentrated semantic ranking also lowers it to balance relevance and diversity.SemTop is the mean of the top K aggregated per-candidate semantic scores; the diagnostics set one β per task while rankings act on individual candidates.Source: §3.3 · p. 4

3.4 Data Selection and Retraining

• For each target validation set, CIGA ranks training candidates by its fused scores, selects the Top-k% subset, and retrains the original base model on it.The pipeline comprises warm-up fine-tuning, layer selection, dynamic feature fusion, subset selection, and retraining.Source: §3 · p. 3, §3.4 · p. 4

4.1 Experimental Setup

• Experiments use two backbones, five target benchmarks, Top-5% subsets, and mean pairwise Jaccard overlap across five surface-form variants; α = 0.7 and γ = 1/3 are fixed across tasks.The values were chosen on TyDiQA; the evaluation compares semantic, gradient, and information-based selection baselines.Source: §4.1 · p. 4

4.2.2 Downstream Performance Results

• 2.11 and 1.04 percentage points: CIGA exceeds Random in macro-average on LLaMA-2-7B and Qwen3-4B-Base, respectively, and improves robustness without sacrificing downstream utility.Gains are significant against all evaluated selection baselines on LLaMA-2-7B and against Random, Cos, and SelectIT on Qwen3-4B-Base (p < 0.05).Source: §4.2.2 · p. 5

4.2.1 Selection Consistency Across Variants

• 58.33% and 48.68% average overlap rates on LLaMA-2-7B and Qwen3-4B-Base improve by 9.27 and 8.65 percentage points over the best global-gradient baseline.On HellaSwag, CIGA’s overlap rates are 49.29% and 36.60%, versus LESS at 22.64% and 18.47%.Source: Table 1 · p. 5, §4.2.1 · p. 5

4.2.2 Downstream Performance Results

• Semantic and gradient signals complement each other across tasks; using 5% of training data, CIGA ranks highest overall, while Full-data fine-tuning averages lower than Random on Qwen3-4B-Base.On LLaMA-2-7B, Full retains a slightly higher average than CIGA, though CIGA exceeds Full on TyDiQA and MMLU.Source: §4.2.2 · pp. 5–6

4.2.3 Ablation Experiment

• CIGA-G, using only the consistent-layer Grad-Score, exceeds LESS and TracIn on both backbones’ average Jaccard similarity and downstream averages.Its Jaccard similarities are 56.44% and 46.96%, and downstream averages are 47.15% and 73.26% on LLaMA-2-7B and Qwen3-4B-Base, respectively.Source: §4.2.3 · p. 6
• Adding dynamic semantic fusion raises average Jaccard similarity by 1.89 and 1.72 percentage points while maintaining or slightly improving downstream performance.The consistent-layer Grad-Score provides the main stability foundation, with semantic information calibrating the gradient-only ranking.Source: §4.2.3 · p. 6
• CIGA consistently selects middle-region layers across tasks rather than relying on a fixed absolute layer index.Multi-layer aggregation improves over single-layer settings by only 0.19 and 0.22 percentage points on LLaMA-2-7B and Qwen3-4B-Base. Increasing variants from three to five raises Jaccard similarity by 1.96 and 3.09 percentage points on LLaMA-2 and Qwen3, respectively, while downstream averages fluctuate by less than 0.4 percentage points on both models.Source: §4.2.3 · p. 6

4.2.4 Data Selection Efficiency

• CIGA reduces total runtime from 74.1 to 21.4 GPU hours, a 71.1% reduction and 3.46× speedup over LESS and TracIn.The comparison uses one A100 (80GB); CIGA uses one checkpoint and single-layer gradients, while LESS and TracIn aggregate gradients from four checkpoints.Source: §4.2.4 · p. 6

5.1 Overcoming Surface Form Bias

• Global-gradient features show substantial dispersion across surface-form variants of the same validation data, indicating sensitivity to wording and formatting.The analysis visualizes validation samples and semantics-preserving variants on GSM8K with LLaMA-2-7B.Source: §5.1 · p. 6
• CIGA’s consistent influential-layer gradients cluster variants of the same sample more tightly, partially mitigating surface-form noise even without semantic similarity.The resulting attribution focuses more on the underlying content.Source: §5.1 · p. 7

5.2 Consistent Influential Score Analysis

• CIGA’s consistent influential score peaks around Layer 15, where it aligns with strong downstream performance in the middle-layer region.The score jointly considers gradient strength and stability across surface forms.Source: §5.2 · p. 7
• High consistent influential scores and strong downstream performance concentrate in middle layers, supporting a stable middle-layer region rather than a fixed absolute index.The proposed explanation is that middle layers balance task-relevant information, gradient strength, and surface-form stability; the authors describe this as plausible.Source: §5.2 · pp. 7–8

• CIGA improves selection consistency under surface-form variations while preserving downstream capabilities and achieving the highest average among data-selection methods.It uses one warm-up checkpoint and one selected layer, offering a more efficient alternative to multi-checkpoint global-gradient selection.Source: §6 · p. 8
• CIGA combines a layer-selection algorithm with dynamic feature fusion to integrate gradient attribution signals and semantic alignment features.Source: §6 · p. 8
• The paper characterizes surface-form bias as interference from minor semantics-preserving changes to validation examples in global-gradient data-value evaluation.Source: §6 · p. 8

• CIGA requires forward and backward passes across the entire candidate corpus and storage of corresponding parameter gradients.At pools of tens of millions of samples, this incurs substantial temporal and computational overhead.Source: Limitations · p. 9
• The authors identify gradient-feature compression and efficient approximate computation as directions for addressing the computational bottleneck.Source: Limitations · p. 9
• Accurate CIGA selection requires a high-quality validation set from the target task to match gradient and semantic features.For novel tasks without an available validation set, constructing a reference dataset entails additional costs.Source: Limitations · p. 9

A.1 Dataset Composition and Scale

• The candidate training corpus aggregates widely adopted collections, with Table 7 summarizing each subset’s approximate volume and primary domain.Source: §A.1 · p. 11, Table 7 · p. 12

A.2 Detailed Baseline Descriptions

• The baselines include semantic retrieval, gradient attribution, and information-based selection methods.Semantic matching ranks cosine similarity; gradient methods use parameter gradients, while information-based methods use model feedback such as perplexity, confidence, or generation difficulty.Source: §A.2 · p. 11
• LESS aggregates gradient similarities across multiple saved checkpoints, whereas this study uses a single checkpoint when following its gradient-computation and saving approach.Source: §A.2 · p. 11

A.3 Evaluation Protocol and Prompt Variants

• Selection consistency is the Jaccard similarity between Top-k% subsets from different variants, while downstream performance uses task-specific metrics after retraining.The metrics are TyDiQA F1, MMLU and HellaSwag accuracy, and BBH and GSM8K exact match.Source: §A.3 · p. 12
• Five validation variants preserve task semantics and question-answer content while changing only semantically irrelevant wording or layout.CIGA, LESS, and TracIn use the same variants for consistency evaluation.Source: §A.3 · p. 12

A.4 Training Parameters

• LoRA uses rank 128, scaling factor alpha 512, and dropout 0.1.Source: §A.4 · p. 12
• Each model trains for one epoch with a maximum sequence length of 2048, using AdamW at a learning rate of 2e-5.Source: §A.4 · p. 12
• Warm-up and selected subsets each contain 5% of the candidate pool, and core experiments run on four 80GB NVIDIA A100 GPUs.Source: §A.4 · p. 12

B.1 Layer Selection Hyperparameter

• At α = 0.7, TyDiQA F1 peaks near 56.2%, while α values 0.1 and 0.4 stagnate around 54.3%.A larger α = 1.0 can suppress useful task-related gradient signals and degrade performance.Source: §B.1 · p. 12, TyDiQA · p. 14

TyDiQA

• Across other tasks, α = 0.7 gives the best or second-best performance in most cases and the highest macro-average.The authors select α = 0.7 on TyDiQA and keep it fixed for other tasks and both backbones.Source: TyDiQA · p. 14

B.2 Feature Fusion Hyperparameter

• For γ greater than 1/4, performance stays in a narrow high-performing range; setting γ to 1/4 degrades performance.The authors select γ = 1/3 based on TyDiQA F1 and keep it fixed; this choice yields the highest five-task macro-average.Source: §B.2 · p. 14

B.3 Selection Ratio

• On Qwen3-4B-Base, the macro-average ranges from 72.53% to 73.52% as the selection ratio varies from 1% to 10%, generally improving with more data.Source: §B.3 · p. 14

C Declaration of Generative AI Tool Usage

• The generative AI tools assisted with English editing, translation of non-English samples, and LaTeX table formatting, but not the core scientific contributions or conclusions.Source: C Declaration of Generative AI Tool… · p. 14

D Case Study

• CIGA retrieves a single-turn step-by-step reasoning sample for the BBH Boolean Expressions task, while Cos, LESS, and TracIn select samples with structural mismatches.Cos selects an unrelated instruction-following task, LESS a noisy multi-turn dialogue, and TracIn an answer lacking the necessary reasoning steps.Source: D Case Study · p. 14, Table 10 · p. 16
• Table 10 presents the top-1 training samples selected by different methods for the BBH evaluation task.Source: Table 10 · p. 16
• The evaluation instance asks for a Boolean expression to be evaluated and provides a step-by-step reasoning response.Source: D Case Study · p. 16