PaperlayerView exact revision on arXiv

Source-linked AI summary

LESS: Selecting Influential Data for Targeted Instruction Tuning

Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, Danqi Chen

arXiv:2402.04333v3cs.CLcs.AIcs.LG

TL;DR

Targeted instruction tuning needs relevant data for specialized capabilities when mixed datasets may hinder them. LESS uses optimizer-aware low-rank gradient similarity search, and its selected 5% often outperforms full-dataset training across diverse downstream tasks.

  • Problem

    Targeted instruction tuning asks how to select relevant fine-tuning data from general instruction datasets using only a handful of capability examples.

  • Method

    LESS uses optimizer-aware influences and reusable low-dimensional gradient features to select instruction data by similarity to target examples.

  • Results

    Training on a LESS-selected 5% often outperforms full-dataset training across diverse downstream tasks, while selected data transfers across model scales and families.

  • Takeaways & Limitations

    LESS selects data reflecting target reasoning and skill types rather than primarily surface-form cues, and smaller models can select data for larger models.

  • Takeaways & Limitations

    LESS's first-order influence approximation ignores interactions among multiple datapoints, while methods modeling such interactions are too computationally expensive for language models.

Abstract

from arXiv · show

Instruction tuning has unlocked powerful capabilities in large language models (LLMs), effectively using combined datasets to develop generalpurpose chatbots. However, real-world applications often require a specialized suite of skills (e.g., reasoning). The challenge lies in identifying the most relevant data from these extensive datasets to effectively develop specific capabilities, a setting we frame as targeted instruction tuning. We propose LESS, an optimizer-aware and practically efficient algorithm to effectively estimate data influences and perform Low-rank gradiEnt Similarity Search for instruction data selection. Crucially, LESS adapts existing influence formulations to work with the Adam optimizer and variable-length instruction data. LESS first constructs a highly reusable and transferable gradient datastore with low-dimensional gradient features and then selects examples based on their similarity to few-shot examples embodying a specific capability. Experiments show that training on a LESS-selected 5% of the data can often outperform training on the full dataset across diverse downstream tasks. Furthermore, the selected data is highly transferable: smaller models can be leveraged to select useful data for larger models and models from different families. Our qualitative analysis shows that our method goes beyond surface form cues to identify data that exemplifies the necessary reasoning skills for the intended downstream application.

1. Introduction

LESS addresses targeted instruction tuning by selecting data relevant to a specific capability from large instruction datasets, rather than relying on surface-form cues. It adapts influence estimation for Adam and variable-length data, enabling efficient, transferable, and interpretable selection that can outperform full-data training with 5% of the data.

  • Motivation: Mixed instruction-tuning datasets can hinder specialized capabilities, motivating selection of relevant data from a large collection using only a handful of target-capability examples.The paper frames this setting as targeted instruction tuning, including applications requiring reasoning skills.
  • Approach: LESS selects data by prioritizing examples that directly minimize target-task loss, using optimizer-aware gradient influence rather than surface-form features.The method performs Low-rank gradiEnt Similarity Search for target applications.
  • Approach: LESS adapts classical gradient influence features to instruction tuning with Adam and variable-length instruction data.The paper presents these optimization insights and influence formulations as potentially independently useful.
  • Efficiency: LESS uses LoRA and random projections to build a reusable, low-dimensional gradient datastore that supports efficient selection for new target tasks.The datastore is designed to make gradient features easily manipulable and reusable.
  • Results: 5% of the data selected by LESS often outperforms training on the full dataset across MMLU, TYDIQA, and BBH, with effectiveness across model scales and families.Small models’ gradient features can select data that performs strongly in larger models and different model families; qualitative analysis links selections to target reasoning and skills.

2. Preliminaries: Influence Formulation

The section formulates data influence by approximating training dynamics to estimate how a training datapoint affects held-out loss. It aggregates per-step influence across training and motivates selecting examples with aligned training and validation gradients, while noting dataset-dependent checkpoints across epochs.

  • Influence Formulation: Influence is estimated using a first-order approximation of training dynamics to measure a training datapoint’s effect on held-out data.The formulation follows Pruthi et al. (2020).
  • Influence Formulation: This training-dynamics formulation is distinguished from influence functions as discussed later in the paper.The distinction is explicitly deferred to Section 7.
  • Per-step influence: Under batch-one SGD, the parameter update is θt+1 − θt = −ηt∇ℓ(z; θt), which enables a Taylor expansion of validation loss.The exposition assumes learning rate ηt and training example z at step t.
  • Trajectory influence: Trajectory influence aggregates the influence of a datapoint at every training step that uses it, naturally summing contributions across epochs.Because each datapoint is used once per epoch, the aggregation is expressed over the training epochs.
  • Data selection with influence: Selecting z to maximize ⟨∇ℓ(z′; θt), ∇ℓ(z; θt)⟩ is intended to produce a larger decrease in validation loss on z′.This repurposes the influence formulation for data selection rather than mislabeled-data identification.
  • Data selection with influence: Across multiple epochs, later checkpoints θi depend on the dataset selected for training, complicating direct computation of InfSGD.The dependency arises because checkpoints after the first epoch reflect the selected dataset.

3. LESS: Estimating the Influence of Instructions

LESS adapts influence-based instruction selection to Adam optimization and addresses sequence-level gradient abnormalities that otherwise favor shorter instructions. It formalizes instruction losses over completion tokens and uses gradient similarity to target capability-inducing examples.

  • 3. LESS: Estimating the Influence of Instructions: LESS identifies Adam optimization and abnormal sequence-level gradients as the two main challenges for adapting influence-based instruction selection.Prior formulations focus on SGD, whereas LLM instruction tuning traditionally uses Adam; averaged gradients can also favor shorter instructions.
  • 3. LESS: Estimating the Influence of Instructions: For an instruction s and completion c, LESS defines the datapoint loss as token-wise cross-entropy averaged across completion tokens.The notation uses z=(s,c), with θ_t denoting the model at training step t and θ_i the post-epoch checkpoint.
  • 3.1. Extension to Adam: Under Adam, first-order dynamics select z by maximizing ⟨∇ℓ(z′; θ_t), Γ(z, θ_t)⟩, but Γ requires prior-training moment terms.The Adam update uses first- and second-moment terms m and v, creating additional circularity in the selection procedure.
  • 3.2. Addressing Abnormalities in Sequence-Level Gradients: Individual token influence could enable fine-grained selection, but estimating it is extremely expensive because it requires computing token-level quantities.A faithful sequence-level adaptation would choose which tokens to train on within each sequence.
  • 3.2. Addressing Abnormalities in Sequence-Level Gradients: Sequence gradient norms are negatively correlated with completion length, causing Equation (1)'s influence to heavily upweight shorter training instructions.This arises because example gradients average many token gradients and is a known issue for varying-length sequences.

4. LESS: Estimating Influences Efficiently

LESS makes influence-based data selection efficient by combining LoRA warmup training with projected low-dimensional Adam gradient features stored for reuse across target tasks. It then ranks candidate examples against few-shot validation subtasks to construct the selected training set.

  • Step 2: Projecting the gradients: The method stores projected gradient features for every candidate datapoint and model checkpoint in a reusable gradient datastore.This datastore supports selection for different target tasks and validation subtasks.
  • Setting: LESS selects Dtrain ⊂ D to reduce loss on a fixed few-shot validation set Dval and thereby perform well on held-out test data.Dval may contain multiple subtasks, such as multilingual question answering, with only a few examples per subtask.
  • Step 2: Projecting the gradients: LESS computes Adam updates from LoRA gradients and applies random projections that preserve inner products while reducing feature dimensionality.The memory-efficient projection implementation uses d = 8192 in practice.
  • Step 1: Warmup training with LoRA: LoRA warmup freezes pretrained weights, adds low-rank adapters, and uses a random 5% subset of D for N training epochs with checkpointing.In LLAMA-2-7B, the resulting LoRA gradient is less than 2% the size of the model parameters.
  • Step 3: Data selection: For each validation subtask, LESS aggregates influence scores across checkpoints, scores each candidate by its maximum subtask influence, and selects the highest-scoring examples for Dtrain.The selected subset is then used to train the target model MT.

5. Experiments

Experiments evaluate LESS across diverse instruction-tuning datasets, models, and tasks using 5% selected subsets. LESS consistently improves over random and other selection baselines, often outperforming full-data training and transferring across model scales and families.

  • Experimental setup: Evaluation covers MMLU, TYDIQA, and BBH, spanning multiple-choice, multilingual extraction, and challenging reasoning tasks.MMLU includes 57 tasks, while TYDIQA covers nine typologically diverse languages and BBH contains 27 challenges.
  • Experimental setup: LESS warmups on a random 5% subset, computes 8192-dimensional gradient features, and selects the highest-scored 5% for target-model training.Warmup training uses N = 4 epochs, and both warmup and final training use LoRA.
  • Main results: 2 to 5 points: LESS consistently outperforms random selection across all tested models and evaluation datasets.Table 2 reports results for LLAMA-2-7B, LLAMA-2-13B, and MISTRAL-7B, including LESS-T selected with LLAMA-2-7B’s gradient store.
  • Main results: 5% of the data often outperforms the full dataset, with the effect more evident for stronger base models such as LLAMA-2-13B and MISTRAL-7B.The authors hypothesize that full datasets contain irrelevant or detrimental examples for specific target tasks.
  • Transfer and robustness: LESS-T selected with LLAMA-2-7B outperforms random selection for LLAMA-2-13B and MISTRAL-7B, with a relatively small gap versus model-specific LESS.Additional Pythia experiments show LESS consistently beats random selection across scales, and Pythia-14M can select data for larger Pythia models.
  • Baseline comparisons: LESS consistently outperforms BM25, DSIR, and RDS, whose word-frequency, n-gram, or representation-based methods show minimal improvement over random selection.The baseline methods are acknowledged to be less computationally expensive.

6. Analysis

LESS is analyzed through its computational costs and design choices, alongside qualitative evidence about the reasoning-relevant data it selects. Gradient computation dominates one-time costs, while warmup size, checkpoints, and projection dimension trade selection quality against resource use.

  • Computational cost: Gradient feature computation is LESS’s most expensive step, scaling linearly with |D|, N, and d, while datastore memory scales linearly with d.Warmup training is also relatively expensive, but the main computational expense is incurred only once and actual selection requires minimal computation.
  • Design choices: Increasing Dwarmup and N improves influence-estimation accuracy but raises warmup-training cost, while increasing d improves projected-feature utility but increases memory and compute.The default warmup set is usually a random 5% of D.
  • Warmup training: Using pre-trained LLAMA-2-7B or LLAMA-2-7B-CHAT selection models significantly hurts LESS performance, potentially because of input-distribution shifts or training-dynamics differences.For these models, the analysis computes InfSGD because pre-training optimization states Γ are unavailable.
  • Checkpoint count: One LoRA checkpoint outperforms random selection but underperforms four checkpoints, suggesting that early instruction-tuning gradient features change substantially.Using fewer checkpoints remains better than random selection but is less effective overall.
  • Qualitative analysis: LESS selects an English open-book question-answering example resembling TYDIQA, whereas BM25 and RDS select Bengali examples based on surface task matching.BM25 selects mask-recovery examples and RDS selects hate-speech-detection examples from the Bengali few-shot data.

7. Related Work

Prior work emphasizes curating high-quality, diverse instruction data and selecting compact training subsets through coreset, gradient-based, attribution, and influence-function methods. These approaches support data selection and interpretability, while differing in whether they enable counterfactual reasoning.

  • Curating high-quality instruction tuning data: Instruction-tuning data comprises task-based datasets curated from traditional NLP tasks and open-ended instruction-following datasets spanning broad topics.The passages identify these as the two general types of instruction-tuning data.
  • Curating high-quality instruction tuning data: Increasing data quality and diversity rather than quantity has been shown to more effectively induce instruction-following abilities.
  • Coresets and gradient-based data selection: Data selection is framed as coreset selection, seeking a subset whose training performance resembles that of the full dataset.The related work cites coreset and gradient-based selection studies pursuing this objective.
  • Data attribution and influence functions: The paper’s influence formulation has supported identifying mislabeled examples, analyzing memorization, and deriving interpretability insights, but does not permit counterfactual reasoning.Influence functions, by contrast, can counterfactually reason about model behaviors and training data.
  • Data attribution and influence functions: First-order influence approximations have been found effective for data selection in transfer-learning settings.

8. Discussion and Future Work

LESS combines optimizer-aware influence estimation with reusable low-dimensional gradient features for efficient, interpretable data selection, including cross-model transfer. Future work includes context-driven test-time adaptation and selecting data to optimize differentiable objectives beyond cross-entropy.

  • Contributions: LESS uses optimizer-aware influences and a reusable datastore of low-dimensional gradient features to enable efficient data selection.The paper presents LESS as a data selection algorithm and describes its datastore as effective and reusable.
  • Contributions: Experiments demonstrate LESS’s effectiveness against relevant baselines and show that smaller models can select data for training larger models.The cross-model selection capability is reported in connection with Table 2.
  • Future Work: Future work could apply LESS to test-time adaptation using data selected from users’ contexts and demonstrations, or optimize differentiable metrics such as toxicity and harmfulness surrogates.This would extend selection beyond cross-entropy loss.
  • Future Work: The finding that more tuning data can degrade task performance may be connected to catastrophic forgetting in LLMs.The paper relates this observation to catastrophic forgetting behaviors reported by Lin et al. (2023b).

Limitations · A. Training

LESS has training-related limitations involving warmup cost, completion-token gradient aggregation, the imperfect link between validation loss and task performance, and linear approximations that overlook datapoint interactions.

  • A. Training: LESS requires a warmup fine-tuning phase to obtain useful gradient features aligned with the target data distribution.Using gradients directly from pretrained models such as LLAMA-2-7B or LLAMA-2-7B-CHAT is less effective, while warmup adds complexity and computational load.
  • A. Training: The method represents an entire sequence using the average gradient across completion tokens.This surrogate can become ambiguous and less effective for longer training or validation sequences, including open-ended generation.
  • A. Training: Completion-token gradient averaging can introduce anomalies into the data-selection pipeline.The paper highlights this issue in Section 3.2, particularly for longer sequences.
  • A. Training: Minimizing validation cross-entropy loss does not monotonically improve instruction-tuning task performance.Instruction-tuning tasks often require long-form generations, making the relationship between perplexity and performance more obscure than in base LLMs.
  • A. Training: LESS’s first-order data-selection approximation ignores the effects of adding multiple datapoints together.Duplicate points may receive equally high scores and appear to provide double the improvement even when that is unlikely.
  • A. Training: The linearity assumption may fail because influences across datapoints do not necessarily add linearly.Preliminary theory discusses when additive influences may hold, while more sophisticated algorithms model datapoint interactions more flexibly.

A.1. Training Datasets … D.1. Comparing SGD and Adam Optimizers for Instruction Tuning

The paper uses diverse human-created instruction datasets and a standardized LoRA training setup, evaluates targeted capabilities with task-specific metrics, and compares several selection baselines. Although SGD uses less memory, Adam consistently performs better across the three datasets, motivating optimizer-aware influence estimation.

  • A.1. Training Datasets: The training corpus comprises four human-annotated or human-authored datasets—FLAN V2, COT, DOLLY, and OPEN ASSISTANT 1—with varied formats, lengths, and tasks.FLAN V2 and COT derive from existing NLP datasets, whereas DOLLY and OPEN ASSISTANT 1 contain open-ended human-written answers.
  • A.2. Training Details: All experiments use LoRA with linear warm-up and cosine decay, a peak learning rate of 2 × 10−5, batch size 128, and four training epochs.The LoRA configuration uses rank 128, α 512, dropout 0.1, and learned matrices for all attention matrices.
  • A.2. Training Details: Each experiment runs three trials with distinct random seeds, while LESS selects different subsets after warmup training on varied data subsets.Optimization seeds remain consistent across experiments.
  • B. Evaluation Details: Evaluation uses 5-shot accuracy for MMLU, 1-shot macro-averaged F1 across 11 TYDIQA languages, and average 3-shot exact match for BBH.TYDIQA uses the gold-passage setup, and BBH provides chain-of-thought reasoning in each in-context example.
  • C. Baseline Details: Stochastic baselines are evaluated with three runs using different random seeds, reporting average performance and standard deviation.BM25 is the exception because it does not involve stochasticity.
  • C.1. BM25: BM25 uses the rank bm2511 package and aggregates scores across subtasks as in Equation (2), replacing gradient similarities with BM25 scores.
  • C.2. DSIR: DSIR selects data using n-gram features but is unsuitable here because limited target-distribution data produces a sparse n-gram table.The paper attributes this sparsity to a trade-off between bucket size and estimation accuracy for source and target distributions.
  • D.1. Comparing SGD and Adam Optimizers for Instruction Tuning: Adam consistently outperforms SGD across all three datasets, despite SGD’s lower memory usage and the influence formulation initially applying only to SGD.The comparison uses a random subset containing 25% of the data.

D.2. Selecting Data for SGD, SignGD, and Adam Gradients … F.1. Does instruction tuning exhibit kernel behavior?

Across ablations and extensions, LESS’s optimizer-aware influence formulation generally matches or outperforms alternatives, remains effective across model sizes and tasks, and relies on approximations whose relationship to kernel behavior is nuanced and difficult to verify directly.

  • D.2. Selecting Data for SGD, SignGD, and Adam Gradients: Adam influence generally matches or outperforms SGD and SignGD for selecting the top 5% of instruction data.LESS obtains the Adam update Γ from optimizer states during a short warmup period, while early Adam updates can be approximated by SignGD.
  • D.3. Comparison to Base Models, and Training with Validation Data: Instruction tuning generally enhances performance, but LESS-selected data can underperform the base model on some tasks, while validation-data training rarely helps.A notable example is MMLU, where the 5% LESS-selected subset performs slightly below the base model.
  • D.4. Detailed Results on Using Different Projection Dimensions: Even small projected gradient dimensions outperform random selection, while larger dimensions yield better performance.The ablation evaluates projection dimensions 1024, 2048, 4096, and 8192 when selecting the top 5% after warmup on all training data.
  • D.5. Results on Pythia Models: LESS almost always surpasses random 5% selection across Pythia models, and small selectors such as 14M models can effectively select data for larger models.LESS-T, using one model to select data for another, is effective within the Pythia family, even when the selector cannot solve TydiQA itself.
  • D.6. Results on GSM8K and TruthfulQA: LESS consistently outperforms random selection on the out-of-domain GSM8K and TruthfulQA benchmarks.GSM8K uses an 8-shot chain-of-thought setting, while TruthfulQA uses 10 randomly selected validation examples.
  • E. Nuances in Algorithm Design: Under the first-order approximation, selecting examples as if batch size were 1 is equivalent to selecting for larger batches because the shared 1/B factor does not change rankings.This assumes interactions between examples within a batch are negligible; LESS also averages epoch-wise learning rates to avoid time-of-sampling effects.
  • F. Comparison to “Kernel Behavior” (Malladi et al., 2023; Jacot et al., 2018).: LESS depends on sufficiently accurate first-order loss-dynamics approximations, which are conceptually related to but formally distinct from kernel behavior.Kernel behavior concerns output-logit gradients, whereas LESS’s approximation concerns loss gradients with respect to parameters.
  • F.1. Does instruction tuning exhibit kernel behavior?: Gradient features change substantially at the start of instruction tuning, and directly verifying kernel behavior is computationally too challenging.LESS therefore benefits from gradients collected throughout warmup rather than only from the pretrained model; kernel construction scales with dataset class count because it is required for every logit.

G. Comparison to Data Models (Ilyas et al., 2022) … K.2. Examples with Negative Gradient Similarity Scores

The paper compares LESS with datamodel-based selection, analyzes sequence-level gradient artifacts and objective limitations, and uses gradient-similarity and qualitative analyses to characterize selected and negatively transferring examples.

  • G. Comparison to Data Models (Ilyas et al., 2022): LESS and datamodel-based selection both frame data selection as optimization, approximating meta-optimization rather than relying on predefined usefulness or surface-form features.Both methods approximate training dynamics and require computationally efficient preliminary training, while differing in their proxy or selection models.
  • G. Comparison to Data Models (Ilyas et al., 2022): Selecting sequences instead of tokens requires adjusting the influence formulation and averaging outputs over sequences, with equal maximal sequence lengths simplifying the pre-training setting.This sequence-versus-token issue appears in both LESS and Engstrom et al. (2024), despite their different selection formulations.
  • H. Sequence Level Gradients in Instruction Tuning: Sequence-level gradients exhibit completion-length artifacts: gradient norms and Adam update norms decrease as completion length increases.The paper argues this motivates sequence-level influence adjustments and distinguishes the issue from a failure of the influence formulation.
  • H.1. Theoretical View: Token-wise selection could address conflicting helpful and unhelpful gradients within sequences, but this fine-grained approach is currently computationally intractable for LLMs.The theoretical discussion considers concentration under independent noise but notes that real tokens are semantically correlated, complicating suitable noise models.
  • I. Loss vs Accuracy: Validation-loss minimization does not always increase task accuracy, although LESS still improves accuracy over relevant baselines.The paper suggests alternative objectives may yield better accuracy and leaves their exploration for future work.
  • J. Gradient Similarity Analysis: Gradient-similarity rankings contain a high-similarity subset followed by a sharp decline and then a smooth decrease; MMLU has no examples with negative similarity scores.TydiQA and BBH, unlike MMLU, include examples with negative gradient similarity scores.
  • K.1. Top Selected Examples for BBH and MMLU: For MMLU, gradient-based retrieval selects problems with similar difficulty, whereas BM25 relies on token overlap and representation-based retrieval may omit required math reasoning.For BBH, the selected examples come from COT and DOLLY, with representation and gradient-based approaches differing in their returned examples.
  • K.2. Examples with Negative Gradient Similarity Scores: Examples with the lowest gradient similarity scores do not reveal clear patterns explaining the observed negative transfer to the target task.The finding motivates further investigation rather than providing a simple qualitative explanation for negative transfer.

K.3. Subset Distribution of Selected Examples

Selected-example distributions vary substantially across methods, but the amount drawn from a subset does not predict performance. LESS instead favors subsets whose examples match the target task format, selecting FLAN V2 for MMLU, DOLLY for TYDIQA, and COT for BBH.

  • Subset distribution: Different methods select markedly different amounts of data from each training subset, without a monotonic relationship between subset selection volume and performance.This indicates that useful examples may exist across subsets, making relevance identification more important than selecting the largest share from any subset.
  • LESS preferences: LESS predominantly selects FLAN V2 data for MMLU, aligning with MMLU’s predominance of multiple-choice questions.The selection is consistent with the target task’s question format.
  • LESS preferences: For TYDIQA, LESS strongly prefers DOLLY examples, many of which follow an open-book question-answering format.This preference matches the format observed in the target task examples.
  • LESS preferences: For BBH, LESS primarily selects examples from the COT dataset, a preference described as aligned with the target task.The supplied passage ends before specifying the full alignment rationale.
Loading 2402.04333v3…