PaperlayerView exact revision on arXiv

Source-linked AI summary

A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)

Nihal V. Nayak, Paula Rodriguez-Diaz, Neha Hulkund, Sara Beery, David Alvarez-Melis

arXiv:2602.14696v2cs.LG

TL;DR

Targeted instruction selection lacks clear evidence about which representations and algorithms matter, complicating budgeted adaptation of LLMs. The paper disentangles these components, compares them across settings, and unifies selection methods through approximate distance minimization. Gradient-based representations provide the most reliable query-distance signal, but downstream gains depend on budgets and candidate pools.

  • Problem

    Targeted instruction selection research is fragmented, with entangled design choices, inconsistent results, and missing baselines, leaving effective techniques unclear.

  • Method

    The paper disentangles data representations from selection algorithms, compares them across models, datasets, and budgets, and develops a distance-minimization perspective with generalization bounds.

  • Results

    Gradient-based representations provide the strongest distance-to-query signal; greedy round-robin tends to perform best at small budgets, but gains are candidate-pool and budget dependent.

  • Takeaways & Limitations

    Representation quality and selection performance should be evaluated separately, because query-distance reliability does not uniformly translate into downstream improvement.

  • Takeaways & Limitations

    The generalization result assumes a symmetric, convex, bounded loss satisfying the triangle inequality and a specified parametric form.

Abstract

from arXiv · show

Instruction fine-tuning of large language models (LLMs) often involves selecting a subset of instruction training data from a large candidate pool, using a small query set from the target task. Despite growing interest, the literature on targeted instruction selection remains fragmented and opaque: methods vary widely in selection budgets, often omit zero-shot baselines, and frequently entangle the contributions of key components. As a result, practitioners lack actionable guidance on selecting instructions for their target tasks. In this work, we aim to bring clarity to this landscape by disentangling and systematically analyzing the two core ingredients: data representation and selection algorithms. Our framework enables controlled comparisons across models, tasks, and budgets. We find that only gradient-based data representations choose subsets whose similarity to the query consistently predicts performance across datasets, models, and candidate pools. While no single method dominates, gradient-based representations paired with greedy round-robin selection often perform best on average at low budgets, but these gains diminish at larger budgets. Finally, we unify several existing selection algorithms as forms of approximate distance minimization between the selected subset and the query set, and support this view with new generalization bounds. More broadly, our findings provide critical insights and a foundation for more principled data selection in LLM fine-tuning. The code is available at https://github.com/dcml-lab/targeted-instruction-selection.

1. Introduction

The paper addresses fragmented, difficult-to-interpret targeted instruction selection by disentangling data representations from selection algorithms. It systematically compares these components and finds reliable query-distance signals from gradient-based representations, while selection performance varies with budget and setting.

  • Motivation: Targeted instruction selection is motivated by the cost of constructing instruction datasets and adapting LLMs under limited data or compute budgets.The task is to select useful examples from a large candidate pool for a target task.
  • Research gap: Existing methods vary in representations, similarity metrics, algorithms, and baselines, leaving their performance drivers and operating conditions unclear.The literature also reports inconsistent empirical results and often omits zero-shot comparisons.
  • Approach: The proposed framework isolates data representation from selection algorithm, enabling controlled comparisons across models, datasets, and budgets.It also interprets several selection methods as approximate distance minimization and introduces generalization bounds for this perspective.
  • Findings: Gradient-based representations provide the strongest distance-to-query signal for query loss, but this signal does not consistently yield downstream gains.Even strong methods can fail to improve over zero-shot inference in some regimes.
  • Findings: Greedy round-robin tends to work best at small budgets, whereas optimal-transport methods offer modest larger-budget gains in some candidate-pool settings.The reported trends are candidate-pool dependent, and random subsets can match or outperform popular methods as budgets increase.

2. Targeted Instruction Selection

Targeted instruction selection chooses a budgeted subset from a large candidate pool to reduce expected loss on a target task, using a query set as a training-time proxy. The objective is useful but combinatorially expensive, motivating efficient selection methods.

  • Problem formulation: A candidate pool D contains instruction examples, while a query set Q is drawn from the target-task distribution and guides subset selection.The model is trained on the selected subset rather than the full candidate pool.
  • Problem formulation: The goal is to choose B examples and train the model only on them to minimize expected loss on the target task.The per-example loss is defined as ℓ(θ; z) := ℓ(fθ(x), y).
  • Problem formulation: Because the target task is unavailable during training and selection, the query set serves as a proxy for the selection objective.The proxy objective selects a subset using information available from Q.
  • Pipeline: The framework separates encoding the query and candidates from the subsequent budgeted selection step.Figure 1 depicts representations first and algorithmic selection second.
  • Computational challenge: The exact optimization is impractical and combinatorially expensive, motivating compute-efficient instruction selection methods.This computational barrier is central to why approximate algorithms are studied.

3. A Disentangled View of Instruction Selection

The paper separates targeted instruction selection into data representation and selection algorithm, then surveys alternative representations and algorithms under a common similarity-based framework. This separation enables component-level analysis while motivating distance-based interpretations of selection.

  • 3.1. Disentangling Data Representation and Selection Algorithm: The disentangled view separates data representation from the selection algorithm to isolate their individual effects.The framework is designed to make prior methods comparable by separating these components.
  • 3.1. Disentangling Data Representation and Selection Algorithm: Data representations encode candidate and query instruction-response pairs as feature vectors intended to capture performance-predictive candidate-query distances.The selection algorithm then uses these representations to choose B candidates.
  • 3.2. Data Representation: RDS+ represents samples using position-weighted means of base-model hidden states, while EMBED uses an off-the-shelf sentence encoder with lower computational cost.EMBED’s smaller representations significantly reduce FLOPs relative to RDS+.
  • 3.2. Data Representation: LESS represents samples with low-dimensional gradient features derived from an optimization-aware influence approximation.It averages cosine similarity between query gradients and candidate Adam update vectors across training checkpoints.
  • 3.3. Selection Algorithm: Greedy round-robin repeatedly selects the highest-similarity candidate for each query sample until budget B is exhausted.Selected candidates are removed before the process continues.
  • 3.3. Selection Algorithm: Doubly greedy assigns each candidate its maximum similarity to any query point and selects the top-B candidates.Its score is computed from a similarity matrix.
  • 3.3. Selection Algorithm: KNN-Uniform assigns uniform mass to each query’s K nearest neighbors and selects candidates with the largest summed mass.K is chosen using a trade-off between alignment and diversity.
  • 3.3. Selection Algorithm: KNN-KDE reduces near-duplicate effects by weighting neighboring candidates according to inverse density estimates, while UOT explicitly transports query mass and penalizes marginal deviations.UOT can ignore outliers and less relevant samples for the target task.

4. Experimental Setup

The experiments compare representations and selection algorithms across target tasks using Llama 2 7B and controlled distance-quantile protocols. They evaluate both query-set loss and downstream performance through task-level correlations.

  • Experimental setup: The primary setup uses Llama 2 7B, selected training subsets, cosine similarity or distance, and multiple target tasks.Additional models are evaluated in an appendix.
  • Query-loss evaluation: Figure 2 stratifies candidates into 10 representation-specific distance quantiles, selects 500 examples per quantile with RR, and reports query cross-entropy loss and Spearman correlation.Quantile 1 is closest to the query and quantile 10 is farthest.
  • Evaluation comparison: LESS (RR) shows stronger distance-performance trends than RDS+ (RR) and EMBED (RR) in both query-loss and downstream analyses.LESS has a strong positive loss-distance trend and a strong negative downstream-performance trend across most tasks.
  • Downstream evaluation: Figure 3 applies the same quantile construction and training protocol while measuring downstream task performance and Spearman correlation.The evaluation compares how subset-query distance relates to downstream outcomes.

5. Experiments and Analysis

The experiments test whether subset distance to the query predicts outcomes, then compare data representations and selection algorithms across budgets. Gradient-based LESS representations provide the most reliable query-loss signal, while downstream and algorithmic advantages depend on task, budget, model, and candidate pool.

  • Subset distance and performance: Only LESS-created distance quantiles show strong, predictable correlation with query loss across target tasks.RDS+ and EMBED have very low Spearman correlation, and their closest quantiles do not consistently yield the lowest loss.
  • Subset distance and performance: LESS is the only representation whose subset distance predictably correlates with both query loss and downstream performance.This pattern is less consistent for downstream performance with newer, over-trained models and when Dolci Instruct is the candidate pool.
  • Data representations: LESS achieves the lowest query loss across target tasks under fixed greedy round-robin selection, while RDS+ and EMBED can underperform Random at larger budgets.The query-loss curves use budgets of 500, 1,000, 2,500, 5,000, and 10,000 examples.
  • Data representations: LESS performs best on BBH, TyDiQA, and MMLU-Pro, whereas RDS+ performs best on GSM8K and Random remains competitive on Codex.These downstream comparisons use Llama 2 7B trained on subsets selected with greedy round-robin.
  • Selection algorithms: Selection algorithms exhibit a budget trade-off: greedy round-robin performs best at smaller budgets, while UOT and KNN-KDE improve relative performance at larger budgets.RR’s gains generally diminish as more examples are added; UOT achieves the lowest query loss on three of five datasets.
  • Selection algorithms: Selection trends generalize across models for particular target tasks but are candidate-pool dependent, and DG consistently underperforms across three of five datasets.With Dolci Instruct, no selection algorithm consistently shows strong performance; LESS representations also require more computation than model-based representations.

6. A Unifying View: Instruction Selection as Set Distance Minimization

Targeted instruction-selection methods can be unified as approximate minimization of a distance between selected examples and the query set. This view yields generalization bounds while predicting diminishing benefits over random sampling as budgets increase.

  • 6. A Unifying View: Instruction Selection as Set Distance Minimization: Greedy, density-based, and optimal-transport methods approximately minimize a distributional distance between the selected subset S and query set Q.Their inductive biases differ through the distance or approximation used.
  • 6. A Unifying View: Instruction Selection as Set Distance Minimization: Reducing Dist(S, Q) tightens a generalization bound on downstream loss, establishing a shared theoretical objective for selection algorithms.The bound connects subset-query distance to target-task performance.
  • 6.1. Minimizing Subset-Query Distance Tightens a Generalization Bound: The target loss is upper-bounded by the 1-Wasserstein distance W1(ˆPS, ˆPQ) between selected-subset and query empirical distributions.The result views selection as two-stage domain adaptation from S to Q and then Q to T.
  • 6.1. Minimizing Subset-Query Distance Tightens a Generalization Bound: Only W1(ˆPS, ˆPQ) depends directly on S, so minimizing it tightens the bound, whereas W1(ˆPQ, ˆPT) imposes an irreducible query–test mismatch.The query–test term is fixed when Q and T are fixed.
  • 6.1. Minimizing Subset-Query Distance Tightens a Generalization Bound: Distance minimization alone does not guarantee strong target performance because training error and ideal joint error can remain large.Failure can arise from noisy or inconsistent subsets, insufficient candidate-pool coverage, or an inadequate hypothesis class.
  • 6.2. Diminishing Returns of Query-Aware Selection as Budget Increases: Query-aware selection is computationally expensive, while uniform sampling is nearly cost-free and becomes increasingly competitive as budget B grows.The theoretical comparison uses a Wasserstein-optimal subset against an i.i.d. random sample of equal size.
  • 6.2. Diminishing Returns of Query-Aware Selection as Budget Increases: On MMLU-Pro, LESS variants approach random-sampling performance as B increases, qualitatively matching the predicted diminishing gap.The figure compares loss gaps across budgets and uses the B^-1/d rate as a reference.
  • 6.2. Diminishing Returns of Query-Aware Selection as Budget Increases: The leading budget-dependent bound scales as B^-1/d, so high-dimensional representations can require B on the order of ε^-d before random sampling becomes competitive.Query-aware selection can therefore be especially beneficial at smaller budgets.

B. Target Tasks

The experiments evaluate diverse reasoning, coding, multilingual question-answering, and academic-professional reasoning tasks, while excluding saturated or costly benchmarks. The section also introduces optimal transport and LESS’s gradient-based influence representations used for selection.

  • Target tasks: The target-task suite includes BBH, Codex, GSM8K, TyDiQA, and MMLU-Pro, spanning reasoning, coding, multilingual QA, and professional-domain evaluation.BBH emphasizes difficult reasoning; Codex measures coding correctness; GSM8K tests multi-step arithmetic; TyDiQA tests multilingual QA; MMLU-Pro targets advanced reasoning.
  • Target tasks: Codex uses 16 query samples from a 164-example dataset, with the remaining examples forming the downstream test set.The downstream metric is pass@10.
  • Dataset scope: MMLU and SQuAD are excluded because they are largely saturated, while AlpacaEval is excluded because GPT-based evaluation makes large-scale comparison prohibitively expensive.These exclusions are specific to the main experiments.
  • Optimal transport: Optimal transport compares probability distributions through their geometry, and entropy regularization enables efficient solution with Sinkhorn–Knopp for discrete problems.Unbalanced optimal transport additionally relaxes exact marginal constraints by penalizing deviations from prescribed marginals.
  • LESS representations: LESS represents examples using projected query gradients and candidate Adam updates, then aggregates checkpoint-wise cosine similarities weighted by average learning rates into influence scores.The resulting influence matrix supplies similarities between query and candidate examples for selection.

G. Training Details and Hyperparameters

The experiments train on a preprocessed candidate pool under a fixed supervised fine-tuning setup, then examine how distance-based subsets, representations, and budgets relate to query and downstream performance. LESS most consistently tracks query-loss distance, while subset-performance differences within the closest quantile are small.

  • Training setup: The candidate pool contains about 198K examples after removing responses exceeding the 2048-token maximum sequence length.Candidate and query examples receive the chat template during instruction selection.
  • Downstream performance: LESS (RR) has a stronger negative downstream-performance correlation on average, but performance differences among subsets in the closest quantile are small.For downstream performance, more negative Spearman correlation is better.
  • Distance sub-quantiles: The closest distance quantile is subdivided into 10 sub-quantiles, from closest to farthest, and 500 examples are selected from each for Llama 2 7B training.This tests whether representations distinguish among highly similar candidate subsets.
  • Distance sub-quantiles: LESS (RR) shows high query-loss Spearman correlation across target tasks, whereas RDS+ (RR) and EMBED (RR) show weak or often negative correlations across distance sub-quantiles.The result indicates that LESS better differentiates similar subsets by query loss.
  • Selected-subset differences: LESS (RR) selects shorter sequences, whereas RDS+ (RR) and EMBED (RR) select longer sequences and have higher Jaccard indices than LESS (RR).The comparison uses selected subsets at a budget of 10,000 samples.

J. Cheaper Proxies for LESS

The proxy-model experiments test whether smaller models can compute LESS representations for selecting data used to train Llama 2 7B. Proxy models can match or exceed larger-model baselines, but their suitability varies and requires further investigation.

  • Motivation: LESS representations are expensive because they require forward and backward passes over all candidate samples.The study therefore evaluates smaller proxy models for representation computation.
  • Results: At higher budgets, smaller proxy models outperform the baseline LESS (RR, Llama 2 7B) on three of five target tasks, but Pythia-160M performs as poorly as Random on several tasks.Proxy models are therefore viable but not uniformly effective.
  • Results: LESS (RR, SmolLM-135M) and LESS (RR, SmolLM2-135M) match or outperform Random across all target tasks and sometimes outperform LESS (RR, Llama 2 7B).SmolLM2-135M uses 2 trillion pre-training tokens, compared with 600B for SmolLM-135M.
  • Results: LESS (RR, Llama 3.2 3B) outperforms LESS (RR, Llama 2 7B) on three of five target tasks, indicating a trade-off between proxy and downstream model sizes.The comparison concerns how well proxy models approximate instruction selection for larger models.
  • Implication: Proxy models may dramatically reduce selection cost, but choosing suitable proxies requires more thorough investigation.The paper does not establish a general rule for proxy-model choice.

K. KNN-Uniform and KNN-KDE with L2 Distance

The study tests KNN-Uniform and KNN-KDE with L2 distances using fixed Llama 2 7B LESS representations, and compares these results with cosine-distance variants. The resulting performance trends are similar across distance choices.

  • Setup: KNN-Uniform and KNN-KDE are evaluated with L2 distances while keeping Llama 2 7B LESS representations fixed.The experiment follows the original implementation’s L2-distance choice.
  • Setup: The L2 setup rescales checkpoint representations by average learning rate, concatenates them across epochs, and normalizes the resulting vectors by their L2 norm.Distances are then computed between query and candidate representations.
  • Results: KNN-KDE and KNN-Uniform with L2 show performance trends similar to cosine-distance variants across budgets.The result further validates the section’s conclusions about selection-algorithm choice.
  • Theory: The theoretical appendix assumes strong convexity, smooth data dependence, and Lipschitzness at the empirical-risk minimizer for its stability results.Under these assumptions, the Wasserstein stability bound relates target-loss differences to W1 distance between empirical dataset measures.
  • Theory: A high-probability bound controls the Wasserstein distance between a uniformly sampled subset and the full dataset under bounded support and dimension d ≥ 3.The proof combines an expectation bound with concentration from McDiarmid’s inequality.

L.2. Theorem 6.1

Theorem 6.1 bounds downstream test loss using selected-set, query-set, and query-to-test Wasserstein distances, plus dataset-dependent and combined-error terms. The resulting decomposition identifies which distance selection can reduce and which mismatch remains independent of the selected subset.

  • The theorem assumes a symmetric, convex, bounded loss with triangle inequality, strong convexity of LS, Lipschitz continuity of LT, and Lipschitz gradients in z.
  • W1(PQ, PT) measures query-to-test mismatch and imposes an irreducible limit on how informative Q is about T.
  • Theorem 6.1 upper-bounds LT(θS) by LS(θS), two 1-Wasserstein distances, and a dataset-size-dependent constant.The bound is LT(θS) ≤ LS(θS) + W1(PS, PQ) + W1(PQ, PT) + ζ.
  • Minimizing W1(PS, PQ) is therefore a principled selection objective because it tightens the bound on LT(θS).

M. Model Ablations with Tulu V2

Across model ablations with Tulu V2, gradient-based LESS representations most reliably align subset-query distances with query loss, while downstream rankings vary by model, task, and budget. Greedy round-robin is often strongest at low budgets, but zero-shot can still outperform tuned models in some settings.

  • Distance quantile experiment: LESS (RR) creates subsets whose distance quantiles strongly correlate with query loss across the evaluated models.This relationship weakens for over-trained SmolLM3 3B Base and Qwen3 4B Base on BBH and Codex.
  • Effect of data representation across subset budgets: LESS with greedy round robin performs best across models on most target tasks, although no fixed representation-algorithm pair always wins.
  • Effect of data representation across subset budgets: Zero-shot inference outperforms instruction-tuned models on some datasets and models, including MMLU Pro with Qwen3 4B Base.The authors suggest insufficient candidate-pool examples or possible pretraining-task leakage as explanations.
  • Effect of selection algorithm across subset budgets: LESS (RR) achieves the strongest BBH results with Llama 3.2 3B and Qwen3 4B Base, while LESS (UOT) often performs best on higher-budget MMLU-Pro with Olmo 3 7B Base.
  • Multitask instruction selection: Multitask selection results for Llama 3.2 3B closely follow single-task results, suggesting that the single-task insights generalize to multitask selection.

O. Model Ablations with Dolci Instruct

With Dolci Instruct, LESS remains a reliable predictor of query loss and often correlates with downstream metrics, but downstream method rankings depend strongly on the task, budget, model, and candidate pool. Random selection becomes competitive at higher budgets.

  • Results: LESS creates subsets whose distance to the query strongly correlates with query loss across model-task pairs.
  • Results: Subset-query distance often correlates with downstream metrics under Dolci Instruct, with LESS showing stronger correlations than other representations.
  • Results: RDS+ (RR) and EMBED (RR) often match or outperform LESS (RR) on BBH, Codex, and GSM8K, whereas LESS (RR) consistently wins on TyDiQA across models.
  • Results: No LESS-based selection algorithm dominates across datasets, budgets, or models.
  • Results: Random selection is competitive at higher budgets, consistent with benefits from a well-curated candidate pool such as Dolci Instruct.
Loading 2602.14696v2…