PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Curation Alone Can Stabilize In-context Learning

Ting-Yun Chang, Robin Jia

arXiv:2212.10378v2cs.CL

TL;DR

ICL is highly sensitive to which training examples are sampled, motivating methods that stabilize performance without changing the ICL algorithm. The paper scores examples individually and selects stable subsets, improving average accuracy over the full training set while finding that these examples are not especially diverse or low in perplexity.

  • Problem

    ICL performance varies substantially with the choice and order of randomly sampled training examples.

    Source: §1 · p. 1

  • Method

    CondAcc ranks examples by conditional dev-set accuracy, while Datamodels learn linear regressors estimating how example presence influences LLM outputs.

    Source: §1 · p. 1, §3.2 · p. 3

  • Results

    Across five tasks and two LLMs, CondAcc and Datamodels improve average accuracy over ALL by 7.7% and 6.3%, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Curating a smaller training subset can make randomly sampled ICL prompts substantially more stable and accurate without prompt retrieval or calibration.

    Source: §1 · p. 1, §8 · p. 9

    The methods require substantial inference computation and memory, with one setup exceeding 500 GPU hours.

    Source: Limitations · p. 10

Abstract

from arXiv · show

In-context learning (ICL) enables large language models (LLMs) to perform new tasks by prompting them with a sequence of training examples. However, it is known that ICL is very sensitive to the choice of training examples: randomly sampling examples from a training set leads to high variance in performance. In this paper, we show that carefully curating a subset of training data greatly stabilizes ICL performance without any other changes to the ICL algorithm (e.g., prompt retrieval or calibration). We introduce two methods to choose training subsets -- both score training examples individually, then select the highest-scoring ones. CondAcc scores a training example by its average dev-set ICL accuracy when combined with random training examples, while Datamodels learns linear regressors that estimate how the presence of each training example influences LLM outputs. Across five tasks and two LLMs, sampling from stable subsets selected by CondAcc and Datamodels improves average accuracy over sampling from the entire training set by 7.7% and 6.3%, respectively. Surprisingly, the stable subset examples are not especially diverse in content or low in perplexity, in contrast with other work suggesting that diversity and perplexity are important when prompting LLMs.

1 Introduction

ICL can perform new tasks without parameter updates, but its performance is highly sensitive to sampled examples and prompt order. The paper shows that curated training subsets stabilize and improve ICL, with CondAcc and Datamodels outperforming random selection.

1 Introduction

  • The paper studies ICL with pretrained LLMs, which perform tasks from labeled-example prompts without parameter updates.ICL avoids task-specific training and uses one model across tasks, supporting easier deployment than finetuning alternatives.Source: §1 · p. 1
  • ICL is sensitive to training-example choice, their prompt order, and small prompt-format changes, despite strong few-shot performance.Standard ICL randomly samples a few training examples to construct each prompt.Source: §1 · p. 1
  • CondAcc scores examples by average dev-set ICL accuracy with random companions, whereas Datamodels use learned linear weights to estimate each example’s effect on outputs.Both methods select the highest-scoring examples to form a stable subset.Source: §1 · p. 2
  • The methods outperform ALL, which uses the entire training set without selection, by 7.7% and 6.3%, respectively, in average accuracy across multiple sampled prompts.The stable subset examples generalize to out-of-distribution test data.Source: §1 · p. 2
  • Stable subsets are not especially diverse, low-perplexity, or unusually long, challenging explanations based on diversity and perplexity.The analysis considers sequence length, perplexity, and diversity in raw-text and embedding spaces; it finds no abnormal sequence lengths or high perplexities and no greater diversity than random subsets.Source: §1 · p. 2

2 Problem Setups

The paper seeks a subset larger than the prompt size whose random K-example prompts reliably perform well on held-out data. It evaluates labeled and unlabeled setups, including cases where candidate examples may have incorrect labels.

2 Problem Setups

  • The goal is to select E ⊂ Dtr with |E| > K so random K-example prompts from E generally perform well on Dtest.Prompts concatenate K labeled training examples, and predictions condition on that prompt plus a test input.Source: §2 · p. 2
  • The labeled setup studies which examples consistently produce high ICL accuracy using a large labeled training set and a small labeled dev set.The large labeled test set is reserved for evaluating selection methods.Source: §2 · p. 2
  • The unlabeled setup uses a large unlabeled input set and a small labeled dev set, pairing each input with every possible label.This setup may select examples with incorrect labels, while the labeled test set is used only for evaluation.Source: §2 · p. 2

3 Methods

The methods construct many random prompts, estimate each training example’s importance, and select high-scoring examples per class. CondAcc uses conditional accuracy, while Datamodels model output margins from example presence and prompt position.

3 Methods

  • The pipeline builds M random K-example prompts, evaluates them on the dev set, scores examples with CondAcc or Datamodels, and selects the highest-scoring examples per class.The selected subset contains E/C top-scoring examples from each of C classes.Source: §3 · pp. 2–3, §3.3 · p. 4

3.1 CondAcc

  • CondAcc scores an example by expected dev-set accuracy conditioned on its appearing in randomly sampled prompts.Each example is observed repeatedly across different prompt orders and companion examples; the score is related to Data Shapley values (Ghorbani and Zou, 2019).Source: §3.1 · p. 3

3.2 Datamodels

  • Datamodels address CondAcc’s order-insensitivity and confidence limitations by learning linear regressors from example presence and prompt index to LLM outcomes.The outcome is the correct-label logit minus the highest incorrect-label logit, and each model is trained per dev example with mean-squared error.Source: §3.2 · p. 3
  • A positive datamodel weight indicates that an example at a prompt position encourages correct answers; scores aggregate positive effects across dev examples and positions.Counting positive weights works better empirically than averaging all datamodel weights.Source: §3.2 · p. 4
  • Datamodels accurately approximate outcomes for unseen prompts outside DICL across tasks, supporting their use as an analyzable proxy for inference-time ICL behavior.The approach requires repeated LLM inference on different prompts rather than retraining the target model.Source: §3.2 · pp. 3–4

4 Experiment

The experiments evaluate curated subset selection across five classification tasks and multiple LLMs, using repeated prompt sampling to measure accuracy and stability against several baselines.

4.1 Setups

  • The study covers five classification tasks, with main experiments on GPTJ-6B and OPT-13B and additional results on GPT-Neo-2.7B and OPT-6.7B.The stable subset size is E = 20; binary-task prompts use K = 4 without class balancing, while multiclass prompts are class-balanced.Source: §4.1 · p. 4

4.2 Evaluation and Baselines

  • Evaluation samples 50 prompts from each selected subset and reports average accuracy, standard deviation, and worst accuracy on the test set.The same evaluation framework is used to assess whether a selected subset is more stable than the full training set.Source: §4.2 · p. 4
  • CONDACC and DATAMODELS are compared with ALL, CALIB, RANDOM, ONESHOT, TOPPROMPTS, and unlabeled-data variants.ALL uses the full training set; ONESHOT scores examples with one-shot ICL, while TOPPROMPTS unions examples from the highest-scoring prompts.Source: §4.2 · pp. 4–5
  • Table 1 reports average accuracies across all tasks and identifies CONDACC and DATAMODELS as the best overall selection methods.The table also includes the unlabeled UN-CONDACC method, which is reported to outperform the gold-labeled ALL baseline.Source: Table 1 · p. 5

5 Results

Across the main benchmarks and additional settings, curated subsets improve ICL accuracy and stability, including unlabeled, single-label, and out-of-distribution prompts.

5.1 Main Results

  • 7.7% and 6.3% average-accuracy gains over ALL are achieved by CONDACC and DATAMODELS, respectively, across the main experiments.The methods also achieve higher average and worst-case accuracy across tasks, with lower variance on most tasks, and outperform CALIB in 8/10 setups.Source: §5.1 · p. 5

Does UN-CONDACC benfit from gold labels?

  • UN-CONDACC outperforms UN-ALL and gold-labeled ALL by 10.2% and 5.7% on average, respectively, despite some selected examples having wrong labels.Correcting wrong labels on BoolQ decreases average and worst accuracy, suggesting that mislabeled selected examples can sometimes perform better.Source: §5.1 · p. 6, Does UN-CONDACC benfit from gold labels? · p. 6
  • Using four curated examples can outperform using 16 or 24 randomly sampled examples on SST-2 and AGNews.This comparison supports the benefit of curated examples relative to simply increasing the number of random shots.Source: Does UN-CONDACC benfit from gold labels? · p. 6

5.2 Single-Label Prompts

  • CONDACC and DATAMODELS substantially outperform majority guessing when prompts contain four examples sharing only one label.In contrast, ALL and TOPPROMPTS perform near chance because single-label prompts bias predictions toward that label; selected examples still provide useful task-definition signal.Source: §5.2 · p. 6, Table 2 · p. 6

5.3 Out-of-Distribution Tasks

  • CONDACC and DATAMODELS outperform baselines on OOD IMDb and BoolQ Contrast Set evaluations using prompts selected from SST-2 and BoolQ, respectively.The result is consistent with selected examples generalizing beyond the source tasks rather than merely overfitting them.Source: §5.3 · p. 6, Table 3 · p. 6
  • Good-subset examples are not outliers with abnormally long sequences or high perplexity.Figure 3 plots each example’s CONDACC importance score against sequence length and perplexity.Source: Figure 3 · p. 7

6 Analysis

The analysis examines which properties distinguish stable training examples, finding limited roles for sequence length, perplexity, and diversity while revealing mixed cross-LLM transfer.

6.1 Sequence Length and Perplexity

  • Good examples avoid the tail of very long sequences, although accuracy has little sequence-length correlation except a slight negative correlation for very long examples.The analysis suggests abnormally long training examples can hurt ICL performance.Source: §6.1 · p. 7
  • Good examples are not high-perplexity outliers, and perplexity alone has no correlation with accuracy across tasks and LLMs.The authors contrast this finding with concurrent work linking lower prompt perplexity to better performance.Source: §6.1 · p. 7

6.2 Diversity

  • Stable subsets are not more diverse than random subsets, and both CONDACC and DATAMODELS select tightly clustered examples in embedding space.This pattern appears across DIV-I, DIV-F, and the Datamodels embedding visualization.Source: §6.2 · pp. 7–8, Figure 4 · p. 8

6.3 Do LLMs find the same stable examples?

  • Stable-example scores and subset overlap show mixed transfer across LLMs, with moderate correlation for some OPT pairs but little correlation for many others.Different LLMs generally identify few overlapping stable examples.Source: §6.3 · p. 8, Table 4 · p. 8
  • Four SST-2 examples shared by GPTJ-6B and OPT-13B achieved 88.6 ± 3.7, 89.8 ± 3.0, and 87.3 ± 5.2 accuracy across models’ 4! permutations.The reported accuracies are for GPTJ, OPT-6.7B, and OPT-13B, respectively.Source: §6.3 · p. 8, Table 5 · p. 8

7 Discussion and Related Work

The discussion positions stable-subset selection as a task-level alternative to instance-dependent retrieval and connects it to data valuation for in-context examples.

7.1 Prompt Retrieval

  • For classification tasks, the paper identifies task-level examples that work across test inputs, avoiding per-example prompt retrieval.This contrasts with retrieval methods that select similar training examples for each test input, which are important for semantic parsing.Source: §7.1 · p. 9

7.2 The influence of in-context labels

  • The method finds subsets of single-label examples that perform well, while prior work shows label proportions can bias predictions and label correctness varies by task.The authors also report that some wrong-label examples can yield surprisingly good performance.Source: §7.2 · p. 9

7.3 Data Valuation

  • CONDACC resembles Data Shapley and DATAMODELS adapts Datamodels to attribute LLM performance to in-context examples without parameter updates.The paper differs from concurrent work by modeling example positions and focusing on few-shot K ∈ {3, 4}.Source: §7.3 · p. 9

8 Conclusion

The paper concludes that curating stable subsets can reduce ICL sensitivity without changing the underlying prompting algorithm, while the selected examples do not follow simple diversity, length, or perplexity heuristics.

8 Conclusion

  • The two proposed methods achieve substantially higher average and worst-case accuracy across different setups by identifying stable training subsets.CONDACC is intuitive and easy to implement, while DATAMODELS provides informative weights for further analysis.Source: §8 · p. 9
  • Using a suitable subset of 20 examples makes ICL less sensitive to training-example choice and prompt order.The conclusion frames this as selecting the pool from which prompts are randomly sampled.Source: §8 · p. 9
  • Stable subsets contain no especially long or high-perplexity outliers and are no more diverse than random training subsets.These findings motivate future guidelines for finding or writing better training examples.Source: §8 · p. 9

Limitations

The paper’s main limitations are substantial inference cost and restricted experimental scope across task type, model size, shot count, and prompt templates.

Limitations

  • The methods require repeated ICL evaluations over roughly 50,000 prompts, with the most expensive setup exceeding 500 GPU hours on an RTXA6000.The authors’ preliminary study indicates that at least 10,000 randomly sampled prompts may be needed for good performance.Source: Limitations · p. 10
  • The study covers only classification tasks, models up to 13B parameters, fixed shot counts, and fixed prompt templates.Applicability to generation tasks and models larger than 100B remains unstudied.Source: Limitations · p. 10

A.1 Connection with Data Shapley

CONDACC is tightly connected to Data Shapley: its conditional-accuracy score determines the Shapley value through a monotonically increasing affine transformation.

A.1 Connection with Data Shapley

  • Data Shapley values are defined from expected dev-set ICL accuracy over permutations of K-shot subsets, with zero valuation for subsets whose size differs from K.The resulting value measures each training example’s contribution to the ICL predictor under the paper’s K-shot setup.Source: §A.1 · p. 13
  • The first Shapley term equals the expected conditional accuracy when example e_i appears in a prompt, while the second is the corresponding expectation when it does not.The analysis denotes the latter quantity as t(i) and the overall random-prompt accuracy as A.Source: §A.1 · p. 13
  • The intermediate algebra rewrites the Shapley expression using the overall accuracy A and conditional-accuracy score s_ca(i), isolating the example-dependent term.This establishes why selecting examples by CONDACC is equivalent to selecting them by their Data Shapley values in this setup.Source: §A.1 · p. 13
  • The paper shows that s_shap(i) is a monotonically increasing affine function of s_ca(i), making CONDACC and Data Shapley rank examples identically.N_tr, K, and A are constants independent of i, so the ordering is preserved.Source: §A.1 · p. 13

A.2 Relation to Prompt Tuning

The appendix relates curated random prompts to prompt tuning, validates datamodel predictions, and reports broader experiments showing that curated subsets remain effective across models, tasks, and shot counts.

A.2 Relation to Prompt Tuning

  • Unlike soft Prompt Tuning embeddings learned through backpropagation, this method randomly samples a fixed sequence of selected training examples for all test inputs.The approach does not access LLM parameters, avoiding Prompt Tuning’s memory cost and training instability and fitting API-only settings.Source: §A.2 · p. 13

A.3 Evaluating Datamodels

  • Datamodels are evaluated on 5,000 newly sampled prompts containing unseen combinations of training examples, using correlations and L1 distance against ground-truth LLM outcomes.Figure 5 visualizes 10,000 sampled predicted-versus-ground-truth outcomes.Source: §A.3 · p. 13, Table 6 · p. 13, Figure 5 · p. 17
  • Datamodels accurately approximate LLM outcomes across tasks and models, suggesting that example presence and prompt index capture dominant effects on ICL predictions.The models use only these two simple features, while label-pattern-specific training improves prediction accuracy.Source: §A.3 · pp. 13–14, §A.4 · p. 14

A.5 Implementation Details

  • The appendix uses GPT-Neo-2.7B, GPTJ-6B, OPT-6.7B, and OPT-13B, with most evaluations run on a single 48GB RTX A6000 GPU.Data collection costs hundreds of GPU hours, but subsequent DATAMODELS training takes about five seconds on a CPU and CONDACC requires no training.Source: §A.5 · p. 14

A.6 More Experiments

  • CONDACC and DATAMODELS substantially outperform four baselines across additional LLMs and the MNLI task, although MNLI performance remains poor overall.The experiments use 50 sampled prompts; MNLI is not pursued further because ICL performs poorly there.Source: §A.6 · p. 14, Table 9 · p. 15

A.7 Why not Instruction-Finetuned LLMs?

  • Instruction-tuned LLMs are excluded because multitask training creates leakage into the evaluated datasets.The cited examples include T0, FLAN-T5, and OPT-IML, each trained on several tasks used in the paper.Source: §A.7 · p. 14

A.8 Larger Number of Shots

  • Using 8–24 shots improves over 3–4 shots on most tasks, but MAXSHOT surpasses CONDACC on only two of five tasks.The comparison supports curated examples over merely increasing the number of randomly sampled examples.Source: §A.8 · p. 16, Table 12 · p. 16
  • Across tasks, accuracy shows no observed correlation with perplexity, while example quality is assessed per class rather than solely by globally highest or lowest scores.The figures also visualize accuracy against sequence length and datamodel embeddings, where class information is encoded in the embeddings.Source: Figure 6 · p. 18, Figure 7 · p. 19, Figure 8 · p. 20
Loading 2212.10378v2…