Finding Support Examples for In-Context Learning

1 Introduction

• LENS targets a limited list of informative, representative examples because language models’ maximum input length constrains how many examples can be supplied.The paper calls these selected examples support examples because they provide crucial task information to the language model.Source: §1 · p. 1
• Because ICL runs in inference without gradients or parameter updates, gradient-dependent coreset methods are poorly suited to selecting informative examples (Paul et al., 2021; Toneva et al., 2019; Mirzasoleiman et al., 2020; Killamsetty et al., 2021a,b; Guo et al., 2022).Source: §1 · p. 2
• In-context examples are sensitive to both selection and ordering (Lu et al., 2022; Chang and Jia, 2023), and examples that perform well individually can hurt when combined.The case study illustrates this combinatorial dependency, while enumerating all combinations is infeasible because of combinatorial explosion.Source: §1 · p. 2, Table 1 · p. 2
• LENS defines support-example selection for ICL and uses InfoScore-based progressive filtering followed by diversity-guided search to find examples that depict the task.InfoScore evaluates each example’s in-context informativeness from the language model’s feedback.Source: §1 · p. 2
• Experiments on varied text-classification datasets show LENS significantly outperforms a wide range of baselines, while prior coreset methods yield only marginal gains over random selection.The paper reports comprehensive empirical results for prior coreset methods on ICL, a setting it says had not previously been explored.Source: §1 · p. 2
• Support examples are less order-sensitive than random examples (Lu et al., 2022), depend on ground-truth labels unlike random examples (Min et al., 2022b), and transfer across language models of different sizes and pre-training corpora.Source: §1 · p. 2

2 Background:In-Context Learning

• A causal language model predicts a test input from concatenated labeled examples, which provide the task supervision.The setup considers a language model, a set of n examples, and a test input.Source: §2 · p. 3
• Classification labels are often mapped to one or more words in the model’s vocabulary, such as “great” and “terrible” for positive and negative sentiment.The definition omits the verbalizer, special tokens, and prompting templates for simplicity.Source: §2 · p. 3
• Because the context length limits n, researchers typically randomly sample examples, but ICL is sensitive to example choice and random examples can yield unstable, inferior performance (Brown et al., 2020; Zhang et al., 2022a; Lu et al., 2022; Chen et al., 2022).The paper therefore focuses on selecting a small, informative support set from the full dataset.Source: §2 · p. 3

3.1 Informative Examples Filtering

• InfoScore measures an example’s task-level in-context informativeness by summing how much it changes the LM’s probability of the ground-truth label across training examples.It compares the probability conditioned on the candidate example and input with the probability conditioned on the input alone.Source: §3.1 · p. 3
• Progressive filtering expands the score set as candidates shrink, allocating more computation to promising examples and achieving O(N * log_ρ N) complexity.The process begins with a small sampled score set, filters the dataset iteratively, and refines scores until a small candidate set remains.Source: §3.1 · pp. 3–4

3.2 Diversity-Guided Example Search

• Diversity-guided search substitutes examples to balance informativeness with diversity, avoiding candidates whose similar in-context features would make their effects redundant.The feature vector encodes each example’s contribution to a score set, and the filtering-stage computations can be reused.Source: §3.2 · p. 4
• Beam search combines diversity-based substitutions with random shuffling, then keeps the top-performing permutations on a small validation set to reduce local-optimum risk.The search starts from candidate permutations and iteratively refines and evaluates them.Source: §3.2 · pp. 4–5

4.1 Experimental Settings

• Experiments cover eight text-classification datasets across three task families, using GPT2-L and averages over four prompt templates and ten seeds, with and without calibration.LENS uses m = 500, λ = 1, beam size B = 8, and 10 diversity-search iterations.Source: §4.1 · p. 5

4.2 Main Results

• LENS significantly outperforms baselines across all datasets with and without calibration, and is more robust than Random-Validation to validation-selected permutations that underperform on test.Random examples slightly underperform Zero-Shot; gradient-based methods perform similarly to Random.Source: §4.2 · pp. 5–6, Table 2 · p. 6

4.3 Analysis

• Across SST-2, Amazon, MR, and Subj, LENS support examples perform more stably across orders than random examples, whose worst orders reach random-guess performance.Most support-example orders remain near the searched-order performance, and few reach random guess.Source: §4.3 · p. 6, Figure 2 · p. 6
• GPT2-L support examples outperform Random on GPT2-M, GPT2-XL, and GPT-Neo-2.7B, including a model with a different pre-training corpus.The authors interpret this as strong transferability and utility for more powerful language models.Source: §4.3 · p. 7, Table 3 · p. 7
• Removing ground-truth labels strongly degrades support-example performance, although randomly labeled support examples still improve on randomly sampled examples.This contrasts with the reported consistency of random-example performance with or without ground-truth labels.Source: §4.3 · p. 7, Figure 3 · p. 7
• Across hyperparameter settings, LENS generally improves on Random except at B = 0, while m = 100 and B = 1 show slight degradations.B = 0 removes stage 2 and significantly degrades performance; B = 1 reduces stage 2 to diversity-guided greedy search, making it susceptible to local optima.Source: §4.3 · pp. 7–8, Table 4 · p. 7
• Filtering informative examples outperforms random sampling, and performance declines from Filtering (Informative) to Random to Filtering (Uninformative), supporting InfoScore’s informativeness signal.Filtering (Informative) still trails the full method, indicating the need to account for dependencies among examples through diversity-guided search.Source: §4.3 · p. 8, Table 5 · p. 8

5 Related Work

• Prior ICL retrieval methods find relevant examples using semantic or dense signals, language-model feedback, reranking, and diversity.Examples include semantic retrieval, task-target dense retrieval for question answering and dialogue state tracking (Das et al., 2021; Hu et al., 2022), language-model-feedback retrievers for semantic parsing (Rubin et al., 2022), permutation reranking, and diversity (Levy et al., 2022; Ye et al., 2023).Source: §5 · p. 8
• This work instead selects task-representative examples, complementing methods that retrieve examples specific to each test input.Because causal Transformer language models allow task-level representations to be calculated in advance and reused across test inputs, the authors leave a hybrid setting combining both goals for future work (Brown et al., 2020; Zhang et al., 2022a; Vaswani et al., 2017).Source: §5 · p. 8
• Active-learning approaches select unlabeled examples for annotation, while this work selects support examples from annotated datasets.Zhang et al. (2022b) learn an active selector with offline reinforcement learning; Su et al. (2022) propose graph-based vote-k annotation and retrieve from annotated examples. The paper motivates its setting by the availability of massive annotated datasets and large language models’ annotation ability (Ye et al., 2022; Meng et al., 2022).Source: §5 · p. 8

6 Conclusion

• LENS uses a two-stage filter-then-search method to find support examples for in-context learning from an annotated dataset.The method first selects informative examples individually, then searches for example permutations that fully depict the task.Source: §6 · p. 8
• InfoScore and progressive filtering select informative examples individually before diversity-guided search iteratively refines and evaluates selected examples.The search aims to find example permutations that fully depict the task.Source: §6 · p. 8
• Experiments show that LENS significantly outperforms extensive baselines, and further analyses find that each component contributes critically to the improvements.The analyses also shed light on principles of support examples and in-context learning.Source: §6 · p. 8

Limitations

• Experiments mainly use GPT2-L because of computation limits; exploring more language models is left to future work.The paper analyzes cross-LM transferability of support examples in section 4.3.Source: Limitations · p. 9
• The filter-then-search framework studies how to find in-context support examples, while broader principles of in-context learning remain future work.Source: Limitations · p. 9
• Because filtering relies on language-model feedback, selected examples may reflect model biases; debiasing is identified as future research.The passage cites Bender et al. (2021) on language-model bias.Source: Limitations · p. 9

A Baselines

• Random & Validation selects the best random examples on validation data using either 100 examples and 640 sets or 1,000 examples and 100 sets.The first setting matches the proposed method’s stage-2 validation size and number of searched and evaluated permutations.Source: A Baselines · p. 13
• Geometry-based baselines use Herding and K-Center Greedy, while uncertainty-based baselines include Least Confidence, Entropy, Margin, and CAL.Herding greedily reduces the distance between coreset and dataset centers; K-Center Greedy minimizes the largest distance to a non-coreset example.Source: A Baselines · p. 13
• Error/loss-based baselines include Forgetting and GraNd, while gradient-matching methods include CRAIG and GradMatch.Forgetting counts examples misclassified after earlier correct classification; GraNd uses expected gradient norm, and GradMatch adds regularization to CRAIG and improves its greedy algorithm.Source: A Baselines · p. 13
• Other coreset baselines use submodular functions for informativeness and diversity, or bilevel optimization with validation-set outer objectives and log-likelihood inner objectives.Examples include Facility Location, Graph Cut, and Glister.Source: A Baselines · p. 13
• To adapt traditional coreset methods to in-context learning, the authors fine-tune GPT2-L with last pooling for five epochs and use final-layer gradients as example features.The fine-tuning uses the whole dataset; the gradient feature follows Guo et al. (2022).Source: A Baselines · p. 13

B.1 Baseline Details

• Coreset baselines use GPT2-L with last pooling, fine-tuned on the full dataset for 5 epochs, to obtain metrics such as gradients or forgetting times.Following Guo et al. (2022), example features are the gradients of the final fully connected layer’s parameters.Source: §B.1 · p. 13
• For weighted-subset baselines such as CRAIG or GradMatch, the setup uses their selected examples directly because few methods weight in-context examples for ICL.Source: §B.1 · p. 13

• Support examples are selected separately for each prompting template, and Eq. (3) is computed without calibration in both calibrated and uncalibrated experiments.Source: §B.2 · p. 14
• The progressive filtering factor is set for four iterations, while the initial score-set size targets about 10K language-model forwards.Source: §B.2 · p. 14
• Most datasets use label-balanced selection with eight examples, or a nearby divisible count; DBPedia instead uses four examples under label-unbalanced selection.For balanced datasets, filtering retains the same number per label, stage 2 starts with balanced labels, and each updated example has the same label as the one it replaces.Source: §B.2 · p. 14
• On SST-2, grid search selected m=500, diversity weight λ=1, beam size B=8, and 10 search iterations.The searched values were m∈{500,1000}, λ∈{0.5,1,2}, B∈{4,8,16}, and 5, 10, or 15 iterations.Source: §B.2 · p. 14

B.3 Experimental Details

• For most analyses, performance is averaged over four prompting templates without calibration; the order-sensitivity analysis instead shows the distribution for one specific template.Source: §B.3 · p. 14

Progressive Filtering

• The filtering stage computes pairwise Eq. 3 N*l times, where N is the training-set size.The passage expresses the count as N*l*ρ/ρ = N*l.Source: Progressive Filtering · p. 14

B.3.1 The Complexity of Our Method

• Filtering takes O(N log_ρ N) time over a training set of size N, with log_ρ N iterations as the dataset is repeatedly reduced to 1/ρ of its previous size.Source: Progressive Filtering · p. 14

Progressive Filtering

• The experiments usually set ρ between 2 and 3 according to dataset size.Source: Progressive Filtering · p. 14
• Diversity-guided search updates B candidate permutations B times per iteration and evaluates them on a fixed-size validation set sampled from the remaining training data.The updates reuse filtering-stage intermediate results and do not compute with the LLM; the passage gives search complexity as B ∗ B.Source: Progressive Filtering · p. 14

C Prompting Templates

• Appendix C presents the prompting verbalizers and templates.Source: C Prompting Templates · p. 14

D Dataset Split and Statistics

• For Amazon, AGNews, and DBPedia, experiments use randomly sampled subsets of 30,000 training and 2,000 test examples, with overall statistics reported in Table 6.The dataset split follows Min et al. (2022a).Source: D Dataset Split and Statistics · p. 14
• SST-2 uses the verbalizers {great, terrible} with four prompting templates.The templates are “A [VERBALIZER] one,” “It was [VERBALIZER],” “All in all [VERBALIZER],” and “A [VERBALIZER] piece,” each preceded by [INPUT].Source: D Dataset Split and Statistics · p. 16
• SST-5 uses {great, good, okay, bad, terrible}; Amazon uses the same verbalizers, while MR uses {great, terrible}, and all reuse SST-2’s templates.“Same as SST-2” is specified for the SST-5, Amazon, and MR templates.Source: D Dataset Split and Statistics · p. 16
• TREC uses six verbalizers and four templates; a further setup uses {World, Sports, Business, Technology} with the same templates as TREC.The TREC verbalizers are {Description, Entity, Expression, Human, Location, Number}.Source: D Dataset Split and Statistics · pp. 16–17