Source-linked AI summary
In-context Example Selection with Influences
Tai Nguyen, Eric Wong
TL;DR
ICL performance is highly sensitive to example choice, and existing selection signals lack consensus. The paper uses in-context influences to rank examples and analyze their effects, finding stronger selection performance than baselines and a gap of up to 16.3% between negative and positive examples. The framework also quantifies recency bias in example ordering.
Problem
ICL performance varies with example selection, while prior selection signals have varying effectiveness and lack consensus about which signals matter most.
Source: §1 · p. 1Method
Results
Influence-based selection outperforms several baselines in positive and negative selections across 8 language models and 9 natural language tasks, with gaps up to 16.3%.
Source: §1 · p. 2Takeaways & Limitations
Influences identify both beneficial and problematic examples and support analysis of ICL phenomena such as recency bias in example ordering.
Source: §1 · p. 2, §6 · p. 11The framework predicts performance for examples in a fixed training set, not arbitrary examples constructed on the fly.
Source: §6 · p. 11
Abstract
from arXiv · showhide
In-context learning (ICL) is a powerful paradigm emerged from large language models (LLMs). Despite its promises, ICL performance is known to be highly sensitive to input examples. In this work, we use $\textit{in-context influences}$ to analyze few-shot ICL performance directly from the in-context examples. Our proposed influence-based example selection method can identify both positive and negative examples, outperforming several baselines when evaluated on 9 SuperGLUE tasks. Our analysis uncovers up to a $16.3\%$ performance gap between using the most negative in-context examples compared to the most positive. In a case study, we apply our influence-based framework to quantify the phenomena of recency bias in example ordering for few-shot ICL.
1 Introduction
ICL performance varies substantially with the examples and their ordering, motivating direct measurement of example effects. The paper proposes influence-based selection and reports stronger performance estimates, large positive–negative gaps, and applications to recency bias.
1 Introduction
- ICL performance is highly variable because example order, templates, and selection influence outcomes, motivating methods that identify more effective prompts.Prior mitigation methods include model calibration and template engineering, while selection methods use signals such as similarity, perplexity, and sensitivity.Source: §1 · p. 1
- In-context influences rank examples by their measured impact on task performance and support offline selection for optimizing classification accuracy or quantifying example position effects.The framework is designed to analyze and select examples directly from in-context examples.Source: §1 · p. 1
- On 8 language models and 9 natural language tasks, influence-based selection outperforms all other baselines for estimating ICL performance in positive and negative selections.The method is evaluated across 9 natural language tasks and 8 language models.Source: §1 · p. 2
- Up to 16.3% separates prompts built from the top influence bin from those built from the bottom influence bin on LLaMA-13B.The comparison demonstrates a substantial gap between the most positive and most negative examples.Source: §1 · p. 2
- The framework generalizes beyond classification accuracy to combinations of performance metrics, models, and tasks, including analysis of recency bias in example ordering.The paper applies the framework to quantify recency bias as a case study.Source: §1 · p. 2
2 In-context Influences
The method estimates each example’s effect on ICL performance by evaluating many randomly sampled prompting subsets, then uses those influences to select examples. It requires only model forward passes, while subset size is constrained by the context window.
2.1 Cost Analysis & Hyperparameters
- Computing in-context influences needs no gradient updates or end-to-end retraining, making each sampled subset’s validation evaluation proportional to an LLM forward pass.The method can operate with only query access to the model and is cheaper than retraining-based influence frameworks.Source: §2 · p. 3, §2.1 · p. 4
2 In-context Influences
- For each random k-example subset, the method evaluates the prompt over the validation set to obtain f(S′), then repeats sampling until examples receive repeated coverage.The resulting dataset stores subsets with their measured validation performance.Source: §2 · p. 3
- An example’s influence is the difference between average validation performance for prompts including it and prompts omitting it.Higher influence means higher average improvement when the example is included, when f measures validation performance.Source: §2 · p. 3
- Influence estimates become more accurate as subset coverage grows, with experiments showing each example seen at least 30 times on average.A sufficiently large M is defined by good coverage for every example.Source: §2 · p. 3
- Influence-based selection collects prompting runs on random subsets, computes each example’s influence, and selects the k examples with the largest scores.The pipeline consists of subset collection, influence calculation, and inference using selected examples.Source: §2 · pp. 3–4
2.1 Cost Analysis & Hyperparameters
- The subset size k equals the number of in-context examples and is limited by the context window, so it is fixed per model and task.The paper chooses the maximal number of examples that fit while accounting for model and example lengths.Source: §2.1 · p. 4
3 Experiments
The experiments evaluate influence-based example selection across 9 tasks and multiple language models, comparing it with non-influence baselines. Influence methods generally rank highest for both positive and negative selection, with performance varying by task type and shot count.
3 Experiments
- The evaluation covers 9 language tasks, three model families, and 72 task-model combinations, using balanced label-class sampling and standard prompt templates.The models include LLaMA, OPT, and GPT-Neo families.Source: §3 · pp. 4–5
3.1 Influence-based methods
- Influence-based methods are evaluated by ranking positive or negative influence examples, while datamodels fit a linear predictor of validation performance from sampled prompt subsets.The datamodel parameters are treated as influence estimates for selecting examples.Source: §3.1 · p. 5
3.2 Non influence-based methods
- Baselines include random, best-set, one-shot, semantic-similarity, and perplexity-based selection, with methods ranked by average single-task accuracy across 7 seeds.Source: §3.2 · pp. 5–6
3.3 Results
- Influence (+) and IC Datamodels (+) outperform non-influence baselines overall, achieving average ranks of 2.98 and 2.96 across models and tasks.Source: §3.3 · p. 6, Table 1 · p. 6
- Influence (-) achieves the highest negative-selection rank at 2.90, showing that the method can identify low-performing examples.These examples can be avoided or studied to understand why they are ineffective.Source: Table 2 · p. 6, §3.3 · p. 7
- Influence methods work better on multi-choice than binary tasks, while label bias, task properties, model capabilities, and OPT scaling may explain weaker binary-task results.For PIQA and WIC, small gaps between positive and negative influence suggest limited capture of example helpfulness.Source: §3.3 · p. 7
4 Analysis
The analysis evaluates influence-based selection across cost, model agreement, example properties, shot count, and ordering, finding useful positive and negative signals and measurable recency bias.
4.1 Cost Comparison
- Influence-based selection realizes favorable gains over other methods at a fraction of the 20M-token budget and scales beyond that budget.One-shot selection scales linearly with the train and development sets, whereas Influence (+) has more flexible compute-budget scaling.Source: §4.1 · p. 7
4.2 Do models agree on high-influence examples?
- Across seven individual models, the most positive and negative influence examples rarely overlap, with at most four shared examples across all tasks.Within model families, agreement is higher, suggesting influence signals reflect family-specific properties such as architecture, training data, stability, or tokenization.Source: §4.2 · pp. 7–8, Figure 3 · p. 8
4.3 Negative vs. Positive Examples
- Influence identifies negative examples that appear unnatural or mislabeled, including a PIQA ordering problem and a falsely annotated WIC instance.The analysis also finds little to no association with input length, perplexity, or similarity distance, with R2 = 0.0.Source: §4.3 · pp. 8–9, Table 3 · p. 8
- 16.3% accuracy separates the best and worst in-context examples on OpenBookQA with LLaMA-13B.Influence-percentile bins generally show increasing validation performance as selected examples become more positive, with BoolQ on LLaMA-7B as an exception.Source: §4.3 · p. 9, Figure 4 · p. 9, Figure 5 · p. 9
4.4 How do in-context influences generalize across k-shot?
- Influence-based selection is most effective at larger shot counts, generally k >= 8, while it can underperform other methods in one-shot and very few-shot settings.Its performance steadily improves with increasing k, unlike one-shot and random selection, which sometimes decline.Source: §4.4 · pp. 9–10, Figure 5 · p. 9
4.5 Case study: Example Ordering
- A four-shot ordering study confirms recency bias: absolute influence differs by 2% between positions 0 and 3, and the latest position has the widest influence spread.Influence estimates and their spread increase as examples move later in the prompt.Source: §4.5 · p. 10, Figure 6 · p. 10, Figure 7 · p. 10
5 Related Work
Prior work explains ICL behavior through example similarity, sensitivity, perplexity, retrieval, editing, ordering, templates, and training-data influence. This paper differs by using in-context influences to select examples from a development set without test-time prompt editing or retrieval.
5 Related Work
- Earlier methods select examples using semantic similarity, sensitivity, or prompt perplexity, while other approaches use retrieval or reinforcement learning for prompt editing.These methods target example choice or prompt construction through different signals and procedures.Source: §5 · pp. 10–11
- Unlike prompt retrieval or editing methods, the in-context influence framework identifies development-set examples that generalize to unseen evaluation without test-time editing or retrieval.Chang and Jia study a related influence-based approach with different positional modeling and shot-count settings.Source: §5 · pp. 10–11
- ICL is also affected by majority bias, recency bias, prompt templates, input-label correctness, example diversity, and the computations underlying in-context learning.These findings motivate studying influences beyond example selection alone.Source: §5 · p. 11
- Training-data influence methods trace model outputs to examples using gradients, retraining, or group-effect analyses, providing conceptual foundations for in-context influence.The present work adapts this influence perspective to examples placed directly in the prompt.Source: §5 · pp. 10–11
6 Conclusion
The paper proposes in-context influences for selecting and analyzing ICL examples, reporting strong selection results, problematic-example identification, shot scaling, and recency-bias analysis. Its main limitation is that performance is predicted only for examples in a fixed training set.
6 Conclusion
- Influence-based selection outperforms all baselines for both positive and negative selections, with stronger results on multi-choice than binary-classification tasks.The framework also identifies problematic examples, scales with k-shot, and generalizes across model families.Source: §6 · p. 11
- The framework is limited to predicting ICL performance from a fixed training set, excluding newly constructed prompts and examples not present there.Future work could extend influence prediction to on-the-fly examples and tasks beyond classification, including generation and summarization.Source: §6 · p. 11
7 Full Results
The supplementary results document influence distributions, positive-versus-negative performance gaps, and how selection methods behave as the number of shots increases.
7.1 Influence distribution
- Table 4 reports mean test-accuracy differences between the top and bottom 20 influence-percentile bins for each model-task pair.The disparity between the two groups is clear but varies by model and task.Source: Table 4 · p. 14, §7.1 · p. 14
- Figure 9.4 visualizes influence-estimate distributions across all models and tasks and shows fine-grained selection behavior as influence increases.The figure supports inspection of how performance changes across influence regions.Source: §7.1 · p. 14
7.2 Choice of k-shot for in-context influences
- Figure 9.4 compares example-selection methods as the number of shots increases.This supplements the analysis of k-shot scaling for influence-based selection.Source: §7.2 · p. 14
8 Discussion
Linear in-context datamodels predict few-shot ICL performance on unseen example subsets, while OPT results show erratic behavior that may partly reflect training accidents and loss divergences.
8.1 Can linear datamodels predict in-context learning?
- Strong linear trends across models and tasks show that fitted in-context datamodels predict few-shot ICL performance on unseen example subsets.SuperGLUE-WSC is hardest to predict, plausibly because its smallest test set produces high variance.Source: §8.1 · p. 14
8.2 Erratic behavior with OPT models on SuperGLUE
- OPT exhibits erratic SuperGLUE behavior, and hardware failures or loss divergences could partially explain signals in its in-context influence estimates.Source: §8.2 · pp. 14–15
9 Implementation Details
The experiments use autoregressive language models and a RoBERTa-large similarity encoder, with fixed and varied random seeds across several benchmark datasets.
9.1 Models
- Autoregressive models are loaded from HuggingFace checkpoints in 16FP half precision, while the Similarity baseline uses a RoBERTa-large sentence encoder.Source: §9.1 · p. 15
- Experiments default to seed=42 and use six additional seeds for random example ordering.Source: §9.1 · p. 15
9.2 Datasets
- Datasets include SuperGLUE's five binary classification tasks, PIQA's two AI2 Arc tasks, Hellaswag, and OpenBookQA.The benchmark setup samples datasets through HuggingFace's datasets module, with dataset sizes and context-window shot counts reported separately.Source: §9.2 · p. 15
9.3 Prompts
The experiments use the full prompt formats specified for the paper.
9.3 Prompts
- Full prompt formats used in the paper are documented in Table 9.Source: §9.3 · p. 15
9.4 Hardware
Experiments run on NVIDIA A100 and RTX A6000 GPUs, with supplementary tables and figures documenting datasets, prompts, influence analyses, and selection results.
9.4 Hardware
- Experiments run on NVIDIA A100 and NVIDIA RTX A6000 GPUs.Source: §9.4 · p. 15
LLaMA-13B
- Supplementary documentation includes model, dataset, and prompt-template tables, with k defined by the demonstrations fitting within the 2048-character context limit.Source: Table 6 · p. 16, Table 9 · p. 25
9.4 Hardware
- Linear in-context datamodels predict ICL performance on arbitrary example subsets, and influence distributions show how widely high-influence points occur.Source: Figure 8 · p. 17, Figure 9 · p. 18
- Test accuracy generally increases across increasing influence percentile bins, with linear trends in most models and tasks except some cases such as WSC.Source: Figure 10 · p. 19
LLaMA-13B
- Tables report full positive- and negative-selection results across models and seven seeds, alongside positive and negative LLaMA-7B influence examples.Source: Table 7 · p. 21, Table 8 · p. 23, Table 10 · p. 26