PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering

Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, Lingpeng Kong

arXiv:2212.10375v2cs.CLcs.AI

TL;DR

ICL is highly sensitive to the organization of in-context examples, while common practice often samples them randomly. This paper introduces self-adaptive ICL, a two-stage select-then-rank framework for finding example organizations per test instance. Across eight NLP datasets, it reports a 40% relative improvement over the ICL random baseline and identifies both substantial promise and practical limitations.

  • Problem

    ICL performance depends strongly on prompt quality, and existing methods have largely used corpus-level organizations rather than searching for an optimal organization for each test sample.

  • Method

    Self-adaptive ICL formulates example selection and ordering as a search problem, filtering candidates before ranking organizations with an information-theoretic approach grounded in compression.

  • Results

    40% relative improvement over the Random baseline was achieved, with consistent superiority over corpus-level baselines and state-of-the-art performance reported across evaluations.

  • Takeaways & Limitations

    Self-adaptive example organization substantially improves ICL over common random selection and may have further potential with better search methods.

  • Takeaways & Limitations

    The approach trades efficiency for effectiveness and requires a high-quality retrieval set, while TopK selection is limited on tasks with large or infinite label spaces.

Abstract

from arXiv · show

Despite the surprising few-shot performance of in-context learning (ICL), it is still a common practice to randomly sample examples to serve as context. This paper advocates a new principle for ICL: self-adaptive in-context learning. The self-adaption mechanism is introduced to help each sample find an in-context example permutation (i.e., selection and ordering) that can derive the correct prediction, thus maximizing performance. To validate the effectiveness of self-adaptive ICL, we propose a general select-then-rank framework and instantiate it with new selection and ranking algorithms. Upon extensive evaluation on eight different NLP datasets, our self-adaptive ICL method achieves a 40% relative improvement over the common practice setting. Further analysis reveals the enormous potential of self-adaptive ICL that it might be able to close the gap between ICL and finetuning given more advanced algorithms. Our code is released to facilitate future research in this area: https://github.com/Shark-NLP/self-adaptive-ICL

1 Introduction

ICL is powerful but prompt-sensitive, and prior work has focused more on templates than on organizing examples. The paper proposes self-adaptive, instance-level example search through a two-stage framework and reports strong performance and further potential.

  • ICL performance can vary from almost random to state-of-the-art depending on prompt quality.
  • Prior research has extensively searched for templates, while comparatively little work has searched for the best selection and ordering of in-context examples.
  • Corpus-level methods can be sub-optimal because one universal example organization is unlikely to suit every test sample and can create majority bias.
  • Self-adaptive ICL searches for a good-performing example organization for each test sample without using a validation dataset.
  • The proposed two-stage framework filters candidate examples heuristically, then ranks their organizations using algorithms supported by the Minimal Description Length principle.
  • The framework achieves state-of-the-art performance, finds instance-level methods generally more robust than corpus-level methods, and suggests further gains may narrow the ICL–finetuning gap.

2 Related Work

Prior ICL prompt-search research has mainly optimized templates at the corpus level, while in-context example search remains comparatively underexplored. Existing example-selection methods use similarity and diversity heuristics but leave some of their effectiveness unexplained.

  • Template-search methods optimize prompts using PLMs, heuristics, reinforcement learning, genetic algorithms, or manual design.
  • Corpus-level prompt-search methods apply one selected template to every test example, making a universally strong template nearly impossible under the No Free Lunch perspective.
  • In-context example search has received little attention despite its substantial influence on ICL performance.
  • Earlier example-search work considered small permutation spaces, nearest-neighbor similarity, and diversity-aware retrieval.

3 Problem formulation

The paper defines ICL prediction through a template-wrapped context and verbalizer, then formulates self-adaptive ICL as finding an example organization that drives the correct prediction for each input.

  • A causal PLM predicts target y for input x from a template-wrapped context c and template T(x), expressed as p(y|x) = P(V(y)|c, T(x)).
  • The context c concatenates k template-wrapped input-output examples, while the verbalizer V maps classification labels to vocabulary words.
  • When k = 0, ICL reduces to zero-shot prompting.
  • Self-adaptive ICL seeks an organization of c that produces the correct y for each x while maximizing task performance, forming a combinatorial optimization problem.

4 Method

The method addresses self-adaptive ICL through a two-stage selection-then-rank search over example organizations. It uses heuristic or diversity-aware selection to reduce candidates, then ranks organizations by information-compression objectives and related interpretations.

  • The framework first selects a small candidate set, then ranks candidate example combinations and permutations to reduce the otherwise intractable search space.Selection filters less useful examples; ranking chooses the best organization according to an information-theoretic objective.
  • TopK selects examples nearest to each test sample in embedding space, while VoteK additionally penalizes similarity to previously selected examples to increase diversity.DPP is also evaluated as a diversity-oriented set-selection method.
  • The ranking module applies the Minimal Description Length principle, favoring organizations that losslessly compress testing samples while accounting for model-description cost.Because all organizations use the same model without parameter updating, the model-description term can be ignored during ranking.
  • Without testing labels, the method replaces exact label codelength with an expected codelength surrogate and models the instance-level label prior using p(y_i|c, x).The instance-level likelihood can vary across samples, unlike a corpus-level prior.
  • Even after selection, exhaustive ranking remains infeasible: 10 candidates with 8-example organizations yield 1.8 million choices, so the current method randomly samples 10 permutations.The authors leave better approximations to optimal ranking for future work.
  • The compression objective also connects to entropy minimization and cross-entropy, interpreting selected organizations as producing confident predictions and standard classification loss when validation labels are available.The paper further relates successful label compression to mutual information and distinguishes its instance-level, validation-free setting from prior methods.

5 Experiments

Experiments evaluate the framework across eight NLP datasets against prompting, corpus-level, and instance-level baselines. Results show that example organization, including selection, ordering, and label alignment, strongly affects ICL performance, with consistent gains from the proposed method.

  • Experimental Setup: Eight NLP datasets are evaluated using GPT2-XL, with TopK+MDL retrieving 30 candidates and ranking 10 sampled organizations of 8 examples each.Experiments cover sentiment classification, natural language inference, multiple-choice question answering, and topic classification.
  • Experimental Setup: The evaluation compares prompting, corpus-level methods, and instance-level methods, including validation-based, validation-free, retrieval, and random baselines.For restricted-test datasets, corpus-level methods use 10% of training data for validation, while other methods use the original validation set.
  • Main Results: ICL methods outperform prompting in most cases, but poor example organizations can reduce performance below prompting, as observed on SST-5.These results establish that both selecting and ordering in-context examples matter for downstream accuracy.
  • Main Results: 40% relative improvement over the Random baseline demonstrates consistent superiority over corpus-level methods and substantial room for improved example search.The Random baseline represents the common practice of randomly organizing in-context examples.
  • Main Results: 17% relative improvement over TopK+LocalE shows gains over an instance-level baseline and supports MDL as a ranking method.The proposed method also improves over instance-level baselines on most evaluated datasets.
  • Impact of Label in ICL: Higher bias rates, meaning more in-context examples sharing the test label, correlate positively with performance, while corrupted labels produce performance drops.The label analysis indicates that in-context labels substantially influence ICL outcomes.

6 Analysis

The analysis examines how self-adaptive ICL behaves under different retrieval settings, ranking choices, context sizes, model scales, PLMs, and prompting templates. Results show broad gains over TopK, while revealing trade-offs between performance, efficiency, and retrieval quality.

  • 6.1 When a large set of annotated examples is not available: Our method consistently outperforms TopK in few-shot settings, and performance steadily increases as the candidate pool grows.Experiments use 16 to 1024 randomly sampled candidate examples on SST2 and SNLI.
  • 6.2 Impact of selection methods: TopK selection outperforms random, DPP, and VoteK on average, but the alternatives perform better on multi-choice Commonsense QA.TopK is strongest mainly for simple classification tasks with limited label space.
  • 6.3 Accuracy of ranking method: A perfect ranking oracle substantially outperforms TopK+MDL, indicating remaining performance headroom in selection and ranking.The oracle selects an organization yielding the correct prediction whenever any tested organization does so.
  • 6.3 Accuracy of ranking method: Lower MDL generally corresponds to higher accuracy, supporting MDL as a criterion for searching in-context example organizations.The analysis reports average MDL values alongside accuracies across four representative datasets.
  • 6.4 Impact of hyperparameter: +3.19 in accuracy is achieved over TopK on SST2 with a window size of two, while larger windows steadily improve performance at greater computational cost.A window size of 50 requires forward passes for the test set 50 times.
  • 6.4 Impact of hyperparameter: Increasing the number of in-context examples generally improves accuracy, but the organization of examples determines whether additional examples help.The random baseline reaches a plateau from N = 8, whereas the proposed method continues to improve on average as N increases to 32.
  • 6.4 Impact of hyperparameter: Our method consistently improves over baselines across model sizes and significantly outperforms TopK across four tested PLMs.Model-scale experiments use OPT models from 350M to 175B, while results across PLMs are averaged over datasets.
  • 6.5 Robustness: Our method remains robust across different prompting templates, although template choices still produce large performance variances.The robustness evaluation uses SST2 and SNLI with three templates per task.

7 Conclusion

The paper presents self-adaptive ICL as a new instance-level paradigm for organizing in-context examples. Its select-then-rank framework and information-theory-driven ranking substantially outperform common ICL practice and expose further research opportunities.

  • 7 Conclusion: Self-adaptive ICL searches for example selection and ordering separately for each test instance instead of applying one organization universally.The framework uses a two-stage select-then-rank procedure with an information-theory-driven ranking algorithm.
  • 7 Conclusion: Empirical results show that self-adaptive ICL significantly outperforms common ICL practice by a large margin.
  • 7 Conclusion: The analysis identifies open problems in method design and suggests substantial potential for improving self-adaptive ICL.

8 Limitation

The paper identifies two limitations of self-adaptive ICL: computational costs rise with broader search, and performance depends on high-quality retrieval sets. TopK also performs poorly on tasks with large label spaces.

  • 8 Limitation: Self-adaptive ICL must trade efficiency for effectiveness because its large search space increases deployment cost.Balancing this trade-off is an important decision when deploying the method.
  • 8 Limitation: Performance gains shrink when the retrieval set is smaller, creating a dependence on high-quality retrieval for maximizing performance.Such retrieval sets may not always be available for unseen tasks in practice.
  • 8 Limitation: TopK is effective mainly for simple NLU tasks with limited label spaces and performs poorly on QA, multi-choice, and NLG tasks.The paper identifies better selection methods as a direction for adapting ICL to more tasks.

A Datasets

The paper’s dataset information is provided in Table 3.

  • A Datasets: Dataset information is detailed in Table 3.

B Impact of hyperparameters

The paper examines how performance changes with the number of in-context examples and with the window size used for ranking permutations.

  • The results of changing the number of examples and window size are presented in Figures 6 and 7, respectively.
  • Figure 6 reports results while adjusting the number of in-context examples.
  • Figure 7 reports evaluation results for different window sizes, defined as the number of permutations to be ranked.

C Templates

The paper documents the task templates used in its experiments, including placeholder conventions for replacing template fields with inputs or answers.

  • The templates used in the paper are detailed in Table 4.
  • Table 4 specifies templates for the paper’s tasks.
  • Placeholders such as <X> and <A> are replaced by real inputs or answers in Commonsense QA.
Loading 2212.10375v2…