PaperlayerView exact revision on arXiv

Source-linked AI summary

Compositional Exemplars for In-context Learning

Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, Lingpeng Kong

arXiv:2302.05698v3cs.CLcs.AIcs.LG

TL;DR

In-context example selection is unstable and prior methods often overlook interactions among examples. CEIL models subsets with conditional DPPs and contrastive learning, outperforming prior selectors across broad benchmarks while transferring across models and datasets. Its scope is bounded by reliance on training data and a two-stage candidate-retrieval procedure.

  • Problem

    ICL performance is highly sensitive to example selection, while prior selectors overlook inter-example relationships and redundancy under limited prompt length.

  • Method

    CEIL formulates selection as end-to-end subset optimization, using conditional DPPs for joint example probabilities and contrastive learning to train the retriever.

  • Results

    Across 12 benchmarks covering 7 classification and generation tasks, CEIL clearly beats previous competitive methods and shows transferability and compositionality.

  • Takeaways & Limitations

    CEIL offers an effective and efficient way to adapt black-box large language models to downstream tasks through learned exemplar selection.

  • Takeaways & Limitations

    For compositional tasks, CEIL's retriever has seen compositional data during standard dataset training, and candidate selection uses a two-stage retrieval framework because full subset modeling is exponential.

Abstract

from arXiv · show

Large pretrained language models (LMs) have shown impressive In-Context Learning (ICL) ability, where the model learns to do an unseen task via a prompt consisting of input-output examples as the demonstration, without any parameter updates. The performance of ICL is highly dominated by the quality of the selected in-context examples. However, previous selection methods are mostly based on simple heuristics, leading to sub-optimal performance. In this work, we formulate in-context example selection as a subset selection problem. We propose CEIL (Compositional Exemplars for In-context Learning), which is instantiated by Determinantal Point Processes (DPPs) to model the interaction between the given input and in-context examples, and optimized through a carefully-designed contrastive learning objective to obtain preference from LMs. We validate CEIL on 12 classification and generation datasets from 7 distinct NLP tasks, including sentiment analysis, paraphrase detection, natural language inference, commonsense reasoning, open-domain question answering, code generation, and semantic parsing. Extensive experiments demonstrate not only the state-of-the-art performance but also the transferability and compositionality of CEIL, shedding new light on effective and efficient in-context learning. Our code is released at https://github.com/HKUNLP/icl-ceil.

1. Introduction

In-context learning enables large pretrained language models to generalize from a few demonstrations, but performance is highly sensitive to example selection. CEIL addresses this by selecting interacting, diverse subsets rather than independent examples.

  • 1. Introduction: In-context learning uses limited demonstration examples to help pretrained language models generalize to unseen tasks.The approach provides task information through examples without parameter updates.
  • 1. Introduction: Different in-context example selections can shift performance from nearly random to near state-of-the-art.This instability makes example selection a central robustness issue for ICL.
  • 1. Introduction: CEIL models the joint probability of an entire example set with a conditional determinantal point process, capturing inter-example relationships.Its objective incorporates language-model scoring through contrastive learning while retaining polynomial-time MAP inference.
  • 1. Introduction: CEIL is evaluated on 12 classification and generation datasets spanning 7 NLP tasks, including sentiment analysis, question answering, code generation, and semantic parsing.The evaluation covers both classification and generation settings.
  • 1. Introduction: CEIL substantially surpasses conventional learning-free and learning-based selection approaches, achieving state-of-the-art in-context learning performance.The experiments also report transferability across language models and datasets, compositionality, and particular effectiveness with few examples.

2. Preliminary

In-context learning infers new tasks from context by conditioning language-model generation on input-output demonstrations, avoiding parameter updates. DPPs provide a subset model that represents diversity and supports efficient inference.

  • 2.1. In-context Learning: In-context learning infers new tasks from context, distinguishing gradient-free contextual learning from gradient-based parameter updates.For a test input, the language model generates an output conditioned on selected input-output examples.
  • 2.1. In-context Learning: The generation procedure avoids updating language-model parameters when adapting to a new task.This is attractive because parameter updates can be expensive and impractical.
  • 2.1. In-context Learning: ICL performance can range from almost random to comparable with state-of-the-art systems depending on retrieved-example quality.Prior retrievers model example selection through P(e_i | x_test), using either off-the-shelf or fine-tuned methods.
  • 2.2. Determinantal Point Processes: Determinantal point processes model subsets sampled without replacement using a positive semidefinite kernel matrix.The kernel entries are computed from item features through a kernel function.
  • 2.2. Determinantal Point Processes: DPP subset probabilities depend on the determinant of the restricted kernel matrix and its normalization by det(L + I).The determinant can be interpreted geometrically as the volume formed by mapped item vectors, favoring high-quality and dissimilar items.
  • 2.2. Determinantal Point Processes: Many DPP inference tasks, including marginalization, conditioning, sampling, and MAP inference, remain polynomial-time despite exponentially many possible subsets.This tractability motivates DPPs for subset-selection problems.

3. Model

CEIL selects in-context examples as a jointly modeled subset, balancing relevance and diversity through a conditional DPP trained against language-model preferences. At inference, it efficiently retrieves an exemplar set with MAP inference while keeping the language model frozen.

  • Model overview: CEIL models the joint probability of an entire in-context example set instead of retrieving examples independently.The conditional DPP captures inter-relationships among exemplars and is trained with contrastive learning.
  • Conditional DPP: The conditional DPP combines relevance to the test input with diversity among selected examples.Its kernel uses relevance scores for each example while the determinant represents diversity.
  • Conditional DPP: CEIL uses learnable embedders for inputs and exemplars, with linear-kernel similarity forming the DPP scoring function.The input and exemplar embedders produce representations used to compute relevance and pairwise similarity.
  • Contrastive training: Contrastive training ranks better exemplar subsets above worse subsets using language-model-derived quality scores.The fine-grained pair-wise margin loss reflects differences in subset quality and avoids the full normalization calculation.
  • Training data: A two-stage procedure narrows the candidate space before sampling distinct subsets, making training feasible despite the exponential subset space.A retriever first selects n candidates from N examples, followed by nonreplacement random sampling.
  • Inference: During inference, MAP inference selects the best exemplar subset by jointly considering relevance and diversity.The NP-hard optimization is approximated with greedy inference over KNN-retrieved candidates, with Cholesky updates reducing per-iteration complexity to O(nK).

4. Experiments

Experiments across 12 datasets and 7 tasks show that CEIL improves in-context example selection through example interaction, with transferability and compositionality across settings.

  • 4.4. Main Results: CEIL substantially outperforms learning-free baselines, achieving over 20% absolute improvement on NLI tasks and around 10% on most other tasks.Commonsense reasoning is an exception, where other retrievers perform comparably to random selection.
  • 4.4. Main Results: CEIL consistently outperforms the learning-based EPR retriever across tasks without adding parameters or deployment cost.The result supports modeling interactions between in-context examples during learning.
  • 4.5. Compositionality: CEIL retrieves compositional exemplars, improving performance on difficult compositional semantic-parsing splits relative to competing retrievers.Cross-domain SMCalFlow-CS improvements exceed those on the single-domain split, and CEIL improves all difficult splits considered.
  • 4.6. Transferability: CEIL transfers across datasets and language models, with a retriever trained on one LM or dataset often improving performance on another.Transfer to NLI tasks is broadly effective, whereas NLI-trained retrievers transfer poorly to most non-NLI tasks.
  • 4.7. Analysis: Pair-wise margin loss outperforms InfoNCE on generation tasks by approximately 4% on GeoQuery and 6% on MTOP.The advantage is attributed to the finer-grained contrastive signal on harder generation tasks.
  • 4.7. Analysis: DPP-MAP improves more than TOPK for learning-based retrievers, while retrieving 50 examples takes 36s versus 30s for TOPK on SST5.The authors recommend choosing between the algorithms according to task needs and inference cost.

5. Related Work

Prior work studies in-context example selection using retrieval heuristics and metrics, while DPPs provide a framework for selecting diverse, high-quality subsets.

  • ICL performance can range from near-random to near-state-of-the-art depending on the selected examples.
  • DPPs model diversity and quality jointly, supporting diverse subset selection across summarization, recommendation, detection, and classification applications.

6. Conclusion

The paper formulates in-context example selection as end-to-end subset optimization with CEIL, combining DPPs and contrastive learning. Across 12 benchmarks and 7 tasks, CEIL outperforms competitive methods and shows transferability and compositionality.

  • CEIL models the probability of the entire in-context example subset with DPPs and learns selection through contrastive learning.
  • Experiments on 12 benchmarks spanning 7 classification and generation tasks show that CEIL clearly beats previous competitive methods.
  • The learned retriever transfers across language models and datasets and exhibits compositionality on compositional tasks.
  • CEIL provides an effective and efficient approach for adapting black-box large language models to downstream tasks.

A.1. Datasets

The experiments use 12 datasets spanning classification, reasoning, question answering, code generation, and semantic parsing, with task-specific prompts and evaluation metrics.

  • SST-5 evaluates five-class sentiment classification, while MRPC evaluates semantic equivalence for sentence pairs.
  • MNLI, QNLI, CMSQA, and HellaSwag evaluate textual entailment, question-answer verification, commonsense multiple choice, and grounded commonsense reasoning.
  • WebQs and NL2Bash pair questions or English descriptions with answers or Bash commands, while GeoQuery maps geography questions to Prolog queries.
  • Prompts and examples for the experimental datasets are documented in an appendix table.
  • Break, MTOP, and SMCalFlow evaluate meaning-representation or semantic-parsing outputs involving ordered steps, nested intent-slot queries, or executable dataflow programs.

A.2. Experimental Setup for Compositionality

The compositionality setup supplies few-shot context examples while comparing retrievers on single-domain exemplars, subject to model- and context-length constraints.

  • All few-shot examples are included in context to provide compositional symbols, while retrievers select single-domain exemplars.
  • Inference latency and evaluation metrics are reported while varying the number of examples at inference time.
  • Codex evaluations limit in-context examples to 16 for fair comparison across k-C splits.

B.1. Varying n at Inference Time

Increasing the candidate-set size n generally improves performance by expanding exploration, but its marginal impact diminishes as n grows. CEIL can outperform baselines using far fewer in-context examples, reducing attention computation.

  • Effect of n: Increasing n generally improves performance by expanding the exploration space and increasing the chance of finding a better subset.This trend was observed across multiple datasets, alongside inference-latency measurements on SST-5.
  • Effect of n: The impact of n diminishes as n increases because examples beyond the top 200 are typically not selected from the top 800 candidates.Relevance receives greater weight than diversity in selection, limiting the contribution of lower-ranked candidates.
  • Comparison across example counts: CEIL mostly outperforms EPR and TOPK-BERT with 4 or 1 example instead of 32, respectively, reducing quadratic attention computations.The comparison is shown in additional experiments varying the number of in-context examples.

C. Limitation

CEIL’s main limitation is its task-specific training requirement and slower dataset-construction scoring stage. Its transferability remains an early-stage result, motivating unified multitask retriever training.

  • Training requirements: CEIL requires task-specific data to train its retriever, despite outperforming learning-free methods.This requirement follows from its learning-based approach.
  • Computational cost: CEIL’s dataset-construction scoring stage is slower than EPR because it scores in-context example subsets rather than single examples.
  • Transferability: Retriever transferability remains in an early research stage, with multitask tuning proposed to enable direct application to new tasks.The proposed direction would avoid retraining the retriever with new task data.
Loading 2302.05698v3…