Source-linked AI summary
A Survey on In-context Learning
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, Zhifang Sui
TL;DR
ICL research has expanded rapidly, but its mechanisms, sensitivity to demonstration design, efficiency, scalability, and generalization remain challenging. This survey synthesizes definitions, techniques, analyses, applications, challenges, and future directions, concluding that the field needs broader, more current evidence and continued work on long-context, efficient, and scalable ICL.
Problem
ICL studies face unresolved questions about how the capability works, while performance depends on demonstration design and efficiency, scalability, and low-resource generalization remain challenging.
Method
The paper comprehensively surveys ICL definitions, training and prompt-design techniques, analyses, applications, challenges, and potential research directions.
Results
The survey organizes current ICL methods and analyses, including training, demonstration reformatting and ordering, scoring functions, applications, and identified challenges.
Takeaways & Limitations
The survey provides a roadmap for ICL research, emphasizing long-context learning, efficiency and scalability, stronger low-resource generalization, and investigation of how ICL works.
Takeaways & Limitations
The survey may overlook valuable contributions, covers studies using outdated models, and leaves several future directions for later work.
Abstract
from arXiv · showhide
With the increasing capabilities of large language models (LLMs), in-context learning (ICL) has emerged as a new paradigm for natural language processing (NLP), where LLMs make predictions based on contexts augmented with a few examples. It has been a significant trend to explore ICL to evaluate and extrapolate the ability of LLMs. In this paper, we aim to survey and summarize the progress and challenges of ICL. We first present a formal definition of ICL and clarify its correlation to related studies. Then, we organize and discuss advanced techniques, including training strategies, prompt designing strategies, and related analysis. Additionally, we explore various ICL application scenarios, such as data engineering and knowledge updating. Finally, we address the challenges of ICL and suggest potential directions for further research. We hope that our work can encourage more research on uncovering how ICL works and improving ICL.
1 Introduction
In-context learning lets LLMs predict from a query paired with a few natural-language demonstrations, without updating model parameters. This survey reviews ICL progress, techniques, applications, challenges, and future directions.
- LLMs exhibit ICL across complex tasks, including mathematical reasoning, as an emerging capability associated with increasing model and data scale.
- ICL forms a prompt from a few natural-language demonstrations and concatenates it with a query for prediction.The model infers the pattern in the demonstrations rather than updating parameters.
- Unlike supervised or few-shot parameter training, ICL is training-free and adapts pretrained LLMs through demonstrations and templates.This can reduce computational costs for adapting models and support large-scale language-model-as-a-service applications.
- ICL performance is sensitive to prompt templates, demonstration selection and ordering, while demonstration conciseness and computational efficiency remain active concerns.
- The survey organizes ICL research, summarizes its taxonomy and findings, and highlights challenges and potential directions as a roadmap for future work.
2 Definition and Formulation
The paper defines ICL as prediction from a pretrained language model conditioned on a small demonstration set, then distinguishes it from related learning paradigms. Its formulation covers task-specific and cross-task demonstrations and selects the highest-scoring candidate answer.
- ICL allows language models to learn a task from only a few demonstration examples.
- Given query x, candidate answers Y, and demonstration set C, a pretrained model predicts the candidate with the maximum score.C may include an optional task instruction and k demonstrations written using natural-language task templates.
- ICL may be task-specific when demonstrations share a task, or cross-task when examples have different instructions.
- ICL differs from few-shot learning because it runs directly on pretrained LLMs without updating model parameters.The survey places ICL as a subclass of prompt tuning in which demonstrations are part of the prompt.
3 Model Training
The survey describes pretraining and warmup as two training-stage strategies for strengthening ICL before inference. These methods reorganize data or adapt models to demonstration-based task formats.
- Specialized training before inference can enhance the ICL capabilities that LLMs already demonstrate directly.
- Pretraining: Pretraining methods reorganize corpora into related contexts or use meta-distillation so models reason across demonstrations more efficiently.Meta-distillation uses distilled demonstration vectors and is described as improving efficiency without compromising effectiveness.
- Warmup: Model warmup adds an optional continual-training stage between pretraining and ICL inference, modifying or adding parameters before deployment.
- Warmup: Warmup strategies continually fine-tune models on multiple-task demonstrations, align raw text with ICL formats, or replace natural-language labels with arbitrary symbols.
- Warmup: Instruction tuning on more than 60 datasets verbalized through natural-language templates improves instruction following and boosts zero-shot and few-shot ICL.
4 Prompt Designing
Prompt designing improves ICL by organizing demonstrations, formatting task instructions, and selecting scoring functions that convert model predictions into answer likelihoods. The surveyed strategies address which examples to use, how to order or reformat them, and how to score candidate answers.
- 4 Prompt Designing: Prompt designing covers demonstration organization and instruction formatting as the main inference-time principles of ICL.
- 4.1.1 Demonstration Selection: Demonstration selection uses either unsupervised predefined metrics or supervised task-specific methods to identify useful ICL examples.
- 4.1.1 Demonstration Selection: Unsupervised selection commonly retrieves nearest neighbors using sentence-embedding distance metrics such as L2 distance or cosine similarity.
- 4.1.1 Demonstration Selection: Supervised selectors address heuristic limitations by training task-specific retrievers or modeling example selection with concept inference and reinforcement learning.
- 4.1.1 Demonstration Selection: Selection effectiveness is model-dependent: MDL performs best on GPT-2, whereas Top-k performs best on the other three evaluated models.
- 4.1.2 Demonstration Reformatting: Demonstration reformatting can generate examples with LLMs or encode them using special positional embeddings to improve how models follow demonstrations.
- 4.1.3 Demonstration Ordering: Ordering is sensitive to example arrangement, motivating proximity-based, entropy-based, and simple-to-complex ordering strategies.
- 4.2 Instruction Formatting: Complex reasoning tasks motivate instruction-based demonstration formats, while Direct, PPL, and Channel scoring functions estimate candidate-answer likelihoods with different constraints and costs.
5 Analysis
The survey analyzes factors associated with ICL performance and theories of why ICL works, spanning pretraining, inference, Transformer mechanisms, and algorithmic interpretations. It also notes that current analyses are concentrated on simple tasks and small models.
- 5 Analysis: The analysis section organizes prior work around factors influencing ICL performance and learning mechanisms explaining why ICL works.
- 5.1 Influencing Factors: Pretraining-corpus diversity affects ICL, with source-domain variety and a task-diversity threshold linked to stronger performance on unseen tasks.
- 5.1 Influencing Factors: Model scale, training duration, and architecture-related information flow are reported as factors associated with emergent or potentially suboptimal ICL behavior.
- 5.1 Influencing Factors: Inference performance depends on input-label settings and demonstration properties, including diversity, simplicity, order, and similarity to the query.
- 5.2 Learning Mechanisms: Mechanistic studies connect ICL to Transformer attention, including induction heads and demonstration label words that aggregate and distribute information for prediction.
- 5.2 Theoretical Interpretation: Theoretical interpretations include implicit Bayesian inference, gradient descent, algorithmic learning, ability decoupling, and information-theoretic error bounds.
- 5.2 Theoretical Interpretation: Most analytical studies focus on simple tasks and small models, leaving extension to extensive tasks and large models as a next step.
6 Application
ICL supports traditional and emerging NLP applications through natural-language demonstrations and flexible context, with surveyed uses including data engineering, model augmentation, and multimodal extensions.
- 6 Application: ICL applies to traditional NLP, complex reasoning, and compositional generalization through demonstrations that guide task reasoning.
- 6 Application: The survey identifies data engineering, model augmentation, and knowledge updating as emerging and prevalent ICL application scenarios.
- 6 Application: In data engineering, ICL generates relatively high-quality data at lower cost than human annotation and noisy automatic annotation.
- 6 Application: ICL research extends beyond text into vision, vision-language, and speech applications.
7 Challenges and Future Directions
The survey highlights efficiency, generalization, and long-context limitations in ICL and identifies these areas as directions requiring further investigation. More demonstrations do not reliably improve performance, while scarce annotated data constrains low-resource applications.
- Efficiency and Scalability: More demonstrations increase computational cost and compete with the maximum input length, creating efficiency and scalability challenges.
- Efficiency and Scalability: Methods that compact demonstrations or accelerate inference may trade off performance or require model-parameter access unavailable for closed-source models.
- Generalization: ICL generalization is constrained by scarce high-quality annotated demonstrations in low-resource languages and tasks.
- Generalization: Leveraging high-resource data for low-resource tasks is identified as an appealing direction because annotated-data availability differs substantially across settings.
- Long-context ICL: Increasing demonstration counts in long-context ICL does not necessarily improve performance and may be detrimental, revealing weaknesses in extended-demonstration comprehension.
8 Conclusion
This survey comprehensively reviews in-context learning, covering techniques, analyses, applications, challenges, and future research directions.
- The survey examines advanced ICL techniques, analytical studies, relevant applications, critical challenges, and potential directions for future research.
Limitations
The survey is broad but acknowledges coverage and recency limitations, while identifying important directions for future ICL research.
- The extensive literature, especially on demonstration design and ICL analysis, may have caused the survey to overlook valuable contributions.
- Long-context ICL, efficiency, and scalability remain future research directions rather than fully addressed aspects of this survey.
- Many surveyed studies used models that were not up to date, limiting the immediacy of their experimental insights for practitioners.
A.1 Training
The survey reviews how training, demonstration design, scoring functions, and analytical perspectives shape ICL, including extensions beyond text.
- Training: Pretraining and warmup introduce objectives closer to downstream ICL formats, potentially improving capability before inference.
- Training: Warmup is optional, but its gains plateau as training data scales, suggesting adaptation may require only limited data.
- Demonstration Organization: Demonstration selection improves ICL, while corpus-level selection and efficient ordering remain underexplored challenges.
- Demonstration Organization: Chain-of-thought demonstrations decompose complex reasoning into intermediate steps, and LLM-generated demonstrations can reduce human template-writing effort.
- Scoring Function: Candidate-answer probabilities are efficient scoring signals but restrict template design and remain sensitive to the demonstration surface.
- Analysis: Analytical studies examine ICL factors and mechanisms, but most remain limited to simple tasks and small models.
- In-context Learning Beyond Text: Beyond text, properly formatted data and architecture designs are key to activating ICL, while textual design findings require domain-specific validation.
B Experimental Detail
Experiments evaluate scoring functions and ICL performance across multiple models and datasets, while prior benchmark evidence shows remaining gaps versus fine-tuning.
- Experimental Setup: Experiments use 8 demonstrations across GPT-2, GPT-J, LLaMA3-8B-Instruct, and Qwen2-7B-Instruct on five classification and reasoning datasets.
- Traditional Tasks: GPT-3 matched state-of-the-art fine-tuning on COPA and ReCoRD with 32 random demonstrations, but remained behind fine-tuning on most NLU tasks.
- Efficiency: Efficiency results measure inference latency for different scoring functions, including scoring time, using milliseconds and ratios to direct inference.
- Stability: Stability results measure performance variance across five random seeds, each selecting 8 examples from 5,000 training examples, with a 10,000-example test set.
- Traditional Tasks: Scaling the number of demonstrations produced only limited improvement, leaving room for ICL to reach fine-tuning performance on traditional NLP tasks.
C.2 New Challenging Tasks
Researchers have developed challenging benchmarks and analyses to probe ICL beyond broad capability demonstrations, including reasoning, generalization, and multimodal settings.
- BIG-Bench spans linguistics, chemistry, biology, social behavior, and other tasks for evaluating ICL across diverse capabilities.
- 65% of BIG-Bench tasks were surpassed by the best models relative to average human-rater results through ICL.
- BBH contains 23 challenging tasks where state-of-the-art model performance remains far below human performance.
- ICL reasoning studies evaluate formal deduction, multilingual chain-of-thought reasoning, and planning over actions and change.
- OpenICL addresses inconsistent implementations and evaluations by providing a flexible toolkit for unified ICL assessment.
- Speech ICL extends the paradigm beyond NLP through audio-code representations for zero-shot cross-lingual synthesis and speech-to-speech translation.