Source-linked AI summary
In-Context Learning with Many Demonstration Examples
Mukai Li, Shansan Gong, Jiangtao Feng, Yiheng Xu, Jun Zhang, Zhiyong Wu, Lingpeng Kong
TL;DR
Context-length limits in existing PLMs leave many-demonstration instruction tuning, ICL, and long-range language modeling under-explored. EVALM combines efficient causal EVA attention with circular position embedding and incremental encoding to train on 8k-token lines and extrapolate to 256k-length contexts. EVALM with many-shot instruction tuning achieves 4.1% higher accuracy on average than same-size OPT, at 12k input length on average.
Problem
Context-length limits in existing PLMs leave many-demonstration instruction tuning, ICL, and long-range language modeling under-explored.
Source: §1 · p. 1Method
EVALM combines efficient causal EVA attention with circular position embedding and incremental encoding to train on 8k-token lines and extrapolate to 256k-length contexts.
Source: §3.1 · pp. 2–3, §5 · p. 8Results
Takeaways & Limitations
More demonstrations can improve ICL performance under many-shot instruction tuning, and the effect is more evident with longer instructions.
Source: §1 · p. 2, §5 · p. 8The experimented EVALM is relatively small compared with existing large-scale models, and its causal EVA attention has quadratic sequence-length complexity.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Large pre-training language models (PLMs) have shown promising in-context learning abilities. However, due to the backbone transformer architecture, existing PLMs are bottlenecked by the memory and computational cost when scaling up to a large context size, leaving instruction tuning and in-context learning of many demonstration examples, as well as long-range language modeling under-explored. In this study, we propose a long-range language model EVALM based on an efficient transformer mechanism. EVALM is trained with 8k tokens per batch line and can test up to 256k-lengthed contexts with extrapolation, 128 times to the limit of existing PLMs (e.g. GPT3). Based on EVALM, we scale up the size of examples efficiently in both instruction tuning and in-context learning to explore the boundary of the benefits from more annotated data. Experimental results on a diverse set of tasks show that EVALM achieves 4.1% higher accuracy on average, and the average length of achieving the best accuracy score over tasks is around 12k. We find that in-context learning can achieve higher performance with more demonstrations under many-shot instruction tuning (8k), and further extending the length of instructions (16k) can further improve the upper bound of scaling in-context learning.
1. Introduction
The paper addresses context-length limits that leave large-sample instruction tuning and in-context learning under-explored. It introduces EVALM to study whether more demonstrations improve ICL, reporting higher average accuracy with many-shot instruction tuning.
1. Introduction
- Existing PLMs’ context limits leave scaling instruction tuning and ICL to large numbers of samples under-explored.The paper asks how effectively more demonstration examples can improve ICL performance.Source: §1 · p. 1
- EVALM is a long-range language model trained at similar cost to OPT, intended to scale instruction tuning and ICL.The paper uses it to test the effects of more demonstrations in ICL.Source: §1 · p. 1
- 4.1% higher accuracy on average with many-shot instruction tuning, at an average input length of 12k.The comparison is reported for EVALM with many-shot instruction tuning.Source: §1 · p. 2
- The benefit from more demonstrations is more evident with longer instructions, but the increasing trend is not endless.The paper reports this pattern for many-shot instruction tuning and ICL.Source: §1 · p. 2
2. Related Work
Prior work establishes ICL as a paradigm, develops efficient attention approaches for longer inputs, and explores increasing the number of demonstrations. The paper situates its work at the intersection of these lines and the context-length limits of current PLMs.
2. Related Work
- Decoder-based causal language models show language-modeling and ICL ability, but their computing overhead makes them difficult for nonprofits and smaller labs to create or use.The passage also notes that the overhead prevents encoding longer inputs.Source: §2 · p. 2
- Efficient attention research spans sparse matrices, memory compression, low-rank methods, kernel-based linear attention, state-space models, and CUDA re-implementation.H3 is described as an efficient pre-trained language model whose training sequence length remains 2048.Source: §2 · p. 2
- ICL has been applied across NLP tasks, while studies report a positive correlation between the number of examples and ICL performance.Work has also explored packing or distilling more examples through continued pre-training and instruction tuning.Source: §2 · p. 2
- The input-length limits of current PLMs still restrict directly feeding more in-context examples into a model.This constraint follows the prior work discussed on scaling the number of examples.Source: §2 · p. 2
3. EVALM
EVALM combines efficient causal attention with position extrapolation and incremental encoding to support longer contexts and more demonstrations. The section describes its architecture, training setup, and method for encoding and reusing ICL demonstrations.
3. EVALM
- EVALM is proposed as a long-range language model, with incremental encoding for scaling ICL shot count beyond the usual context limit.The section is organized around its architecture, pre-training and instruction tuning, and ICL procedure.Source: §3 · p. 2, §3.3 · p. 4
3.1. Architecture
- Causal EVA compresses remote key-value chunks and attends over those compressed features alongside local features.Circular positional embedding uses positions modulo the learned embedding size, and is presented as supporting extrapolation and interaction with distant features.Source: §3.1 · pp. 2–3
- EVALM addresses extrapolation with an architecture intended to support the practical “train short, test long” approach.The passage identifies the cost of training on very long sequences and limited long-range pre-training data as challenges.Source: §3.1 · p. 3
3.2. Pre-training & Instruction Tuning
- EVALM is pre-trained on the Pile and instruction-tuned with task examples converted into natural-language instruction sequences.Many-shot instruction tuning concatenates instructions up to 8192 tokens per batch line; MSIT+ extrapolates the total length to 2 × 8192.Source: §3.2 · pp. 3–4
3.3. In-Context Learning with EVALM
- For ICL, demonstration examples become instruction sequences concatenated into a prompt, and the model selects the candidate answer with the highest likelihood.Incremental encoding updates and caches EVA states; shared demonstration states can be reused across test inputs, reducing the extra overhead of scaling the number of shots.Source: §3.3 · p. 4
4. Experiments
Experiments compare EVALM and OPT across diverse tasks, instruction-tuning strategies, and in-context learning settings. They examine how tuning and demonstration length affect performance, extrapolation, and inference efficiency.
4.1. Experimental setting
- Experiments use FLAN instruction tuning with one-shot IT, MSIT up to 8,192 tokens per batch line, and MSIT+ up to 2 × 8,192 tokens.Many-shot ICL uses randomly selected demonstrations shared across the test set; OPT is the main baseline, chosen for comparable size and training setup.Source: §4.1 · p. 5
4.2. Main Results
- At both 350M and 1.3B scales, EVALM outperforms OPT in zero-shot and many-shot settings, with best scores at about 10 times OPT’s average shot number.The reported comparison covers diverse tasks; Table 1 presents the overall results.Source: §4.2 · p. 5, Table 1 · p. 6
- MSIT, especially MSIT+, yields the greatest growth from zero-shot to many-shot performance, while larger EVALM and OPT models make more progress than their 350M counterparts.The authors suggest that MSIT aligns language modeling with many-shot ICL; they also speculate that learning many tasks can harm small models’ zero-shot performance.Source: §4.2 · p. 5
4.3. Analysis on MSIT
- On Trec, MSIT+ accuracy rises with demonstration length and peaks near 15k, while untuned and one-shot EVALM peak within 128 shots, around 2k, before dropping.Similar trends appear on AgNews, but not for OPT; the authors say increasing instruction-tuning shot numbers further improves the ICL scaling upper bound.Source: §4.3 · p. 6, Figure 2 · p. 7, Figure 3 · p. 7
- Even with MSIT+, accuracy drops rapidly once demonstrations reach 20k, with model size and reliance on extrapolation offered as possible reasons.The paper notes that scaling demonstration sizes in instruction tuning and ICL remains under-explored and calls for further study of advanced ICL algorithms.Source: §4.3 · pp. 6–7
4.4. Discussion
- On PG-19, OPT perplexity rises steeply beyond 2,048 input tokens, while MSIT+ EVALM reaches lower perplexity than vanilla EVALM and the vanilla model around 16k.The authors connect this comparison to why OPT benefits less from MSIT than EVALM, especially MSIT+.Source: §4.4 · p. 7, Figure 4 · p. 7
- Under Top-k ICL, MSIT’s advantage is less pronounced than with randomly selected demonstrations, though MSIT+ has a positive effect on most tasks.The authors suggest overlapping goals and different demonstration ordering as possible explanations; Top-k selects examples per test instance, whereas instruction-tuning examples are randomly ordered.Source: §4.4 · pp. 7–8
- Reusing incremental states saves redundant computation as the number of shots grows, while EVALM achieves better ICL performance than same-size OPT with lower training costs.The paper attributes EVALM’s training-cost efficiency to causal EVA and deduplicated training data, and says instruction tuning costs significantly less than pre-training.Source: §4.4 · p. 8
5. Conclusions & Future Work
EVALM combines long-context modeling with techniques for efficient many-shot instruction tuning and in-context learning. Across diverse tasks, many-shot instruction-tuned EVALM improves average accuracy, while the authors identify limitations in model scale, sequence complexity, and consistent gains from additional examples.
5. Conclusions & Future Work
- EVALM is trained with 8k tokens and extrapolates to 256k-length contexts, using incremental encoding and circular position embeddings.The authors describe this as enabling investigation of increased instruction-tuning and in-context demonstration counts.Source: §5 · p. 8
- EVALM with many-shot instruction tuning and MSIT+ outperforms same-size OPT by 4.1% accuracy on average.The paper also reports that longer instructions make the benefits of larger demonstrations more evident.Source: §5 · p. 8
- EVALM remains relatively small, EVA has quadratic sequence-length complexity for causal modeling, and performance gains from marginal examples are inconsistent.The authors identify these as limitations and future-work directions.Source: §5 · p. 8
A.1. Data Processing
The pretraining corpus is built from a filtered subset of the Pile, with near-duplicate removal and heterogeneous data blending. The selected sources emphasize general natural-language tasks and exclude other subsets that are domain-specific or noisy.
A.1. Data Processing
- The corpus pipeline filters and deduplicates Pile data, then blends the result into heterogeneous batches.The resulting corpus is described in Table 5.Source: §A.1 · p. 13, Table 5 · p. 13
- The selected Pile subsets include BookCorpus2, Books3, DM Mathematics, Project Gutenberg, HackerNews, OpenSubtitles, OpenWebText2, Pile-CC, USPTO, and Wikipedia.Other subsets were excluded as unsuitable for the general-language-task scope or relatively noisy.Source: §A.1 · p. 13
- Content filtering standardizes whitespace, removes non-ASCII characters, and filters documents using four text-based criteria.The criteria are flagged harmful words, stop-word ratio, word/character repetition ratio, and specific-character ratio.Source: §A.1 · p. 13
- Fuzzy deduplication uses 100-bit document fingerprints and LSH to connect document pairs with Jaccard similarity above 0.95.Connected components identify near-duplicate groups; one high-quality document is retained from each group according to predefined priority.Source: §A.1 · p. 13
A.2. Training Details
The appendix identifies the pretraining implementation and points to its hyperparameter table, while the instruction-tuning experiments follow FLAN settings and train models for five epochs.
A.2. Training Details
- EVALM pretraining uses metaseq, with its hyperparameters listed in Table 6.The supplied passage does not specify the individual hyperparameter values.Source: §A.2 · p. 14, Table 6 · p. 14
B. Instruction Tuning Details
- Instruction tuning follows FLAN, excludes the test task's dataset cluster, and trains each model for five epochs on selected FLAN datasets.AgNews is treated as a classification task, so only AgNews itself—not its full summarization cluster—is excluded.Source: B. Instruction Tuning Details · p. 14
C.1. In-context learning details
The appendix reports the tested in-context shot counts, a memory comparison setup, and supplementary best-shot results by dataset.
C.1. In-context learning details
- Memory consumption is compared for EVALM-350M and OPT-350M on a single 80G A100.Figure 6 identifies the comparison scope; the supplied passages do not report its outcome.Source: §C.1 · p. 14, Figure 6 · p. 14
- Supplementary results report the number of demonstration shots at which each dataset achieves its best score.The passage identifies the results as Table 7 but does not provide its values.Source: §C.1 · p. 14
C.2. Prompt Template
The section documents the experiment’s prompt templates and label mappings for reproducibility, following template protocols used in GPT3 and other works. It also identifies supplementary results reporting the demonstration-shot count at which each dataset achieves its best score.
C.2. Prompt Template
- The prompt templates and label mappings are listed for reproducing the experiments, following protocols used in GPT3 and other works.Source: §C.2 · p. 14
- Table 7 reports the demonstration-shot count at which each dataset achieves its best score in the many-shot setting.Source: Table 7 · p. 15
- Table 8 presents the prompt templates and label mappings used in the experiments.Source: Table 8 · p. 15