PaperlayerView exact revision on arXiv

Source-linked AI summary

In-Context Learning with Long-Context Models: An In-Depth Exploration

Amanda Bertsch, Maor Ivgi, Emily Xiao, Uri Alon, Jonathan Berant, Matthew R. Gormley, Graham Neubig

arXiv:2405.00200v2cs.CL

TL;DR

The paper asks how ICL behaves when context windows hold thousands of demonstrations and how it compares with retrieval and finetuning. It systematically evaluates these settings and finds that long-context ICL keeps improving, reduces retrieval sensitivity, and can rival finetuning while retaining important scope limitations.

  • Problem

    As context lengths grow, the behavior and efficiency tradeoffs of using thousands of ICL demonstrations instead of retrieval or finetuning remain poorly understood.

  • Method

    The paper studies naive and retrieved ICL, full and parameter-efficient finetuning, long-context models, and properties of ordering, label grouping, and attention.

  • Results

    Performance continues increasing past 2000 demonstrations, with gains up to 50.8 accuracy points from 10 to 1000 demonstrations and performance sometimes approaching or exceeding finetuning.

  • Takeaways & Limitations

    Long-context ICL is an effective alternative to retrieval and finetuning when inference-time computation can be exchanged for reduced finetuning-time cost.

  • Takeaways & Limitations

    The study focuses primarily on open-source Llama-2-family models and classification tasks, so generalization across model families and tasks remains open.

Abstract

from arXiv · show

As model context lengths continue to increase, the number of demonstrations that can be provided in-context approaches the size of entire training datasets. We study the behavior of in-context learning (ICL) at this extreme scale on multiple datasets and models. We show that, for many datasets with large label spaces, performance continues to increase with thousands of demonstrations. We contrast this with example retrieval and finetuning: example retrieval shows excellent performance at low context lengths but has diminished gains with more demonstrations; finetuning is more data hungry than ICL but can exceed long-context ICL performance with additional data. We use the ICL setting to study several properties of both in-context learning and long-context models. We show that long-context ICL is less sensitive to random input shuffling than short-context ICL, that grouping of same-label examples negatively impacts performance, and that the performance boosts do not arise from cumulative gain from encoding many examples together. We conclude that long-context ICL can be an effective tool, and may not require long-context for encoding the demonstration set at all.

1 Introduction

The paper studies ICL when extreme context lengths allow thousands of demonstrations, comparing it with retrieval and finetuning while probing how long-context ICL behaves.

  • The study examines naive ICL, per-example retrieval, full and parameter-efficient finetuning, and models adapted to longer contexts.
  • Performance continues increasing past 2000 demonstrations, approaching or sometimes exceeding finetuning on thousands of examples from the same dataset.
  • At extreme demonstration counts, ICL becomes less sensitive to example order, while retrieval’s advantage over random demonstrations diminishes.
  • Grouping examples with the same label harms long-context ICL as the number of demonstrations grows.
  • Long-context ICL can use local attention for demonstrations and global attention only for the test example with nearly the same performance.

2 Experimental setup

The experiments cover five classification datasets and one generation dataset across short- and long-context language models, using constrained outputs and task-specific evaluation metrics.

  • Datasets: The benchmark includes TREC, TREC-fine, NLU, Banking-77, Clinic-150, and the generation dataset SAMSum.
  • Models: Models include Llama-2 variants with 4k, 32k, and 80k context windows, Mistral-7b-v0.2, and Qwen 2.5-7B.
  • Evaluation: Classification uses constrained decoding to produce valid labels, while finetuning uses a classification head that cannot produce invalid outputs.
  • Evaluation: Evaluation uses accuracy and macro-F1 for classification, BERTScore for SAMSum, and a subsampled test set of 250 examples per dataset.

3 Long-context ICL

Long-context ICL improves substantially with more demonstrations, reduces the value of careful retrieval, and remains competitive with finetuning while shifting inference costs upward.

  • Compared settings: Random-sampling ICL draws the first n examples from shuffled training data, allowing the demonstration encoding to be cached.
  • Compared settings: Retrieval compares each test input with demonstration texts using BM25 or BERTScore-Recall, selecting relevant examples for each test example.
  • 3.2 In-context results: 50.8 points is the maximum accuracy gain from increasing Llama2-80k demonstrations from 10 to 1000, averaging 36.8 points across five datasets.
  • 3.2 In-context results: The retrieval advantage is largest at short contexts, but BM25 and BERTScore-Recall become nearly identical as k increases.
  • 3.2 In-context results: On Banking-77, BM25’s gain falls from 51.5 points at 1-shot ICL to 4.9 points at 1500-shot ICL.
  • 3.2 In-context results: Long-context ICL improves SAMSum performance through at least 250-shot ICL, while retrieval sometimes underperforms random selection.
  • 3.3 Comparison with finetuning: Finetuning has higher upfront cost but lower inference cost, whereas long-context ICL trades finetuning-time cost for increased inference-time computation.
  • 3.3 Comparison with finetuning: With relatively few examples, ICL generally outperforms LoRA; full finetuning with substantially more examples often reaches the highest performance.

4 Properties of long-context ICL

Long-context ICL changes how demonstrations contribute: performance generally keeps improving, sensitivity to order decreases, but grouping labels can sharply hurt results and simple tasks may saturate early.

  • Is it best to use the entire context?: Performance is monotonically increasing or flat at higher demonstration counts on nearly every dataset, without significant degradation.Using the full context window can be computationally costly and unnecessary on some datasets, but is minimally not harmful to performance.
  • Sensitivity to example order: 1000-shot ICL flips fewer than half as many labels under shuffling as 10-shot ICL across all datasets.Some order sensitivity remains, but it weakens substantially with longer context.
  • Label sorting: At 1169-shot ICL, sorting examples by label decreases accuracy by 25.7 percentage points.The result suggests that contextualization across different labels is important and effective only over relatively short distances.
  • Effectiveness for frontier models: Frontier-model performance on Clinic-150 increases beyond the few-shot range but saturates quickly, while invalid-label outputs continue to decline.The authors caution that long-context ICL may have diminishing returns for relatively simple intent-classification tasks at around 1,000 examples.

5 Why does long-context ICL help?

The paper tests whether long-context ICL benefits from richer contextualization or simply from access to more demonstrations. Sparse block attention preserves much of full-attention performance, supporting an in-context retrieval account.

  • Motivation: Long-context ICL is hypothesized to benefit primarily from retrieving among more demonstrations rather than aggregating a complex decision boundary.This predicts that restricting attention between demonstrations should have limited impact if the test example can still attend to them.
  • Block-sparse attention patterns: A sparse pattern with an attention sink and two prior local blocks removes almost all long-range demonstration connectivity without substantially affecting performance.The pattern is a small variation on Star Attention.
  • Restricting contextualization on a fixed example set: Block attention reaches 95% of full-attention performance on Banking-77 with restricted contextualization.The experiment fixes the number of examples per block and compares block size against normal full attention.
  • Increasing examples with fixed contextualization quality: When block size is around 10 examples, adding more locally encoded blocks dramatically increases performance; with smaller blocks, inadequate contextualization limits gains.This separates the effects of total demonstrations k from examples per block b.
  • Conclusion: The results support more relevant examples, rather than better task-boundary learning, as the primary source of long-context gains.Short-context retrieval performance approaches, but never exceeds, very-long-context ICL performance in the cited comparison.

6 Related Work

Related work covers methods for extending context, limitations of long-context models, prior analyses of ICL properties, and comparisons between ICL and finetuning. This paper addresses the many-shot comparison and analyzes long-context behavior across models.

  • Augmenting decoder-only models with long context: Long-context methods include positional-embedding extrapolation, additional long-context finetuning, parallel context windows, structured prompting, and efficient attention mechanisms.Some approaches reuse positional embeddings to encode more demonstrations, with diminishing returns as overlap increases.
  • Critiques of long-context models: Prior studies report that some long-context models underuse middle or longer portions of their intended context windows, whereas this paper observes only modest saturation before the maximum.The authors note that their models generally score highly on middle-of-context retrieval.
  • Properties of in-context learning: Earlier ICL studies examine many-class classification, demonstration selection, task learning versus retrieval, and conditions producing early performance saturation.The paper argues that early saturation may reflect shallow heuristics in particular models rather than an inherent limitation of 7B models.
  • Many-shot prompting: Concurrent work reports many-shot improvements with Gemini 1.5, while this paper differs through multiple open-source models, same-base-model finetuning comparisons, and long-context analysis.The comparison also situates this work relative to prior many-shot prompting results.
  • Comparing in-context learning and finetuning: Previous ICL–finetuning comparisons largely use few-shot settings, and no prior work considered their relative performance with hundreds or thousands of in-context examples.Prior results vary across architectures and tasks, including cases where ICL outperforms finetuning.

7 Conclusion

Long-context ICL can be effective with large demonstration sets, often approaching or exceeding parameter-efficient finetuning while relying mainly on retrieval during prediction. The authors propose caching and reusing long demonstration-set encodings as a third option alongside finetuning and subset retrieval.

  • Long-context ICL performance often approaches or exceeds parameter-efficient finetuning on the same data.
  • Long-context ICL shows reduced dependence on example selection and relatively stable performance across example order.
  • Its effectiveness is largely attributed to retrieval from the long context during prediction rather than cross-attention within the demonstration set during encoding.
  • The authors state that understanding ICL remains incomplete because prior mechanism studies largely used simple tasks with fewer than 10 demonstrations.
  • The authors identify fitting more task data in context, then caching and reusing its encoding, as a potential third paradigm beyond finetuning and subset retrieval.

8 Limitations

The study’s evidence is concentrated on open-source models, especially the Llama-2 family, and primarily on classification tasks, limiting how broadly its trends can be generalized.

  • The study focuses mainly on open-source models, predominantly the Llama-2 family, and classification tasks.Whether the observed trend holds across model families and new task types requires further study.
  • Other non-LoRA parameter-efficient finetuning methods are not considered and may outperform in-context learning.

9 Broader impacts

The authors note that accessible in-context learning can be used in both positive and negative downstream applications. They also identify compute-constrained model customization as a potential benefit and call for informed method selection.

  • In-context learning is particularly vulnerable to dual use because it requires less compute than finetuning or pretraining.
  • Many-shot prompting has independently been observed to jailbreak some models, although the authors neither study nor suggest long-context ICL for jailbreaking.
  • The authors hope long-context ICL expands model-customization options for compute-constrained users.
  • They aim to clarify when finetuning or long-context ICL is appropriate so practitioners can make informed choices.

A Saturation

Saturation is defined at 95% of maximum dataset performance, but its location varies with dataset and label-space size. Longer contexts reveal further gains, although performance often saturates before the available context is exhausted.

  • Saturation: Saturation occurs at the smallest tested example count reaching 95% of the model’s maximum performance.This operational definition identifies when additional demonstrations are unlikely to improve performance meaningfully.
  • Saturation: Saturation generally occurs later for datasets with larger label spaces.Banking-77 and Clinic-150 do not saturate within Llama2’s 4096-token context, which holds roughly 100–162 examples for these datasets.
  • Saturation: Long-context Llama2-80k saturation points generally occur slightly later than Llama2 saturation points but before maximum context length.Thus, using the full context window is often unnecessary for high performance, while longer contexts can still reveal additional improvements.
  • Label-space effects: Datasets with more classes benefit from more in-context demonstrations before saturation, but label-space size does not fully explain saturation timing.Halving the label space reduces useful examples before saturation for the largest-label datasets, though not by half and not consistently for smaller-label tasks.
  • Selection strategy: Random-selection ICL continues improving with additional examples, whereas retrieval-based ICL improves until reaching a saturation point.Retrieval begins with higher short-context performance, but its advantage over random selection diminishes as the number of demonstrations grows.
  • Comparison with finetuning: At small example counts, ICL outperforms finetuning, while several thousand examples allow finetuning to exceed ICL on some datasets.This comparison frames saturation and scaling behavior as a tradeoff between prompting and parameter updates.

E Finetuning

The finetuning experiments examine classification-head initialization and PEFT hyperparameters while emphasizing comparability with ICL. A pretrained-language-model-head initialization helps in few-shot settings, but its advantage largely disappears as training data grows.

  • Evaluation: Finetuning selects the checkpoint with the highest test accuracy after 30 epochs.The authors note that test-set model selection gives finetuning an unfair advantage relative to ICL and may not reflect generalization error.
  • Classification-head initialization: Pretrained language-model-head initialization significantly benefits few-shot finetuning, but its advantage becomes negligible as the training set grows.The comparison uses a PEFT Llama-2-7b model and averages best test accuracy over five random seeds.
  • Hyperparameter tuning: The study avoids extensive task-specific hyperparameter tuning and instead evaluates global PEFT settings on Banking-77.The selected configuration uses LoRA rank r = 8, α = 32, dropout 0.1, learning rate 1e-3, batch size 32, and weight decay 0.01.
  • Scope: Small-data finetuning performance might improve with specialized methods such as T-few, which the study excludes because of additional pretraining cost and its encoder-decoder focus.This leaves the comparison limited to the decoder-only setting examined in the paper.

F Block attention patterns

The block-attention experiments restrict demonstration attention to local blocks, optional sink blocks, or preceding blocks to reduce long-range attention. Combining a sink block with local attention performs best among the restricted patterns, while two local blocks are used in the paper’s experiments.

  • Attention-mask design: The proposed attention masks let demonstrations attend only within limited blocks rather than across the full demonstration sequence.Variants include local-only attention, access to an initial sink block, and access to immediately preceding local blocks.
  • Experimental setup: With 500 demonstrations and block size b = 50 on Banking-77, the experiments compare restricted attention patterns against full attention.The block size refers to examples per block, not individual tokens.
  • Results: Neither a sink block nor local attention alone approaches full-attention performance, although both substantially improve over the naive strategy.The best restricted configuration combines sink and local blocks.
  • Results: One local block recovers 94% of full-attention performance, while the paper uses two local blocks to reduce performance degradation.The chosen pattern combines a sink block with two local blocks.
Loading 2405.00200v2…