Source-linked AI summary
Many-Shot In-Context Learning
Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, Hugo Larochelle
TL;DR
Few-shot ICL is constrained by limited context windows and by the availability of human-generated examples, motivating study of many-shot learning. The paper scales ICL to hundreds or thousands of examples and introduces reinforced and unsupervised variants; many-shot ICL improves performance broadly, can match fine-tuning, and supports adaptation beyond pretraining biases, while its behavior and cross-model generality remain incompletely understood.
Problem
Limited context windows and scarce high-quality human-generated rationales constrain prior few-shot ICL and many-shot applications, especially for complex reasoning tasks.
Method
The paper systematically evaluates ICL across task scales and introduces Reinforced ICL with filtered model-generated rationales and Unsupervised ICL with problems alone.
Results
Many-shot ICL yields significant gains across diverse tasks, can overcome pretraining biases, learn high-dimensional numerical-input functions, and perform comparably to SFT.
Takeaways & Limitations
Many-shot ICL can reduce dependence on human-generated demonstrations and make fine-tuning less essential for a wider range of tasks.
Takeaways & Limitations
The work mainly evaluates many-shot ICL with Gemini 1.5 Pro, and performance can sometimes degrade as more examples are added.
Abstract
from arXiv · showhide
Large language models (LLMs) excel at few-shot in-context learning (ICL) -- learning from a few examples provided in context at inference, without any weight updates. Newly expanded context windows allow us to investigate ICL with hundreds or thousands of examples -- the many-shot regime. Going from few-shot to many-shot, we observe significant performance gains across a wide variety of generative and discriminative tasks. While promising, many-shot ICL can be bottlenecked by the available amount of human-generated examples. To mitigate this limitation, we explore two new settings: Reinforced and Unsupervised ICL. Reinforced ICL uses model-generated chain-of-thought rationales in place of human examples. Unsupervised ICL removes rationales from the prompt altogether, and prompts the model only with domain-specific questions. We find that both Reinforced and Unsupervised ICL can be quite effective in the many-shot regime, particularly on complex reasoning tasks. Finally, we demonstrate that, unlike few-shot learning, many-shot learning is effective at overriding pretraining biases, can learn high-dimensional functions with numerical inputs, and performs comparably to fine-tuning. We also find that inference cost increases linearly in the many-shot regime, and frontier LLMs benefit from many-shot ICL to varying degrees. Our analysis also reveals the limitations of next-token prediction loss as an indicator of downstream ICL performance.
1. Introduction
Many-shot ICL uses hundreds or thousands of in-context examples to move beyond context-window-limited few-shot learning, producing broad performance gains and enabling new forms of adaptation. The paper also introduces reinforced and unsupervised variants to reduce reliance on human-generated data, and analyzes their learning dynamics and limits.
- Scaling ICL: Many-shot ICL evaluates hundreds or thousands of examples across problem solving, question-answering, summarization, translation, planning, reward modeling, reasoning, and sentiment analysis.The evaluation spans both generative and discriminative tasks.
- Scaling ICL: Many-shot ICL produces significant performance gains across a wide range of tasks compared with few-shot ICL.The paper reports large performance jumps when transitioning from the few-shot to many-shot regime.
- Context Length versus Number of Shots: Many-shot ICL can require hundreds of thousands of prompt tokens for maximum performance, although code verification and planning sometimes deteriorate with additional shots.The available number of examples can limit the tested shot count.
- Reinforced and Unsupervised ICL: Reinforced ICL replaces human-written rationales with correctness-filtered model-generated rationales, while unsupervised ICL prompts only with problems.Both approaches reduce dependence on human-generated data; reinforced ICL is more broadly effective in the reported problem-solving evaluations.
- Analysing ICL: With sufficient examples, many-shot ICL can overcome pretraining biases, perform comparably to full fine-tuning, and learn high-dimensional numerical-input functions.The paper studies sequential parity prediction and linear classification as examples of such functions.
- Analysing ICL: Next-token prediction loss may not reliably predict many-shot ICL performance on problem-solving and reasoning tasks.The paper also reports that example order can influence many-shot performance.
2. Scaling In-Context Learning
Many-shot ICL improves performance across diverse tasks as the number of in-context examples increases, though scaling can plateau or reverse depending on the task and model. Experiments span translation, summarization, planning, and in-context code verification.
- Setup: Many-shot ICL uses expanded context windows to provide hundreds or thousands of examples, extending beyond the few-shot regime.The evaluation uses Gemini 1.5 Pro with a 1 million token context length and nested prompts that retain examples from smaller prompts.
- Machine Translation: 15.3% improvement on Bemba and 4.5% on Kurdish occurs with the full 997-example development set relative to the 1-shot Gemini prompt.These gains establish new state-of-the-art results for the evaluated language pairs.
- Abstractive Summarization: XSum performance improves through 50 shots before declining, whereas XLSum performance typically improves monotonically with more XSum examples.Peak many-shot performance is close to specialized models fine-tuned on XSum and XLSum; the 500-shot prompt is approximately 205K tokens.
- Planning: Logistics Domain: Planning success rises from 42% to 62% for a recent 1.5 Pro version, while an older version improves through 10 shots before saturating.The older-model result uses a prompt of approximately 37K tokens, and many-shot performance remains far from state-of-the-art planning approaches.
- Learning Code Verifiers: 128-shot verifier prompting bridges the gap between 77.25% Pass@1 and 90% Pass@4 accuracy for Gemini 1.0 Pro on GSM8K code solutions.The verifier ranks four candidate solutions using normalized Yes-token probabilities, with correct and incorrect solution scores separating as shots increase up to 256.
3. Many-shot Learning without Human-Written Rationales
Reinforced ICL replaces human-written rationales with correctness-filtered model-generated rationales, while Unsupervised ICL prompts only with problems. Both reduce reliance on human-generated data, with Reinforced ICL generally the stronger approach on complex reasoning tasks.
- Reinforced ICL: Reinforced ICL replaces human-written rationales with correctness-filtered, model-generated rationales for in-context learning.It samples multiple rationales from a zero-shot or few-shot chain-of-thought prompt and uses them as demonstrations.
- Unsupervised ICL: Unsupervised ICL removes rationales from many-shot prompts, using unsolved problems plus instructions specifying the desired output format.The prompt contains a preamble, a list of unsolved inputs, and a zero-shot or few-shot output-format instruction.
- Problem-solving: Hendrycks MATH & GSM8K: On MATH500, Reinforced and Unsupervised ICL outperform ground-truth-solution ICL, while Reinforced ICL reaches a plateau around 25 examples without the large-context decline seen in standard ICL.Reinforced ICL is about 5% higher than standard ICL, and many-shot problem-only prompts achieve comparable or superior performance to prompts with solutions.
- Problem-solving: Hendrycks MATH & GSM8K: MATH prompts transfer well to GSM8K, with Reinforced ICL outperforming ICL using ground-truth MATH solutions on the different problem distribution.This tests whether many-shot ICL improves transfer beyond the original MATH task.
- Question Answering: Google-Proof QA (GPQA): 125-shot GPQA prompts with ground-truth or model-generated rationales surpass Claude-3 Sonnet’s 40.4% accuracy, while Reinforced ICL consistently outperforms Unsupervised ICL.Ground-truth-rationale accuracy improves substantially from 5 to 125 shots, but performance degrades at 250 shots; GPQA’s small size produces higher run-to-run variance.
- Algorithmic and Symbolic Reasoning: Big-Bench Hard: On BIG-Bench Hard, Reinforced ICL strongly outperforms Unsupervised ICL, which outperforms the standard human-written 3-shot chain-of-thought prompt across the evaluated tasks.Reinforced ICL generally improves monotonically with more prompts on 7 of 8 tasks and can exceed the human-written prompt even with 3 shots.
4. Analyzing Many-Shot ICL
Many-shot ICL extends beyond few-shot limitations: it can overcome pre-training biases, learn non-natural-language functions, approach fine-tuning performance, and expose important inference and evaluation trade-offs.
- Overcoming Pre-training Biases with Many-Shot ICL: Many-shot ICL improves performance on flipped and abstract sentiment labels, eventually approaching default-label accuracy after initially struggling in the few-shot regime.Confidence for flipped labels first drops, then sharply increases before plateauing, suggesting a period of overcoming pre-training bias.
- Learning Non-Natural Language Tasks: High-Dimensional Functions: Many-shot ICL substantially outperforms random chance and nearly matches k-nearest neighbors on high-dimensional numerical-input functions.The result indicates that many-shot ICL can implement nearest-neighbour search over inputs.
- Learning Non-Natural Language Tasks: High-Dimensional Functions: Test accuracy for sequential parity improves consistently through 8192 shots and surpasses a GPT-2 Medium transformer trained from scratch on 20× more examples.The authors interpret this as evidence that many-shot ICL can implement computations analogous to gradient descent.
- Many-Shot ICL vs. Supervised Fine-Tuning: Many-shot ICL performs comparably to full fine-tuning in low-resource translation, with close performance on Bemba and a slight fine-tuning advantage on Kurdish.The experiments use either 250 examples or the entire 997-example development set.
- Inference Cost of Many-Shot ICL: With KV caching, inference runtime increases linearly with the number of shots rather than quadratically for self-attention.Doubling the number of shots nearly doubles runtime when generated output is much shorter than the prompt.
- Frontier Model Comparisons: Many-shot benefits vary across frontier models and tasks: models scale favorably on Bemba but show little improvement on Kurdish.Gemini 1.5 Flash can match or outperform larger models on some many-shot translation tasks despite worse few-shot performance.
- Long-Context Scaling Laws: NLL decreases predictably with more context but is not a reliable proxy for downstream ICL performance in problem-solving domains.ICL success rates can decline after 125 shots without a corresponding NLL increase, and model-generated outputs can yield better performance despite higher NLL.
5. Related Work
The paper extends prior work on long-context scaling, fine-tuning comparisons, self-generated demonstrations, mathematical-function learning, and input-output generalization by studying many-shot ICL across broader settings.
- Scaling ICL: Prior work largely explored few-shot ICL because earlier language-model context lengths limited the use of hundreds or thousands of examples.This paper studies many-shot prompting with substantially longer contexts and up to thousands of examples.
- Learning Input-Output Relationships with ICL: The paper contributes evidence that many-shot demonstrations can override pre-training biases, complementing concurrent work on overriding preferences learned during RLHF.Its experiments also extend many-shot evaluation to more tasks, examples, and context length than the cited concurrent jailbreak study.
- Long-context Scaling Laws: Long-context studies report declining next-token loss with longer contexts, while this paper shows that loss can continue declining after overall performance plateaus.This motivates caution when using next-token prediction loss to evaluate long-context performance.
- Learning from Self-Generated Data: Unlike prior self-generated-demonstration methods, Reinforced ICL uses randomly sampled model-generated demonstrations without clustering, post-processing heuristics, or test-input access.The approach can apply to problems with reliable reward signals.
- Learning Mathematical Functions with LLMs: The paper differs from prior mathematical-function studies by showing that pretrained LLMs, rather than specially trained in-context transformers, can learn high-dimensional functions.The related work also notes concurrent evidence that pretrained LLMs can perform regression tasks.
- Comparing ICL with Fine-Tuning: Prior comparisons often found full fine-tuning stronger than few-shot ICL, whereas this paper compares both methods using the same number of examples in low-resource translation.The paper reports that many-shot ICL can perform comparably to full fine-tuning.
6. Discussion, Limitations and Future Work
The discussion presents broad gains from many-shot ICL and two methods for reducing dependence on human rationales, while identifying model coverage and unexplained degradation as limitations.
- Discussion: Many-shot ICL yields significant gains across translation, summarization, planning, reward modeling, mathematical problem solving, question-answering, algorithmic reasoning, and sentiment analysis.The paper also reports effectiveness at overriding pretraining biases, learning high-dimensional numerical functions, and performing comparably to SFT.
- Discussion: Reinforced ICL and Unsupervised ICL address the need for large numbers of high-quality human-written rationales in many-shot prompting.The former uses model-generated rationales, while the latter removes rationales and prompts with domain-specific questions.
- Limitations and Future Work: The main evaluation uses Gemini 1.5 Pro, so conclusions about many-shot ICL across long-context models remain limited in model coverage.Preliminary results and concurrent work suggest other LLMs can also benefit, motivating broader future evaluation.
- Limitations and Future Work: The paper does not completely explain why performance can sometimes degrade as more examples are added, including for MATH.The authors report that NLL trends are insufficient to explain this degradation and call for further investigation.
Contribution Statement
The project involved distributed contributions spanning leadership, experimentation, infrastructure, task-specific results, fine-tuning, analysis, writing, and revision.
- Project Leadership: RA led the project, ran most many-shot experiments and analysis, developed Reinforced ICL, coordinated collaborators, and led drafting and revision.RA also handled the NeurIPS rebuttal.
- Collaborative Contributions: Contributors supplied infrastructure, task results, fine-tuning experiments, NLL analysis, writing, editing, and feedback across MATH, GSM8K, BBH, planning, translation, GPQA, and parity.HL suggested the Unsupervised ICL experiments, while several contributors supported collaboration and paper revision.
A.1. Example Prompts
The appendix illustrates prompts spanning translation, summarization, planning, reasoning, code verification, and classification. These examples vary the number of demonstrations and task-specific output instructions.
- Translation prompts specify English–Kurdish example pairs and request only the Kurdish translation.
- Summarization prompts show a news article followed by a short fluent English summary.
- PDDL planning prompts encode a Logistics-domain problem involving cities, locations, vehicles, and a delivery goal.
- The examples include zero-shot GPQA, code-correctness verification on GSM8K, and Inner Monologue prompts for MATH and GSM8K.
- High-dimensional linear classification prompts use 8 shots per class with semantically unrelated labels such as ‘Foo’ and ‘Bar’.
A.2. Prompts for Unsupervised ICL
Unsupervised ICL prompts list domain-specific inputs without pairing every listed example with a rationale or solution. Output-format instructions remain in the prompt while the many-shot expansion increases the number of questions or source sentences.
- The appendix examples include mathematical problems, including a remainder question, an inequality, a parabola-vertex task, and trigonometric calculations.
- Unsupervised MATH and GSM8K prompts list problems, then specify the output format with up to 4 examples.Many-shot versions add problems but do not increase the problem-solution pairs used to describe the format.
- Unsupervised GPQA prompts list questions and add a separate preamble specifying the required output format.Many-shot versions increase only the number of listed questions.
- Unsupervised low-resource translation prompts list English source sentences and retain 1 input-output example for output formatting.Many-shot versions add source sentences rather than additional translated pairs.
A.3. Unsupervised ICL on Machine Translation
For low-resource machine translation, the appendix reports that Unsupervised ICL does not work. Providing only source sentences does not improve task specification.
- Unsupervised ICL does not work for low-resource machine translation.The prompt supplies source sentences without enough information to improve task specification.
A.4. Reinforced ICL: Data-collection Prompt Sensitivity and Iteration 2
The appendix examines reinforced-ICL prompt sensitivity and iteration, synthetic high-dimensional classification data generation, training comparisons, likelihood diagnostics, and summarization hallucinations. These examples show both methodological details and documented boundaries of many-shot prompting.
- A.4. Reinforced ICL: Data-collection Prompt Sensitivity and Iteration 2: Model-generated rationales from a 4-shot Minerva prompt are generally better or comparable to those from a 4-shot Inner Monologue prompt on MATH.A second reinforced-ICL iteration using unsolved problems from a 25-shot prompt can further improve many-shot performance.
- A.5. Linear Classification: Data Generation: Synthetic classification data use an N-dimensional decision boundary and threshold, with K examples per class provided as in-context exemplars.The model must classify unseen points without being given the boundary equation or threshold.
- A.5. Linear Classification: Data Generation: The generation routine documents N as dimensionality, k as exemplars per class, xtrain as 2k examples, and yeval as evaluation labels.
- A.5. Linear Classification: Data Generation: The generator samples integer coefficients and a pivot point, computes t = np.dot(a, p), and creates balanced training and evaluation examples.Training data contain k positive and k negative examples, while evaluation data contain 25 positive and 25 negative examples.
- A.6. Training GPT-2 from scratch on the sequential parity task: Training transformers from scratch on sequential parity do not reach 8192-shot ICL performance until 20× the number of examples.The comparison covers GPT-2 Small- and GPT-2 Medium-sized transformers.
- A.7. Negative Log-Likelihood on Model-Generated Data: Negative log-likelihood is evaluated on model-generated MATH solutions produced and scored with prompts ranging from 4-shot to 500-shot.For solutions generated with a 4-shot prompt, the lowest NLL occurs with 4-shot and 10-shot scoring prompts.
- A.8. Many-Shot ICL: Hallucinations on Summarization: Many-shot XSum summaries begin hallucinating dates and times at higher shot counts, including 250-shot, 500-shot, and other listed settings.The reference summary mentions the justice ministry but does not include those generated dates and times.