Many-Shot In-Context Learning

• Many-shot ICL significantly outperforms few-shot ICL across a wide range of tasks, especially on difficult non-natural-language tasks.The study reports gains using hundreds or thousands of shots, with maximum performance often requiring contexts of hundreds of thousands of tokens.Source: Figure 1 · p. 1, §1 · p. 2
• The study evaluates scaling with Gemini 1.5 Pro across diverse tasks, using up to 8192 shots and contexts as long as 1M tokens.The introduction notes that expanded context windows make many-shot ICL feasible, compared with the few-thousand-token windows of earlier models.Source: §1 · pp. 1–2
• Reinforced ICL uses correctness-filtered, model-generated rationales, while Unsupervised ICL prompts with problems alone; both can outperform few-shot human-rationale ICL on reasoning tasks.Reinforced ICL is more broadly effective across the reported problem-solving tasks.Source: §1 · pp. 2–3
• With sufficient examples, many-shot ICL can overcome pretraining biases, match full fine-tuning comparably, and solve numerical prediction tasks where few-shot ICL struggles.The introduction names sequential parity prediction and linear classification; it also reports that example order can affect performance and next-token loss may not reliably predict ICL performance.Source: §1 · pp. 2–3

2. Scaling In-Context Learning

• Evaluation uses greedy decoding, averages results across random-seed prompts, nests smaller-shot examples within larger prompts, and applies KV caching.The evaluated model is Gemini 1.5 Pro with a 1-million-token context window.Source: §2 · p. 3

2.1. Machine Translation

• chrF2++ improves by 15.3% on Bemba and 4.5% on Kurdish over the 1-shot prompt, establishing new state-of-the-art scores with all 997 dev pairs.Performance improves monotonically as the number of translation pairs increases.Source: §2.1 · p. 3, Figure 3 · p. 4

2.2. Abstractive Summarization

• Many-shot ICL approaches fine-tuned-model performance, but XSum performance declines beyond 50 shots while XLSum improves monotonically with more XSum examples.The model occasionally generated summaries with fabricated dates and times, despite their absence from the in-context summaries.Source: Figure 4 · p. 4, §2.2 · p. 4

2.3. Planning: Logistics Domain

• On Logistics, a recent Gemini 1.5 Pro version scales from 42% to 62% success, but remains far below state-of-the-art planning approaches.The older model version improved quickly through 10 shots and then saturated.Source: Figure 5 · p. 4, §2.3 · p. 5

2.4. Reward Modelling with Many-Shot ICL: Learning Code Verifiers

• At 128 shots, best-of-4 verification bridges the gap between 77.25% Pass@1 and 90% Pass@4 accuracy on GSM8K with Gemini 1.0 Pro solutions.Best-of-4 accuracy rises above pass@1 with 16 or more examples, while verifier Yes-token probabilities for correct and incorrect solutions separate with up to 256 shots. The verifier is prompted with labeled correct and incorrect code solutions and scores candidates using the normalized probability of the Yes token.Source: Figure 6 · p. 5, §2.4 · p. 5

3. Many-shot Learning without Human-Written Rationales

• Reinforced ICL filters model-generated rationales for correct final answers, while Unsupervised ICL supplies unsolved problems without rationales.Reinforced ICL samples multiple rationales from zero- or few-shot chain-of-thought prompts and assumes access to ground-truth answers or correctness checks; incorrect reasoning chains can still reach the right answer. Unsupervised ICL adds a preamble and output-format instruction, and may be less effective when outputs are critical for specifying the task; one hypothesis is that inputs help locate relevant pretrained knowledge.Source: §3 · p. 6

3.1. Problem-solving: Hendrycks MATH & GSM8K

• On MATH500, Reinforced and Unsupervised ICL outperform ICL with ground-truth solutions, and Reinforced ICL transfers best to GSM8K at 25 or more shots.On MATH500, Reinforced ICL plateaus around 25 examples at about 5% above ICL, while ICL peaks around 125 examples and then declines. On GSM8K, Reinforced ICL with MATH prompts outperforms both ground-truth-solution ICL and Unsupervised ICL in the many-shot setting with at least 25 shots.Source: Figure 7 · p. 6, §3.1 · p. 7

3.2. Question Answering: Google-Proof QA (GPQA)

• On GPQA, Reinforced ICL generally outperforms Unsupervised ICL, while ground-truth-rationale ICL improves through 125 shots but degrades at 250 shots.Model-generated rationales perform better than ground-truth rationales up to 25 shots and similarly with more shots; Unsupervised ICL has non-systematic results and generally underperforms Reinforced ICL. At 125 shots, both rationale-based methods surpass Claude-3 Sonnet's 40.4% accuracy, while the zero-shot baseline is 38.8%.Source: Figure 8 · p. 7, §3.2 · p. 8

3.3. Algorithmic and Symbolic Reasoning: Big-Bench Hard

• 83% success rate for Reinforced ICL, versus 77.1% for Unsupervised ICL and 72.1% for human-written CoT, averaged across BIG-Bench Hard tasks.Reinforced ICL outperforms Unsupervised ICL on almost all tasks, which outperforms the standard 3-shot CoT prompt; Reinforced ICL improves with more prompts on 7 of 8 tasks. The evaluation selects 8 of 23 tasks to reduce false positives, and generated rationales beat the human-written 3-shot prompt on some tasks even at 3 shots.Source: Figure 9 · p. 8, §3.3 · p. 8

4.1. Overcoming Pre-training Biases with Many-Shot ICL

• With more shots, flipped and abstract sentiment labels approach default-label accuracy, while flipped-label confidence drops before rising to a plateau, indicating many-shot ICL can overcome pretraining bias.The authors posit the initial drop may reflect an “early ascent” effect: a few shots retrieve an incorrect skill that diminishes as task learning takes effect.Source: §4.1 · p. 9, Figure 10 · p. 9

4.2. Learning Non-Natural Language Tasks: High-Dimensional Functions

• Many-shot ICL nearly matches k-nearest neighbors on high-dimensional classification and, on 20-digit sequential parity, improves through 8,192 shots to surpass GPT-2 trained on 20× more examples.For classification, performance can decline slightly beyond 512 shots at higher dimensions; the authors suggest nearest-neighbor-like search as a plausible mechanism. For parity, they suggest the result indicates computations analogous to gradient descent.Source: §4.2 · p. 10, Figure 11 · p. 10, Figure 12 · p. 10

4.3. Many-Shot ICL vs. Supervised Fine-Tuning

• For low-resource translation, supervised fine-tuning and many-shot ICL perform quite closely on Bemba, while fine-tuning has a slight edge on Kurdish.The experiments use either 250 examples or the entire 997-example dev set; the authors conclude many-shot ICL can be a viable alternative to fine-tuning for some tasks.Source: §4.3 · p. 11, Figure 13 · p. 11

4.4. Inference Cost of Many-Shot ICL

• With KV caching, doubling the number of shots nearly doubles runtime; runtime is nearly constant for few shots, but grows linearly with shots rather than quadratically.When generated output is much shorter than the prompt, each new token still incurs linear cost in prompt length.Source: §4.4 · pp. 11–12, Figure 14 · p. 11

4.5. Comparing Many-Shot Abilities of Frontier LLMs

• Frontier models vary in many-shot gains: Gemini 1.5 Pro improves more rapidly than Claude-3 on Bemba and reaches higher performance at 997 shots, while Gemini 1.5 Flash also surpasses stronger few-shot models with enough shots.Many-shot ICL scales favorably on Bemba for GPT-4-Turbo and Claude-3-Opus but shows little improvement on Kurdish; Flash also outperforms Claude-3 on English-to-Tamil while lagging Pro.Source: §4.5 · p. 12, Figure 15 · p. 12

4.6. Where Do Gains in Many-Shot ICL Stem From?

• On low-resource translation, repeating 25 examples up to 1,000 times performs nearly the same and significantly worse than using distinct examples, so gains mainly come from added information.This experiment separates the effect of distinct samples from simply increasing prompt length.Source: §4.6 · p. 12, Figure 16 · p. 13

4.7. Is Many-Shot ICL Sensitive to Example Ordering?

• On MATH, example orderings vary substantially across subareas, though variation in average performance is smaller, posing a challenge for reliable many-shot results.The study compares ten random orderings of 50 examples against 500 held-out test examples; an ordering that excels in one subarea may do poorly in another.Source: §4.7 · p. 13, Figure 17 · p. 13

4.8. Long-context scaling laws may not predict ICL performance

• NLL generally decreases with more context but does not reliably predict ICL performance: task success can decline while NLL does not rise, and transfer performance can improve despite nearly unchanged NLL.Ground-truth-output NLL can be lower even when model-generated-output ICL performs better; for problem solving, scoring one ground-truth chain of thought may not represent capability because many solutions can be correct.Source: §4.8 · pp. 13–14, Figure 18 · p. 14

Scaling in-context learning

• Brown et al. [8] reported improved in-context learning with up to 64 examples; Li et al. [31] scaled to 2000 using a custom architecture, evaluating models up to 1.3B parameters that were ineffective on complex tasks such as GPQA.Source: §5 · p. 14

Long-context scaling laws

• Many-shot ICL confirms that next-token prediction loss falls with longer contexts, but the loss can keep decreasing after overall performance plateaus.Prior reports [68] [27] [16] and theoretical analysis by Jeon et al. [25] also found lower next-token loss with longer contexts.Source: §5 · p. 14

Learning from self-generated data

• Prior self-generated-data methods [19] [70] [55] generate and filter samples using binary feedback, fine-tune on them, and repeat; this work extends the idea to Reinforced ICL for reasoning and problem-solving.Source: §5 · p. 15

Self-generated data and in-context learning

• Reinforced ICL generates demonstrations using Singh et al.’s [55] procedure, requires no clustering, post-processing heuristics, or test inputs, and applies to problems with reliable reward signals, focusing on randomly sampled model-generated demonstrations for many-shot ICL.Source: §5 · p. 15

Learning Input-Output Relationships with ICL

• Building on mixed findings about whether ICL learns input-output relationships [69] [30] [36], this work extends the trend to longer contexts, showing enough examples can mostly overcome pretraining biases.Source: §5 · p. 15

Learning Mathematical Functions with LLMs

• Unlike prior function-learning studies that trained transformers for ICL [14] [72] [67] [7], this work shows pretrained LLMs can learn high-dimensional functions and perform comparably to kNNs on high-dimensional classification.It complements Vacareanu et al. [58], who found pretrained LLM regression rivaled supervised methods with 500 examples, by studying other synthetic tasks with many more examples; Figure 11 reports the kNN comparison, while Dinh et al. studied fine-tuned GPT-3.Source: §5 · p. 15

Comparing ICL with fine-tuning

• Where few-shot comparisons found mixed results, this work compares full fine-tuning and many-shot ICL using equal example counts for low-resource translation and finds comparable performance.Bertsch et al. [6] concurrently found many-shot ICL generally outperformed LoRA on classification tasks.Source: §5 · pp. 15–16

Exemplar vs. Rule-based ICL generalization

• Although many-shot linear classification supports exemplar-based explanations of ICL [9] [6], sequential parity shows the strongest improvement despite nearest neighbors always predicting the wrong answer.The authors suggest larger, more capable models may be less exemplar-based, but leave this explanation open.Source: §5 · p. 16

Discussion, Limitations and Future Work

• Many-shot ICL significantly improves over few-shot ICL across translation, summarization, planning, reward modeling, math, QA, algorithmic reasoning, and sentiment.To address the difficulty of obtaining many high-quality human-written rationales, the authors introduce Reinforced ICL and Unsupervised ICL.Source: §6 · p. 16
• Unlike few-shot ICL, many-shot ICL overrides pretraining biases, learns high-dimensional functions with numerical inputs, and performs comparably to SFT.Source: §6 · p. 16
• Because evaluation centers on Gemini 1.5 Pro, future work should test more long-context models; concurrent works [6] and preliminary results suggest other LLMs also benefit.The preliminary results involve GPT-4-Turbo, Gemini 1.5 Flash, and Claude-3-Opus.Source: §6 · p. 16
• Many-shot performance may serve as a valuable long-context model quality metric beyond the needle-in-a-haystack test.Source: §6 · p. 16
• The authors do not fully understand why more prompt examples can degrade performance, as with MATH, and find that negative log-likelihood trends do not explain it.They call for future work to investigate the degradation and improve many-shot ICL capabilities.Source: §6 · p. 16

Contribution Statement

• RA initiated and led the project, ran most many-shot experiments and analysis, conceived reinforced ICL, and wrote the initial draft and revision.RA also onboarded collaborators and handled the NeurIPS rebuttal.Source: Contribution Statement · p. 17
• AS contributed MATH and GSM8K experiment infrastructure, ran BBH experiments, co-led fine-tuning experiments, conducted problem-solving NLL analysis, and wrote several sections.Source: Contribution Statement · p. 17
• Other collaborators contributed verifier and planning results, fine-tuning leadership, GPT-4 and Claude-3 results, GPQA, parity baselines, editing, and unsupervised ICL suggestions.HL suggested the unsupervised ICL experiments and, with AF, provided feedback on an early draft; others joined discussions and made minor edits.Source: Contribution Statement · p. 17

A.1. Example Prompts

• The prompt gallery includes a translation test input and an 8-shot-per-class linear-classification example in 16 dimensions, using “Foo” and “Bar” labels [63].The classification example is discussed in §4.2.Source: Figure A.1 · p. 23, Figure A.7 · p. 27
• Examples include a 1-shot summarization prompt for XSum and XLSum and a zero-shot prompt for GPQA.Source: Figure A.2 · p. 23, Figure A.4 · p. 24
• A 1-shot PDDL [17] prompt illustrates a Logistics-domain test example, with trucks moving between locations within cities and airplanes between cities.Each city has one truck and a designated airport location.Source: Figure A.3 · p. 24
• A 2-shot GSM8K code-correctness-verifier prompt uses solutions generated by Gemini 1.0 Pro [15].The caption points to results in §2.4.Source: Figure A.5 · p. 25
• A 4-shot Inner Monologue prompt is shown for MATH and GSM8K.Source: Figure A.6 · p. 26

A.2. Prompts for Unsupervised ICL

• For MATH and GSM8K, prompts list problems and specify the output format with up to 4 examples; many-shot prompts add problems, not problem-solution pairs.The prompt uses a preamble before the problems and another before the output-format examples.Source: Figure A.8 · p. 28
• For GPQA, prompts list questions and specify the output format; many-shot prompts increase the number of questions only.Both the question list and output-format instructions are introduced with preambles.Source: Figure A.9 · p. 29
• For low-resource MT, prompts list source sentences and use 1 input-output example to specify the format; many-shot prompts increase the number of source sentences only.Source: Figure A.10 · p. 29

A.3. Unsupervised ICL on Machine Translation

• Unsupervised ICL does not work for low-resource machine translation, as source sentences alone do not improve task specification.The figure points to Figure A.10 for the experiment’s prompt.Source: Figure A.11 · p. 30

A.4. Reinforced ICL: Data-collection Prompt Sensitivity and Iteration 2

• A second Reinforced ICL iteration using rationales from the best-performing 25-shot prompt appears to further improve MATH many-shot performance; 4-shot Minerva rationales are generally better or comparable to 4-shot InnerMono rationales.The second iteration uses MATH training problems not solved in the first iteration.Source: Figure A.12 · p. 30

A.5. Linear Classification: Data Generation

• The linear-classification data generator supplies balanced labeled examples above and below a hidden threshold, then asks the model to classify unseen N-dimensional points.The decision boundary and threshold are randomly sampled; the prompt does not reveal their equation or value.Source: §A.5 · p. 30

A.6. Training GPT-2 from scratch on the sequential parity task

• Training transformers from scratch on sequential parity did not reach 8192-shot ICL performance until using 20× the number of examples.The comparison covers GPT-2 Small- and Medium-sized models.Source: Figure A.13 · p. 31

A.7. Negative Log-Likelihood on Model-Generated Data

• For model-generated MATH solutions, negative log-likelihood is generally lowest when generation and scoring prompts have similar lengths, while correct and incorrect solutions follow similar curves.For solutions generated with a 4-shot prompt, NLL is lowest with 4-shot and 10-shot scoring prompts and increases as scoring prompts lengthen.Source: Figure A.14 · p. 32

A.8. Many-Shot ICL: Hallucinations on Summarization

• On XSum, an example of many-shot ICL shows hallucinated dates and update times as shot count increases: at 50, 125, 250, and 500 shots, the summaries give varying timestamps.Source: §A.8 · p. 32
• The 1-shot XSum summary adds claims about restricted activities, election-related polling, and political censorship beyond the supplied reference summary.The reference says the justice ministry labeled the polling agency a “foreign agent” and that it could no longer work.Source: §A.8 · p. 32