PaperlayerView exact revision on arXiv

Source-linked AI summary

Many-Shot CoT-ICL: Making In-Context Learning Truly Learn

Tsz Ting Chung, Lemao Liu, Mo Yu, Dit-Yan Yeung

arXiv:2605.13511v3cs.CLcs.AI

TL;DR

The paper asks whether many-shot ICL findings from non-reasoning tasks extend to reasoning-oriented CoT-ICL. Across models and tasks, it finds that reasoning scaling is model- and task-dependent, then frames CoT-ICL as test-time learning and proposes CDS, which yields consistent gains across math and narrative reasoning.

  • Problem

    Prior many-shot ICL research largely studies non-reasoning tasks, leaving the scaling behavior of many-shot CoT-ICL for reasoning unknown.

  • Method

    The paper systematically evaluates ICL across task types, model types, prompt formats, and demonstration counts, then develops CDS from ease-of-understanding and smooth-progression principles.

  • Results

    Many-shot CoT-ICL scales mainly for reasoning-oriented models, remains unstable for non-reasoning models, and exhibits increasing performance variance as demonstrations grow.

  • Takeaways & Limitations

    Many-shot CoT-ICL is better understood as in-context test-time learning, with useful demonstrations that are understandable and smoothly ordered rather than merely similar.

  • Takeaways & Limitations

    Because CoT demonstrations are substantially longer than standard ICL examples, the scaling analysis is limited to at most 128 demonstrations under long-context constraints.

Abstract

from arXiv · show

While many-shot ICL achieves remarkable performance, prior studies of its scaling behavior have mainly focused on non-reasoning tasks. In this work, we study many-shot ICL on reasoning tasks, with a particular focus on many-shot chain-of-thought in-context learning (CoT-ICL). Analyzing across non-reasoning and reasoning tasks and across non-reasoning and reasoning-oriented LLMs, we identify several distinctive properties of many-shot CoT-ICL. We further interpret these findings by viewing many-shot CoT-ICL as in-context test-time learning rather than scaled pattern matching, and suggest two principles: (i) demonstrations should be easy for the target model to understand, and (ii) they should be ordered to support a smooth conceptual progression. Guided by the principle, we propose Curvilinear Demonstration Selection (CDS), a simple ordering method that yields up to a 5.42 percentage-point gain on a math task with 64 demonstrations. Overall, our results reframe the long context window from a retrieval buffer into a structured curriculum for in-context test-time learning.

1. Introduction

The introduction argues that many-shot CoT-ICL behaves differently from established many-shot ICL findings on non-reasoning tasks. It reframes reasoning demonstrations as an in-context test-time learning curriculum guided by understandability and smooth progression.

  • Background: Many-shot CoT-ICL combines expanded-context many-shot ICL with chain-of-thought prompting for complex reasoning without updating model parameters.Many-shot ICL uses dozens to hundreds of demonstrations, while CoT prompting generates intermediate reasoning steps before answers.
  • Research gap: Prior many-shot dynamics mainly concern non-reasoning tasks, leaving unclear whether more reasoning demonstrations reliably improve many-shot CoT-ICL.For non-reasoning tasks, demonstration-order effects reportedly diminish with scale, but the extension to reasoning remained unknown.
  • Findings: Many-shot ICL scales on non-reasoning tasks, whereas reasoning-task CoT-ICL is unstable for non-reasoning LLMs and improves mainly for reasoning-oriented LLMs.The experiments compare non-reasoning and reasoning-oriented models across non-reasoning and reasoning tasks.
  • Findings: Similarity-based retrieval explains non-reasoning scaling but fails for reasoning because question similarity does not guarantee procedural compatibility.This motivates interpreting many-shot CoT-ICL as learning beyond surface pattern matching.
  • Interpretation and method: The paper reframes many-shot CoT-ICL as in-context test-time learning requiring understandable demonstrations and a smoothly sequenced conceptual progression.These principles motivate a practical method for demonstration ordering that advances many-shot reasoning.

2. Related Works

Prior many-shot ICL and demonstration-selection research largely emphasizes non-reasoning tasks and similarity-based retrieval, while CoT methods primarily improve reasoning for individual queries. This work challenges that framing for CoT-ICL by emphasizing smooth learning sequences that support conceptual understanding.

  • Many-shot ICL: Many-shot ICL research has largely studied non-reasoning tasks, leaving its effectiveness on reasoning tasks comparatively neglected.Prior work also found that gains often come from influential examples and that similarity-based retrieval can underperform validation-guided optimization and generation.
  • Chain-of-Thought: CoT prompting decomposes reasoning into intermediate steps, while later methods explore structured paths and search-based trajectory optimization for single queries.These approaches primarily enhance the reasoning process for one query rather than constructing a many-shot learning sequence.
  • Demonstration Selection: Demonstration selection traditionally retrieves examples most similar to the test query, implicitly framing in-context learning as pattern matching.This paradigm parallels embedding-similarity retrieval in retrieval-augmented generation.
  • Demonstration Selection: For CoT-ICL, this work hypothesizes that effective selection should construct a smooth learning sequence that facilitates conceptual understanding rather than retrieve semantically similar examples.The proposed framing shifts retrieval from matching toward learning.

3. Settings

The experimental framework studies many-shot ICL and CoT-ICL across task types, model types, prompting configurations, and demonstration counts under long-context constraints. It compares traditional ICL with CoT-ICL while accounting for the substantially greater context cost of reasoning demonstrations.

  • The framework varies task type, model type, prompt format, and number of demonstrations to study many-shot ICL with and without CoT under long-context constraints.
  • Evaluation covers classification-style and multi-step reasoning tasks using open-ended generation, task-specific answer extraction, normalization, and exact-match scoring.Numerical answers from GSM8K and MATH are compared with ground-truth values or expressions under the same exact-match criterion.
  • The benchmark suite includes SuperGLUE, NLU, TREC, and BANKING77 for non-reasoning tasks, plus GSM8K, MATH, and DetectiveQA for reasoning tasks.For datasets with gold rationales, demonstrations use the provided reasoning chains as their CoT components.
  • Models are grouped into non-reasoning LLMs that produce direct answers and reasoning-oriented LLMs that generate explicit intermediate reasoning tokens.The evaluated reasoning-oriented models are Qwen 3, QwQ, and DeepSeek-R1, with reasoning mode enabled during inference.
  • Traditional ICL uses ordered input–output pairs, whereas CoT-ICL uses ordered input–reasoning-chain–output triples and generates both an intermediate chain and final answer.
  • ∼30× longer CoT demonstrations constrain CoT-ICL to at most a few hundred examples, motivating scaling analysis focused on n ≤128.The comparison comes from the geometry setting versus BANKING77, and the limit captures trade-offs among model type, task type, and context length.

4. Properties of CoT-ICL

Many-shot CoT-ICL scales differently from ordinary ICL: reasoning performance is unstable for non-reasoning LLMs but improves consistently for reasoning-oriented models. Similarity retrieval fails on reasoning tasks, while demonstration order creates strong and growing path dependence.

  • Scaling behavior: Reasoning performance is unstable and often degrades for non-reasoning LLMs as CoT demonstrations increase, even for Llama 3.3 70B.By contrast, non-reasoning tasks improve steadily with more demonstrations.
  • Scaling behavior: QwQ (32B), R1 (685B), and smaller Qwen3 reasoning models improve consistently or near-monotonically as more CoT demonstrations are added.The divergence suggests positive scaling requires mechanisms that use demonstrations as intermediate reasoning signals.
  • Retrieval versus learning: Similarity retrieval helps BANKING77 but fails on reasoning tasks, where most-similar sets are consistently worse than most-dissimilar or original sets.Question-level similarity is therefore a weak proxy for procedural compatibility in reasoning.
  • Retrieval versus learning: Surface matching can produce negative gains because procedurally mismatched CoTs mislead models, whereas reasoning-oriented models can extract higher-level procedures from demonstrations.This explains why positive scaling differs between reasoning-oriented models on reasoning tasks and non-reasoning models on non-reasoning tasks.
  • Order sensitivity: Reasoning-task performance variance increases with more demonstrations across both non-reasoning and reasoning LLMs, unlike the low variance reproduced for non-reasoning tasks.The result indicates strong and growing path dependence on demonstration sequencing.

5. Rethinking ICL: From Pattern Matching to In-Context Test-Time Learning

The section reframes many-shot CoT-ICL as in-context test-time learning, arguing that models absorb procedures from demonstrations rather than merely matching input–output patterns. This view motivates principles of model-understandable demonstrations and smoothly ordered information flow.

  • From Pattern Matching to In-Context Test-Time Learning: Many-shot CoT-ICL behaves unlike simple pattern matching because increasing demonstration counts amplify order sensitivity and similarity-based selection is unreliable on reasoning tasks.The prompt is treated as training data, and the forward pass as gradient-free adaptation.
  • Procedure Absorption: At n = 128, procedurally corrupted rationales cause clear performance drops for both Qwen3-8B and Qwen3-14B despite preserving questions, answers, format, and context length.The corruption replaces all rationales with the same static chain, isolating whether demonstration-specific procedures matter.
  • Design Principles: Demonstrations should be understandable to the target model and arranged to create smooth information flow, because incomprehensible examples cannot supervise adaptation and abrupt transitions destabilize induced procedures.These principles directly explain the order sensitivity observed earlier.
  • Demonstration Understandability: Self-generated CoTs consistently outperform dataset-provided and stronger-model demonstrations for weaker targets, even when some self-generated demonstrations have incorrect final answers.The result supports distributional alignment as a determinant of demonstration usefulness.
  • Ordering Smoothness: Overall r = −0.547, with correlations of −0.545 for geometry, −0.468 for number theory, and −0.628 for counting and probability, showing that smoother orderings tend to improve accuracy.Higher demonstration counts make random permutations more likely to contain sharp conceptual jumps, increasing order variability.

6. Curvilinear Demonstration Selection

Curvilinear Demonstration Selection (CDS) orders many-shot chain-of-thought demonstrations to create a smooth embedding-space trajectory, motivated by the relationship between curvature and performance. It is evaluated across three reasoning tasks and tested for robustness across embedding models and target LLMs.

  • Method: CDS constructs demonstration orderings that avoid abrupt transitions between successive examples by producing a smooth trajectory in embedding space.The method is motivated by the curvature–performance correlation reported in Section 5.2.
  • Evaluation: CDS is evaluated on geometry proof generation, number theory problem solving, and DetectiveQA logical reasoning using reasoning LLMs from the Qwen3 family.The primary experiments use Qwen3 8B and 14B models across multiple demonstration budgets.
  • Method: CDS seeks a permutation of n demonstrations that minimizes total curvature.The objective is defined over the demonstrations’ original embeddings.
  • Method: A TSP-based heuristic balances spatial proximity with a local curvature proxy because exact permutation search is infeasible for prompts with n ≤128.The resulting graph search uses nearest-neighbor initialization, 2-opt local search, tour breaking, and multiple starting points to select a smooth path.
  • Robustness: CDS gains persist when using bge-m3 embeddings and when evaluating the closed-source gpt-5.2 model, especially on geometry and DetectiveQA.Number theory shows smaller margins because baseline accuracies are already high, leaving less room for improvement.

7. Conclusion … A.4. Qualitative Example: When “Similar” Questions Provide Misleading CoT

Many-shot CoT-ICL differs from non-reasoning many-shot ICL: scaling depends on the model and task, similarity retrieval can select procedurally incompatible demonstrations, and ordering effects increase with more CoT examples. The appendix details a cosine-similarity selection procedure and illustrates how semantic similarity can mislead reasoning.

  • 7. Conclusion: Many-shot CoT-ICL does not inherit non-reasoning ICL’s stable scaling and diminishing ordering effects: scaling varies by model and task, retrieval can fail procedurally, and order variance grows.The paper frames CoT-ICL as in-context test-time learning rather than scaled pattern matching.
  • A.1. Candidate pool and data splits: Candidate demonstrations come from each task’s training split, while evaluated queries come from the disjoint test split, preventing query-demonstration overlap.This defines the candidate pool and data-split protocol for similarity-based selection.
  • A.2. Embedding model and similarity: Questions are embedded with Qwen3-Embedding-4B, and candidate demonstrations are ranked against each test query using cosine similarity.Similarity is computed from question embeddings, excluding answers and rationales.
  • A.3. Constructing the most-similar and most-dissimilar sets: For a target of k demonstrations, selection forms top-k most-similar and bottom-k most-dissimilar candidate sets.The most-similar set is presented in descending similarity order, whereas the most-dissimilar set uses ascending similarity order.
  • A.4. Qualitative Example: When “Similar” Questions Provide Misleading CoT: A highly embedding-similar geometry demonstration can induce incorrect intermediate steps when its invariant or decomposition differs from the query.A less similar but structurally closer demonstration instead encourages the correct decomposition and improves accuracy.
  • A.4. Qualitative Example: When “Similar” Questions Provide Misleading CoT: In the illustrative query, the correct solution finds YZ=100 and XW=48 before deriving WZ=64 through area and Pythagorean relationships.The example’s alternative solution also obtains WZ=64 from triangle similarity.
  • A.4. Qualitative Example: When “Similar” Questions Provide Misleading CoT: The retrieved example is misleading because it uses a 30°−60°−90° triangle’s 1:2 ratio and lacks an altitude-to-hypotenuse configuration, so its method does not transfer.This explains why question-level semantic similarity can select incompatible reasoning trajectories.

A.5. Analysis of Similarity in Different LLM Types

The analysis compares similarity-based demonstration selection separately for non-reasoning and reasoning LLMs, reporting averaged performance for original, similar, and dissimilar sets.

  • Figures 9 and 10 compare similarity-based selection separately across non-reasoning and reasoning LLMs.
  • Figure 9 reports performance for original (ori), similarity (sim), and dissimilar (dis) sets averaged across three non-reasoning LLMs.
  • Figure 10 reports performance for original (ori), similarity (sim), and dissimilar (dis) sets averaged across two reasoning LLMs.

B. Statistical Robustness on a New ICL Subset

Repeating the analysis on a newly sampled ICL subset across five random demonstration-ordering seeds shows that the reported trends are robust beyond a single ordering or candidate pool.

  • Statistical robustness: Five random demonstration-ordering seeds were used to compute means and standard deviations on a newly sampled ICL subset.The analysis was explicitly repeated across random seeds and summarized with mean ± standard deviation.
  • Statistical robustness: The observed trends persist beyond a single demonstration ordering.This supports robustness to the ordering seed used in the analysis.
  • Statistical robustness: The observed trends also persist beyond a single candidate pool, strengthening the claims in Figures 7, 8, and 6.The robustness analysis extends the claims across the newly sampled subset rather than relying on one candidate pool.

C. Curvature-based Smoothness: Details and Implementation … C.3. Correlation Protocol

This section specifies how curvature-based ordering smoothness is computed and correlated with performance. Demonstrations are embedded in a fixed format, scored by bounded mean curvature, and evaluated through Pearson correlation across random orderings.

  • C. Curvature-based Smoothness: Details and Implementation: Algorithm 1 takes multiple demonstration orderings and their performance scores, then outputs the correlation coefficient between ordering smoothness and performance.It initializes one smoothness score per ordering before computing the final correlation.
  • C.1. Demonstration Format: Each demonstration is represented as one text string concatenating its Question, Chain-of-Thought, and Answer.The main-paper curvature results use this template unless otherwise stated.
  • C.1. Demonstration Format: For each ordering, the procedure computes local curvature angles from adjacent embedding vectors and stores the resulting angles in a curvature list.The cosine value is bounded to [-1, 1] before applying arccos.
  • C.1. Demonstration Format: The ordering smoothness score is m[j] = 1/(1 + θ̄^(j)), and the procedure returns PearsonCorrelation(m, S).Here, θ̄^(j) denotes the mean curvature angle for ordering j, while S contains the corresponding performance scores.
  • C.2. Embedding Model: Demonstrations are embedded with Qwen3-Embedding-4B, producing vectors e_i = Embed(d_i) ∈ R^d from formatted demonstration strings d_i.The implementation uses the embedding model’s default output representation with vLLM deployment.
  • C.3. Correlation Protocol: The correlation protocol samples multiple random permutations of a fixed set containing n = 128 demonstrations.This fixed demonstration set comes from Section 4.4.
  • C.3. Correlation Protocol: For every permutation O, the protocol computes m(O) = 1/(1 + θ̄(O)), evaluates accuracy under the corresponding prompt, and correlates scores with accuracy.The Pearson correlation is computed over the sampled orderings.

D. CDS: Details and Implementation · D.1. Model Studies

The study isolates demonstration-ordering effects in many-shot CoT-ICL by evaluating reasoning-oriented models with positive demonstration scaling. It controls rationale quality and uses Qwen3 models across three tasks and varying demonstration counts.

  • D. CDS: Details and Implementation: The experiments target reasoning-oriented LLMs whose performance improves with more demonstrations, enabling evaluation of whether better ordering further benefits in-context learning.This setup is intended to isolate demonstration ordering from models lacking in-context learning capacity.
  • D.1. Model Studies: The WSC prompt requires each predicted answer to come from the demonstrations and follow the exact prescribed format.The task asks whether a pronoun refers to a specified span and requires an Answer: [yes | no] response.
  • D.1. Model Studies: The WSC demonstrations present sentence, pronoun, reference-span, and answer fields before a final query requiring the same decision format.The prompt includes multiple indexed examples followed by a query instance.
  • D.1. Model Studies: Figure 11 documents the prompt used for the WSC task in the model studies.It provides the implementation-level prompt reference for the task.
  • D.1. Model Studies: Dataset-provided CoT rationales and answers control for errors or quality degradation from self-generated rationales.The analysis therefore focuses on models that can interpret and leverage the provided CoT.
  • D.1. Model Studies: Qwen3 8B and 14B models are evaluated across varying demonstration counts on geometry, number theory, and DetectiveQA.These models provide a stable platform because they satisfy the study’s scaling and rationale-use criteria.

E. Prompt Formatting for Each Task

The appendix specifies task-specific inference prompts spanning binary classification, fine-grained intent and question labeling, mathematical reasoning, and narrative evidence reasoning. These prompts define demonstration formats and required answer structures, while DetectiveQA additionally filters overlapping test sources to reduce information leakage.

  • E.1. SuperGlue: SuperGlue uses premise-choice demonstrations and requires the model to answer in A or B using the format “Answer: [A | B].”Each demonstration contains a premise, two options, and an answer.
  • E.2. TREC: TREC prompt inference covers 50 fine class labels, presents labeled question demonstrations, and requires the final output format “label: [category_name].”The prompt restricts predictions to the listed categories and asks for the final question’s label.
  • E.3. BANKING77: BANKING77 evaluates 77 fine-grained banking intents with a prompt that restricts predictions to the listed categories and asks for the final query’s intent category.The inference prompt is presented in Figure 14.
  • E.4. NLU: NLU evaluates a conversational-domain dataset with 68 fine-grained intents, using an inference prompt presented in Figure 15.The supplied passage identifies the dataset and prompt location but does not provide the full template.
  • E.5. GSM8K: GSM8K is evaluated on grade school math word problems with an inference prompt presented in Figure 16.The supplied passage specifies the dataset and task type but not the full prompt text.
  • E.6. MATH: MATH covers competition problems across seven mathematical question types, while its prompts use chain-of-thought demonstrations and request step-by-step solutions with boxed final answers.The listed types are counting and probability, prealgebra, geometry, precalculus, number theory, and algebra; the supplied passages also show a summary-answer format for one prompt.
  • E.7. DetectiveQA: DetectiveQA uses annotated derivations labeled “-1” for CoT demonstrations and excludes test instances sharing training data sources to prevent narrative-information leakage.The prompt includes question, context, four options, demonstration derivations, and a required final answer statement.
Loading 2605.13511v3…