From Harm to Help: Turning Reasoning In-Context Demos into Assets for Reasoning LMs

1 INTRODUCTION

• Few-shot CoT is a standard reasoning-evaluation approach, but earlier benchmark demonstrations may constrain modern RLMs, motivating the question of whether higher-quality examples help.The section notes that few-shot CoT is used across tasks including mathematical problem-solving and commonsense reasoning.Source: §1 · pp. 1–2
• Even high-quality, closely matched reasoning demos lower accuracy as shots increase, with semantic copying and strategy-transfer failure identified as mechanisms.High textual similarity can lead models to treat distinct problems as identical and copy intermediate steps; models also struggle to extract and apply useful strategies.Source: Figure 1 · p. 2, §1 · p. 2
• I2S extracts reusable insights from demonstrations and applies them to derive target-specific reasoning; I2S+ optionally self-refines that trace before demo-decoupled solving.The design aims to mitigate strategy-extraction failure and semantic misleading, respectively.Source: §1 · p. 2
• Across open- and closed-source models, I2S and I2S+ improve performance; GPT-4.1 gains +14.0 on AIME’25, while o1-mini gains +2.7 on AIME and +1.7 on GPQA.The passage also reports gains on open-ended General Reasoning tasks.Source: Figure 2 · p. 2, §1 · p. 2

Few-Shot CoT Prompting

• Few-shot CoT prompting elicits step-by-step reasoning through demonstrations; prior work automates exemplar selection and generation, while recent studies question its effectiveness for powerful models.The cited work reports CoT can be detrimental for certain tasks and may enforce formatting rather than improve reasoning.Source: §2 · p. 3

Test-Time Scaling

• Test-time scaling allocates inference-time compute through parallel candidate aggregation or sequential refinement; this work instead studies effective use of reasoning demonstrations.Parallel approaches include majority voting and search, while sequential approaches include self-revision and external feedback.Source: §2 · p. 3

3.1 REASONING DEMONSTRATIONS DEGRADE REASONING MODELS

• The controlled evaluation retrieves relevant demonstrations from OpenThoughts-114k for AIME 2025 and GPQA Diamond, comparing few-shot prompts with zero-shot direct inference.The tested models include Qwen3 variants and DeepSeek-R1-Distilled models.Source: §3.1 · p. 4
• Across all evaluated models and both benchmarks, adding reasoning demonstrations reduces accuracy versus direct inference, with larger shot counts worsening the decline.The decline also affects the 32B distilled model, despite its training on DeepSeek-R1-generated traces.Source: §3.1 · p. 4

3.2 DIAGNOSING IN-CONTEXT REASONING FAILURES

• The analysis separates reasoning-trace generation from final-answer generation and examines how demonstrations condition each stage.The trace is the model’s intermediate scratchpad for forming hypotheses, calculating, and self-correcting.Source: §3.2 · p. 4

3.3 WHY DO REASONING DEMONSTRATIONS BACKFIRE?

• Demonstrations can mislead through surface similarity, prompting copied steps, or thwart transfer when the model mistakes a target’s structural constraint for the example’s different one.The examples contrast copying a heuristic despite a divisibility-by-22 constraint with reducing a triplet-wise seating constraint to pairwise exclusions.Source: §3.3 · pp. 4–5, Figure 3 · p. 5

3.4 A QUICK FIX: “TWO-STEP” INFERENCE

• Two-step inference elicits a trace with demonstrations, removes them, and generates the answer from the target question and trace, improving over naive single-pass one-shot prompting.It does not yet match direct inference.Source: §3.4 · p. 5

4 INSIGHT–REFINE–SOLVE

• I2S addresses semantic misleading and strategy-transfer failure by extracting abstract guidance from demonstrations and applying it to target questions; I2S+ adds iterative refinement.The pipeline breaks the shortcut from demonstrations to answers.Source: §4 · pp. 5–6

4.1 DESIGN PRINCIPLES

• I2S prioritizes reusable strategies over raw chain-of-thought content, using demonstrations as hints that complement rather than override the model’s reasoning.I2S+ lets the model iteratively refine its own target-specific trace.Source: §4.1 · p. 6

4.2 STRUCTURED REASONING TRANSFER

• Structured transfer compares the example and target, derives transferable insights from that comparison and the example trace, then generates a new trace from the target and insights.The example trace is withheld during target-trace generation, discouraging verbatim reuse while supplying useful strategies.Source: §4.2 · p. 6

4.3 ITERATIVE SELF-REFINEMENT

• I2S+ iteratively proposes trace changes, reviews their validity, and applies the best-rated suggestion to improve coherence and correctness while preserving question consistency.The loop can stop after a fixed number of iterations or when no further improvement occurs.Source: §4.3 · pp. 6–7

5.1 SETUP

• The evaluation covers AIME 2025, GPQA Diamond, and balanced 500-question Engineering and General subsets, using five runs and reporting average performance.Models include three open-source RLMs and two closed-source models; demonstrations come from OpenThoughts-114k and GeneralThought-430k, with benchmark questions excluded from retrieval.Source: §5.1 · p. 7

5.2 MAIN RESULTS

• I2S and I2S+ outperform direct inference and SR/PHP on closed-ended and open-ended reasoning; for R1-Distill-Qwen-14B, I2S+ raises AIME’25 from 50.00 to 60.00 (+10.0) and GPQA from 59.50 to 63.03 (+3.5).On general reasoning the gains are smaller, and I2S sometimes matches or slightly exceeds I2S+; the authors suggest LLM-as-judge sensitivity may partly explain this.Source: Table 1 · p. 8, Table 2 · p. 8, §5.2 · p. 8

5.3 HOW EFFICIENTLY DOES EXTRA COMPUTE TURN INTO GAIN?

• With three question-conditioned calls, I2S consistently outperforms majority@3 on AIME’25 and GPQA; against majority@32, I2S+ is mixed across benchmarks and models.The comparison matches question-conditioned calls rather than raw FLOPs, and majority@N is not defined for open-ended generation.Source: §5.3 · p. 8, Table 3 · p. 9

5.4 EFFECT OF REFINEMENT ITERATIONS

• Refinement gains on AIME’25 mostly arrive in the first one or two iterations before saturating, while GPQA patterns vary by model and open-ended changes are marginal or occasionally negative.On AIME’25, gains are +3.33 for 7B and +6.00 for 14B; the 1.7B model peaks at Iter 1 and then regresses.Source: §5.4 · p. 9, Table 4 · p. 9

• Few-shot demonstrations can hurt modern RLMs even when they are high quality and closely matched to the target question.Source: §6 · p. 9
• The analysis identifies semantic misleading and strategy transfer failure as mechanisms behind the degradation.Surface similarity can prompt near-verbatim copying, while useful problem-solving patterns may not be extracted and applied to the target.Source: §6 · p. 9
• I2S extracts transferable insights before target-specific solving, with I2S+ optionally self-refining the reasoning trace.The framework decouples solution generation from demonstrations.Source: §6 · p. 9
• Across closed-ended AIME’25 and GPQA and open-ended General Reasoning, I2S/I2S+ outperform direct answering and baselines across diverse models, including GPT models.Source: §6 · p. 9

A.1 METHODS TO COMPARE

• Direct Inference is zero-shot answering without demonstrations and serves as a baseline for inherent reasoning ability.Source: §A.1 · p. 14
• Self-Refine iteratively critiques an initial solution and revises it using the problem, draft, and generated feedback.The feedback-refinement loop can repeat multiple times.Source: §A.1 · p. 14
• The evaluated Self-Refine setup runs up to four rounds, excludes internal reasoning traces, and reports the highest per-round score.Source: §A.1 · p. 14
• Progressive-Hint Prompting gives a high-level hint first, then increasingly specific hints until the model solves the problem or exhausts them.Source: §A.1 · p. 14
• Majority Voting samples and aggregates parallel outputs, but is naturally suited to closed-ended tasks and is not strictly comparable to sequential scaling.Extending it to open-ended generation generally requires an additional aggregation step.Source: §A.1 · p. 14

A.2 IMPLEMENTATION DETAILS

• Retrieval uses a FAISS index of query–reasoning–answer triples with all-mpnet-base-v2 embeddings, adding the nearest exemplar’s reasoning trace to generator context.Source: §A.2 · p. 14
• Inference uses vLLM with default temperature 0.5, top-p 0.95, and context length up to 32k tokens, with a final-answer fallback for rare format failures.Source: §A.2 · p. 14
• AIME25 and GPQA use LightEval, while open-ended answers are extracted and checked against GPT-4.1-generated references by GPT-4.1 as judge.The judge makes a binary equivalence decision.Source: §A.2 · p. 15

B.1 DISCUSSION FOR ITERATIVE REFINEMENT

• On AIME 2025, refinement raises R1-Distill-Qwen-7B from 48.00 to 51.33, R1-Distill-Qwen-14B from 54.00 to 60.00, and Qwen3-1.7B from 41.32 to 42.65.The respective gains are +3.33, +6.00, and +1.33 points; the authors suggest larger models exploit extra reasoning passes more effectively.Source: §B.1 · p. 15
• Most AIME improvement arrives in the first refinement pass, with performance typically plateauing by iteration 2 or 3.Qwen3-1.7B peaks at iteration 1, while the two R1-Distill-Qwen models reach their best scores at iteration 3 with only tiny increments over iteration 2.Source: §B.1 · p. 15
• On GPQA, refinement gains vary by model: R1-Distill-Qwen-7B peaks at iteration 1, while 14B and Qwen3-1.7B improve through iteration 3.The 7B score slips after its +1.86 iteration-1 gain, whereas 14B improves by +3.43 and Qwen3-1.7B reaches 42.42 by iteration 3.Source: §B.1 · p. 15
• With GPT-4.1 judging open-ended tasks, refinement reliably benefits the 14B model, while smaller models are flat or regress.The 14B model improves by +2.24 on Engineering and +1.76 on General; smaller models show declines or no durable gains.Source: §B.1 · p. 15
• Open-ended refinement is difficult to assess because single-reference grading can penalize valid variants, agreement feedback is noisy, and answer extraction can remove judged context.The authors also report that smaller models may drift from task intent across passes, while larger models more reliably use weak feedback for goal-aligned edits.Source: §B.1 · p. 16
• The authors propose multiple references or tolerant scoring, explicit judge feedback, and revised extraction that preserves essential reasoning as future improvements.These proposals target semantic-equivalence scoring, weak corrective signals, and loss of reasoning context.Source: §B.1 · p. 16

B.2 CONTEXT LENGTH ANALYSIS

• Reasoning demonstrations dominate context length, with maximum traces exceeding 13k tokens on AIME25 and 14k on GPQA.The authors say excessively long contexts risk confusion and degraded reasoning quality, motivating a single-exemplar-per-query design.Source: §B.2 · p. 16, Table 5 · p. 16

B.3 MORE CASE STUDY

• Additional chemistry and physics cases reinforce that semantic misleading and failure to transfer reasoning structure occur across domains.The cases point to overfitting to surface cues and difficulty transferring deeper reasoning schemas.Source: §B.3 · p. 16

B.3.1 SEMANTIC MISLEADING FROM DEMONSTRATION CONTENT.

• In chemistry examples, superficial phrase or sequence overlap led the model to transplant an irrelevant reaction heuristic and overlook target-specific constraints.The cases include applying an alkene reaction sequence to spectroscopy-based compound identification and transferring an SN2 steric-ordering rule to substituted-benzene directing effects.Source: §B.3.1 · p. 17

B.3.2 FAILURE TO TRANSFER UNDERLYING REASONING STRUCTURE.

• In the particle-decay example, the model treated rest-frame lifetime as lab-frame lifetime, omitting time dilation and underestimating decay distance by a factor of γ.The needed chain was τrest = ℏ/Γ, τlab = γτrest with γ = E/m, then d = βcτlab.Source: §B.3.2 · p. 18
• In the transit-probability example, the model kept only a middle fragment, simplified the result to Ptr ∝ 1/P, and predicted a threefold increase rather than the correct factor (9/2)^(1/3) ≈ 1.65.Source: §B.3.2 · p. 18
• Across both cases, structurally sound demonstrations were transferred incompletely, with skipped steps yielding flawed conclusions.The figure caption identifies the illustrated category as failure to transfer underlying reasoning structure.Source: §B.3.2 · p. 18, Figure 8 · p. 18

B.3.3 FROM ERROR TO CORRECTNESS: A CASE STUDY

• Abstracting a quantum-probability problem into normalization, eigenvector identification, projection, and squaring corrected a shortcut and yielded the answer 1/3.The analysis aligned the steps with the formal definition of measurement probability.Source: §B.3.3 · p. 18, Figure 9 · p. 19
• In the spectroscopy case, checking oxygen balance, vinyl hydrogens, and consistency across IR, NMR, and MS led to C6H10O2, a cyclic ester with a double bond.The refined analysis corrected misread signal assignments and reconciled the spectroscopic evidence.Source: §B.3.3 · pp. 18–19, Figure 9 · p. 19

C.1 GENERATION PROMPT / C.2 VERIFICATION PROMPT

• The generation and verification prompts request a final answer in a fixed format and compare it with the gold answer for a Boolean correctness signal.The generation prompt asks for only the final answer; verification returns True or False.Source: §C.1 · p. 20, Generation Prompt · p. 20, §C.2 · p. 20

C.3 ANSWER CONSTRUCTION PROMPT

• The answer-construction prompt uses an example, a comparison, and potentially helpful insights to guide step-by-step reasoning on a new question.It instructs the model to use only insights relevant to the new question.Source: §C.3 · p. 20, Answer Construction Prompt · p. 21

C.4 COMPARISON PROMPT

• The comparison prompt asks the model to identify structural similarities, concepts, and likely reasoning patterns without solving the target problem.Source: §C.4 · p. 21, Comparison Prompt · p. 21

C.5 USEFUL REASONING EXTRACTION PROMPT

• The reasoning-extraction prompt distills an example’s detailed chain of thought into the minimal necessary reasoning path, omitting irrelevant detail and speculation.Source: §C.5 · p. 21, Useful Reasoning Extraction Prompt · p. 22

C.6 TEACHER PROMPT

• The teacher prompt extracts only strategies likely to transfer from an example solution, while instructing the model not to solve the new question.Source: §C.6 · p. 22, Teacher Prompt · p. 22

C.7 SUGGESTION PROMPT / C.8 REVIEW PROMPT / C.9 REFINEMENT PROMPT

• The refinement prompts identify an issue and sanity check, select the best check among candidates, and revise the reasoning path to address it.The revision prompt also asks the model to inspect and fix potential problems in its approach.Source: §C.7 · pp. 22–23, §C.8 · p. 23, §C.9 · p. 24