PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning Shift: How Context Silently Shortens LLM Reasoning

Gleb Rodionov, Roman Garipov, George Yakushev

arXiv:2604.01161v2cs.LG

TL;DR

The paper examines whether reasoning behaviors remain robust when problems appear with irrelevant or otherwise non-isolated context. It evaluates multiple models across long-input, multi-turn, and complex-task settings, finding shorter reasoning traces and suppressed verification, with targeted fine-tuning partially improving robustness.

  • Problem

    The paper addresses limited evidence about how context length and content affect reasoning capabilities when models solve isolated subproblems amid irrelevant information.

  • Method

    The authors evaluate multiple reasoning models on math and scientific reasoning benchmarks across long-input, multi-turn, and complex-task context conditions, measuring accuracy and reasoning tokens.

  • Results

    Across models and benchmarks, non-baseline contexts produce up to 65% fewer reasoning tokens for the same problems, while accuracy drops disproportionately on medium and hard tasks.

  • Takeaways & Limitations

    Irrelevant context can suppress self-verification and uncertainty management, while targeted supervised fine-tuning helps models better maintain reasoning behavior than prompting alone.

  • Takeaways & Limitations

    The evaluated context conditions are relatively simple and synthetic, and the paper does not provide a mechanistic account of the effect.

Abstract

from arXiv · show

Large language models (LLMs) exhibiting test-time scaling behavior, such as extended reasoning traces and self-verification, have demonstrated remarkable performance on complex, long-term reasoning tasks. However, the robustness of these reasoning behaviors remains underexplored. To investigate this, we conduct a systematic evaluation of multiple reasoning models across three scenarios: (1) problems augmented with lengthy, irrelevant context; (2) multi-turn conversational settings with independent tasks; and (3) problems presented as a subtask within a complex task. We observe an interesting phenomenon: reasoning models tend to produce much shorter reasoning traces (up to 65%) for the same problem under different context conditions compared to the traces produced when the problem is presented in isolation. A finer-grained analysis reveals that this compression is associated with a decrease in self-verification and uncertainty management behaviors, such as double-checking. While this behavioral shift does not compromise performance on straightforward problems, it might affect performance on more challenging tasks. Additionally, we show that targeted supervised fine-tuning partially mitigates the adverse effects of irrelevant context. We hope our findings draw additional attention to both the robustness of reasoning models and the problem of context management for LLMs and LLM-based agents.

1 Introduction

Test-time scaling and long-context models expand the resources available for complex reasoning, but it remains unclear how context length and content affect reasoning capabilities. This work asks whether models solve isolated subproblems similarly when irrelevant information surrounds them.

  • Test-time scaling improves LLM performance by allocating more inference compute, enabling adaptation to input complexity without retraining.
  • Long-context models support reasoning, prover-verifier pipelines, and agent workflows within expanded context windows.
  • Input length alone can hurt LLM performance even without distracting information or retrieval-quality differences.
  • The paper asks how context length and content affect reasoning, including whether irrelevant data changes performance on isolated subproblems.
  • Across three non-isolated settings, reasoning models generate fewer reasoning tokens and show reduced self-verification and uncertainty management.
  • The study evaluates whether contextual compression preserves accuracy on easier problems, harms harder tasks, and can be mitigated through targeted supervised fine-tuning.
  • The findings motivate attention to reasoning-model robustness and context management for LLM-based agents.

2 Background

Test-time scaling produces long reasoning traces and sophisticated behaviors, while long-context research examines how models use expanded inputs effectively. Prior work also studies reasoning calibration, trace functions, and methods for managing context and decomposing tasks.

  • Chain-of-Thought reasoning: Test-time scaling and reinforcement learning elicit long Chain-of-Thought trajectories with behaviors such as self-verification and iterative refinement.
  • Chain-of-Thought reasoning: Models may overthink simple problems and underthink hard ones, indicating difficulty calibrating reasoning length to problem complexity.
  • Chain-of-Thought reasoning: Researchers analyze reasoning traces by classifying sentence functions and measuring how steps guide trajectories and affect reasoning outcomes.
  • Long context and context management: Long-context models often process shorter effective contexts than their advertised windows, with limitations involving retrieval, multi-hop tracing, and aggregation.
  • Long context and context management: Context-management approaches include compaction, iterative summarization, external memory, and recursive self-calls over isolated subproblems.

3 Experiments

The experiments compare isolated problems with subtasks, irrelevant long inputs, and multi-turn histories across reasoning models and benchmarks. Across these conditions, added context is associated with shorter reasoning traces and, in some settings, lower accuracy, especially as task difficulty increases.

  • 3.1 Setup: The evaluation compares Baseline, Subtask, Long Input, and Multi-turn conditions, measuring accuracy and generated reasoning tokens across IMOAnswerBench and GPQA-Diamond.The Multi-turn condition evaluates only the second turn; Subtask reports reasoning tokens averaged per task.
  • 3.2 Results: Up to 65% fewer reasoning tokens are generated for the same problems under non-baseline context conditions across both benchmarks.For all models under Long Input on IMOAnswerBench, the paired Wilcoxon signed-rank test reports p < 10^-10.
  • 3.2 Results: Accuracy drops under Long Input are disproportionately concentrated on medium and hard IMOAnswerBench problems, while straightforward problems are relatively benign.The difficulty analysis samples 50 runs per task for 30 selected problems grouped by baseline accuracy.
  • 3.2 Results: 18% shorter reasoning follows even short distractions for Qwen3.5-27B, while larger prompts reduce reasoning by 50% and additional chat turns further reduce it.The reduction is gradual and consistent in both Long Input and Multi-turn setups.
  • 3.2 Results: Qwen3.5-27B shows a 19% reduction in non-thinking response length and a 53% reduction in thinking-mode reasoning length on MATH-500 under Long Input.Non-thinking length changes from 1664 to 1348 tokens, while thinking-mode reasoning changes from 8003 to 3762 tokens.

4 Analysis

The analysis examines why reasoning traces shorten under non-isolated context conditions, finding that context suppresses high-level behaviors such as self-verification and uncertainty management rather than causing obvious task confusion.

  • Trace-length analysis: Non-baseline contexts produce significantly different reasoning-token counts for the same problems.The analysis compares traces across context conditions and investigates whether shortening reflects earlier answers or reduced post-answer verification.
  • Trace-length analysis: Manual inspection finds no evidence that Long Input makes Qwen3-32B misunderstand the query or task.The model briefly acknowledges the irrelevant prefix, labels it irrelevant, and moves on without semantic influence from the inserted content.
  • Behavioral analysis: Sentence-level analysis categorizes traces into setup, planning, retrieval, computation, consolidation, uncertainty management, and answer emission.Transition matrices are used to compare how these reasoning functions unfold under different contexts.
  • Resampling analysis: 46% of Long Input resampled traces finish, compared with 21% under Baseline, alongside fewer self-verification and uncertainty-management words.Words such as “wait,” “alternatively,” and “but” occur less often under Long Input; the same trend appears across other models and Multi-turn settings.
  • Behavioral analysis: Different context conditions can suppress self-verification and uncertainty management for the same reasoning prefixes, making traces significantly shorter.The finding links trace compression to a shift in high-level reasoning patterns rather than only to earlier answer generation.

5 Mitigation approaches

The paper tests prompting and targeted supervised fine-tuning as mitigations for context-induced reasoning shifts. Prompting does not remove shortening, whereas fine-tuning improves reasoning-length stability but does not fully prevent performance degradation.

  • 5.1 Prompting: Prompting for maximum reasoning effort preserves the shortening rate under non-baseline contexts.For GPT-OSS-120B, Long Input produces 13,494 versus 11,876 tokens with the max-effort prompt, still 51% below its 27,525-token max-effort baseline.
  • 5.1 Prompting: 10–15% more tokens appear in Baseline traces with the max-effort prompt, with comparable or slightly improved accuracy.Under non-baseline contexts, the prompt yields a similar relative increase in trace length without reversing context-induced shortening.
  • 5.1 Prompting: Table 4 compares the ratio of traces receiving the highest self-confidence scores across generation and evaluation contexts.Rows encode the context used to generate traces, while columns encode the context used for self-confidence evaluation.
  • 5.2 Fine-tuning: Targeted SFT trains an already SFT-trained reasoning model on samples augmented with irrelevant context.Augmentations include unchanged samples, long Shakespeare fragments, one irrelevant interaction, or multiple irrelevant turns; inserted tokens remain visible to the model but are masked from loss.
  • 5.2 Fine-tuning: The fine-tuned model preserves performance and reasoning length on GPQA-Diamond and MMLU-Pro, while MATH-500 baseline performance is also preserved.These evaluations test whether adaptation to irrelevant context degrades general reasoning capabilities.
  • 5.2 Fine-tuning: 30% fewer reasoning tokens occur for the original Olmo-3-7B-Think-SFT model from Baseline to Multi-turn(32k), while the fine-tuned model stays near baseline lengths.The proposed procedure improves reasoning-length stability across evaluated scenarios but remains insufficient to counteract performance degradation.

6 Discussion

The discussion frames Reasoning Shift as a robustness problem: non-relevant context can suppress uncertainty management and self-verification, with possible performance costs on difficult tasks. The paper does not yet explain the mechanism or develop context-management methods.

  • Limitations: The paper’s context conditions are relatively simple and synthetic, leaving realistic and agentic “in the wild” settings for future work.This limits how directly the findings can be generalized to more realistic context-management scenarios.
  • Limitations: The paper provides empirical evidence for the shift but no mechanistic account of how it arises in internal representations.It also does not cover or develop context-management methods, including recursive self-calls.
  • 6 Discussion: Non-relevant context can suppress high-level uncertainty-management and self-verification behaviors in reasoning LLMs.The paper characterizes these behavioral patterns as fragile across context conditions.
  • 6 Discussion: The behavioral shift may reduce overthinking on easier problems but can degrade performance on more challenging tasks.The discussion distinguishes potential effects by task difficulty rather than treating shorter reasoning as uniformly beneficial or harmful.
  • 6 Discussion: Training on targeted examples enables models to better maintain reasoning behavior amid distracting information, whereas prompting alone is insufficient.This conclusion connects the observed robustness problem with the mitigation results.

A.3 Compute resources

The paper reports compute used for API-based benchmark evaluation and local-GPU analysis experiments, alongside API pricing information for cost estimation.

  • Local-GPU evaluation: Analysis and Olmo evaluations required at most 1000 NVIDIA A100 GPU-hours.Because these runs included exploratory analyses, failed runs, and intermediate ablations, the estimate is reported as a conservative upper bound.
  • Cost estimation: Table 8 lists API prices in USD per million tokens for estimating evaluation costs.The prices follow OpenRouter model pages at the time of the experiments.

B Additional experiment on code generation task

On LiveCodeBench code-generation tasks, reasoning models showed the same context-sensitive shortening observed elsewhere, with substantial reductions in some settings and performance degradation in the Subtask setting.

  • The LiveCodeBench evaluation used the code_generation_lite v6 subset containing 1055 competitive-programming and algorithmic code-generation tasks.
  • Nearly 3.6× fewer reasoning tokens were generated by Qwen3.5-27B in the multi-turn setting than in the baseline setup.
  • Both Qwen3.5-27B and GPT-OSS-120B produced substantially shorter reasoning traces when irrelevant or additional context was introduced.
  • Table 9 reports accuracy and average generated reasoning tokens, with Subtask accuracies separated for the two generated subtasks.
  • GPT-OSS-120B showed noticeable performance degradation in the Subtask setting, which required solving two tasks within one response.

C Subtask evaluation details

The Subtask evaluation measured each subproblem separately, using two judge calls and accounting for whether each problem appeared first or second.

  • Two separate judge calls evaluated the correctness of each subproblem in the Subtask scenario.
  • Each problem appeared once as the first subtask and once as the second subtask.
  • Table 10 reports model accuracy separately for each subproblem in the Subproblem setup.

D.1 Verbalized confidence experiment

The verbalized confidence experiment estimated models’ confidence during unfinished reasoning and compared confidence changes across context conditions and reasoning-prefix lengths.

  • The same reasoning traces received higher self-confidence scores under non-baseline context conditions.
  • Confidence was elicited by intervening during the reasoning process rather than asking in the next conversational turn.
  • The categorical prompt mapped nine verbal confidence classes to probability ranges from 0.1–0.2 through 0.9–1.0.
  • An alternative numerical prompt requested a score from 0 to 99, with models typically selecting 98 or 99.
  • The analysis reported ratios of maximum confidence scores without performing calibration because it focused on statistical changes in confidence.
  • Evaluating the first 50% of each reasoning trace tested whether context-induced confidence inflation emerged early or only after more tokens.
  • For the transition analysis, Long Input increased the probability of transitioning from final-answer emission to the end of thinking, with p = 0.0011.

F Long Input references

The Long Input references provide trace examples and supporting analyses of how inserted context changes reasoning behavior, including earlier termination, distractor discussion, and altered transition patterns.

  • Long Input traces explicitly discussed irrelevant Shakespeare context before returning attention to the mathematical or geometry problem.
  • Table 13 reports average reasoning length and first candidate-answer position across models and settings on MATH-500.
  • The resampling experiment compared traces across Baseline, Long Input, and Multi-turn conditions to examine whether context or trace content drove changes.
  • A transition analysis compared Long Input with Baseline using 500 traces per condition and found increased final-answer-to-end-of-thinking transitions under Long Input.
  • Some traces retained answer checking, including remarks such as “let me check once more” or “let me check another angle.”
  • Figure 5 compares the number of generated tokens for each IMOAnswerBench task between Baseline and Long Input.
Loading 2604.01161v2…