PaperlayerView exact revision on arXiv

Source-linked AI summary

Echoes as Anchors: Probabilistic Costs and Attention Refocusing in LLM Reasoning

Zhuoyuan Hao, Zhuo Li, Wu Li, Fangming Liu, Min Zhang, Jing Li

arXiv:2602.06600v1cs.CL

TL;DR

Large reasoning models often spontaneously repeat the prompt, but prior approaches do not explain whether this repetition is functional or merely wasteful. The paper formalizes EOP’s probabilistic cost, develops training-based and training-free ways to harness it, and reports consistent gains while linking echoing to attention refocusing.

  • Problem

    The paper addresses the underexplored question of whether spontaneous prompt repetition in reasoning traces is a functional strategy or a superfluous artifact.

  • Method

    The paper combines rejection-based conditioning and the Echo Likelihood Gap with attention analysis, ED-SFT, and training-free Echoic Prompting.

  • Results

    ED-SFT and EP yield consistent gains over baselines across multiple math benchmarks, while attention analysis shows EOP refocusing task-critical information in middle layers.

  • Takeaways & Limitations

    The findings reframe early repetition as a beneficial cognitive primitive that can support deliberate design of reasoning processes.

  • Takeaways & Limitations

    The echo-free target distribution is intractable to compute directly, and suffix-only likelihood gaps can be larger for wrong reasoning traces.

Abstract

from arXiv · show

Test-time compute allocation in large reasoning models (LRMs) is widely used and has applications in mathematical problem solving, code synthesis, and planning. Recent work has addressed this problem by scaling self-consistency and parallel thinking, adding generic ``thinking tokens'' and prompting models to re-read the question before answering. Unfortunately, these approaches either inject task-agnostic tokens or mandate heuristics that do not explain -- and often ignore -- the \emph{spontaneous} repetition that many LRMs exhibit at the head of their internal chains. In contrast, we analyze and harness the model's tendency to restate the question, which we term the \emph{Echo of Prompt (EOP)}, as a front-loaded, compute-shaping mechanism. We formalize its probabilistic cost by casting echo removal as rejection-based conditioning and defining the \emph{Echo Likelihood Gap} $Δ\mathcal{L}$ as a computable proxy. This provides the missing theoretical link that links early repetition to likelihood gains and downstream accuracy. However, it does not by itself specify how to exploit EOP. Consequently, we develop \emph{Echo-Distilled SFT (ED-SFT)} to instill an ``echo-then-reason'' pattern through supervised finetuning, and \emph{Echoic Prompting (EP)} to re-ground the model mid-trace without training. While promising, quantifying benefits beyond verbosity is non-trivial. Therefore, we conduct length and suffix-controlled likelihood analyses together with layer-wise attention studies, showing that EOP increases answer to answer-prefix attention in middle layers, consistent with an \emph{attention refocusing} mechanism. We evaluate on GSM8K, MathQA, Hendrycks-MATH, AIME24, and MATH-500 under identical decoding settings and budgets, and find consistent gains over baselines. Code is available at https://github.com/hhh2210/echoes-as-anchors.

1 INTRODUCTION

Large reasoning models often begin their chains by repeating the user’s query, but whether this Echo of Prompt is wasteful or functional has been underexplored. The paper formalizes and analyzes EOP, introduces ED-SFT and EP, and argues that echoing refocuses attention during reasoning.

  • Motivation: Large reasoning models commonly begin reasoning traces by repeating the user’s prompt, a behavior termed Echo of Prompt (EOP).
  • Motivation: The paper asks whether spontaneous EOP is a superfluous training artifact or a functional reasoning strategy.
  • Contributions: The paper reframes EOP as a functional strategy for cognitive self-alignment and a way models structure thought for complex reasoning.
  • Contributions: The probabilistic framework models echo removal through rejection sampling and defines the Echo Likelihood Gap (∆L) to quantify its cost.
  • Contributions: ED-SFT instills echo-then-reason behavior through fine-tuning, while EP reintroduces the prompt during inference without training.
  • Contributions: Attention analysis indicates that EOP refocuses the model on critical problem details and that this refocusing correlates with correctness.

2 RELATED WORK

Prior work improves reasoning efficiency or effectiveness through computational and attention-focused interventions. This paper instead studies spontaneous prompt echoes as an intrinsic, learned refocusing strategy.

  • Computation In Reasoning: Efficiency And Effectiveness: Test-time computation improves reasoning but can create substantial overhead through long and redundant chains of thought.
  • Computation In Reasoning: Efficiency And Effectiveness: Efficiency research reduces waste through techniques such as early exiting and step compression.
  • Computation In Reasoning: Efficiency And Effectiveness: Other methods explicitly restate or re-read questions to re-align models during inference, treating repetition as an instructed heuristic.
  • Attention-Refocusing Mechanisms: Attention-refocusing research addresses positional bias and attention drift with architectural, calibration, reweighting, or evidence-reinjection methods.
  • Attention-Refocusing Mechanisms: This paper distinguishes its approach by treating spontaneous EOP as an intrinsic learned strategy that restates salient prompt content without external guidance.

3 THE PRICE OF AN ECHO: A PROBABILISTIC COST FRAMEWORK

The paper formalizes the probabilistic cost of spontaneous prompt echoes and tests whether that cost relates to reasoning correctness. Likelihood and attention analyses associate echoes with higher correct-trace likelihood and stronger mid-layer attention to the answer prefix.

  • Probabilistic Framework: The framework treats echo presence as probabilistic conditioning and uses rejection sampling to reason about an intractable echo-free distribution.The hypothetical echo-free distribution cannot be computed directly because its partition function requires summing over all echo-free sequences.
  • Likelihood Metrics: The Echo Likelihood Gap ΔL compares average per-token log-likelihood for raw and trimmed traces, with positive values indicating preference for echo-containing traces.The Suffix-only Likelihood Gap separately measures whether the echo makes the subsequent reasoning suffix more probable.
  • Empirical Relationship: Correct GSM8K traces have higher overall gaps than wrong traces: ΔL = 2.5231 versus 2.4421, a difference of +0.0811 nats/token.Logistic regression reportedly confirms ΔL as a significant positive predictor of correctness after controlling for trace length.
  • Empirical Relationship: The suffix-only gap is larger for wrong traces, 1.2938 versus 1.1449, suggesting that local plausibility after an echo does not by itself determine correctness.The paper characterizes this pattern as consistent with confirmation bias, while identifying overall ΔL as the determinative trade-off.
  • Robustness and Cost: Echo prefixes usually consume substantial output budget, with most removed prefixes roughly 200–240 tokens long and mean 219 tokens.Length-stratified analysis finds ΔL positive across trace-length bins and reports improved scoring on shared subsequent reasoning steps.
  • Attention Mechanism: Attention analyses localize the echo effect to layers 7–18, where correct traces attend more strongly to the answer prefix while answer-to-question attention remains closely matched.The reported group difference is approximately 2.87% for answer→answer-prefix versus approximately 0.66% for answer→question, supporting an anchoring interpretation.

4 EMPIRICAL VALIDATION

The paper validates EOP as a reasoning aid through causal reinsertion, ED-SFT, and EP experiments, finding gains under controlled settings and across models and benchmarks.

  • Echo Reinsertion as a Causal Intervention: Forcing a short echo before resuming failed chains yields +10.4 and +7.9 EM points for DeepSeek-R1-Distill-Llama-8B and Qwen3-8B, respectively.The non-reasoning Qwen3-8B-Base shows no improvement, while both branches use identical questions, prefixes, decoding parameters, and random seeds.
  • Performance Gains From Echo-Distilled SFT (ED-SFT): ED-SFT embeds an echo-then-reason pattern by fine-tuning on verified teacher traces with an initial echo segment.Missing echoes are inserted with minimal edits while preserving subsequent reasoning and final answers.
  • Performance Gains From Echo-Distilled SFT (ED-SFT): ED-SFT improves GSM8K by +3.4 points for Qwen3-8B-Base and +2.8 points for instruction-tuned Qwen3-8B.The gains occur on both pretrained and instruction-tuned models.
  • Performance Gains From Echo-Distilled SFT (ED-SFT): ED-SFT generalizes across model architectures, including +3.4 points on MathQA and +2.24 points on Hendrycks-MATH for DeepSeek-distill-llama-8B.These benchmarks differ distributionally from GSM8K, the fine-tuning source.
  • Mechanistic Alignment With Attention Analysis: ED-SFT strengthens answer→answer-prefix attention in layers 7–18, reaching a +3.20 pp correct-versus-wrong gap versus +1.90 pp for the base model and +2.40 pp for normal SFT.The result aligns the training effect with the paper’s proposed attention-refocusing mechanism.
  • Echoic Prompting: Echoic Prompting reintroduces the original question after an initial reasoning chain, and consistently outperforms generic thinking-token TTTS on AIME24 and MATH-500.The comparison uses deterministic decoding and identical budget settings on DeepSeek-R1-Distill-Llama-8B.

5 CONCLUSION

The paper interprets EOP as a beneficial cognitive primitive: attention analysis links it to middle-layer refocusing, while ED-SFT and EP produce consistent benchmark gains and robustness under distribution shifts.

  • 5 CONCLUSION: Attention analysis demonstrates that EOP refocuses attention on task-critical information in middle layers.The paper reports consistent gains over baselines across multiple math benchmarks and improved robustness under distribution shifts.

ETHICS STATEMENT

The authors state that the work follows the ICLR Code of Ethics and uses standard public benchmarks, while acknowledging inherited biases from opaque training data.

  • The authors adhere to the ICLR Code of Ethics.
  • The study uses standard public benchmarks, including GSM8K and MathQA, for fine-tuning and evaluation.
  • The authors acknowledge that the models may inherit biases from their original, opaque training data.

REPRODUCIBILITY STATEMENT

The authors commit to releasing source code and describe reliance on publicly available models, standard benchmarks, and documented data-generation and probe-training procedures.

  • The authors commit to releasing source code for data processing, training, and evaluation upon publication.
  • The study relies on publicly available Qwen3 and DeepSeek models and standard benchmarks such as GSM8K, MathQA, and Hendrycks-MATH.
  • The Echo-Distilled SFT data-generation methodology and MLP-probe design and training are documented in the cited appendix sections.

A.1 THE USE OF LARGE LANGUAGE MODELS

The authors disclose using LLMs for manuscript editing, CoT data annotation, and synthetic SFT-data generation.

  • Writing Assistance: LLMs were used to improve manuscript clarity, grammar, and readability without altering the core scientific contributions.
  • Data Annotation: GPT-4.1 annotated the CoT dataset to train the MLP probe to identify Echo of Prompt instances.
  • Synthetic Data Generation: The gpt-oss-120B model generated the CoT dataset used for Supervised Fine-Tuning.

A.2 ADDITIONAL CROSS-MODEL ATTENTION STATISTICS

The additional statistics report mid-layer answer-to-answer-prefix attention gaps as within-model discriminability measures, not cross-model rankings.

  • Table 6 reports answer→answer-prefix attention gaps for layers 7–18 across model variants.
  • The gap is the difference in average attention percentage points between correct and incorrect reasoning traces.
  • A larger positive gap indicates stronger within-model discriminability.
  • The values should not be used to rank models against one another.

A.3 ON THE ORIGINS OF THE ECHO OF PROMPT

The paper presents EOP as an emergent behavior whose precise formation remains unclear, but hypothesizes that restating the prompt reinforces the task representation and focuses later reasoning on relevant information.

  • EOP is hypothesized to emerge from the model’s implicit need to ground reasoning in the problem statement.
  • Restating the prompt may reinforce the model’s internal task representation and improve focus on relevant information during subsequent reasoning.

A.4 MLP PROBE FOR ECHO DETECTION

The paper uses a lightweight MLP probe and controlled analyses to detect echo prefixes and test whether attention refocusing reflects echo content rather than merely echo length. Across likelihood, attention, and outcome analyses, echo-related signals are associated with correct reasoning and stronger answer-prefix attention.

  • A.4 MLP PROBE FOR ECHO DETECTION: A two-layer MLP probe detects echo prefixes from concatenated embeddings of the question and its first 32 think-content tokens.The probe uses weighted binary cross-entropy and a calibrated hysteresis threshold during inference.
  • A.4 MLP PROBE FOR ECHO DETECTION: Fixed-prefix analyses at K ∈ {32, 64, 128} show that Correct traces consistently direct more attention to the answer prefix than Wrong traces.The positive attention gap persists across all tested prefix lengths, separating content effects from echo-length effects.
  • A.4 MLP PROBE FOR ECHO DETECTION: Normalized Cohen’s d exceeds 0.75 in middle layers 7–18 and peaks at 0.86 for the answer→answer-prefix attention difference.The raw attention difference and normalized effect-size curves track closely across layers.
  • A.4 MLP PROBE FOR ECHO DETECTION: Correct traces route answer-token attribution repeatedly through echo-prefix tokens, which focus attention on key numerical entities and constraints in the echoed prompt.The routing analysis and token-level heatmap provide complementary evidence for the proposed attention-refocusing mechanism.
  • A.4 MLP PROBE FOR ECHO DETECTION: A likelihood-gap regression finds ΔL positively predicts correctness, with β1 ≈ 0.24 and odds increasing by exp(0.24) ≈ 1.27 per 1.0 nat/token increase.The predictor is statistically significant at p ≈ 0.022.
  • A.4 MLP PROBE FOR ECHO DETECTION: EOP-present traces achieve 63.8% accuracy versus 57.2% for EOP-absent traces and show stronger answer-prefix attention even within Correct and Wrong outcomes.The comparison associates spontaneous echoes with both higher accuracy and stronger attention refocusing.
Loading 2602.06600v1…