Source-linked AI summary
Thinking in Latents: Adaptive Anchor Refinement for Implicit Reasoning in LLMs
Disha Sheshanarayana, Rajat Subhra Pal, Manjira Sinha, Tirthankar Dasgupta
TL;DR
Long token-level reasoning improves mathematical problem solving but incurs substantial output and inference costs, while many latent methods still require fixed refinement schedules. AdaAnchor uses latent anchor refinement with stability-based adaptive halting, and across three benchmarks it improves the efficiency–accuracy trade-off while reducing both refinement steps and generated tokens.
Problem
Long token-level reasoning traces increase output length and inference cost, while many latent reasoning methods require a fixed number of refinement steps that must be tuned across settings.
Method
AdaAnchor silently refines learnable latent anchor vectors attached to the input and halts when anchor dynamics stabilize, selecting instance-wise refinement steps under a shared maximum budget.
Results
Across three mathematical word-problem benchmarks, adaptive halting improves accuracy by up to ∼5% over fixed-step refinement, reduces average refinement iterations by ∼48–60%, and cuts generated tokens by ∼92–93% versus token-level reasoning baselines.
Takeaways & Limitations
Stability-aware latent refinement offers a practical accuracy–efficiency trade-off by supporting iterative reasoning with substantially fewer generated tokens and refinement steps.
Takeaways & Limitations
The hand-designed halting criterion may be sensitive to hyperparameters and distribution shifts, while learned anchor semantics remain difficult to interpret.
Abstract
from arXiv · showhide
Token-level Chain-of-Thought (CoT) prompting has become a standard way to elicit multi-step reasoning in large language models (LLMs), especially for mathematical word problems. However, generating long intermediate traces increases output length and inference cost, and can be inefficient when the model could arrive at the correct answer without extensive verbalization. This has motivated latent-space reasoning approaches that shift computation into hidden representations and only emit a final answer. Yet, many latent reasoning methods depend on a fixed number of latent refinement steps at inference, adding another hyperparameter that must be tuned across models and datasets to balance accuracy and efficiency. We introduce AdaAnchor, a latent reasoning framework that performs silent iterative computation by refining a set of latent anchor vectors attached to the input. AdaAnchor further incorporates an adaptive halting mechanism that monitors anchor stability across iterations and terminates refinement once the anchor dynamics converge, allocating fewer steps to easier instances while reserving additional refinement steps for harder ones under a shared maximum-step budget. Our empirical evaluation across three mathematical word-problem benchmarks shows that AdaAnchor with adaptive halting yields accuracy gains of up to 5% over fixed-step latent refinement while reducing average latent refinement steps by 48-60% under the same maximum-step budget. Compared to standard reasoning baselines, AdaAnchor achieves large reductions in generated tokens (92-93%) by moving computation into silent latent refinement, offering a different accuracy-efficiency trade-off with substantially lower output-token usage.
1 INTRODUCTION
AdaAnchor addresses the cost of long token-level reasoning by shifting multi-step computation into latent anchor refinement and adapting the number of refinement steps to instance difficulty. Across three mathematical benchmarks, it improves the efficiency–accuracy trade-off while reducing answer-generation overhead.
- Motivation: Long Chain-of-Thought traces improve reasoning but increase decoding latency, token usage, and serving cost, motivating alternatives that preserve multi-step computation with less verbalization.Token-level efficiency methods remain tied to sparse autoregressive generation, while latent approaches move computation into hidden representations.
- Contribution: AdaAnchor refines latent anchor vectors silently, monitors their stability, and halts when anchor dynamics converge, allocating more computation only to harder instances under a shared maximum-step budget.This avoids requiring a fixed latent step count tuned separately across models and datasets.
2 RELATED WORK
Prior latent reasoning methods shift computation into hidden representations but often use fixed refinement schedules or lack explicit compute control. AdaAnchor combines a compact anchor bottleneck with convergence-based halting to allocate latent computation per example under a shared budget.
- Latent reasoning: Latent reasoning methods replace or reduce textual rationales by performing multi-step computation in hidden representations and emitting only the final answer.Related approaches use continuous hidden states or learn latent trajectories through distillation and self-training.
- Adaptive computation: Many iterative reasoning methods fix the number of refinement steps, risking over-computation on easy instances and under-computation on hard ones.Adaptive Computation Time provides a classic formulation for dynamically learning when to stop iterating (Graves, 2016).
- AdaAnchor: AdaAnchor introduces learnable anchor vectors and convergence-based halting, enabling per-example adaptive compute under a shared maximum-step budget without a separate halting controller.This contrasts with prior methods that may rely on task-specific training signals or lack explicit budget control.
3 METHOD
AdaAnchor performs implicit multi-step reasoning by iteratively refining learnable latent anchors prepended to the input, then decodes only the final answer. Its stability-based halting rule allocates refinement steps per instance, stopping converged cases early under a shared maximum-step budget.
- AdaAnchor framework: AdaAnchor prepends learnable latent anchors to the input and repeatedly refines them through forward passes, enabling silent multi-step computation without intermediate reasoning tokens.The refined anchors serve as a persistent latent state rather than a static prefix.
- Anchor refinement: The method forms an anchor-augmented embedding sequence, extracts hidden states at anchor positions, and feeds updated anchors into the next refinement step.A projection maps anchors into the same embedding space as token embeddings before each forward pass.
- Answer-only decoding: After refinement terminates, AdaAnchor decodes only a short answer continuation conditioned on the refined anchors and original question, reducing token-level generation overhead.The output remains answer-only rather than including a generated rationale.
- Adaptive halting: AdaAnchor measures convergence with cosine-distance change between successive mean anchor representations, where smaller updates indicate that refinement is approaching a fixed point.The stability signal is computed directly from anchor states produced during refinement.
- Adaptive halting: The halting rule stops at the first step whose update magnitude remains below threshold τ for s consecutive iterations, subject to a maximum of Kmax steps.This provides an instance-wise stopping rule while preserving a shared refinement budget.
4 EXPERIMENTS
AdaAnchor is evaluated on three mathematical word-problem benchmarks against token-based and latent baselines under controlled answer-only settings. Adaptive halting preserves or improves accuracy while reducing unnecessary latent refinement steps and output-token usage.
- 4.1 EXPERIMENTAL SETUP: The evaluation uses GSM8K, SVAMP, and MultiArith with answer-only final numeric outputs, measuring accuracy, average output tokens, and average latent steps.Experiments use Qwen2.5-1.5B and Llama-3.2-1B with deterministic decoding and fixed inference settings.
- 4.2 RESULTS: AdaAnchor improves relative accuracy over No-CoT by roughly 23–32% on Qwen2.5-1.5B and 39–64% on Llama-3.2-1B across GSM8K, SVAMP, and MultiArith.Compared with explicit CoT, it reduces generated tokens by about 90–93%.
- 4.2 RESULTS: Adaptive halting uses 48–61% fewer latent refinement steps than fixed K = 8 while maintaining similar accuracy and improving it in several cases.It stops when anchor dynamics stabilize, allocating more steps to harder instances under the same maximum budget.
- 4.3 ABLATION STUDY: Fixed-step accuracy generally improves as K increases from 1 to 8, but gains saturate beyond moderate budgets, motivating convergence-aware termination.Figure 3 reports accuracy as a function of K ∈{1, 2, 4, 8} on Qwen2.5-1.5B.
- 4.3 ABLATION STUDY: Adaptive halting frequently terminates before Kmax = 8 while reserving longer refinement for a smaller fraction of harder examples.The halting distribution supports instance-wise compute allocation without tuning a fixed latent-step hyperparameter per dataset.
5 LIMITATIONS AND FUTURE WORK
AdaAnchor’s stability-based halting and latent anchors have practical limitations involving heuristic termination and interpretability. These constraints motivate learned stopping policies and tools for analyzing anchor dynamics.
- LIMITATIONS: The hand-designed stability criterion may depend on hyperparameters and can halt too early or too late on atypical inputs or under distribution shifts.A learned halting policy or calibrated confidence signals are proposed as future remedies.
- FUTURE WORK: Learned anchor semantics are not directly interpretable, making improvements difficult to attribute to specific latent behaviors compared with explicit rationales.Suggested directions include probing, visualization, structured anchors, and auxiliary objectives aligned with human-interpretable sub-computations.
6 CONCLUSION
AdaAnchor shifts multi-step reasoning into iterative refinement of latent anchor vectors while emitting only answers, using stability-based halting to allocate computation per instance. Across benchmarks, it improves the efficiency–accuracy trade-off and substantially reduces both latent iterations and generated tokens.
- AdaAnchor refines compact learnable anchor vectors in latent space and emits answer-only outputs instead of long intermediate reasoning traces.This shifts multi-step computation away from token-level generation while retaining iterative refinement.
- Stability-based adaptive halting monitors anchor dynamics and stops refinement when updates converge, enabling instance-wise computation under a shared maximum budget.
- Up to ∼5% accuracy gains and ∼48–60% fewer average latent refinement iterations are achieved over fixed-step refinement under the same maximum latent budget.
- ∼92–93% fewer generated tokens are used than with token-level reasoning baselines through answer-only decoding and silent latent computation.