PaperlayerView exact revision on arXiv

Source-linked AI summary

System-1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcuts

Xiaoqiang Wang, Suyuchen Wang, Yun Zhu, Bang Liu

arXiv:2505.18962v3cs.CL

TL;DR

CoT reasoning incurs inference costs from long chains, while existing latent-space methods may allocate similar computation to critical and auxiliary steps. System-1.5 Reasoning uses adaptive depth and step shortcuts in latent space; it matches CoT fine-tuning on mathematical tasks, outperforms it on StrategyQA, and reaches 20.27× inference speedup on GSM8K.

  • Problem

    CoT generates costly long reasoning chains, and latent-space methods may treat critical deductions and auxiliary steps uniformly, limiting adaptive computation allocation.

    Source: §1 · p. 2

  • Method

    System-1.5 Reasoning combines adaptive model-depth and decoding-step shortcuts in latent space and learns shortcut paths through two-stage distillation.

    Source: §1 · p. 3, §5 · p. 10

  • Results

    It matches CoT fine-tuning on mathematical tasks and outperforms it on StrategyQA, while achieving 20.27× inference speedup on GSM8K.

    Source: §3.1 · p. 8

  • Takeaways & Limitations

    The framework supports deliberate reasoning on difficult steps while processing simple steps quickly and skipping trivial ones.

    Source: §1 · p. 3

Abstract

from arXiv · show

Chain-of-thought (CoT) reasoning enables large language models (LLMs) to move beyond fast System-1 responses and engage in deliberative System-2 reasoning. However, this comes at the cost of significant inefficiency due to verbose intermediate output. Recent latent-space reasoning methods improve efficiency by operating on hidden states without decoding into language, yet they treat all steps uniformly, failing to distinguish critical deductions from auxiliary steps and resulting in suboptimal use of computational resources. In this paper, we propose System-1.5 Reasoning, an adaptive reasoning framework that dynamically allocates computation across reasoning steps through shortcut paths in latent space. Specifically, System-1.5 Reasoning introduces two types of dynamic shortcuts. The model depth shortcut (DS) adaptively reasons along the vertical depth by early exiting non-critical tokens through lightweight adapter branches, while allowing critical tokens to continue through deeper Transformer layers. The step shortcut (SS) reuses hidden states across the decoding steps to skip trivial steps and reason horizontally in latent space. Training System-1.5 Reasoning involves a two-stage self-distillation process: first distilling natural language CoT into latent-space continuous thought, and then distilling full-path System-2 latent reasoning into adaptive shortcut paths (System-1.5 Reasoning). Experiments on reasoning tasks demonstrate the superior performance of our method. For example, on GSM8K, System-1.5 Reasoning achieves reasoning performance comparable to traditional CoT fine-tuning methods while accelerating inference by over 20x and reducing token generation by 92.31% on average.

1 Introduction

CoT and latent-space reasoning can spend computation inefficiently by generating verbose chains or treating all steps uniformly. System-1.5 Reasoning adapts computation across latent depth and decoding steps, reporting CoT-comparable performance with faster inference and fewer generated tokens.

1 Introduction

  • CoT inference is costly because it generates long reasoning chains, while latent-space approaches can treat critical deductions and auxiliary steps uniformly.The paper notes that simple problems may still receive redundant, verbose reasoning.Source: §1 · p. 2
  • System-1.5 Reasoning adaptively combines fast and slow latent reasoning paths through model-depth and decoding-step shortcuts.The framework allocates computation along vertical model depth and horizontal decoding steps.Source: §1 · p. 3
  • Depth shortcuts route tokens through lightweight adapter branches or deeper Transformer layers, while step shortcuts pass early-exited hidden states to the next decoding step.This avoids restarting latent computation from the first layer at each step.Source: §1 · p. 3
  • The framework aims to reserve deliberate reasoning for difficult steps, process simple steps quickly, and skip trivial steps.This is presented as a flexible latent-space approach that mirrors different levels of human reasoning effort.Source: §1 · p. 3
  • Training first aligns latent-space behavior with language-space CoT, then distills full-depth reasoning into shortcut paths using reasoning criticality estimates.The second stage tunes router-adapter modules to exit non-critical tokens earlier while preserving hidden-state consistency with the full path.Source: §1 · p. 3
  • Over 20× inference acceleration, CoT-comparable reasoning performance, and 91.0% fewer generated tokens on average are reported for System-1.5 Reasoning.The evaluation includes challenging reasoning datasets such as GSM8K.Source: §1 · p. 3

2 System-1.5 Reasoning

System-1.5 Reasoning dynamically allocates latent-space computation across model depth and decoding steps. It is trained by aligning latent reasoning with language-space CoT, then learning criticality-guided shortcut paths.

2 System-1.5 Reasoning

  • System-1.5 Reasoning combines model-depth and decoding-step shortcuts to allocate latent computation across steps of varying complexity.Depth shortcuts route computation vertically through model layers; step shortcuts reuse hidden states horizontally across decoding steps.Source: §2 · p. 3

2.1 Dynamic Shortcut Architecture

  • The step shortcut reuses hidden states at the same layer to skip decoding steps, passing an intermediate exit directly to the next step.When computation reaches the final layer, the model can feed the hidden state into another reasoning cycle or generate the answer.Source: §2 · p. 3, §2.1 · pp. 4–5
  • At each Transformer layer, a router chooses between deeper standard-layer processing and early exit through a lightweight adapter branch.At inference, the router’s confidence is compared with a depth-exit threshold.Source: §2.1 · p. 4

2.2 Two-Stage Distillation: Language-to-Latent Alignment and Shortcut Learning

  • Training first aligns a latent-space student with a CoT teacher, then freezes the Transformer parameters and trains router-adapter modules using an early-exit loss.The shortcut-learning loss encourages consistency between System-1.5 intermediate hidden states and the System-2 student’s final-layer states.Source: §2.2 · pp. 5–7
  • Atomic-thought decomposition labels independent DAG nodes as non-critical and derived nodes requiring logical integration as critical, guiding earlier or later exits, respectively.The early-exit loss encourages non-critical steps to exit earlier and critical steps to proceed deeper.Source: §2.2 · pp. 5–6

3 Experiments

System-1.5 matches CoT fine-tuning accuracy on mathematical reasoning and exceeds it on StrategyQA, while reporting faster inference and fewer generated tokens. Ablations and test-time scaling analyses examine its training choices and adjustable computation budgets.

3 Experiments

  • The evaluation uses augmented GSM8K and StrategyQA training splits, tests both in-domain, and also tests mathematical generalization on GSM-HARD.Source: §3 · p. 7

3.1 Main Results

  • 92.31% fewer intermediate tokens on GSM8K and 96.15% fewer on StrategyQA accompany latent-space reasoning; average FLOPs per decoding step fall 1.95× versus early-exit methods.Source: §3.1 · p. 7
  • 20.27× GSM8K inference speedup and over 55× StrategyQA acceleration accompany the combined depth exits and shortcut copying across decoding steps.Source: §3.1 · p. 8
  • 46.94% GSM8K and 38.32% GSM-HARD accuracy closely match CoT’s 46.67% and 38.28%, while 48.61% StrategyQA accuracy exceeds CoT’s 47.36%.Source: §3.1 · p. 8

3.2 In-depth Analysis

  • Coconut as the System-2 student produces a significant final accuracy drop; the authors suggest its hard language-to-latent curriculum may limit flexible latent-space modeling.Source: §3.2 · p. 8, Figure 3 · p. 8
  • Joint alignment-and-shortcut learning and full-parameter shortcut learning both degrade performance, with a larger drop for joint learning, which the authors attribute to parameter-group optimization conflicts.Source: §3.2 · pp. 8–9, Figure 3 · p. 8
  • Tuning λdepth and λstep provides test-time budget control; performance is approximately equally sensitive to both, while depth scaling saturates sooner and deeper reasoning requires more training computation.Source: Figure 4 · p. 8, §3.2 · p. 9

4 Related Works

Related work addresses reasoning inefficiency through language-space shortening, latent-space reasoning, conditional computation, and routing among reasoning systems. System-1.5 builds on latent reasoning and conditional computation with shortcuts across both model depth and decoding steps.

  • Language-space methods budget or shorten generated reasoning, while latent-space approaches compress CoT, add latent computation, or mix language and latent thoughts.Source: §4 · p. 9
  • Conditional computation includes system switching, sparse expert activation, and dynamic-depth techniques such as early exit and layer skipping.Source: §4 · pp. 9–10
  • System-1.5 extends latent reasoning and early exits with dynamic shortcuts across decoding steps, allocating computation both horizontally and vertically.Source: §4 · pp. 9–10

5 Conclusion

The conclusion presents System-1.5 as an adaptive latent-space reasoning framework that combines depth and step shortcuts, trained through two-stage distillation guided by reasoning criticality. On reasoning benchmarks, it preserves traditional CoT accuracy while accelerating inference by more than 20×.

  • System-1.5 uses depth and step shortcuts to allocate latent-space computation dynamically, with two-stage distillation learning shortcut paths guided by stepwise reasoning criticality.Source: §5 · p. 10
Loading 2505.18962v3…