PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Reasoning with Hidden Thinking

Xuan Shen, Yizhou Wang, Yufa Zhou, Xiangxi Shi, Pu Zhao, Yanzhi Wang, Jiuxiang Gu

arXiv:2501.19201v2cs.CLcs.AIcs.LG

TL;DR

MLLMs benefit from CoT reasoning, but verbose textual reasoning is inefficient. Heima compresses CoTs into thinking tokens, analyzes and reconstructs the retained reasoning, and achieves comparable or superior zero-shot accuracy with significantly fewer tokens. The method is limited when retention reaches 70%, where generated tokens exceed the baseline.

  • Problem

    Verbose textual CoT improves complex MLLM reasoning but incurs significant computational costs, motivating token reduction during reasoning.

  • Method

    Heima distills CoT stages into compact thinking tokens, analyzes their information gap, and trains pure-LLM interpreters to reconstruct textual reasoning from them.

  • Results

    Heima achieves comparable or superior zero-shot accuracy with significantly fewer tokens across reasoning benchmarks, while interpreters reconstruct coherent reasoning progress.

  • Takeaways & Limitations

    The results support latent reasoning as an efficient MLLM reasoning framework whose compressed representations preserve effective reasoning capability.

  • Takeaways & Limitations

    At a 70% retention ratio, adaptive distillation generates more tokens than the baseline and is no longer effective for compressing reasoning progress.

Abstract

from arXiv · show

Chain-of-Thought (CoT) reasoning has become a powerful framework for improving complex problem-solving capabilities in Multimodal Large Language Models (MLLMs). However, the verbose nature of textual reasoning introduces significant inefficiencies. In this work, we propose Heima (as hidden llama), an effective CoT compression framework that condenses lengthy CoTs into a small set of abstract thinking tokens, preserving essential reasoning while removing redundancy. We then conduct a theoretical analysis from an information-theoretic perspective, quantifying the information gap induced by compression, showing that reasoning capability is preserved when non-trivial mutual information is retained. To further explore and quantify this information gap, we design the adaptive interpreter that maps thinking tokens back to variable-length textual sequences, thereby reconstructing the reasoning process. Experiments across diverse reasoning benchmarks demonstrate that Heima improves reasoning efficiency, while maintaining or even achieving better zero-shot accuracy. Moreover, the interpreter reconstructs coherent reasoning progresses from compressed thinking tokens, revealing that the information gap is minimal and validating the effectiveness of the proposed framework. This work paves the way for scalable latent reasoning models and advances our understanding of efficient reasoning processes in large models. Code: https://github.com/shawnricecake/Heima

1. Introduction

Heima compresses multimodal CoT reasoning into compact thinking tokens, analyzes the resulting information gap, and uses interpreters to reconstruct textual reasoning. Experiments report substantially fewer tokens with comparable or superior zero-shot performance.

  • Motivation: The framework addresses a gap in extending CoT compression from small text-only models to large MLLMs handling general multimodal reasoning.Prior compression work considered GPT-2 and individual text-only reasoning tasks, while related MLLM work focused on other latent representations.
  • Framework: Heima compresses each CoT stage into a compact thinking token, enabling MLLMs to reason in latent space instead of generating verbose textual CoTs.The framework uses step-by-step distillation to map each CoT stage to a single special token.
  • Analysis: Information-theoretic analysis shows that reasoning effectiveness is preserved when compression retains non-trivial mutual information.The analysis quantifies the information gap between textual CoTs and thinking tokens.
  • Interpretation: Adaptive interpreters decode thinking tokens into variable-length textual sequences and reconstruct reasoning progress using pure LLMs.The interpreters receive compressed representations and questions, then reconstruct textual reasoning without directly receiving visual inputs.
  • Results: Experiments show substantially improved reasoning efficiency, with far fewer generated tokens and comparable or superior performance across zero-shot reasoning benchmarks.The framework also reconstructs reasoning progress that captures key visual information despite the interpreters lacking visual input.

2. Related Work

Prior reasoning methods improve capabilities by generating additional textual tokens or adding inference-time overhead. Latent reasoning research reduces textual reasoning in limited settings, motivating broader latent representations for multimodal models.

  • Chain-of-Thought Reasoning: Many CoT methods improve reasoning by generating additional textual tokens, increasing computational costs for large generative models.This motivates methods that reduce tokens during the reasoning process.
  • Reasoning Acceleration: Parallel-generation approaches accelerate reasoning through templates or Jacobi decoding but introduce additional overhead during inference.
  • Latent Reasoning: Prior latent-reasoning work compresses CoTs for GPT-2 on math datasets, but its effectiveness is unverified beyond small models and limited task coverage.The cited evaluation does not establish performance for large multimodal models or general reasoning tasks.
  • Motivation: These limitations motivate developing latent representations for MLLMs rather than relying only on textual reasoning or narrow task settings.

3. Methodology

Heima compresses textual CoT reasoning into compact thinking tokens through progressive distillation, then uses interpreters and information-theoretic analysis to study preserved reasoning information. At inference, the model generates answers from far fewer intermediate tokens, while the framework evaluates whether compressed representations retain task-relevant information.

  • Heima Framework: The updated training data replaces textual CoT stages with stage-specific special tokens that are shared across samples at the same reasoning stage.The model is fine-tuned to predict these thinking tokens instead of verbose CoTs.
  • Heima Framework: Heima distills each textual CoT stage into a single thinking token through progressive stage-by-stage training.The first s stages use thinking tokens while later stages retain textual reasoning; a final recovery stage optimizes interactions among thinking tokens.
  • Efficient Reasoning: At inference, Heima produces the final answer using only K_i intermediate thinking tokens instead of the original sum of textual CoT lengths.This reduces generation length and supports lower memory usage and faster generation.
  • Information-Theoretic Compression: Compression cannot increase information about Y, but reasoning remains effective when the compressed tokens retain nontrivial task-relevant mutual information.The information gap I(Y; CoTs | X, <CoTs>) measures information lost during compression.
  • Interpreter Design: Separate pretrained-LLM interpreters reconstruct variable-length textual reasoning from the hidden states associated with each thinking token.The interpreters use explanatory prompts and next-token prediction; they are trained on text-only questions and cannot read images.
  • Interpreter Design: Comparing reconstructed reasoning with original CoTs provides an empirical test of whether compression preserves essential reasoning capability.Close semantic alignment is treated as evidence that the compression-induced information gap is minimal.

4. Experimental Results

Heima maintains comparable or superior zero-shot performance while substantially reducing generated reasoning tokens across multimodal benchmarks and model families. Interpreter reconstructions support a small information gap, while ablations identify effective and ineffective compression settings.

  • Main Results: Heima retains most LLaVA-CoT performance while using as little as 6% of tokens on certain datasets, and exceeds the baseline on MMBench.It also outperforms Llama3.2-11B-Vision-Instruct in average accuracy while using fewer tokens.
  • Main Results: Heima outperforms Llama3.2-11B on instance and logical reasoning while using less than 10% of tokens, while preserving most mathematical reasoning capability.These results are attributed to progressive distillation in the detailed MMStar evaluation.
  • Main Results: Heima achieves comparable accuracy to the LoRA-fine-tuned CoT model with as little as 6% of generated tokens on the LLaVA model family.Consistent results across model architectures support the reported generalization of the framework.
  • Interpretability Analysis: Interpreter reconstruction is most successful for summary stages, followed by caption and reasoning stages, across lexical and GPT-4o similarity evaluations.Results are consistent across the Llama3 and LLaVA model families.
  • Ablation Study: A single thinking token per CoT stage achieves the best performance in the token-count ablation.Adaptive retention ratios from 10% to 90% show irregular accuracy fluctuations rather than a discernible trend.
  • Ablation Study: At a 70% retention ratio, generated tokens exceed the baseline, indicating that adaptive distillation is ineffective for compressing reasoning progress at that setting.The baseline LLaVA-CoT generates 181 tokens on average.
  • Ablation Study: Distinct interpreters are crucial for summary and caption reconstruction, but interpreters are used only for latent-space analysis and not efficient reasoning.The ablation examines whether each CoT stage requires a separate interpreter.

5. Conclusion

Heima compresses multimodal chain-of-thought reasoning into compact thinking tokens and uses interpreters to reconstruct reasoning progress. Experiments report comparable or superior zero-shot accuracy with fewer tokens, while reconstructions indicate preserved reasoning capability and a minimal information gap.

  • 5. Conclusion: Heima accelerates MLLM reasoning by distilling each CoT into a compact thinking token and analyzing the resulting information gap.Interpreters reconstruct reasoning progress from the compressed representations.
  • 5. Conclusion: Heima achieves comparable or superior zero-shot accuracy with significantly fewer tokens across the reported experiments.The conclusion characterizes the framework as efficient and robust.
  • 5. Conclusion: Successful reconstructions indicate that the information gap is minimal and reasoning capability is preserved.The paper states that future work will extend Heima to larger models.

A. Training Hyperparameters

The appendix provides hyperparameters for progressive distillation, additional recovering, and adaptive interpretation training.

  • A. Training Hyperparameters: Training hyperparameters are provided for progressive distillation, additional recovering, and adaptive interpretation training.The progressive-distillation hyperparameters are listed in Table A1, while additional recovering uses Table A2.

B.1. Detailed Results on MMStar

On MMStar, Heima outperforms Llama3.2-11B on instance and logical reasoning while using less than 10% of the tokens. Progressive distillation preserves most mathematical reasoning capability.

  • Heima outperforms Llama3.2-11B on both instance reasoning and logical reasoning tasks.
  • Heima uses less than 10% of the tokens used by the comparison system.
  • Progressive distillation preserves the majority of Heima’s reasoning capabilities for mathematical problems.

B.2. Detailed Evaluation of Interpreters

The section reports detailed evaluation metrics for the interpreters in Table A5.

  • Table A5 contains detailed evaluation results for the interpreters.
  • The evaluation covers the metrics used to assess the interpreters.
  • The reported results are presented as an appendix table for interpreter evaluation.

B.3. Results with LLaVA Model Family

This section combines interpreter results, training settings, MMStar task definitions, and ablations for thinking-token counts and retention ratios.

  • Three interpreters trained with Vicuna-7B are evaluated.
  • Table A3 reports hyperparameters for adaptive interpretation training.
  • Table A4 defines MMStar categories including coarse perception, fine-grained perception, instance reasoning, and logical reasoning.
  • Ablations examine different numbers of thinking tokens and retention ratios.

B.5. Ablation Study for Number of Decoders

The decoder ablation evaluates interpreter configurations across three stages and reports detailed results for interpreter counts, thinking-token counts, and retention ratios.

  • Decoder ablation: A single interpreter performs well on reasoning but poorly on summary and caption stages.
  • Decoder ablation: The results highlight the necessity of separate interpreters for summary and caption stages.
  • Interpreter configurations: Tables A6 and A9 report evaluation results for interpreters trained with Vicuna-7B and LLaMA-3.1-8B-based configurations.
  • Ablation settings: Additional ablations vary the number of thinking tokens and retention ratios.
  • Evaluation layout: The decoder comparison includes summary, caption, and reasoning columns.

C. Prompts for GPT-4o Evaluation

The evaluation uses GPT-4o to rank how closely reconstructed reasoning aligns with ground-truth Chain-of-Thought. It compares cleaned reconstructed and reference reasoning across caption, summary, and reasoning stages using a five-level similarity scale.

  • GPT-4o evaluates reconstructed reasoning by comparing it with the ground-truth CoT after special-token removal.The evaluation is framed as a ranking process for reconstructed reasoning similarity.
  • Rank 1 indicates little overlap and different themes, whereas Rank 5 indicates strong alignment with the ground-truth CoT.
  • The GPT-4o instruction prompts for this reconstruction evaluation are provided in Prompt A1 and Figure A1.
  • The evaluator takes an image, question, reconstructed CoT, ground-truth CoT, and stage type as input, then outputs a similarity rank from 1 to 5.Stage types are caption, summary, and reasoning.
Loading 2501.19201v2…