Source-linked AI summary
Latent Visual Reasoning
Bangzheng Li, Ximeng Sun, Jiang Liu, Ze Wang, Jialian Wu, Xiaodong Yu, Hao Chen, Emad Barsoum, Muhao Chen, Zicheng Liu
TL;DR
Current multimodal reasoning remains confined to language space, limiting capture of visual details relevant to queries. LVR enables hybrid reasoning over visual tokens and text, achieving strong performance across visual question answering benchmarks, especially perception-intensive tasks.
Problem
Current multimodal language models often fail to capture visual details most relevant to text queries despite sophisticated visual encoders.
Method
LVR combines latent reasoning over visual tokens with standard text generation, trained using reconstruction loss, next-token prediction, and a GRPO extension for latent reasoning.
Results
LVR achieves strong and generally state-of-the-art performance across diverse visual question answering benchmarks, with substantial gains on perception-intensive tasks.
Takeaways & Limitations
Joint reasoning over latent visual and textual spaces is a promising direction for fine-grained visual understanding and perception.
Takeaways & Limitations
LVR is trained exclusively on single-image data and does not achieve top performance on Relative Reflect, which requires reasoning over multiple images.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have achieved notable gains in various tasks by incorporating Chain-of-Thought (CoT) reasoning in language spaces. Recent work extends this direction by leveraging external tools for visual editing, thereby enhancing the visual signal along the reasoning trajectories. Nevertheless, these approaches remain fundamentally constrained: reasoning is still confined to the language space, with visual information treated as static preconditions. We introduce Latent Visual Reasoning (LVR), a new paradigm that enables autoregressive reasoning directly in the visual embedding space. A visual encoder first projects images into visual tokens within a joint semantic space shared with the language model. The language model is then trained to generate latent states that reconstruct key visual tokens critical for answering the query, constituting the process of latent visual reasoning. By interleaving LVR with standard text generation, our model achieves substantial gains on perception-intensive visual question answering tasks. In addition, we adapt the GRPO algorithm to conduct reinforcement learning on latent reasoning, further balancing LVR and textual generation. We show that LVR substantially improves fine-grained visual understanding and perception, achieving 71.67% on MMVP compared to 66.67% with Qwen2.5-VL. Code base and model weights will be released later.
1 INTRODUCTION
Prior multimodal reasoning methods operate in text space or use external visual tools, while LVR reasons over visual and textual representations jointly by alternating latent visual reasoning with text generation.
- Text-space Chain-of-Thought reasoning extends MLLMs beyond perception by structuring responses to complex multimodal queries.
- External editing methods highlight, zoom, crop, or otherwise manipulate images, then re-encode the outputs as visual enhancements during textual reasoning.
- LVR reconstructs question-relevant visual tokens through the LLM’s last hidden states and alternates this latent process with autoregressive next-text-token prediction.
- LVR training combines reconstruction loss with next-token prediction for supervised finetuning and adapts GRPO to latent reasoning for reinforcement learning.
- LVR unifies latent visual reasoning with language-space generation to integrate visual and textual signals throughout multimodal reasoning.
- LVR achieves strong performance across visual question answering benchmarks requiring fine-grained visual understanding and perception.
2 RELATED WORKS
Related work includes text-space reasoning, external visual tools, and latent-space reasoning in language models; LVR’s pipeline combines visual encoding, latent processing, and text generation.
- Think about Images: Text-space multimodal reasoning uses supervised finetuning and reinforcement learning to improve visual perception and multimodal mathematical reasoning.
- Think with Images: External-tool approaches use zooming, cropping, OCR, chart parsing, and drawing interfaces to locate or process question-relevant image regions.
- Latent Visual Reasoning: LVR extends latent reasoning from language-model representations to joint reasoning over text and visual tokens.
3 LATENT VISUAL REASONING
LVR enables MLLMs to reason jointly over visual and text tokens by reconstructing query-relevant visual semantics and interleaving latent reasoning with text generation. It combines supervised reconstruction and language modeling with reinforcement learning, while using decoding strategies to regulate latent-reasoning length.
- Overview: LVR reconstructs visual semantics relevant to an image–question pair and combines the resulting latent visual thoughts with original inputs to guide textual responses.The model enters latent reasoning mode after generating a special start token and propagates hidden states as input embeddings until stopping.
- Architecture: The architecture uses a vision encoder, LLM backbone, and multimodal projector to align visual and textual features in a shared semantic space.Visual features are produced from the image, while the question is embedded through the LLM’s language layers.
- Motivation: LVR addresses the text-centric output limitation of standard MLLMs by directly reasoning over visual semantics rather than only discrete vocabulary tokens.In conventional MLLMs, visual information can guide reasoning, but autoregressive outputs remain constrained to text tokens.
- Supervised Fine-Tuning: Supervised fine-tuning jointly trains visual reconstruction and next-token prediction, using annotated regions of interest to supervise latent visual semantics.SFT selects visual patches corresponding to each ROI and constrains latent hidden states to approximate ground-truth visual embeddings with an MSE objective.
- Supervised Fine-Tuning: The overall SFT objective is a weighted sum of visual reconstruction and next-token prediction losses, with λ_LVR balancing the two signals.The text-generation component uses standard cross-entropy to maximize the likelihood of the ground-truth response sequence.
- Reinforcement Learning: GRPOlatent adapts reinforcement learning to latent reasoning by replaying recorded latent hidden states during teacher-forcing evaluation of text-token importance ratios.Rewards depend only on output accuracy and response format, removing intermediate-output constraints and the need for pre-annotated ROI boxes.
- Decoding Strategies: Fixed Token achieves the best decoding performance, whereas Mode Switching Loss fails to work as intended.Decoding is challenging because the model may not reliably determine when to exit latent reasoning; Latent End Token is also reported as unstable.
4 EXPERIMENT
LVR is evaluated across perception-intensive and vision-centric benchmarks against multiple MLLM baseline categories, with results showing strong performance across most tasks and further gains from latent-reasoning reinforcement learning. The experiments also identify limitations in multi-image reasoning and unrestricted variable-length decoding.
- Evaluation benchmarks: LVR is evaluated on fine-grained visual detail, spatial reasoning, perception robustness, and diverse vision-centric tasks using standardized benchmark metrics.The evaluation includes V∗Bench, MMVP, Counting, IQ-Test, JigSaw, Relative Reflectance, and Spatial Relation.
- Main results: LVR achieves state-of-the-art results across most benchmarks and outperforms existing “Think about Images” and “Think with Images” approaches under the reported decoding settings.All open-source baselines use the same backbone MLLMs as LVR, supporting a controlled comparison.
- Main results: 2.7% improvement on V∗ D.A. and 5.3% improvement on V∗ R.P. show LVR’s gains in visual detail search and relative spatial reasoning.On MMVP, LVR reconstructs target visual semantics and surpasses PixelReasoner in both compared categories despite PixelReasoner’s external cropping tools.
- Main results: LVR also leads on Counting, IQ-Test, JigSaw, and Spatial Relation, covering object enumeration, geometry puzzles, visual reconstruction, and relative-position understanding.These results span detection, visual-dependent logical reasoning, image reconstruction, and spatial relation understanding.
- Limitations: LVR does not achieve top performance on Relative Reflect because evaluation requires multi-image reasoning while training uses exclusively single-image data.The authors propose cross-image data augmentation as future work.
- RL results: GRPOlatent further improves LVR beyond supervised fine-tuning, while trigger-token rewards encourage latent reasoning and their removal destabilizes training toward purely textual responses.The reinforcement-learning stage replays latent reasoning steps during policy-gradient loss computation, while rewards remain based on generated text.
5 CONCLUSION
LVR unifies latent reasoning over visual tokens with standard text generation. This hybrid approach, trained with SFT and GRPO-based reinforcement learning, shows substantial gains on perception-intensive benchmarks.
- LVR unifies latent reasoning over visual tokens with standard text generation.
- The approach extends the Vision–Projector–LLM structure and uses Supervised Finetuning plus GRPO-based reinforcement learning.
- Experiments show substantial gains on perception-intensive benchmarks.
6 ETHICS STATEMENT
The ethics statement highlights risks related to data quality, intended use, privacy, bias, fairness, and generalizability. It calls for responsible data practices, transparency, accountability, and safeguards for vulnerable populations.
- The framework’s impact depends heavily on data quality and intended use.
- Responsible deployment requires legally and ethically sourced training data and safeguards for individual data rights.
- Biases in supervised finetuning data or reinforcement learning rewards may affect underrepresented groups and undermine fairness and generalizability.
- The statement emphasizes transparency, open-source releases for accountability, and protective measures for vulnerable populations.
A.1 USAGE OF LLMS
LLMs supported coding and writing tasks in the research project. Their uses included helper-function generation, data-loader implementation, proofreading, and LATEX-formatting suggestions.
- LLMs were used for coding assistance and writing support.
- They generated helper functions and implemented data loaders.
- They also proofread text and suggested LATEX formatting.