Source-linked AI summary
Imagine while Reasoning in Space: Multimodal Visualization-of-Thought
Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulić, Furu Wei
TL;DR
Complex spatial reasoning exposes limitations of text-only Chain-of-Thought prompting. MVoT addresses this by interleaving generated visualizations with verbal reasoning, and token discrepancy loss improves visual generation; experiments show competitive cross-task performance and stronger results in challenging scenarios.
Problem
CoT and VoT rely on purely textual reasoning representations that can fail to capture intricate visual patterns and spatial layouts in complex multimodal tasks.
Method
MVoT enables multimodal reasoning by generating image visualizations interleaved with verbal reasoning traces, using token discrepancy loss in an autoregressive MLLM.
Results
MVoT achieves competitive performance across dynamic spatial reasoning tasks and outperforms traditional CoT by over 20% in challenging scenarios.
Takeaways & Limitations
MVoT demonstrates that visual thoughts can complement verbal reasoning, with combined MVoT and CoT predictions reaching nearly 100% accuracy on Maze and MINIBEHAVIOR and 92% on FROZENLAKE.
Takeaways & Limitations
Generated visualizations may reconstruct task-irrelevant details, and explicitly generating them introduces computational overhead during inference.
Abstract
from arXiv · showhide
Chain-of-Thought (CoT) prompting has proven highly effective for enhancing complex reasoning in Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs). Yet, it struggles in complex spatial reasoning tasks. Nonetheless, human cognition extends beyond language alone, enabling the remarkable capability to think in both words and images. Inspired by this mechanism, we propose a new reasoning paradigm, Multimodal Visualization-of-Thought (MVoT). It enables visual thinking in MLLMs by generating image visualizations of their reasoning traces. To ensure high-quality visualization, we introduce token discrepancy loss into autoregressive MLLMs. This innovation significantly improves both visual coherence and fidelity. We validate this approach through several dynamic spatial reasoning tasks. Experimental results reveal that MVoT demonstrates competitive performance across tasks. Moreover, it exhibits robust and reliable improvements in the most challenging scenarios where CoT fails. Ultimately, MVoT establishes new possibilities for complex reasoning tasks where visual thinking can effectively complement verbal reasoning.
1 Introduction
MVoT addresses the weakness of text-only reasoning in complex spatial tasks by interleaving verbal and visual thoughts within multimodal reasoning traces. Implemented with token discrepancy loss, it achieves competitive performance and stronger robustness than CoT in challenging scenarios.
- Motivation: CoT improves general reasoning but deteriorates significantly on complex spatial reasoning tasks.
- Motivation: Existing multimodal approaches rely on textual proxies, staged visual extraction, or external tools, which can leave intricate visual patterns and spatial layouts insufficiently represented.
- Results: Over 20%, MVoT outperforms traditional CoT in challenging scenarios while maintaining competitive performance across three dynamic spatial reasoning tasks.
- MVoT: MVoT unifies text and vision in reasoning traces by generating image visualizations alongside verbal thoughts.
- Implementation: Token discrepancy loss bridges separately trained tokenizer spaces to improve the quality of visual reasoning traces in an autoregressive MLLM implementation.
2 Multimodal Visualization-of-Thought (MVoT)
MVoT generates interleaved verbal and visual thoughts so multimodal models can reason through image visualizations rather than text alone. Its autoregressive formulation and token discrepancy loss support multimodal generation while addressing visual-token alignment.
- Formulation: MVoT combines verbal reasoning steps with image visualizations, conditioning each subsequent step on the preceding text and visual thoughts.
- Formulation: Training uses multimodal rationales containing interleaved verbal and visual thoughts followed by a final answer.
- Scope: MVoT can extend beyond discrete-image-token autoregressive MLLMs to architectures and modalities supporting interleaved multimodal generation.
- Architecture: A unified Transformer processes concatenated text and image token sequences produced by separate tokenizers.
- Training: Token discrepancy loss penalizes visual-token predictions that are distant from their labels in visual embedding space, aiming to improve generated-image quality.
3 Spatial Reasoning Tasks
The paper evaluates dynamic spatial reasoning in grid-based environments that require models to locate objects, track environmental changes, and predict action outcomes. The three tasks increase from maze navigation to object interaction and more detailed frozen-lake environments.
- The evaluation uses MAZE, InstallingAPrinter from MINIBEHAVIOR, and FROZENLAKE as controllable grid-based spatial reasoning tasks.These tasks test dynamic object location, evolving environments, and action-conditioned outcome prediction.
- MAZE: MAZE requires following a sequence of actions through an image-described maze and predicting the final destination among locations A, B, C, and D.
- MINIBEHAVIOR: InstallingAPrinter requires locating, carrying, and activating a printer, then predicting task success, object placement, or missing objects.Its action space extends maze navigation with environment interaction while using simple symbolic representations.
- FROZENLAKE: FROZENLAKE requires predicting the consequence of actions in a grid-based lake while avoiding holes and reaching the goal.Compared with the other tasks, it contains more diverse image details and a more complex environment with more key entities.
4 Experiments
Experiments evaluate MVoT across three spatial reasoning tasks using interleaved multimodal training and compare it with Direct, CoT, and GPT-4o variants. MVoT achieves competitive or better accuracy, remains more robust as environments become complex, and provides interpretable verbal and visual reasoning traces.
- Experimental Setup: The experiments use newly collected datasets for three dynamic spatial reasoning tasks and evaluate systems with multiple prompting and training variants.The tasks include MAZE, MINIBEHAVIOR, and FROZENLAKE; dataset statistics cover varying complexity in actions and patterns.
- Performance: MVoT surpasses Direct by 7% on FROZENLAKE and exceeds 90% accuracy on both MAZE and MINIBEHAVIOR.The comparison also reports that GPT-4o performs worse with and without CoT prompting on these tasks.
- Interpretability: MVoT provides verbal and visual intermediate reasoning states, offering clearer interpretability than Direct and reducing reliance on textual coordinate descriptions.The visualizations are intended to mitigate errors from inaccurate text-based captions and make reasoning states more directly trackable.
- Performance: MVoT reaches 92.95% on MAZE, 95.14% on MINIBEHAVIOR, and 85.60% on FROZENLAKE, with higher FROZENLAKE accuracy than Direct and CoT.CoT exceeds 95% on MAZE and MINIBEHAVIOR but performs worse than Direct on FROZENLAKE.
- Robustness: MVoT maintains over 83% task performance on increasingly complex FROZENLAKE environments, while CoT declines as grid size and hole count increase.CoT performance drops from 0.9401 on a 3 × 3 grid to 0.3911 on a 6 × 6 grid.
- Ablations: Interleaved visualizations improve MAZE and MINIBEHAVIOR performance by over 10%, whereas the Interleaved paradigm drops on FROZENLAKE; MVoT yields higher and more consistent gains.The authors attribute the FROZENLAKE drop to the complexity of its visual cues and use visual-token supervision in MVoT.
- Qualitative Analysis: The qualitative analysis compares FROZENLAKE visualizations with and without token discrepancy loss and examines reconstructed images from tokenized inputs.These comparisons target visualization quality and image reconstruction behavior.
- Plug-in Use: Providing GPT-4o with MVoT visual thoughts improves accuracy by over 15% across all tasks.The visual thoughts are generated by the fine-tuned Anole 7B implementation of MVoT and supplied after GPT-4o generates verbal thought.
5 Discussions and Ablations
The paper analyzes visualization quality, token discrepancy loss, and how MVoT relates to CoT in dynamic spatial reasoning. It also identifies visualization errors and evidence that the two reasoning paradigms can complement each other.
- Visualization Quality: MVoT recursively generates multimodal thoughts, unlike training with the previous golden image, so the discussion emphasizes visualization quality.
- Visualization Metrics: Visualization Accuracy measures whether intended next-action modifications are correctly rendered in the corresponding grid.
- Visualization Metrics: Visualization metrics also measure unintended patterns, consecutive correct visualizations, and the proportion of correct visualizations across action sequences.
- Token Discrepancy Loss: Token discrepancy loss produces highly accurate, minimally redundant visualizations, averaging 95% correct consecutive visualizations during recursive reasoning.
- Complementarity with CoT: Combining CoT and MVoT reaches nearly 100% upper-bound accuracy on Maze and MINIBEHAVIOR and 92% on FROZENLAKE.
- Tokenization: Separate visual and language tokenization systems create embedding discrepancies that can degrade generated image quality.
6 Related Work
Related work covers multimodal chain-of-thought methods, spatial reasoning benchmarks, and generative world models. These areas frame MVoT within broader efforts to reason over visual environments and dynamics.
- Multimodal Chain-of-Thought Reasoning: Multimodal CoT research converts images into text, graphs, or bounding boxes, or uses external tools in ReAct-style reasoning pipelines.
- Multimodal Spatial Reasoning: Multimodal spatial reasoning concerns spatial relationships among objects, their movements, and interactions with environments.
- Video Generation World Model: World-model research predicts future states, while video-generation models simulate and comprehend real-world dynamics such as physics and motion.
7 Conclusion
The paper concludes that MVoT supports multimodal reasoning by combining verbal and visual thoughts, while token discrepancy loss improves visualization quality. MVoT is robust across complex tasks, and its complementarity with CoT supports hybrid reasoning.
- Conclusion: MVoT uses multimodal-native generative models to elicit reasoning through interleaved verbal and visual thoughts.
- Conclusion: Token discrepancy loss addresses embedding misalignment and alleviates redundant patterns and inaccurate visual thought generation.
- Conclusion: MVoT outperformed textual reasoning baselines across varied tasks while showing greater robustness to state complexity and enhanced interpretability.
- Conclusion: The complementary strengths of MVoT and CoT indicate promise for hybrid multimodal reasoning approaches.
Limitation
The paper identifies task-irrelevant visual details and inference-time computational overhead as limitations of MVoT. It proposes guidance techniques and compact image representations as future directions.
- Limitations: MVoT visualizations may reconstruct task-irrelevant background patterns while overlooking intended alterations.
- Limitations: Explicitly generating visualizations introduces computational overhead during inference.
- Future Improvements: The paper proposes diffusion-style guidance to improve visualization focus and compact image representations to reduce token and inference costs.
B.1 Dataset Collection
The datasets are constructed from grid-based spatial environments with task-specific trajectories, actions, and outcome labels. Each collection procedure retains or generates examples to support dynamic reasoning evaluation.
- MAZE uses depth-first-search generation to create size-3–6 mazes with randomized seeds and navigation paths.Repetitive paths are removed to prevent train–development knowledge leakage.
- MAZE examples pair three random destination candidates with the true destination for each maze.
- MINIBEHAVIOR examples come from INSTALLINGAPRINTER grid environments of sizes 7–10, retaining action sequences that successfully complete the task.The environments are generated with stable-baseline3-trained reinforcement-learning agents.
- MINIBEHAVIOR modifies layouts probabilistically when action sequences repeat or environments were previously seen.The supplied passage specifies a 40% chance of altering the layout, but the remainder is truncated.
- FROZENLAKE uses Gym environments and Q-table agents to collect successful action sequences while retaining selected unsuccessful sequences involving holes or incomplete navigation.Unsuccessful hole cases are saved as-is or with appended random actions at equal probabilities.
B.2 Dataset Statistics
The datasets vary in entities, actions, visualization patterns, grid sizes, and environmental complexity. Their statistics are documented through distributions of options, grid sizes, and FROZENLAKE key entities.
- Dataset statistics: Additional dataset details are provided beyond the statistics in Table 1.
- MAZE: MAZE represents the start with a red dot, labels destination candidates A–D, and visualizes valid four-direction paths incrementally with red arrows.
- MINIBEHAVIOR: MINIBEHAVIOR includes an agent, printer, and table, with actions for movement, pickup, dropping, and toggling; only the agent’s current position is visualized.
- FROZENLAKE: FROZENLAKE depicts an elf, gift, and holes in comic-style illustrations, using four-direction movement while visualizing only the current state.
- Dataset statistics: Tables 5 and 6 report train–development distributions for task options and grid sizes, while Table 7 reports how FROZENLAKE environmental complexity changes with grid size.
C Experiments
The experiments use specified training and inference configurations, including MI300X hardware, GPT-4o hosted on Azure, and augmentation during MVoT training to improve visualization robustness.
- Configurations and prompts: Tables 8–12 document MVoT and GPT-4o hyperparameters, prompting templates, and responses across task and system variants.
- Experimental setup: All models were trained on MI300X GPUs, with configuration details provided for the experimental settings.
- Experimental setup: GPT-4o inference uses the 2024-07-01 version hosted on Azure, with inference parameters specified separately.
- Training augmentation: MVoT training applies repeated tokenization and detokenization to input images, randomly selecting between 0 and 10 iterations.The augmentation targets robustness and noise introduced during image reconstruction.
D.1 Task Performance
MVoT maintains performance as spatial environments become more complex, while CoT and GPT-4o decline on larger FROZENLAKE grids. Token discrepancy loss further improves visualization accuracy and reduces redundancy.
- Task performance: CoT and GPT-4o performance declines as FROZENLAKE grid size increases, reflecting greater environmental complexity.
- Task performance: MVoT maintains consistent performance across grid sizes and tasks, unlike CoT-based approaches that struggle to generalize in larger settings.The reported explanation is MVoT’s integration of verbal and visual thoughts.
- Visualization metrics: Fine-grained visualization metrics for MAZE and MINIBEHAVIOR remain consistent across grid sizes.
- Visualization quality: Token discrepancy loss improves visualization accuracy and reduces redundancy compared with MVoT models trained without the loss.
- Visualization quality: Figures 7 and 8 qualitatively illustrate improved MAZE and MINIBEHAVIOR visualizations after introducing token discrepancy loss.
- Failure and success example: A FROZENLAKE example shows CoT producing incorrect hole coordinates while MVoT succeeds by leveraging visualized thought.