Source-linked AI summary
SwimBird: Eliciting Switchable Reasoning Mode in Hybrid Autoregressive MLLMs
Jintao Tong, Shilin Yan, Hongwei Xue, Xiaojun Tang, Kunyu Shi, Guannan Zhang, Ruixuan Li, Yixiong Zou
TL;DR
Existing multimodal CoT methods rely on rigid patterns and can struggle when visual reasoning is needed or when latent thoughts interfere with textual logic. SwimBird combines token- and embedding-level autoregression with curated multimodal training data to select reasoning modes adaptively. It achieves state-of-the-art results across text-centric reasoning and challenging vision-dense tasks.
Problem
Existing multimodal CoT designs use fixed reasoning modes and visual-thought lengths, creating modality mismatch across heterogeneous queries and limiting vision-dense or text-based reasoning.
Method
SwimBird combines next-token prediction for textual thoughts with next-embedding prediction for visual thoughts and trains on SwimBird-SFT-92K covering three reasoning patterns.
Results
SwimBird achieves state-of-the-art performance across text-centric reasoning and challenging vision-dense tasks, with gains across fine-grained perception, general VQA, and multimodal reasoning benchmarks.
Takeaways & Limitations
Query-adaptive mode selection lets one model use textual, visual, or interleaved reasoning while dynamically allocating the latent visual token budget.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have made remarkable progress in multimodal perception and reasoning by bridging vision and language. However, most existing MLLMs perform reasoning primarily with textual CoT, which limits their effectiveness on vision-intensive tasks. Recent approaches inject a fixed number of continuous hidden states as "visual thoughts" into the reasoning process and improve visual performance, but often at the cost of degraded text-based logical reasoning. We argue that the core limitation lies in a rigid, pre-defined reasoning pattern that cannot adaptively choose the most suitable thinking modality for different user queries. We introduce SwimBird, a reasoning-switchable MLLM that dynamically switches among three reasoning modes conditioned on the input: (1) text-only reasoning, (2) vision-only reasoning (continuous hidden states as visual thoughts), and (3) interleaved vision-text reasoning. To enable this capability, we adopt a hybrid autoregressive formulation that unifies next-token prediction for textual thoughts with next-embedding prediction for visual thoughts, and design a systematic reasoning-mode curation strategy to construct SwimBird-SFT-92K, a diverse supervised fine-tuning dataset covering all three reasoning patterns. By enabling flexible, query-adaptive mode selection, SwimBird preserves strong textual logic while substantially improving performance on vision-dense tasks. Experiments across diverse benchmarks covering textual reasoning and challenging visual understanding demonstrate that SwimBird achieves state-of-the-art results and robust gains over prior fixed-pattern multimodal reasoning methods.
1 Introduction
Prior multimodal CoT methods use rigid reasoning patterns that can mismatch a query’s needs, while SwimBird enables input-conditioned switching among textual, visual, and interleaved reasoning modes.
- Motivation: Textual CoT improves symbolic manipulation, numerical calculation, and logical analysis but does not fully transfer to vision-dense spatial tasks.Such tasks require dense perception and accurate intermediate visual states, which language may represent unreliably.
- Motivation: Fixed text-only, vision-only, or interleaved templates can mismatch query requirements, causing redundant modality steps or weakening either visual or symbolic reasoning.The mismatch arises when visual thoughts are forced onto text-centric queries or visual problems are restricted to textual reasoning.
- Approach: SwimBird combines next-token prediction for textual thoughts with next-embedding prediction for continuous visual thoughts in one hybrid autoregressive formulation.This interface supports reasoning traces that use text-only, vision-only, or interleaved vision–text patterns.
- Approach: SwimBird-SFT-92K is constructed through systematic curation and covers text-only, vision-only, and interleaved vision–text reasoning patterns.The dataset is designed to address training-data bias associated with rigid prior reasoning patterns.
- Adaptive computation: SwimBird dynamically allocates visual-thought length, using more latent computation for vision-dense queries and avoiding redundant visual thoughts for text-centric problems.This removes the fixed-budget constraint that limits fixed-pattern baselines across diverse query types.
2 Related Works
Multimodal reasoning has progressed from direct answer generation to explicit textual chains and latent visual reasoning, while SwimBird’s formulation supports both discrete and continuous reasoning tokens.
- Explicit multimodal reasoning: Early MLLMs primarily addressed visual question answering through direct answer generation, whereas later models added explicit reasoning chains for complex multimodal problems.These textual chains improved handling of mathematical word problems, scientific diagrams, and multi-hop visual reasoning.
- Latent visual reasoning: Latent visual reasoning uses continuous embeddings trained with visual reconstruction objectives as an alternative computational substrate for visual thinking.Examples include hidden states approximating helper images or reconstructing cropped image regions.
- SwimBird: SwimBird unifies discrete textual tokens and continuous latent visual tokens through next-token and next-embedding prediction.At inference, it can generate text-only, variable-length vision-only, or interleaved vision–text traces conditioned on the input.
3 Method
SwimBird unifies textual and visual thought generation in a hybrid autoregressive model, enabling text-only, vision-only, and interleaved reasoning. Its dynamic visual-token allocation and curated three-mode SFT data support query-adaptive multimodal reasoning.
- Hybrid Autoregressive Modeling: SwimBird predicts textual thoughts with next-token modeling and visual thoughts with next-embedding modeling in one hybrid autoregressive formulation.Text spans use shifted cross-entropy, while visual spans use MSE against target embeddings derived from intermediate thinking images.
- Hybrid Autoregressive Modeling: Training instances may contain text-only, vision-only, or interleaved reasoning segments, with modality-specific losses applied only to activated modes.This unified objective lets the model learn all three reasoning patterns without unnecessary supervision.
- Hybrid Autoregressive Modeling: Special delimiters mark visual-thought spans and are generated autoregressively during inference to control switching between textual and latent visual reasoning.The delimiters specify when the model should produce continuous embeddings instead of textual tokens.
- Dynamic Latent Token Budget: SwimBird uses a resolution-aware latent-token budget that varies visual-token capacity with image resolution, preserving detail for dense images while improving efficiency on simpler cases.Independent pixel or patch budgets bound variable visual-token counts for question and intermediate thinking images.
- Dynamic Latent Token Budget: During vision-only and interleaved inference, latent span length is dynamically determined as the model generates embeddings until emitting </latent>.The resulting variable-length computation is intended to match perceived query difficulty.
- Switchable Reasoning SFT Dataset Construction: SwimBird-SFT-92K combines filtered multimodal reasoning data with 50K text-only CoT instances to cover text-only, vision-only, and interleaved patterns.Stage 2 contributes 42K samples after pass@8-based filtering and mode labeling, while Stage 3 adds 50K textual traces.
4 Experiments
SwimBird is evaluated across visual understanding, general VQA, multimodal reasoning, and analyses of its adaptive reasoning behavior and training choices. Results show strong performance while dynamically matching reasoning modes and latent-token capacity to task demands.
- Fine-grained Visual Understanding: SwimBird achieves 85.5 on V* Bench, 79.0 on HR-Bench 4K, and 74.9 on HR-Bench 8K, exceeding Qwen3-VL-8B-Instruct.It also outperforms Thyme and DeepEyesV2 on these benchmarks without relying on complex tool pipelines.
- General VQA and Multimodal Reasoning: SwimBird reaches 71.2 on MMStar, 73.1 on RealWorldQA, 49.5 on WeMath, 67.2 on DynaMath, and 65.8 on MathVerse_MINI.The reported results include gains over strong open-source and agentic multimodal models, including Qwen2.5-VL-32B-Instruct on MMStar.
- Maximum Latent Token Budget: Increasing Nmax from 16 to 32 improves HRBench4K from 76.4 to 79.0 and HRBench8K from 71.4 to 74.9.The study fixes Nmin = 2 and reports no further benefit from expanding Nmax to 64 or 128.
- MSE Loss Weight Coefficient: Setting λvis = 0.2 provides balanced performance, whereas λvis = 0.1 weakens HRBench8K supervision and λvis = 0.5 improves HRBench8K but degrades RealWorldQA.The reported values are 71.8 on HRBench8K for λvis = 0.1, 75.9 for λvis = 0.5, and 72.0 on RealWorldQA for λvis = 0.5.
- Reasoning-Mode Distribution: Mode selection follows benchmark difficulty: text-only reasoning dominates DynaMath and MathVerse_MINI, while vision-only and interleaved reasoning are frequent on V* Bench and HR-Bench.Vision-only reasoning increases from HR-Bench 4K to 8K, while WeMath uses a more balanced mixture of all three modes.
- Reasoning-Mode Cases: Qualitative cases show vision-only reasoning for spatial cube folding, text-only reasoning for arithmetic, and interleaved reasoning for localized phone-number reading.The examples use dynamically allocated latent lengths of N=18 and N=24 for the visual cases.
5 Prompt
The system prompt explicitly enables SwimBird to reason in textual, visual, or interleaved modes. It uses structured tags and lets the model choose the mode combination based on the query.
- Mode Instructions: The prompt defines text-only, vision-only, and interleaved reasoning patterns for multimodal thinking.Textual thoughts use <reason> tags, while visual thoughts use <latent> tags.
- Adaptive Selection: The model is instructed to dynamically choose the most appropriate reasoning mode or combination for each input query.This prompt operationalizes query-adaptive switching between textual and visual thinking.
6 Conclusion
SwimBird addresses rigid multimodal reasoning patterns with switchable modes, hybrid autoregressive generation, adaptive latent-token allocation, and multi-pattern supervision. Experiments report state-of-the-art performance on both text-centric reasoning and vision-dense tasks.
- Conclusion: SwimBird switches among text-only, vision-only, and interleaved vision–text reasoning while dynamically allocating the latent visual-token budget.Its hybrid autoregressive paradigm supports both discrete textual and continuous visual reasoning.
- Conclusion: SwimBird-SFT-92K uses systematic curation and mode labeling to support effective multi-mode training.The dataset covers the reasoning patterns needed for adaptive mode selection.
- Conclusion: Extensive experiments show state-of-the-art performance on text-centric reasoning and challenging vision-dense tasks.The reported conclusion covers both reasoning regimes rather than a single benchmark category.