Source-linked AI summary
Mantis: A Versatile Vision-Language-Action Model with Disentangled Visual Foresight
Yi Yang, Xueqi Li, Yiyang Chen, Jin Song, Yihan Wang, Zipeng Xiao, Jiadi Su, You Qiaoben, Pengfei Liu, Zhijie Deng
TL;DR
VLA models face sparse action supervision, costly visual forecasting, information loss from compression, and neglected language supervision. Mantis decouples visual foresight from the backbone, and reports 96.7% success on LIBERO alongside stronger real-world instruction following and generalization than π0.5.
Problem
Sparse action signals may underuse VLA capacity, direct prediction of high-dimensional visual states is costly, compression loses fine-grained motion information, and some methods overlook language supervision.
Source: §1 · p. 2Method
Mantis uses meta queries and a Diffusion Transformer head, with current visual input supplied through a residual connection, to learn latent-action cues for explicit action prediction.
Source: §1 · p. 2Results
Mantis achieves 96.7% success on the LIBERO benchmark and outperforms π0.5 in real-world instruction following and generalization to out-of-domain instructions.
Source: §1 · p. 2Takeaways & Limitations
Mantis-ATE reduces inference calls by up to 50% while maintaining comparable task success rates.
Source: §1 · p. 2Minor motion rollbacks occur in real-world scenarios due to missing robot state inputs.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Recent advances in Vision-Language-Action (VLA) models demonstrate that visual signals can effectively complement sparse action supervisions. However, letting VLA directly predict high-dimensional visual states can distribute model capacity and incur prohibitive training cost, while compressing visual states into more compact supervisory signals inevitably incurs information bottlenecks. Moreover, existing methods often suffer from poor comprehension and reasoning capabilities due to the neglect of language supervision. This paper introduces Mantis, a novel framework featuring a Disentangled Visual Foresight (DVF) to tackle these issues. Specifically, Mantis decouples visual foresight prediction from the backbone with the combination of meta queries and a diffusion Transformer (DiT) head. With the current visual state provided to the DiT via a residual connection, a simple next-state prediction objective enables the meta queries to automatically capture the latent actions that delineate the visual trajectory, and hence boost the learning of explicit actions. The disentanglement reduces the burden of the VLA backbone, enabling it to maintain comprehension and reasoning capabilities through language supervision. Empirically, pretrained on human manipulation videos, robot demonstrations, and image-text pairs, Mantis achieves a 96.7% success rate on LIBERO benchmark after fine-tuning, surpassing powerful baselines while exhibiting high convergence speed. Real-world evaluations show that Mantis outperforms $\pi_{0.5}$, a leading open-source VLA model, particularly in instruction-following capability, generalization to unseen instructions, and reasoning ability. Code and weights are released to support the open-source community.
… ALLaVA
The examples include an arithmetic question, a question about a magazine cover, and a request for help.
… ALLaVA
- The reply says three cubes minus one cup equals two.The question asks how many cubes remain after subtracting the number of cups.Source: … ALLaVA · p. 1
- Asked who is on the magazine cover, the reply identifies Iron Man.Source: … ALLaVA · p. 1
- A speaker says they are thirsty and asks for a hand.Source: … ALLaVA · p. 1
1. Introduction
Mantis addresses gaps in visual and language supervision for VLA models with Disentangled Visual Foresight and a progressive training recipe. It achieves 96.7% success on LIBERO and reports stronger real-world instruction following and generalization than π0.5.
1. Introduction
- Sparse action signals may underuse VLA capacity, while direct visual forecasting is costly and compressed visual supervision loses fine-grained motion information.Existing methods may also overlook language supervision needed for context understanding and reasoning.Source: §1 · p. 2
- Mantis uses meta queries and a Diffusion Transformer head to decouple visual foresight from the backbone, with a residual connection that helps queries capture inter-frame dynamics as latent actions.Learnable action queries extract information from input and latent-action queries to generate explicit actions.Source: §1 · p. 2
- A progressive training recipe incorporates modalities incrementally, while the disentangled design frees backbone capacity for language supervision and helps preserve semantic understanding and reasoning.Pretraining uses human manipulation videos, robot demonstrations, and image-text pairs.Source: §1 · p. 2
- 96.7% success on the LIBERO benchmark, with faster convergence than previous visual foresight approaches.Mantis also surpasses several strong baselines.Source: §1 · p. 2
2. Related Work
Related work describes limits in VLA language understanding and in approaches that augment action learning with visual signals. It contrasts visual foresight, track guidance, and latent-action supervision with Mantis’s decoupled visual foresight.
2.1. Vision-Language-Action Models
- VLA models use pretrained vision-language representations to map instructions and visual observations to actions, but robot-specific training can degrade instruction following and reasoning.Explicit language assistance may accumulate errors, while implicit language supervision lacks visual foresight for frame-level planning.Source: §2.1 · pp. 2–3
2.2. Vision-Augmented Action Learning
- Visual foresight predicts future frames to inform action prediction, but pixel-level forecasting can add distracting information, increase training costs, and risk confusing motion with appearance changes.The overview groups explicit frame generation and implicit joint video-generation/action training as visual foresight approaches.Source: §2.2 · p. 3, Figure 1 · p. 3
- Track guidance uses compact representations such as keypoint tracks to capture physical dynamics, but compression can bottleneck information and imprecise tracking can increase action errors.Source: §2.2 · p. 3, Figure 1 · p. 3
- Latent-action supervision predicts actions learned from inter-frame differences, but requires an additional quantization model; Mantis instead decouples foresight from the backbone for compact auxiliary information.Source: §2.2 · p. 3
3. Methodology
Mantis combines a backbone, a disentangled visual foresight head, and an action head, trained through staged modality integration and paired with adaptive temporal ensembling at inference.
3.1. Model Overview
- Latent-action queries condition a DVF head to predict future frames, while an action head uses latent-action and action-query context to generate actions.The current visual state enters the DVF head through a residual connection, so the queries capture inter-frame dynamics; multi-gap queries guide predictions across different time intervals.Source: §3.1 · p. 4
3.2. Model Specification
- Qwen2.5-VL serves as the backbone, Sana as the DVF head, and a DiT-based head predicts actions.The DVF head is omitted during action inference to reduce computational overhead; the primary camera receives higher input resolution than the wrist camera.Source: §3.2 · p. 4
3.3. Progressive Training Recipe
- Training moves from video-only future-frame prediction to vision-action joint training, then language-supervised multimodal mixing.Stage 1 uses human manipulation videos while freezing the backbone; Stage 2 aligns the visual time gap with the action chunk size and jointly optimizes visual and action losses; Stage 3 unfreezes the backbone and adds language loss.Source: §3.3 · p. 5
3.4. Adaptive Temporal Ensemble
- ATE selects instruction-relevant target patches and visually changing dynamic patches, enabling temporal ensembling when they overlap and disabling it otherwise for efficiency.Figure 4 uses darker colors for higher attention values but lighter colors for higher cosine similarity; its parameters are τtarget = 1 and τdynamic = 12.Source: §3.4 · p. 5, Figure 4 · p. 5
4. Experiments
Mantis is evaluated on LIBERO and in real-world scenarios, with implementation and training details reported alongside benchmark, convergence, and ablation results. The experiments report strong simulation performance, faster convergence than entangled visual foresight, and better real-world ID and OOD instruction performance than π0.5.
4.2. Simulation Experiments
- Mantis achieves the highest average success rate and outperforms existing baselines on 3 of 4 LIBERO task suites.Evaluation covers four suites, with 10 tasks and 50 trials per task.Source: Table 1 · p. 6, §4.2 · p. 6
4.1. Implementation Details
- Pretraining uses human manipulation videos, robot episodes, and language-supervised multimodal datasets; LIBERO fine-tuning runs 30 epochs without language supervision.The reported setup uses 220K SSV2 videos, 76K DROID episodes, and 38 multimodal datasets.Source: §4.1 · p. 6
4.2. Simulation Experiments
- Mantis converges relatively quickly on LIBERO Spatial, while UnifiedVLA remains at zero success rate for its first ten fine-tuning epochs.Mantis's convergence is comparable to OpenVLA and UniVLA; the comparison fine-tunes models for 20 epochs.Source: Figure 5 · p. 6, §4.2 · p. 7
4.3. Real World Experiments
- Mantis consistently outperforms π0.5 on ID and OOD instructions across all three real-world scenarios.The scenarios include instructions involving world knowledge and arithmetic reasoning.Source: §4.3 · p. 7, Figure 6 · p. 7
4.4. Ablation Study
- Mantis-ATE reduces inference count by nearly 50% while maintaining comparable performance to standard Mantis.The comparison measures inference count and success rate across four LIBERO suites.Source: §4.4 · pp. 7–8, Figure 8 · p. 8
- The pretrained-DVF variant achieves the highest success rate, followed by vanilla-DVF, flawed-DVF, and no-DVF.The variants test DVF, its residual connection, and pretraining on human and robot videos.Source: §4.4 · p. 8, Table 2 · p. 8
- The experiments assess language-supervision ablations on multimodal benchmarks and in real-world settings, with results reported in Appendix D.Mantis has 5.8 billion parameters across its backbone, DVF head, action head, and VAE.Source: §4.1 · p. 5, §4.4 · p. 8
5. Conclusion, Limitations and Future Work
The paper concludes that DVF improves performance and convergence speed in simulation, while language supervision is effective in real-world experiments. It identifies missing robot-state inputs as a source of minor motion rollbacks and reports that ATE reduces inference count by more than 40% with negligible selection overhead.
5. Conclusion, Limitations and Future Work
- The paper identifies missing robot-state inputs as causing minor motion rollbacks in real-world scenarios and proposes richer inputs, such as 3D point clouds, as future work.It also names inference-speed optimization as future work.Source: §5 · p. 8
A. Adaptive Temporal Ensemble
- ATE reduces inference count by more than 40%, while dynamic- and target-patch selection overhead is negligible relative to backbone computation.Dynamic-patch identification, cross-modal attention aggregation, and thresholding contribute to selection overhead.Source: A. Adaptive Temporal Ensemble · p. 12
B. Implementation Details
Appendix B reports visual-input resolutions and language-supervision training details, as well as benchmark comparisons. It also identifies Figure 9 as a comparison between Mantis and Mantis-LU.
B. Implementation Details
- Video frames and primary-camera images are cropped and resized to 512 × 512 pixels, while wrist-camera images are 256 × 256 pixels.Source: B. Implementation Details · p. 12
- Mantis achieves superior performance on 2 of 3 VQA and multimodal-understanding benchmarks, with only a marginal decrease from its original backbone performance.Source: Table 3 · p. 12
- Figure 9 compares Mantis with Mantis-LU; the supplied passage does not state the comparison outcome.Source: Figure 9 · p. 12
- Stage 3 uses 38 language-supervision datasets covering visual question answering, OCR, embodied planning, and general-purpose tasks, while excluding several specialized domains.The datasets are sourced from LLaVA-OneVision-1.5-Instruct [1].Source: B. Implementation Details · p. 12
C. Real World Experiments
The real-world appendix describes scenario-specific data collection and fine-tuning for Mantis and π0.5, with different language-supervision settings.
C. Real World Experiments
- Each of the three scenarios contributes 100 teleoperated demonstrations, with 25 episodes per task.Source: C. Real World Experiments · p. 12
- Mantis and π0.5 are fine-tuned on the combined four-task dataset for each scenario, using the same learning-rate settings as pretraining stage 1.Both models are trained for 10 epochs.Source: C. Real World Experiments · p. 12
- Mantis receives LLaVA-Instruct supervision during fine-tuning, whereas π0.5 receives no language supervision.The vision loss weight is α = 0.1, and Mantis's language loss weight is β = 0.1.Source: C. Real World Experiments · p. 12
D. Language Supervision Ablations
The ablations show that language supervision preserves Mantis’s multimodal understanding and supports instruction generalization: the unsupervised variant remains reasonably strong on ID instructions but performs poorly on OOD instructions.
D. Language Supervision Ablations
- Mantis achieves the best performance on two of three VQA and multimodal-understanding benchmarks, with only a marginal drop relative to the original Qwen2.5-VL backbone.The comparison included other VLA models and the original backbone as a reference.Source: D. Language Supervision Ablations · p. 12
- Mantis-LU retains reasonably strong performance on ID instructions but performs poorly on OOD instructions, indicating language supervision is crucial for instruction generalization.Source: D. Language Supervision Ablations · p. 12
- Mantis language supervision uses 38 datasets covering general visual-language tasks while excluding specialized domains.The datasets are sourced from LLaVA-OneVision-1.5-Instruct instruction-tuning data [1].Source: Table 4 · p. 13
- Each of three scenarios uses four ID instructions and four corresponding OOD instructions to evaluate world knowledge, reasoning, and understanding of human intent.Source: Table 5 · p. 13
- Mantis outperforms π0.5 on both ID and OOD instructions in task-level success counts.Source: Table 6 · p. 13