Source-linked AI summary
BagelVLA: Enhancing Long-Horizon Manipulation via Interleaved Vision-Language-Action Generation
Yucheng Hu, Jianke Zhang, Yuanfei Luo, Yanjiang Guo, Xiaoyu Chen, Xinshu Sun, Kun Feng, Qingzhou Lu, Sheng Chen, Yangang Zhang, Wei Li, Jianyu Chen
TL;DR
Existing VLA models often emphasize linguistic planning or visual forecasting separately, leaving a gap in guiding actions for complex, long-horizon manipulation. BagelVLA interleaves language planning, visual prediction, and action generation, using Residual Flow Guidance for low-latency predictive features. The paper reports substantial improvements over baselines across simulation and real-world settings, especially for long-horizon tasks.
Problem
Recent VLA methods typically treat linguistic planning and visual forecasting separately, limiting their joint use for complex, long-horizon manipulation.
Method
BagelVLA interleaves textual planning, visual forecasting, and action generation in a unified transformer, using two-stage training and Residual Flow Guidance.
Results
BagelVLA substantially outperforms existing baselines in simulation and shows strong real-world generalization to unseen instructions and diverse environments.
Takeaways & Limitations
Explicitly coupling linguistic planning with predictive visual representations can improve robustness and instruction-following in long-horizon manipulation.
Takeaways & Limitations
Complete Denoise has high inference latency from total denoising steps N1+N2 and may accumulate visual errors.
Abstract
from arXiv · showhide
Equipping embodied agents with the ability to reason about tasks, foresee physical outcomes, and generate precise actions is essential for general-purpose manipulation. While recent Vision-Language-Action (VLA) models have leveraged pre-trained foundation models, they typically focus on either linguistic planning or visual forecasting in isolation. These methods rarely integrate both capabilities simultaneously to guide action generation, leading to suboptimal performance in complex, long-horizon manipulation tasks. To bridge this gap, we propose BagelVLA, a unified model that integrates linguistic planning, visual forecasting, and action generation within a single framework. Initialized from a pretrained unified understanding and generative model, BagelVLA is trained to interleave textual reasoning and visual prediction directly into the action execution loop. To efficiently couple these modalities, we introduce Residual Flow Guidance (RFG), which initializes from current observation and leverages single-step denoising to extract predictive visual features, guiding action generation with minimal latency. Extensive experiments demonstrate that BagelVLA outperforms existing baselines by a significant margin on multiple simulated and real-world benchmarks, particularly in tasks requiring multi-stage reasoning.
1 Introduction
BagelVLA unifies linguistic planning, visual forecasting, and action generation to address the limitations of separating these capabilities in long-horizon manipulation. Its interleaved architecture and Residual Flow Guidance support predictive, instruction-aligned control, with reported gains over baselines in simulation and real-world generalization.
- Motivation: Recent VLA models often separate linguistic planning and visual forecasting, limiting unified reasoning for complex manipulation tasks.Some methods emphasize high-level planning without visual forecasting, while others predict visuals but struggle with logical reasoning.
- Framework: BagelVLA interleaves textual planning, future visual-state prediction, and action generation within one unified transformer architecture.The model decomposes instructions, predicts the future visual state, and then generates actions aligned with the instruction.
- Training: A two-stage training strategy combines general multimodal data with annotated robotic datasets before coupling language, visual dynamics, and control through an action expert.The first stage transfers general reasoning and visual prediction to embodied settings; the second fine-tunes the full model for control.
- Efficient visual guidance: Residual Flow Guidance uses the current observation as a structural prior and single-step denoising to capture future visual dynamics with minimal latency.The resulting predictive visual features guide action generation without the computational cost of full image synthesis.
- Results: BagelVLA substantially outperforms existing baselines in simulation and generalizes to unseen instructions and diverse environments in real-world experiments.The reported improvements are particularly strong for long-horizon tasks, where baseline methods often fail under varied object arrangements.
2 Related Works
Related work spans VLA policies that map vision-language inputs to actions and unified multimodal models that jointly understand and generate across modalities. These approaches differ in action representation, modality integration, and the balance between expressiveness and efficient alignment.
- Vision-Language-Action models: RT-2 and OpenVLA use discrete action tokens compatible with VLMs for direct vision-language-to-action mapping.The passage notes that discrete tokens can limit expressiveness in continuous control.
- Vision-Language-Action models: Octo, 3D Diffuser Actor, and π0 use continuous action representations via diffusion models to capture multimodal distributions.
- Unified multimodal learning: Bagel, Chameleon, LMFusion, and Show-o develop unified multimodal architectures for joint understanding and generation across language and vision.Their designs include interleaved datasets, token-based mixed-modal input/output, fused crossmodal alignment, and unified multimodal processing.
3 Methodology
BagelVLA formulates long-horizon manipulation as interleaved planning over language, visual forecasts, and actions. Its unified transformer architecture and denoising schemes connect these modalities while reducing the latency of visual foresight.
- Interleaved Planning: Global instructions are decomposed into subtasks, future keyframes are predicted, and action chunks are generated from both the textual plan and visual forecast.This factorization explicitly models the task’s logical dependency rather than directly mapping a global instruction to actions.
- Unified Architecture: BagelVLA uses a Mixture-of-Transformers architecture with language, generation, and action experts connected through self-attention.The LLM and generation experts are initialized from Bagel, while a smaller transformer is added as the action expert.
- Multimodal Experts: The understanding expert autoregressively generates the subtask, while the generation expert denoises a keyframe conditioned on observations and textual information.The action expert processes proprioceptive and action modalities and learns action chunks with Flow Matching.
- Conditioning Schemes: Complete Denoise separately generates a fully denoised keyframe before action generation, requiring the action expert to observe the completed visual prediction.Training appends the ground-truth keyframe after the denoising sequence to provide this condition.
- Conditioning Schemes: Joint Denoise synchronizes keyframe and action denoising, allowing inference to generate both within N steps instead of separate denoising sequences.The action component attends to intermediate noisy keyframes during training.
- Conditioning Schemes: Residual Flow Guidance uses the current observation as a prior and conditions actions on single-step visual denoising, reducing foresight latency.The authors select Single-step Denoise with RFG as the default after ablation and hypothesize that the prior focuses world-modeling on manipulation changes.
4 Experiment
Experiments across simulation and real-world manipulation tasks show that BagelVLA’s interleaved linguistic planning and visual forecasting improve long-horizon performance, generalization, and action planning. Ablations further identify single-step denoising and RFG as effective mechanisms for balancing performance with inference efficiency.
- Evaluation Setup: BagelVLA was evaluated in Calvin, Robotwin, real-world basic tasks, and real-world long-horizon tasks spanning multiple manipulation skills and planning settings.The simulation evaluation includes Calvin and Robotwin, while the real-world evaluation uses the Aloha-AgileX bimanual platform.
- Simulation Results: BagelVLA outperforms all baselines on both the Calvin ABC-D split and Robotwin tasks, achieving an average completion length of 4.41 on Calvin ABC-D.The Robotwin evaluation includes Clean and Randomized settings with unseen instructions.
- Real-World Long-Horizon Results: BagelVLA achieves nearly 90% planning accuracy on real-world long-horizon tasks, measuring correct motion trends across intermediate subtasks.The evaluated tasks include stacking cubes in an instructed order and calculating and placing symbolic blocks.
- Ablation Results: Single-step denoising outperforms complete and joint denoising in task success and inference speed, while RFG improves performance over naive single-step denoising with low latency.RFG incorporates the initial frame into noise initialization, providing stronger prior information for action generation; it also produces high-quality future frames with few denoising steps.
- Planning Ablations: Textual planning yields a 21% success-rate improvement in Robotwin and substantial gains on real-world long-horizon tasks.The reported comparison attributes these gains to incorporating language planning into the interleaved framework.
- Planning and Generalization: Visual planning improves action-planning accuracy in both simulation and real-world environments, while pre-training improves semantic generalization on unseen-object tasks.The pre-trained baseline achieves a significantly higher success rate on pick-and-place OOD tasks than the without-pretraining variant.
5 Conclusion
BagelVLA is a unified framework for long-horizon manipulation that interleaves linguistic planning, visual forecasting, and action generation. Its results suggest this coupling improves robustness and instruction-following.
- BagelVLA interleaves linguistic planning, visual forecasting, and action generation within a single transformer system.
- The framework builds on Bagel’s multimodal backbone, adds an action expert, and uses two-stage training to transfer reasoning and visual dynamics into embodied control.
- Residual Flow Guidance captures task-relevant future dynamics with substantially reduced computational costs, addressing visual-foresight latency.
- The reported results suggest that coupling linguistic planning with predictive visual representations improves robustness and instruction-following in long-horizon manipulation.
A Details of Model Architecture
This section provides the architecture specification for BagelVLA. The model architecture is detailed in a dedicated table.
- The architecture of each BagelVLA expert is detailed in the model-architecture table.
B Dual Denoise Flow-Matching Implementation Details
The implementation uses interleaved multi-task training to learn planning, visual dynamics, and action generation while preventing cross-modal information leakage. It combines general multimodal and robotic data with scenario-specific downstream robot data.
- Dual flow-matching implementation: A unified multi-task attention mask enables simultaneous computation of multiple task losses from one input sequence.
- Dual flow-matching implementation: The training design prevents information leakage between modalities and aligns training with conditions encountered during inference.
- Training stages: Stage one finetunes the Understanding and Generation Experts for sub-task planning and keyframe prediction while co-training on general QA data.
- Training stages: The pretraining data include 2.56M QA pairs, 310k human-hand episodes, 382k open-source robot episodes, and 4.5k self-collected real-robot episodes.
- Training stages: Stage two finetunes the entire model simultaneously on three planning tasks using downstream robot data with action labels.
- Scenario-specific finetuning: Robotwin contributes 50 tasks with 50 episodes each, totaling 2.5k episodes, while Aloha uses 3k short-horizon and 1.5k long-horizon episodes.
C.3 Implementation Details about task Annotation
The annotation procedure supplies missing subtask and global-task descriptions for robotic datasets. It uses prompt templates and automated video processing to create planning-related supervision.
- For datasets without subtask annotations, the authors use a prompt template and Seed-1.5-VL-thinking to process videos or image sequences for pretraining.
- For datasets lacking overall task descriptions, another prompt template extracts a global task description for planning or keyframe prediction.
D Training and Evaluation Details
The paper uses distinct training and evaluation settings across Calvin, RoboTwin, and real-robot scenarios, varying data, compute, camera inputs, action horizons, and robustness tests.
- Calvin ABC-D Simulation Environment: Calvin ABC-D training used 8 A800 GPUs for 30,000 steps, with action chunks of 10 and two camera views predicting a third view.Evaluation used 1,000 length-5 tasks from the D-split and reported mean task completion length.
- Robotwin Simulation Environment: RoboTwin training used 2,500 clean demonstrations on 8 A800 GPUs for 60,000 steps, with action chunks of 16 sampled every 3 steps.The effective action horizon was 48.
- Robotwin Simulation Environment: RoboTwin evaluation ran 100 trials on 50 tasks in Clean and Randomized settings with unseen instructions, reporting success rate.All three camera views were inputs, and the primary view image was predicted.
- Real-Robot Tasks: Real-robot training used 32 A800 GPUs for 50,000 steps, with action chunks of 24, three camera views, and primary-view image prediction.Evaluation randomized initial positions, added distractors, and used unseen target objects for OOD tasks.
E Detailed Results in Simulation Environments
The paper points to detailed simulation results and real-robot demonstrations covering both basic manipulation and long-horizon planning tasks.
- Real-Robot Results: The real-robot evaluation is organized into Basic Tasks and Long-Horizon Planning Tasks, with demo videos presented for each category.These tasks assess both basic manipulation and planning capabilities.
F.1 Basic Tasks
The task suite evaluates robustness, precise manipulation, semantic instruction following, and long-horizon planning through varied basic and multi-stage robotic tasks.
- Basic Tasks: Robustness testing includes novel objects, distractors, background and object-color changes, and varied initial object positions.These conditions evaluate generalization beyond fixed task scenes.
- Basic Tasks: Basic manipulation tasks include pick-and-place, unseen-object transfer, watering a flower, stacking cubes, inserting flowers into a vase, and stacking bowls by color sequence.They probe grasping, alignment, position robustness, and language following.
- Basic Tasks: Pour Fries requires opening a carton and pouring toy fries onto a plate while determining the next action from current progress.The task tests manipulation accuracy and long-horizon capability.
- Basic Tasks: Sweep Rubbish requires grasping a broom, sweeping randomly placed tissue into a dustpan, and putting down the broom.The model must assess progress and increase sweeping speed so the tissue rolls into the dustpan.
- Basic Tasks: Additional tasks include pressing buttons in a specified color sequence and opening and closing a drawer to evaluate semantic following and manipulation accuracy.The button task is a simple long-horizon task, whereas drawer operation primarily evaluates precision.
- Long-Horizon Planning Tasks: Long-horizon planning tasks are Stack Cubes in Requested Order and Calculate and Place Symbol Blocks.Both require the model to choose successive actions under a language instruction.
- Long-Horizon Planning Tasks: Stack Cubes in Requested Order requires arranging scattered multicolored cubes into instructed shapes and sequences across structures of one to three layers.The task requires interleaved planning at each step and poses a semantic-following challenge for methods without explicit planning.
- Long-Horizon Planning Tasks: Calculate and Place Symbol Blocks requires assembling an instructed arithmetic equation, computing its result, and placing the corresponding symbol blocks.Partially arranged scenes require autonomous next-block selection and add a chain-of-thought reasoning requirement.
G More Interleaved Planning Visualizations on diverse robotic Tasks
The supplementary material presents visualizations, demonstrations, and denoising comparisons for BagelVLA’s interleaved planning across simulation and real-world tasks.
- Conditioning Schemes: Figure 7 contrasts complete denoising, joint denoising, and single-step denoising conditioning schemes for image prediction and action generation.Complete denoising uses N1 + N2 steps, joint denoising uses N steps, and single-step denoising conditions actions on the first image-prediction step.
- Simulation Results: Tables 7 and 8 contain simulation evaluation results for RoboTwin 2.0 and Calvin ABC→D, respectively.The supplied table descriptions identify success-rate reporting for RoboTwin and detailed benchmark results for Calvin.
- Demonstrations: Figures 8 and 9 provide demo videos for BagelVLA on basic tasks and long-horizon planning tasks.The demonstrations cover both ordinary manipulation and extended planning scenarios.
- Interleaved Planning Visualizations: Figure 10 visualizes interleaved planning in diverse robotic tasks, where global instructions and current observations support immediate-subtask identification and goal-image prediction.The visualization emphasizes the transition from a global instruction to a predicted visual target.
- Residual Flow Guidance: Figure 11 compares naive single-step denoising with RFG across denoising steps in real-world and simulation scenarios.RFG preserves backgrounds and achieves high-quality generation with very few steps, supporting lower interleaved-generation latency.
- Prompt Materials: Figures 12 and 13 show prompts used for Seed-1.5-VL-thinking.The passages identify the prompt figures but do not specify their contents.