Source-linked AI summary
Interleave-VLA: Enhancing Robot Manipulation with Interleaved Image-Text Instructions
Cunxin Fan, Xiaosong Jia, Yihang Sun, Yixiao Wang, Jianglan Wei, Ziyang Gong, Xiangyu Zhao, Masayoshi Tomizuka, Xue Yang, Junchi Yan, Mingyu Ding
TL;DR
Text-only instructions limit VLA generalization and flexibility for unseen robotic scenarios. Interleave-VLA adapts existing VLAs to interleaved image-text instructions and trains them with an automatically constructed embodied dataset, achieving 2× generalization gains and flexible zero-shot instruction following.
Problem
Text-only VLA instructions provide insufficient multimodal grounding for unseen scenarios and limit instruction flexibility.
Method
Interleave-VLA adds lightweight interleaved-input support to existing VLAs and trains it with an automatically constructed dataset from Open X-Embodiment.
Results
2× improvement in generalization across simulation and real-world experiments, with zero-shot support for hand-drawn sketches, cropped images, and Internet photos.
Takeaways & Limitations
Interleaved visual instructions provide flexible zero-shot interfaces and mitigate attentional hallucinations through visual grounding and modality diversity.
Takeaways & Limitations
Interleaved inputs increase computational demands because they produce longer image token sequences.
Abstract
from arXiv · showhide
The rise of foundation models paves the way for generalist robot policies in the physical world. Existing methods relying on text-only instructions often struggle to generalize to unseen scenarios. We argue that interleaved image-text inputs offer richer and less biased context and enable robots to better handle unseen tasks with more versatile human-robot interaction. Building on this insight, Interleave-VLA, the first robot learning paradigm capable of comprehending interleaved image-text instructions and directly generating continuous action sequences in the physical world, is introduced. It offers a natural, flexible, and model-agnostic paradigm that extends state-of-the-art vision-language-action (VLA) models with minimal modifications while achieving strong zero-shot generalization. Interleave-VLA also includes an automatic pipeline that converts text instructions from Open X-Embodiment into interleaved image-text instructions, resulting in a large-scale real-world interleaved embodied dataset with 210k episodes. Comprehensive evaluation in simulation and the real world shows that Interleave-VLA offers two major benefits: (1) improves out-of-domain generalization to unseen objects by 2x compared to text input baselines, (2) supports flexible task interfaces and diverse instructions in a zero-shot manner, such as hand-drawn sketches. We attribute Interleave-VLA's strong zero-shot capability to the use of instruction images, which effectively mitigate hallucinations, and the inclusion of heterogeneous multimodal datasets, enriched with Internet-sourced images, offering potential for scalability. More information is available at https://interleave-vla.github.io/Interleave-VLA-Anonymous/
1 INTRODUCTION
Text-only VLA instructions limit multimodal grounding and instruction flexibility in unseen scenarios. Interleave-VLA addresses this with interleaved image-text inputs, a large-scale dataset, and evidence of stronger generalization.
- Text-only VLA training limits instruction flexibility and prevents models from leveraging the richer semantics and grounding of interleaved multimodal signals.
- The authors construct an automated interleaved dataset from Open X-Embodiment and propose Interleave-VLA for generating continuous actions from interleaved inputs.The dataset contains 210k episodes and 13 million frames.
- Interleave-VLA adapts existing VLAs with separator tokens, scalable training, and an inference interface supporting text, camera crops, web images, and sketches.
- Interleaved image-text instructions mitigate hallucinations caused by ambiguous contexts by providing less-biased visual grounding.
- Modality diversity alleviates hallucinations from training-distribution biases, improving both interleaved and text-only evaluation.
- Interleave-VLA delivers 2× gains in out-of-domain generalization to novel objects and supports zero-shot visual instructions such as hand-drawn sketches.
2 RELATED WORK
Related work spans interleaved vision-language models and vision-language-action models. Interleave-VLA connects these areas by extending text-oriented VLA processing toward interleaved inputs.
- Interleaved Vision-Language Models: Interleaved vision-language models have progressed from image-text pairs to arbitrarily interleaved sequences of images and text.
- Vision Language Action Models: Most prior VLA models condition robotic policies on visual observations and text-only instructions.
- Interleave-VLA: Interleave-VLA is presented as an extension of the Text-VLA paradigm to support interleaved inputs.
3 INTERLEAVE-VLA AND OPEN INTERLEAVED X-EMBODIMENT DATASET
Interleave-VLA defines action generation from states containing observations, proprioception, and interleaved instructions, then adapts, trains, and deploys VLA models with multimodal data. Its dataset is automatically created by parsing instructions, detecting objects, and verifying crops.
- Problem Formulation: Interleave-VLA conditions each action on visual observations, proprioception, and an ordered sequence of text and image tokens.When all instruction tokens are textual, the formulation reduces to standard Text-VLA.
- Interleave-VLA Paradigm: The paradigm comprises an adaptation module, scalable interleaved-data training, and a versatile inference interface.
- Adaptation: Special separator tokens and an updated input processor enable interleaved inputs while leaving the core VLA architecture unchanged.
- Training: Training uses a large-scale interleaved embodied dataset without changing the base model’s hyperparameters or flow-matching objective.
- Inference: Inference accepts text, robot camera crops, web images, and hand-drawn sketches, including image styles differing from the training data.
- Dataset Construction: The automatic dataset pipeline parses instructions, detects and crops target objects, and verifies data quality, with OWLv2 achieving 82.6% accuracy.
- Dataset Construction: The released dataset integrates 11 Open X-Embodiment sources into a diverse real-world cross-embodiment resource.
4 EXPERIMENTS
Experiments compare Interleave-VLA with Text-VLA across simulation, real-robot, flexibility, and data-diversity settings. Interleaved instructions improve semantic generalization and reduce attentional hallucinations while supporting diverse task interfaces.
- Simulation comparison: 2× better performance on semantically out-of-domain tasks is achieved by full Interleave-VLA than Text-VLA in SimplerEnv.Interleave-VLA (Partial) already outperforms Text-VLA, while the full version further improves generalization.
- Simulation comparison: Interleave-VLA addresses attentional bias, diffused attention, and attention leakage by supplying explicit visual grounding through instruction images.These failure modes arise when Text-VLA attends to distractors, spreads attention across the scene, or leaks attention into irrelevant background regions.
- Inference flexibility: Interleave-VLA supports goal-image matching, multi-image instruction following, and other flexible task interfaces in VIMA-Bench.Its unified image-text format is used across diverse instruction specifications.
- Data diversity: Combining Internet images with task-specific robot-observation crops yields the best overall performance and stronger generalization.Internet-only images have limited task relevance, whereas cropped-only images lack diversity.
5 CONCLUSION
The paper presents Interleave-VLA as a minimally modified approach for training VLAs on interleaved image-text instructions. It reports 2× generalization gains, diverse zero-shot instruction handling, and a computational limitation from longer image-token sequences.
- Conclusion: Interleave-VLA converts Open X-Embodiment data into 210k episodes and 13 million frames for interleaved instruction training.The automatic pipeline addresses the lack of real-world interleaved robotic datasets.
- Conclusion: 2× improvement in generalization is achieved across simulation and real-world experiments with minimal VLA modifications.The approach also handles hand-drawn sketches, cropped images, and Internet photos without additional finetuning.
- Limitations: Training with interleaved inputs increases computational demands because image-token sequences are longer.The paper identifies image-token compression as a possible efficiency direction.
A INTERLEAVE-VLA IMPLEMENTATION DETAILS
Interleave-VLA adapts existing VLA models through interleaved image-text inputs rather than major architectural changes. Implementations on π0 and OpenVLA support the model-agnostic design and improve generalization across architectures.
- Scope: Interleave-VLA targets VLM-based VLAs and has been instantiated with π0 and OpenVLA.The paper notes that extension to other sequence-modeling VLA approaches remains future work.
- Architecture: Interleave-VLA requires minimal modifications and changes only the input format in the architecture comparison.The paradigm preserves existing VLA architectures while accepting interleaved image-text instructions.
- Input processing: Interleaved image-text instructions are enabled by adding special boundary tokens around image embeddings in the input sequence.<BOI> and <EOI> delimit image segments, allowing image and text tokens to alternate.
- Model compatibility: 2–3× gains over π0 are reported for Interleave-VLA, while OpenVLA adaptation achieves more than double the original OpenVLA performance.These results support compatibility with different VLA and VLM backbones.
B RELIABILITY ANALYSIS OF THE INTERLEAVED DATASET GENERATION PIPELINE
The dataset-generation pipeline combines QwenVL+SAM and Owlv2 to detect instruction-relevant objects. Their complementary strengths reduce detection errors substantially compared with either method alone.
- Pipeline design: QwenVL+SAM handles open-world objects, while Owlv2 performs better in cluttered or occluded scenes.The pipeline uses their complementary behavior across scene conditions.
- Reliability evaluation: 4.4% combined error rate is achieved, versus 22.1% for QwenVL+SAM and 17.4% for Owlv2 individually.The evaluation sampled 200 generated examples and counted a failure when any key object was undetected.
C.1.1 SIMPLERENV EVALUATION TASKS
The SimplerEnv evaluation covers familiar and challenging out-of-domain manipulation tasks, including visual and semantic generalization. It compares Interleave-VLA with established VLA baselines under checkpoint variability.
- Evaluation scope: Out-of-Domain tasks vary environments, tablecloths, lighting, objects, and object categories to test visual and semantic generalization.Examples include unseen environments, backgrounds, lighting conditions, known-category objects, and completely unseen categories.
- Baselines: The experiment compares Interleave-VLA adapted from π0 with π0, RT-1-X, and Octo-Base using official or repository-based evaluation implementations.The models follow the SimplerEnv evaluation protocol, with π0 reimplemented for direct BridgeData V2 evaluation.
- Training setup: Both Interleave-VLA and π0 use 5e-5 learning rate, global batch size 1024, and approximately 30 training epochs.Inputs include one observation image, interleaved instruction tokens, one proprioceptive token, and four action tokens.
- Evaluation caveat: Table 1 reports the best checkpoint performance, although results can vary substantially across checkpoints, especially for precise tasks such as stacking cubes.This variability is noted even among checkpoints that appear mostly converged.
C.1.3 SIMPLERENV EVALUATION RESULTS
Interleave-VLA matches Text-VLA on familiar tasks and improves generalization on unseen tasks, while interleaved instructions also improve object identification in VIMA-Bench and real-robot settings. The reported comparisons include visual, semantic, and memory-independent evaluations.
- SimplerEnv results: Interleave-VLA consistently surpasses π0 across all SimplerEnv Out-of-Domain generalization tasks, with co-training yielding further improvements.The co-trained model shows especially strong gains on semantic generalization and evidence of cross-embodiment transfer.
- SimplerEnv results: The co-trained model omits Unseen Environment evaluation because its scene overlay comes from RT-1 data included in co-training, creating a Google Robot–WidowX embodiment mismatch.The mismatch causes incorrect actions during WidowX evaluation.
- VIMA-Bench: VIMA-Bench evaluation excludes tasks requiring historical memory because Interleave-VLA operates in a memory-independent first-order Markov setting.The policy models future actions conditioned on the current observation, language command, and proprioceptive state.
- VIMA-Bench: Interleave-VLA improves correct-object identification by 2× on memory-independent VIMA-Bench tasks compared with text-only descriptions.The interleaved format helps distinguish objects with desired textures and shapes among distractors.
- Real-robot results: On real-robot Lift and Pick&Place tasks, Interleave-VLA consistently outperforms π0 in semantic generalization by 2×.Pretraining on the Open Interleaved X-Embodiment Dataset further improves performance, including on the FANUC robot.
D SCALABILITY OF INTERLEAVE-VLA WITH THE OPEN INTERLEAVED X-EMBODIMENT DATASET
Interleave-VLA scales through pretraining and co-training on the Open Interleaved X-Embodiment Dataset. Pretraining is especially important in low-data settings, while co-training adds gains for semantic generalization in higher-data regimes.
- Pretraining for low-data regimes: Pretraining on the Open Interleaved X-Embodiment Dataset is essential for strong real-robot performance in low-data regimes.The FANUC evaluation contains only 60 demonstrations per task.
- Evaluation measures: Tables 8 and 9 report Lift and Pick&Place results using successful trials and correct-object manipulation counts.Each object is evaluated over 12 trials, with # Succ and # Acc reported separately.
- Pretraining for low-data regimes: Cross-embodiment pretraining enables effective generalization to FANUC despite FANUC not appearing in the pretraining data.The larger and more diverse dataset supports more robust representations than the small task-specific demonstrations alone.
- Co-training for high-data regimes: Co-training with additional Open Interleaved X-Embodiment datasets further improves semantic generalization beyond the extensive BridgeData V2 dataset.The results are presented for the SimplerEnv Out-of-Domain setting.
E TASK FLEXIBILITY AND EMERGENT GENERALIZATION DETAILS
Interleave-VLA supports diverse user-created and Internet-sourced image-text instruction formats without additional fine-tuning. Its emergent generalization extends from object crops to hand-drawn sketches.
- Evaluation setup: The flexibility evaluation reuses the SimplerEnv Interleave-VLA model without additional fine-tuning.It tests diverse user input styles during deployment across three manipulation tasks.
- Instruction formats: The evaluated instructions include hand-drawn sketches, Internet images, and human-captured object crops.These formats specify eggplants, carrots, colored blocks, blocks, and spoons for placement tasks.
- Emergent generalization: Interleave-VLA generalizes to sketches, a novel input style not encountered during training, as well as unseen Internet images and object crops.The paper characterizes these capabilities as emergent generalization beyond typical novel-object and novel-environment evaluations.
F OPEN INTERLEAVED X-EMBODIMENT DATASET DETAILS
The Open Interleaved X-Embodiment Dataset combines diverse skills and data from 11 Open X-Embodiment sources, with actions standardized into a unified 7D pose format.
- Dataset composition: The dataset integrates data from 11 sources within Open X-Embodiment.
- Action standardization: Actions across datasets use a unified 7D pose format covering xyz position, Euler orientation, and gripper state.
- Source distribution: RT-1 contributes the largest share at 41.01%, followed by Bridge at 28.25% and BC-Z at 20.34%.