PaperlayerView exact revision on arXiv

Source-linked AI summary

HALO: A Unified Vision-Language-Action Model for Embodied Multimodal Chain-of-Thought Reasoning

Quanxin Shou, Fangqi Zhu, Shawn Chen, Puxin Yan, Zhengyang Yan, Yikun Miao, Xiaoyi Pang, Zicong Hong, Ruikai Shi, Hao Huang, Jie Zhang, Song Guo

arXiv:2602.21157v2cs.RO

TL;DR

VLA models can struggle with long-horizon and out-of-distribution tasks, and existing reasoning approaches do not unify textual reasoning with visual foresight. HALO combines textual reasoning, visual subgoal prediction, and action prediction through specialized experts, and reports stronger performance in simulation and real-world settings, including on RoboTwin.

  • Problem

    Many VLA models lack explicit multimodal reasoning and foresight, while existing textual or visual reasoning approaches do not provide a unified framework for both.

    Source: §1 · p. 1

  • Method

    HALO uses a Mixture-of-Transformers architecture for textual reasoning, visual subgoal prediction, and EM-CoT-conditioned action prediction, supported by synthesized training data and a staged training recipe.

    Source: §1 · p. 2

  • Results

    HALO performs more strongly in simulation and real-world settings, reaching an 80.5% average success rate on RoboTwin and surpassing π0 by 34%.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The authors position HALO as a scalable, generalizable paradigm for human-like VLA reasoning in out-of-distribution and real-world settings.

    Source: §5 · p. 9

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have shown strong performance in robotic manipulation, but often struggle in long-horizon or out-of-distribution scenarios due to the lack of explicit mechanisms for multimodal reasoning and anticipating how the world will evolve under action. Recent works introduce textual chain-of-thought or visual subgoal prediction within VLA models to reason, but still fail to offer a unified human-like reasoning framework for joint textual reasoning, visual foresight, and action prediction. To this end, we propose HALO, a unified VLA model that enables embodied multimodal chain-of-thought (EM-CoT) reasoning through a sequential process of textual task reasoning, visual subgoal prediction for fine-grained guidance, and EM-CoT-augmented action prediction. We instantiate HALO with a Mixture-of-Transformers (MoT) architecture that decouples semantic reasoning, visual foresight, and action prediction into specialized experts while allowing seamless cross-expert collaboration. To enable HALO learning at scale, we introduce an automated pipeline to synthesize EM-CoT training data along with a carefully crafted training recipe. Extensive experiments demonstrate that: (1) HALO achieves superior performance in both simulated and real-world environments, surpassing baseline policy pi_0 by 34.1% on RoboTwin benchmark; (2) all proposed components of the training recipe and EM-CoT design help improve task success rate; and (3) HALO exhibits strong generalization capabilities under aggressive unseen environmental randomization with our proposed EM-CoT reasoning.

1. Introduction

HALO addresses the limited reasoning and visual foresight of existing VLA models with a unified process for textual reasoning, visual subgoals, and action prediction. It combines specialized experts with synthesized training data, and reports stronger results in simulation and real-world settings.

1. Introduction

  • Existing VLAs often lack explicit task reasoning and visual foresight, a limitation that matters in long-horizon and out-of-distribution scenarios.Prior approaches add textual reasoning or visual subgoals, but do not unify both capabilities.Source: §1 · pp. 1–2
  • HALO sequences textual task reasoning, visual subgoal prediction, and action prediction conditioned on the resulting embodied multimodal chain of thought.The subgoal images provide fine-grained visual guidance.Source: §1 · p. 2, Figure 1 · p. 2
  • A Mixture-of-Transformers architecture assigns reasoning, visual foresight, and action prediction to specialized experts that collaborate through shared self-attention.HALO also preserves each expert’s generative workflow, using autoregressive text generation and diffusion-based visual-state and action prediction.Source: §1 · p. 2
  • An automated pipeline pairs planned subtasks with subgoal images and explicit reasoning, while training combines diverse pretraining data with EM-CoT- and VQA-augmented fine-tuning.The training recipe includes general VQA, egocentric and robotic videos, and heterogeneous robot trajectories.Source: §1 · p. 2
  • HALO reaches an 80.5% average success rate on RoboTwin, surpassing π0 by 34%.Experiments cover simulated and real-world environments; the paper also reports generalization to unseen scenarios.Source: §1 · pp. 2–3

2. Related Work

Related work extends VLA models with action prediction, future-state generation, and intermediate reasoning, but the cited approaches leave gaps in explicit multimodal reasoning or tightly couple text and image generation.

Vision-Language-Action Models

  • OpenVLA and FAST extend pretrained VLMs with discrete action tokens, while the π0 series adds a separate flow-based action expert.The passage cites OpenVLA, FAST (Pertsch et al., 2025), and π0 (Black et al., 2024; Intelligence et al., 2025).Source: §2 · p. 3
  • Unified VLA architectures jointly model understanding, future visual states, and actions, but the cited approaches lack explicit textual or visual reasoning.The passage describes unified and mixed architectures for image or video generation alongside action prediction.Source: §2 · p. 3

Textual and Visual Reasoning for Robotic Intelligence

  • Prior work explores textual reasoning, visual subgoals, or both; ManualVLA couples text and image generation in one autoregressive model, which may degrade VLM reasoning.HALO instead uses a Mixture-of-Transformers architecture for joint textual reasoning, visual generation, and action prediction.Source: §2 · p. 3

3. Method

HALO formulates manipulation as a sequence of textual reasoning, visual foresight, and action prediction, implemented with collaborating modality-specialized experts. Its EM-CoT data pipeline and two-stage training recipe provide supervision for that sequence.

3.1. Problem Formulation

  • HALO conditions action chunks on generated textual reasoning and predicted visual subgoals rather than mapping observation histories directly to actions.The formulation jointly models textual reasoning, visual foresight, and action prediction as a deliberate sequence.Source: §3.1 · pp. 3–4

3.2. Unified Architecture

  • Three Mixture-of-Transformers experts handle textual reasoning, visual generation, and action prediction, interacting through shared self-attention.Special tokens route generation to the visual or action expert, while structured masks control information flow.Source: §3.2 · p. 4

3.3. EM-CoT Data Pipeline

  • The EM-CoT pipeline extracts motion primitives, uses a VLM to generate task plans and subtask reasoning, then selects each subtask’s terminal frame as its visual goal.These aligned textual and visual annotations are synthesized from raw robot trajectories.Source: §3.3 · p. 4, Figure 2 · p. 5

3.4. Training Recipe

  • Versatile pretraining combines VQA, visual generation, and action prediction; visual generation uses OXE trajectories and SSv2 videos, while action prediction uses OXE.The pretraining objective assigns higher weights to manipulation-intensive tasks.Source: §3.4 · pp. 5–6
  • EM-CoT fine-tuning trains a thought-foresight-action sequence with joint reasoning, visual-subgoal, and action losses, while retaining general VQA data.The VQA data is included to mitigate catastrophic forgetting of general multimodal capabilities.Source: §3.4 · p. 6

4. Experiments

Experiments evaluate HALO’s performance and generalization in simulation and real-world settings, the contributions of its training recipe and EM-CoT, and the quality of its generated reasoning. They include simulation and real-world task evaluations alongside ablation and qualitative analyses.

4.1. Experiment Settings

  • Simulation fine-tuning uses 2,500 expert demonstrations, while real-world fine-tuning uses 320; pre-training runs for 90k steps.Fine-tuning runs for 110k simulation steps and 80k real-world steps; inference uses action chunks of 16 and 50, respectively.Source: §4.1 · p. 7

4.2. Simulation Results

  • HALO reaches 80.46% average success on Easy tasks and 26.44% on Hard tasks, surpassing π0 by 34.1% and 10.1%, respectively, on RoboTwin 2.0.The reported relative performance gaps over π0 are 73.5% on Easy tasks and 62.0% on Hard tasks.Source: §4.2 · p. 7
  • In randomized Hard settings, Diffusion Policy’s success rate falls to 0.6%, while HALO reports multi-fold gains on fine-grained tasks such as Blocks Ranking Size and Stamp Seal.The passage also reports that π0 and RDT-1B show some robustness but plateau below HALO.Source: §4.2 · p. 7

4.3. Ablation Study

  • Removing visual-generation data from versatile pre-training degrades performance, with a reported reduction of over 50% on Hard tasks; removing further data types worsens performance.Without pre-training, performance is reported as 0% on Hard tasks.Source: Table 2 · p. 7, §4.3 · p. 8
  • Removing either visual subgoal images or textual reasoning degrades EM-CoT performance, while using both achieves the highest accuracy, particularly on Hard tasks.Table 2’s Panel B compares visual subgoal images (V) and textual reasoning (T).Source: Table 2 · p. 7, §4.3 · p. 8

4.4. Qualitative Results of EM-CoT

  • Qualitative rollouts show textual reasoning and visual subgoals aligning with ground truth in clean settings and target identification under unseen domain randomization.The passage describes target identification and trajectory planning despite substantial changes in object appearance and backgrounds.Source: §4.4 · p. 8, Figure 5 · p. 8

4.5. Real-World Results

  • HALO outperforms π0 and π0.5 across real-world tasks in both basic and generalization settings, where each task is evaluated over 50 trials.The generalization settings include visual distractions, lighting and background variation, and novel objects.Source: §4.5 · p. 8, Figure 6 · p. 8, Figure 7 · p. 9

5. Conclusion

The conclusion presents HALO as a unified VLA model for embodied multimodal chain-of-thought reasoning, combining textual reasoning, visual foresight, and action prediction. It also highlights its automated training-data synthesis pipeline and results in out-of-distribution and real-world settings.

5. Conclusion

  • HALO combines textual reasoning, visual foresight, and action prediction through a Mixture-of-Transformers architecture, with an automated pipeline for synthesizing EM-CoT training data.The conclusion describes the model as a scalable and generalizable human-like VLA reasoning paradigm, supported by experiments and ablations in out-of-distribution and real-world settings.Source: §5 · p. 9

A.1. LLM Backbone

The appendix describes HALO’s language-model backbone and the encoders, workflow tokens, and attention-mask figures used in its implementation.

A.1. LLM Backbone

  • HALO uses Qwen-2.5-1.5B LLM as its initial backbone.The appendix points to Table 3 for the model configuration.Source: §A.1 · p. 12

A.2. Multimodal Encoders

  • Separate encoders process visual understanding and generation, while linear projections encode continuous actions and decode them back to action dimensions.The visual pathway uses a ViT initialized with SigLIP2 for understanding and a pre-trained FLUX VAE for generation; the VAE parameters remain frozen during training.Source: §A.2 · p. 12

A.3. Special Tokens

  • Special tokens mark the start and end of subtasks, plans, moves, actions, and vision phases.Source: §A.3 · p. 12

A.4. Attention Mask

  • The appendix directs readers to Figure 3 for the fine-tuning attention mask and Figure 8 for the pre-training attention mask.Source: §A.4 · p. 12

B. Details on EM-CoT Data Pipeline

The EM-CoT data pipeline extracts frame-wise action labels and descriptions, creates temporally structured textual annotations, and assigns visual goal images to frames.

B.1. Action Primitives Extraction

  • Action primitives are extracted from bimanual gripper states and end-effector poses, with frame-wise action labels and natural-language descriptions as output.The algorithm uses idle periods to segment actions and labels movement direction from displacement.Source: §B.1 · p. 13
  • Action labels are converted into natural-language descriptions by looking up a sentence for each arm at each frame.The algorithm specifies a per-arm description lookup for each frame’s label.Source: §B.1 · p. 13

B.2. Prompts for EM-CoT

  • The annotation pipeline prompts a VLM to narrate the task, divide it into goal-oriented subtasks, and align frames with first-person reasoning.The authors state that this staged process is intended to produce temporally coherent, semantically clear annotations.Source: §B.2 · p. 13

B.3. Visual Subgoal Extraction

  • Visual subgoal extraction takes frame annotations and images as input, identifies a goal frame for each subtask, and broadcasts goal images to frames.The algorithm describes the output as a goal image sequence aligned with the input frame sequence.Source: §B.3 · p. 15

C. Training Implementation

HALO is pretrained and then fine-tuned on 32 Nvidia H100 GPUs, with separate fine-tuning schedules for simulation and real experiments.

C. Training Implementation

  • Pretraining runs for 90k steps on 32 Nvidia H100 GPUs with a sequence length of 40k per rank.The accompanying Table 4 is identified as the source of detailed training hyperparameters.Source: C. Training Implementation · p. 16, Table 4 · p. 16
  • Fine-tuning uses 32 Nvidia H100 GPUs and a sequence length of 27k, with 110k steps for simulation and 80k for real experiments.Table 4 is described as providing the detailed configurations for the two-stage training process.Source: C. Training Implementation · p. 16, Table 4 · p. 16

D. Complete Result on RoboTwin2.0 Benchmark

The section directs readers to Table 5 for the complete RoboTwin 2.0 benchmark results.

D. Complete Result on RoboTwin2.0 Benchmark

E. Details on Real-World Experiment

The real-world experiments use a head-camera-only robot setup and four manipulation tasks, with a multi-task EM-CoT corpus and evaluations across unseen environmental variations.

E. Details on Real-World Experiment

  • Real-world experiments use a COBOT Magic robot with the Mobile ALOHA design and its head camera as the sole visual input.The setup requires policies to reason and act from an egocentric perspective.Source: E. Details on Real-World Experiment · p. 16
  • The four tasks test tool-mediated sweeping, semantic grounding in bimanual cup nesting, inter-arm screwdriver handover, and multi-step drawer manipulation.The task set includes placing a lemon in a drawer after opening it, then closing the drawer.Source: E. Details on Real-World Experiment · p. 16
  • The authors collect 80 trajectories per task, process them through the automated EM-CoT pipeline, and aggregate the data for multi-task fine-tuning.Source: E. Details on Real-World Experiment · p. 16
  • Generalization evaluations vary visual distractions, desk-lamp illumination, tablecloth color, and the sweeping tool, replacing the broom with a sponge.Source: E. Details on Real-World Experiment · p. 16

Related papers

Loading 2602.21157v2…