PaperlayerView exact revision on arXiv

Source-linked AI summary

Emma-X: An Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning

Qi Sun, Pengfei Hong, Tej Deep Pala, Vernon Toh, U-Xuan Tan, Deepanway Ghosal, Soujanya Poria

arXiv:2412.11974v2cs.ROcs.AIcs.CLcs.CV

TL;DR

Robotic control methods can be task-specific, and existing VLAs face challenges in grounded planning and long-horizon spatial reasoning. EMMA-X combines grounded task reasoning, look-ahead spatial guidance, and a hierarchical dataset with 60,000 manipulation trajectories. The authors report improvements over baselines, especially on tasks requiring long-horizon spatial reasoning.

  • Problem

    Robotic control methods can be task-specific, while VLAs face challenges in long-horizon spatial reasoning and grounded task planning.

    Source: §1 · pp. 1–2

  • Method

    EMMA-X trains an embodied action model on a hierarchical dataset with grounded task reasoning and 2D gripper positions and 3D spatial movements, using trajectory segmentation based on gripper states and arm motion.

    Source: §1 · pp. 2–3, §3.1 · p. 5

  • Results

    EMMA-X shows significant improvements over existing baselines, especially on tasks requiring long-horizon spatial reasoning.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    The reported gains are strongest in tasks requiring long-horizon spatial reasoning.

    Source: §5 · p. 9

    EMMA-X has higher latency than OpenVLA and generates approximately 10 times more tokens during reasoning.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Traditional reinforcement learning-based robotic control methods are often task-specific and fail to generalize across diverse environments or unseen objects and instructions. Visual Language Models (VLMs) demonstrate strong scene understanding and planning capabilities but lack the ability to generate actionable policies tailored to specific robotic embodiments. To address this, Visual-Language-Action (VLA) models have emerged, yet they face challenges in long-horizon spatial reasoning and grounded task planning. In this work, we propose the Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning, Emma-X. Emma-X leverages our constructed hierarchical embodiment dataset based on BridgeV2, containing 60,000 robot manipulation trajectories auto-annotated with grounded task reasoning and spatial guidance. Additionally, we introduce a trajectory segmentation strategy based on gripper states and motion trajectories, which can help mitigate hallucination in grounding subtask reasoning generation. Experimental results demonstrate that Emma-X achieves superior performance over competitive baselines, particularly in real-world robotic tasks requiring spatial reasoning.

1 Introduction

Robot control must generalize beyond fixed tasks while connecting visual-language understanding to embodiment-specific action. EMMA-X combines grounded task reasoning with look-ahead spatial guidance and a hierarchical manipulation dataset.

1 Introduction

  • Reinforcement-learning controllers often target narrowly defined tasks in fixed environments, while VLMs are not designed to generate policies for specific robotic embodiments.Existing VLAs also struggle with long-horizon spatial reasoning about how a robot should move next.Source: §1 · pp. 1–2
  • EMMA-X predicts future gripper positions and 3D movement plans to guide immediate actions toward longer-term task objectives.The model uses future gripper positions as checkpoints for a high-level movement plan.Source: §1 · p. 2
  • ECoT generates subtask reasoning from text alone, whereas EMMA-X uses segmented visual images to ground reasoning in the robot’s current state.The paper describes an ECoT pot-cover mismatch and says segmentation-based visual input reduces Gemini’s hallucination issues.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • EMMA-X is a 7B-parameter model fine-tuned from OpenVLA with grounded reasoning data, and its hierarchical BridgeV2-based dataset contains 60,000 manipulation trajectories.The dataset includes grounded task reasoning, 2D gripper positions, and 3D spatial movements.Source: §1 · pp. 2–3

2 Problem Formulation

The formulation learns robot actions from expert demonstrations, then structures tasks into states, segments, and subtasks with state-level labels that support reasoning during inference.

2.1 Policy Imitation Learning

  • The imitation-learning objective is to learn a policy πθ(a | s, T) that mimics expert actions from demonstrations of states, instructions, and actions.Each demonstration contains a sequence of image states, a natural-language task instruction, and the expert action at each state.Source: §2.1 · p. 3
  • The VLA policy outputs a 7-dimensional action encoding end-effector velocity, orientation, and gripper close-open action.The components are Cartesian velocity (x, y, z), orientation (roll, pitch, yaw), and gripper action.Source: §2.1 · p. 3

2.2 Hierarchical Policy Imitation

  • Hierarchical policy imitation decomposes a general task into finer-grained states, segments, and subtasks.Segments contribute to subgoals, and subtasks combine segments to achieve specific subgoals.Source: §2.2 · p. 3
  • A segment is a consecutive sequence of states contributing to a subgoal, while a subtask consists of segments and a task consists of subtasks.States represent scenes; the trajectory’s segments and subtasks form progressively larger units toward the overall objective.Source: §2.2 · p. 3
  • The hierarchical VLA predicts actions at each state by decomposing tasks into subtasks so end-effector motion aligns with subgoal intent.Source: §2.2 · p. 3
  • The dataset labels each state with its subtask, which the paper says is necessary for chain-of-thought training.Source: §2.2 · p. 3
  • At inference, the model generates reasoning chains containing subtasks and spatial information derived from visual scenes.Source: §2.2 · p. 3

3 Methodology

EMMA-X combines trajectory segmentation, hierarchical planning data generation, and OpenVLA-based model training. Its pipeline uses segmented demonstrations to produce grounded reasoning and look-ahead spatial guidance before action-policy prediction.

3 Methodology

  • EMMA-X combines gripper- and motion-based trajectory segmentation, hierarchical planning data generation, and training from OpenVLA.The design includes grounded task reasoning, 2D gripper positions, and 3D spatial movements.Source: §3 · pp. 3–4

3.1 Trajectory Segmentation

  • HDBSCAN clusters end-effector trajectories using spatial, orientation, and temporal differences, with additional breakpoints when gripper state changes.Segmenting consecutive states gives Gemini more context for assigning subtask labels than annotating individual frames.Source: §3.1 · pp. 4–5

3.2 Data Generation

  • For each segment, the dataset includes grounded subtask reasoning, a 2D gripper position, and a 3D movement plan toward the next segment’s first state.The 3D movement plan is not directly executed as a robot action.Source: §3.2 · pp. 5–6

3.3 EMMA-X

  • EMMA-X is a 7B-parameter VLA fine-tuned from OpenVLA; at inference, it reasons about the scene and subtask before predicting the next robot action policy.Real-robot inputs include the task description, current observation, and real-time detected 2D gripper position.Source: §3.3 · p. 6

4 Experiments

Experiments evaluated EMMA-X on real-world WidowX tasks against several baselines and found gains in overall performance, especially on spatial-relation and out-of-domain instruction tasks. Ablations showed contributions from segmentation, look-ahead spatial reasoning, and grounded CoT reasoning.

4.1 Implementation Details

  • The hierarchical reasoning data pipeline augmented approximately 60,000 BridgeData-v2 trajectories, and OpenVLA was fine-tuned on the augmented data for three epochs.OpenVLA is a 7B VLA model; its 7-dimensional action policy was tokenized into discrete policy tokens.Source: §4.1 · pp. 6–7

4.4 EMMA-X Improves Policy Generalization

  • EMMA-X increased task success rate by 24.17% and half-success rate by 26.25% over OpenVLA; spatial-relation h_Succ gains were 35% over OpenVLA and 29% over ECoT, with gains also on OOD instructions.Figure 4 reports category-level results across real-world robot tasks; Table 1 covers 12 WidowX-250 manipulation tasks.Source: Table 1 · p. 7, Figure 4 · p. 7, §4.4 · p. 8

4.2 Robot Setup and Metrics

  • Evaluation used a robot arm with a single third-person camera feed and natural-language instruction to test in-domain, OOD-object, spatial-relationship, and OOD-instruction generalization tasks under identical real-world setups.Each task was conducted over 10 trials; success received a score of 1, failure a score of 0, and half-success received 0.5 when its goal-and-difficulty criteria were met.Source: §4.2 · p. 7

4.3 Baselines

  • The real-world comparison included OpenVLA, OpenVLA fine-tuned on BridgeV2, and ECoT fine-tuned on BridgeV2 with generated chain-of-thought data.The OpenVLA fine-tuning baseline used the same number of epochs and training setting as EMMA-X.Source: §4.3 · p. 7

4.5 Analysis

  • Segmentation based only on gripper open-close positions caused performance drops of 10% to 50%, with spatial-reasoning performance declining by 50%.The authors report that the distance metric introduced for segmentation is crucial to the process.Source: §4.5 · p. 8
  • Removing future-position prediction caused 30% to 45% performance drops, while removing the movement plan caused 25% to 40% drops; spatial-reasoning declines were 45% and 35%, respectively.The reported comparison suggests future end-effector position prediction is more critical than the 3D movement plan.Source: §4.5 · p. 9
  • Removing grounded CoT while retaining spatial reasoning caused a 43% to 55% performance drop, with spatial-reasoning performance declining more than under explicit spatial-reasoning ablations.The authors conclude that spatial reasoning alone is insufficient.Source: §4.5 · p. 9
  • Fine-tuning OpenVLA on BridgeV2 degraded its performance by 5% to 30%, with the worst performance on OOD instructions.The authors hypothesize that the decline reflects overfitting because BridgeV2 was already part of OpenVLA's pretraining data.Source: §4.5 · p. 9

5 Conclusion

The conclusion presents EMMA-X as a 7B-parameter embodied multimodal action model with grounded reasoning and spatial guidance. It reports improvements over existing baselines on tasks requiring long-horizon spatial reasoning.

5 Conclusion

  • The 7B-parameter model is designed to enhance spatial reasoning and task planning for robotic policy generation.Source: §5 · p. 9
  • EMMA-X shows significant improvements over existing baselines on tasks requiring long-horizon spatial reasoning.Source: §5 · p. 9
  • Its hierarchical embodiment dataset includes grounded reasoning, 2D gripper positions, and 3D spatial movements.The conclusion also says trajectory segmentation reduces hallucination in task reasoning by grounding reasoning in visual images.Source: §5 · p. 9

Limitations

The authors identify higher inference latency, limited generalization, and possible gripper-detection inaccuracies as limitations of EMMA-X.

Limitations

  • EMMA-X inference latency is higher than OpenVLA, primarily because it generates approximately 10 times more tokens.The authors suggest predicting policies across a segment and regenerating only when a prediction significantly deviates from the expected movement plan.Source: Limitations · p. 9
  • The authors identify generalization capability as a limitation and suggest scaling training to a larger subset of OXE to broaden task and robot coverage.Source: Limitations · p. 9
  • SAM-based gripper-position detection can be inaccurate when the gripper is partially occluded or outside the image frame.The authors propose a more robust robot-hand detection and segmentation model as a possible remedy.Source: Limitations · p. 9

A Related Work

Related work spans generalist robot policies, vision-language-action models, and reasoning for robotics. The authors position EMMA-X against limitations in long-horizon spatial reasoning and visual grounding.

Generalist Robot Policies

  • Generalist policies such as Octo and OpenVLA pursue multi-task robotics across tasks and embodiments using compositional or end-to-end designs.The section describes large and diverse datasets as a promising strategy for improving policy performance.Source: A Related Work · p. 12

Vision-Language-Action Models

  • VLAs encode robot actions in VLM vocabularies, with RT-2, RT-2-X, and OpenVLA presented as examples of approaches demonstrating generalization (Brohan et al., 2023b; Collaboration et al., 2023; Kim et al., 2024).The section says current VLAs underuse language and vision-language models’ capacity to reason through complex tasks.Source: A Related Work · p. 12

Reasoning for Robotics

  • ECoT trains a VLA to generate chain-of-thought reasoning, but the section identifies gaps in long-horizon spatial reasoning and visual grounding (Zawalski et al., 2024).EMMA-X is described as integrating grounded chain-of-thought reasoning to predict future states and transitions for robotic action policies.Source: A Related Work · p. 12

B Prompts and Sample Outputs

The appendix presents a Gemini prompt template and sample outputs for segment-level subtask reasoning. The example breaks a pot-cover task into ordered subtasks with justifications.

B Prompts and Sample Outputs

C Half Success Criteria

Magenta prompts are used to report the half-success results in Table 2.

C Half Success Criteria

  • Magenta prompts are used to report the half-success results in Table 2.The supplied passage does not specify the half-success criteria themselves.Source: C Half Success Criteria · p. 13

D Segmentation Statistics

The section reports average segment and trajectory lengths and specifies the translational, rotational, and gripper commands in the motion plan template.

D Segmentation Statistics

E Motion Plan Template

F Pseudo Code for Training EMMA-X and Running Inference

The pseudocode maps robot trajectories to grounded subtasks and reasons for supervised fine-tuning, then repeatedly controls the robot until task completion.

F Pseudo Code for Training EMMA-X and Running Inference

Loading 2412.11974v2…