PaperlayerView exact revision on arXiv

Source-linked AI summary

MolmoAct: Action Reasoning Models that can Reason in Space

Jason Lee, Jiafei Duan, Haoquan Fang, Yuquan Deng, Shuo Liu, Boyang Li, Bohan Fang, Jieyu Zhang, Yi Ru Wang, Sangho Lee, Winson Han, Wilbert Pumacay, Angelica Wu, Rose Hendrix, Karen Farley, Eli VanderBilt, Ali Farhadi, Dieter Fox, Ranjay Krishna

arXiv:2508.07917v4cs.RO

TL;DR

Robotic foundation models often map perception and instructions directly to control, limiting flexibility and robustness. MolmoAct introduces a structured action-reasoning pipeline that connects depth-aware perception, spatial planning, and action prediction, and it consistently outperforms strong baselines across simulation and real-world settings while remaining fully open.

  • Problem

    Existing robotic vision-language-action models remain brittle and opaque, with limited flexibility and robustness across tasks, scenes, and embodiments.

  • Method

    MolmoAct integrates perception, planning, and control through depth perception tokens, visual reasoning traces, and autoregressive action prediction.

  • Results

    MolmoAct consistently outperforms strong vision-language-action baselines across simulation and real-world settings, including 75% success when steered via visual traces.

  • Takeaways & Limitations

    MolmoAct provides an open blueprint for explainable, spatially grounded, steerable robotic agents and reproducible action-reasoning research.

  • Takeaways & Limitations

    2D visual-trace steering lacks explicit depth and can produce imprecise motion along the camera’s depth axis on more complicated tasks.

Abstract

from arXiv · show

Reasoning is central to purposeful action, yet most robotic foundation models map perception and instructions directly to control, which limits adaptability, generalization, and semantic grounding. We introduce Action Reasoning Models (ARMs), a class of robotic foundation models that integrate perception, planning, and control through a structured three-stage pipeline. Our model, MolmoAct, encodes observations and instructions into depth-aware perception tokens, generates mid-level spatial plans as editable trajectory traces, and predicts precise low-level actions, enabling explainable and steerable behavior. MolmoAct-7B-D achieves strong performance across simulation and real-world settings: 70.5% zero-shot accuracy on SimplerEnv Visual Matching tasks, surpassing closed-source Pi-0 and GR00T N1.5; 86.6% average success on LIBERO, including an additional 6.3% gain over ThinkAct on long-horizon tasks; and in real-world fine-tuning, an additional 10% (single-arm) and an additional 22.7% (bimanual) task progression over Pi-0-FAST. It also outperforms baselines by an additional 23.3% on out-of-distribution generalization and achieves top human-preference scores for open-ended instruction following and trajectory steering. Furthermore, we release, for the first time, the MolmoAct Dataset -- a mid-training robot dataset comprising over 10,000 high quality robot trajectories across diverse scenarios and tasks. Training with this dataset yields an average 5.5% improvement in general performance over the base model. We release all model weights, training code, our collected dataset, and our action reasoning dataset, establishing MolmoAct as both a state-of-the-art robotics foundation model and an open blueprint for building ARMs that transform perception into purposeful action through structured reasoning. Blogpost: https://allenai.org/blog/molmoact

1 Introduction

MolmoAct addresses brittleness and opacity in robotic vision-language-action models by introducing structured spatial reasoning between perception, planning, and control. Its open architecture produces explainable behavior and supports adaptation, generalization, and steerability across robotic tasks.

  • The model predicts depth-aware perception tokens, visual reasoning traces for spatial plans, and low-level action sequences.
  • Robotic vision-language-action models remain brittle and opaque, struggling to transfer across tasks, scenes, and embodiments.
  • MolmoAct introduces Action Reasoning Models that integrate perception, planning, and control through a structured reasoning pipeline.
  • MolmoAct’s visual traces provide an explainable view of decisions and can be edited to steer actions more reliably than ambiguous language commands.
  • MolmoAct consistently outperforms competitive baselines across LIBERO and SimplerEnv while generalizing to diverse environments and task perturbations.
  • The authors release model weights, training code, and action reasoning data to support reproducibility and community research.

2 MolmoAct

MolmoAct is an open Action Reasoning Model that extends vision–language modeling with structured spatial reasoning for robotic manipulation. It represents depth, plans motion as visual traces, and predicts executable actions while using staged training and diverse robot data.

  • Overview: MolmoAct extends a vision–language backbone into an open Action Reasoning Model for robotic manipulation.It builds on Molmo and is trained with OXE data plus the in-house MolmoAct Dataset.
  • Training and data: Training combines action reasoning, trajectory-conditioned, auxiliary, and multimodal web data, with MolmoAct additionally mid-trained on its collected robot dataset.The overall pre-training mixture contains 26.3M samples, while the sampled subset increases auxiliary depth and trace data proportions.
  • Action reasoning pipeline: MolmoAct generates depth perception tokens, visual reasoning traces, and action tokens in sequence, grounding low-level control in inferred depth and planned motion.Its pipeline uses images and instructions to produce spatially structured intermediate representations rather than relying on language-only reasoning.
  • Action reasoning pipeline: Depth perception tokens provide a discrete, explainable summary of the scene’s depth map and condition downstream trajectory and action generation.A specialist depth estimator produces the target depth string, while MolmoAct learns to predict it autoregressively from RGB observations.
  • Steerability: Visual reasoning traces let users steer behavior by drawing editable motion cues directly on camera images instead of relying solely on ambiguous language prompts.The overlaid trace conditions closed-loop action generation and is intended to support precise, scalable steering across tasks and phrasing.

3 Data Curation and Generation

MolmoAct curates diverse robot and multimodal data into action-reasoning examples that align depth perception, visual traces, and actions. Its dedicated dataset contains 10,689 high-quality trajectories spanning 93 household manipulation tasks.

  • Action Reasoning Data: Action reasoning data appends depth perception tokens and visual reasoning traces to action tokens, conditioned on language and robot observations.This converts conventional robot episodes into sequences combining RGB observations, instructions, ground-truth actions, depth representations, and visual traces.
  • Action Reasoning Data: Each robot timestep contains an RGB observation, language instruction, and ground-truth action specified in end-effector or joint space.
  • Depth Perception Tokens: Depth perception tokens are generated by encoding observation images through a VQVAE trained on 10 million tabletop-manipulation depth maps.DepthAnything-v2 provides depth maps, which are used to train the VQVAE and encode observations into latent representations.
  • Visual Reasoning Traces: Visual reasoning traces aggregate gripper coordinates across an episode, with Molmo prompted to point to the gripper in each observation frame.Single-arm and bimanual embodiments use different prompts to identify the relevant gripper.
  • MolmoAct Dataset: 10,689 high-quality trajectories cover 93 manipulation tasks performed by a single-arm Franka robot across home and tabletop environments.The average trajectory length is 112 timesteps, and collection involved five full-time operators over two months.

4 Training Recipe

MolmoAct uses staged training that combines action-reasoning, auxiliary robot, and multimodal web data, then mid-trains on its dedicated dataset and post-trains for new tasks and embodiments. The LIBERO benchmark reports an 86.6% overall average success rate.

  • Training Stages: MolmoAct’s training recipe has three stages: pre-training, mid-training on the MolmoAct Dataset, and post-training for downstream tasks and embodiments.
  • Pre-training: Pre-training combines action-reasoning data, auxiliary robot data, and multimodal web data, using a total mixture of up to 26.3M samples.
  • Pre-training: Pre-training uses 256 H100s for 100k gradient steps with batch size 512, requiring around 9,728 GPU hours.
  • Evaluation: 86.6% is MolmoAct’s highest overall average success rate on LIBERO across Spatial, Object, Goal, and Long-horizon categories.
  • Mid-training: Mid-training formulates the MolmoAct Dataset into 1M action-reasoning samples and 1M trajectory-conditioned action samples.The additional trajectory-conditioned data is reported as beneficial for overall performance and action steering.

5 Experimental Evaluation

MolmoAct is evaluated across zero-shot simulation, post-training adaptation, out-of-distribution generalization, dataset ablations, instruction following, and interactive steerability. Across these settings, it consistently surpasses strong baselines in simulation and real-world task progression.

  • 5.1 MolmoAct After Pre-training: 70.5% success rate was achieved zero-shot by MolmoAct-7B-D-Pretrain on SimplerEnv visual matching, surpassing GR00T N1.5, π0, π0-FAST, and Magma.Fine-tuning on the same RT-1 subset increased performance to 71.6%, exceeding Magma by 3.2%.
  • 5.2 Fast Adaptation of MolmoAct in Post-training: 86.6% average success rate was achieved on LIBERO, with MolmoAct-7B-D exceeding ThinkAct by 6.3% on the long-horizon LIBERO-Long suite.The evaluation used LoRA fine-tuning and included simulation and real-world embodiment tests.
  • 5.4 Effect of the MolmoAct Dataset on MolmoAct Performance: 5.5% average improvement across three real-world tasks resulted from mid-training with the MolmoAct Dataset compared with the counterpart without mid-training.Without mid-training, MolmoAct-7B-D-Pretrain still surpassed π0-FAST by 14.8% and OpenVLA by 10.9%.
  • 5.5 Instruction Following of MolmoAct: MolmoAct-7B-D-Pretrain achieved the highest Elo rating in simulation instruction following and also outperformed all baselines in visual-trace evaluation.It won 58% of pairwise comparisons against SpatialVLA and 81% against OpenVLA.
  • 5.6 Steerability of MolmoAct: 75% success rate was achieved when MolmoAct-7B-D was steered with visual traces, outperforming open-ended language steering by 33% and π0-FAST by 29%.The steering evaluation used ambiguous instructions, visual trace sketches, and interactive natural-language commands.

6 Related Work

Related work has improved robotic manipulation through broad datasets, VLM backbones, hierarchical reasoning, and language-conditioned planning. MolmoAct distinguishes its approach by grounding reasoning directly in spatial scene representations that support precise, visualizable manipulation and steering.

  • 6 Related Work: Large, diverse robotic datasets have shifted manipulation research beyond narrow single-task specialists toward policies spanning scenes, tasks, and embodiments.VLAs commonly pretrain vision-language backbones on web data before fine-tuning action heads for robot control.
  • 6.2 Robot reasoning and planning with language: High-level reasoning has been integrated into robotic policies to improve long-horizon performance and generalization, often with separate low-level modules executing the resulting plans.Approaches differ in whether reasoning is supplied by LLMs, VLMs, subtasks, visual subgoals, or other intermediate representations.
  • 6.2 Robot reasoning and planning with language: MolmoAct supports steering through natural language and an interactive visual reasoning-trace sketch interface, improving explainability and behavioral diagnosis.This provides a dual-modality control interface rather than relying solely on a fixed action policy.
  • 6.2 Robot reasoning and planning with language: Compared with Emma-X and prior steering methods, MolmoAct uses fuller 3D scene context and generalizes steering to novel spatial configurations, unseen objects, and ambiguous instructions.Prior methods described in the passage may use predicted gripper positions, 2D trajectories, or architecture-specific steering mechanisms.
  • 6.3 Embodied reasoning for robotic manipulation: Unlike methods with latent, textual, or subgoal reasoning, MolmoAct grounds each reasoning step directly in the scene for manipulation-oriented precision.Its spatial reasoning can be decoded and visualized on the image plane and within the 3D environment.

7 Conclusion

MolmoAct integrates perception, planning, and control by reasoning in space, producing spatially coherent and steerable robot behavior. Its architecture combines multi-scale visual encoding with a language-model backbone and structured intermediate representations.

  • 7 Conclusion: MolmoAct integrates perception, planning, and control through depth perception tokens, visual reasoning traces, and action prediction.The resulting behaviors are explainable, spatially coherent, directly executable, and steerable through trajectory editing.
  • Model Architecture: The architecture combines preprocessing, a ViT image encoder, a vision–language connector, and an autoregressive LLM.The preprocessor creates low- and high-resolution crops, while the connector maps pooled patch features into the LLM embedding space.
  • Model Variants: MolmoAct instantiates model variants by pairing different vision encoders with either OLMo-2-1124-7B or Qwen2.5-7B.MolmoAct-7B-D uses SigLIP2 with Qwen2.5-7B, while MolmoAct-7B-O uses OpenAI CLIP with OLMo-2-1124-7B.
  • Image Encoding and Cropping: Multi-scale cropping and overlapping grids preserve fine-grained image details and border context for visual encoding.Overlapping pixels are excluded from forwarded tokens, and the approach empirically improves performance.
  • Feature Aggregation: Attention pooling in 2 × 2 patch windows reduces sequence length while preserving local spatial structure.Features from selected ViT layers are concatenated before pooling and projection into the LLM embedding space.

B Training Details

The training pipeline combines distributed implementation details, depth-token initialization, action-token discretization, and evaluations in simulation benchmarks. It uses multiple model configurations and records the associated architecture and training settings.

  • Training Implementation: MolmoAct training uses PyTorch, FSDP, scaled dot-product attention, and mixed-precision computation with bfloat16.Layer normalization and rotary position embeddings are computed in fp32.
  • Training Implementation: Global loss normalization avoids bias when weighted examples contain different numbers of loss tokens across devices.This correction is especially important when the global batch is much larger than device-local batches.
  • Parameter-Efficient Fine-Tuning: LoRA adapter parameters are replicated across GPUs and synchronized with gradient hooks rather than sharded under FSDP.Because LoRA adds few parameters, this has negligible memory and communication overhead.
  • Depth Token Training: Depth perception tokens are introduced by replacing tokenizer padding tokens and unfreezing the tokenizer embedding and language-model head.MolmoAct-7B-O first pads its tokenizer and language-model head to a multiple of 512 because OLMo2 has fewer than 130 padding tokens.
  • Simulation Evaluation: The evaluation covers SimplerEnv Visual Matching and Variant Aggregation, plus the four LIBERO task suites.LIBERO uses a Franka Panda arm, front and wrist views, language instructions, and delta end-effector pose actions.

D.3 Evaluation on Real-world Post-training

The real-world post-training evaluation measures fine-tuning efficiency on six manipulation tasks across single-arm and bimanual Franka setups. Models are trained to convergence and assessed with task-progression scores.

  • Evaluation Setup: The evaluation includes three single-arm tasks and three bimanual tasks using Franka robot setups.Single-arm tasks are put bowl in sink, wipe table, and table bussing; bimanual tasks are set table, lift tray, and fold towel.
  • Evaluation Setup: The single-arm Franka is mounted on a movable platform, whereas the bimanual setup is fixed to a tabletop.Sensing uses an Intel RealSense D405 wrist camera and D435 front-facing camera.
  • Baselines: MolmoAct is benchmarked against OpenVLA and π0-FAST with each model trained until convergence.Baseline implementations use their official training procedures and default configurations.
  • Progression Metrics: Task progression scores decompose each task into staged milestones such as grasping, moving, opening or closing the gripper, and completing placement.The six task descriptions specify milestone scores ranging from partial progress to 1.

D.4 Evaluation on Generalization in Real-world

Real-world generalization is evaluated across multiple object-placement tasks under in-distribution and perturbed conditions. The perturbations vary language, object positions, distractors, and object identities.

  • Evaluation Tasks: The generalization evaluation uses three object-placement tasks involving cans, cups, bananas, and colored plates.All models are trained jointly on a multi-task set containing the task permutations.
  • Generalization Conditions: Performance is evaluated in one in-distribution setting and four perturbation settings.The perturbations are language variation, spatial variation, distractor objects, and novel objects.
  • Progression Metrics: Task progression measures moving toward the correct object, picking it up, moving toward the correct plate, and placing it there.The progression milestones receive scores of 0.25, 0.5, 0.75, and 1.
  • Evaluation Procedure: The evaluation compares fully converged baseline models using their official implementations and default configurations.The full evaluation details are reported in Table 21.

D.5 Evaluation on the Effect of MolmoAct Dataset for MolmoAct Mid-training

The evaluation measures MolmoAct’s mid-training contribution through repeatable real-world tasks, open-ended language following, and visual-trace steering. Comparisons use converged baseline implementations with documented task-progression criteria.

  • MolmoAct Dataset evaluation: Three real-world tasks—close_lid, rotate_pot, and pour_tea—each use 50 demonstrations and 10 evaluation trials per model.Object locations were pre-marked to improve repeatability across MolmoAct, MolmoAct without its dataset, OpenVLA, and π0-FAST.
  • MolmoAct Dataset evaluation: Task progression is scored at staged milestones, including closing the lid, rotating the pot to 90 degrees, and pouring tea into a cup.The task-specific progression scales are defined for close_lid, rotate_pot, and pour_tea.
  • Language instruction following: Language-following evaluation uses 29 filtered open-ended prompts from five customized SimplerEnv scenes, producing 200-step rollouts rated by 100 participants.The rollouts compare MolmoAct, OpenVLA, and SpatialVLA in an arena-style interface.
  • Trajectory steering: Visual-trace steering is evaluated on pick_up_bowl, where users modify instructions or sketch five image points to guide action generation.Progression measures target direction, grasping the correct bowl, and lifting the bowl.
  • Evaluation protocol: Training data includes regular demonstrations and alternative paths toward the same target, while baseline models use official default configurations and are fully converged.The alternative-path collection is intended to condition behavior on language while preserving task completion.

E Data Details

The MolmoAct Dataset combines two external camera views with one wrist camera view, using environment-dependent camera configurations and ranked external views.

  • Camera configuration: Each dataset example contains two external camera views and one wrist camera view.Home-environment camera configurations may vary across tasks, whereas tabletop configurations remain fixed.
  • Camera configuration: External views are ranked by scene clarity and whether the robot occludes the scene, then labeled primary and secondary cameras.The ranking prioritizes visibility of the robot and objects during task execution.

F Data Examples

The data examples cover action reasoning, auxiliary perception and trace supervision, multimodal web data, mid-training robot data, and post-training demonstrations. Prompts and visual reasoning traces are illustrated across these sources.

  • Pre-training data: Pre-training examples include Action Reasoning Data, Auxiliary Visual Reasoning Trace data, Auxiliary Depth Perception Tokens data, and Trajectory-conditioned Action Data.These sources illustrate reasoning traces, depth tokens, and trajectory-conditioned supervision.
  • Pre-training data: Multimodal Web Data is also shown among the randomly selected examples used during pre-training.The examples are presented alongside the robotics-specific pre-training sources.
  • Mid-training data: Mid-training examples come from the MolmoAct Dataset’s Home and Tabletop environments.These examples are listed separately from the pre-training and post-training sources.
  • Post-training data: Post-training examples include Single Arm Franka, Bimanual Franka, and Rainbow demonstrations.Prompts are shown in bold, and visual reasoning traces are annotated with yellow lines.

G Limitations and Potential Solutions

The limitations concern camera visibility, trace quality and coverage, two-dimensional steering, inference speed, and depth-token resolution. The authors pair these constraints with practical directions including wider-field cameras, richer data, 3D traces, faster inference, and more depth tokens.

  • Camera occlusion: Occluding the end-effector in the front camera can degrade visual trace prediction and overall performance.The model processes multiple views but relies primarily on the front camera for spatial reasoning; fisheye cameras and SLAM are proposed.
  • Steerability robustness: Robust steerability depends on precise, diverse 2D traces and abundant, high-quality post-training data spanning many action compositions.The authors recommend inducing motion variants while still completing tasks so image-space traces correspond to resulting actions.
  • Trace quality and diversity: Bounding-box detectors can collapse trace points toward box centers, require task-specific fine-tuning, and transfer poorly across embodiments.VLM-based point annotations are described as producing more accurate, non-degenerate traces and improving steerability.
  • Trace dimensionality: 2D trace steering lacks explicit depth, causing unintended or imprecise out-of-plane translation during more complicated tasks.The authors hypothesize that reusing predicted depth-perception tokens could lift traces into 3D.
  • Inference speed: A mismatch between inference and data-collection control frequencies can arise from communication latency and predicting more reasoning tokens.Future directions include VLM optimization and smaller models for edge or local execution.
  • Depth precision: The fixed set of 100 depth tokens may be insufficient for fine-grained manipulation requiring higher-resolution depth estimation.Increasing the number of depth tokens could improve spatial reasoning and task performance.
Loading 2508.07917v4…