PaperlayerView exact revision on arXiv

Source-linked AI summary

MolmoAct2: Action Reasoning Models for Real-world Deployment

Haoquan Fang, Jiafei Duan, Donovan Clay, Sam Wang, Shuo Liu, Weikai Huang, Xiang Fan, Wei-Chuan Tsai, Shirui Chen, Yi Ru Wang, Shanli Xing, Jaemin Cho, Jae Sung Park, Ainaz Eftekhar, Peter Sushko, Karen Farley, Angad Wadhwa, Cole Harrison, Winson Han, Ying-Chun Lee, Eli VanderBilt, Rose Hendrix, Suveen Ellawela, Lucas Ngoo, Joyce Chai, Zhongzheng Ren, Ali Farhadi, Dieter Fox, Ranjay Krishna

arXiv:2605.02881v2cs.RO

TL;DR

Real-world deployment of VLA models remains limited by incomplete spatial reasoning, slow inference, fragmented data, and restricted openness. MolmoAct2 addresses these gaps with an open embodied-reasoning backbone, new multi-embodiment datasets, redesigned action modeling, and adaptive reasoning, outperforming strong baselines across seven simulation and real-world benchmarks.

  • Problem

    VLA models remain limited by incomplete spatial representations, heavyweight reasoning that impedes real-time interaction, and restricted openness for adaptation.

  • Method

    MolmoAct2 combines a spatially specialized VLM, open multi-embodiment datasets, a redesigned action architecture, and adaptive depth reasoning for efficient inference.

  • Results

    Across 7 simulation and real-world benchmarks, MolmoAct2 outperforms every strong baseline, while Molmo2-ER surpasses GPT-5 and Gemini Robotics Embodied Reasoning models.

  • Takeaways & Limitations

    The results support MolmoAct2 as an open VLA candidate for practical deployment across simulation and real-world robot tasks.

  • Takeaways & Limitations

    The post-training recipe uses K = 8 only during fine-tuning because GPU memory constraints prevented its use elsewhere.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models aim to provide a single generalist controller for robots, but today's systems fall short on the criteria that matter for real-world deployment. Frontier models are closed, open-weight alternatives are tied to expensive hardware, reasoning-augmented policies pay prohibitive latency for their grounding, and fine-tuned success rates remain below the threshold for dependable use. We present MolmoAct2, a fully open action reasoning model built for practical deployment, advancing its predecessor along five axes. We introduce MolmoER, a VLM backbone specialized for spatial and embodied reasoning, trained on a 3.3M-sample corpus with a specialize-then-rehearse recipe. We release three new datasets spanning low-to-medium cost platforms, including MolmoAct2-BimanualYAM, 720 hours of teleoperated bimanual trajectories that constitute the largest open bimanual dataset to date, together with quality-filtered Franka (DROID) and SO100/101 subsets. We provide OpenFAST, an open-weight, open-data action tokenizer trained on millions of trajectories across five embodiments. We redesign the architecture to graft a flow-matching continuous-action expert onto a discrete-token VLM via per-layer KV-cache conditioning. Finally, we propose MolmoThink, an adaptive-depth reasoning variant that re-predicts depth tokens only for scene regions that change between timesteps, retaining geometric grounding at a fraction of prior latency. In the most extensive empirical study of any open VLA to date, spanning 7 simulation and real-world benchmarks, MolmoAct2 outperforms strong baselines including Pi-05, while MolmoER surpasses GPT-5 and Gemini Robotics ER-1.5 across 13 embodied-reasoning benchmarks. We release model weights, training code, and complete training data. Project page: https://allenai.org/blog/molmoact2

1 Introduction

MolmoAct2 is presented as a fully open action-reasoning model designed for practical real-world deployment, addressing limitations in reasoning latency, openness, embodiment coverage, and control. It advances MolmoAct through five contributions spanning embodied reasoning, datasets, action tokenization, architecture, and adaptive inference, with broad simulation and real-world evaluation.

  • Five Contributions: MolmoAct2 improves over MolmoAct along five axes: embodied-reasoning VLM, open datasets, multi-embodiment tokenizer, VLA architecture, and adaptive reasoning.These contributions target stronger reasoning, broader training coverage, reproducible action learning, continuous control, and efficient inference.
  • Embodied Reasoning: 3.3M samples train Molmo2-ER for spatial and embodied reasoning using a specialize-then-rehearse recipe.The model targets metric distances, free space, cross-view object tracking, and scene geometry.
  • Open Training Data: 720 hours of teleoperated trajectories make MolmoAct2-BimanualYAM the largest open bimanual dataset to date.The dataset spans tabletop and household tasks on the YAM platform.
  • Action Tokenization: MolmoAct2-FAST Tokenizer is an open-weight, open-data FAST implementation trained and released with millions of trajectories across five embodiments.It supports reproducible discrete-action VLA training across embodiments.
  • VLA Architecture: The new architecture conditions each continuous action-expert layer on keys and values from the corresponding VLM layer.This connects discrete reasoning grounding with the smooth, continuous, high-frequency trajectories required for robot control.
  • Evaluation: 7 environment benchmarks and 8 real-world YAM tasks underpin an evaluation in which MolmoAct2-Think outperforms every strong baseline across simulation and real-world settings.Ablations show further gains from MolmoAct2-Think, alongside added interpretability for diagnosis and performance.

2 Molmo2-ER

Molmo2-ER is a VLM backbone specialized for embodied reasoning and the spatial, metric, temporal, and viewpoint-grounded skills needed for robot control. It is trained on a 3.3M-sample embodied corpus using a two-stage specialize-then-rehearse recipe that preserves general language and multimodal capabilities.

  • Model objective: Molmo2-ER specializes Molmo2 for embodied perception skills including scene understanding, pixel-accurate pointing, multi-image and egocentric reasoning, exocentric correspondence, and video reasoning.These skills support downstream action reasoning for robot control.
  • Training corpus: 3.3M samples span six capability pillars: embodied QA, pointing, detection, video QA, multi-image and ego–exo reasoning, and abstract embodied reasoning.The corpus combines complementary datasets and supervision sources, including simulator ground truth, annotated real scans, and template-generated data.
  • Training recipe: 20K steps of embodied specialization start from the Molmo2–4B mid-training checkpoint, add 8% Tulu-3 text-only data, and use sequence length 4,200 with global batch size 64.The text-only mixture is used to preserve language competence while moving the model onto the embodied data manifold.
  • Training recipe: 1.5K additional steps jointly refine the model by interleaving embodied data with Molmo2’s original multimodal mid-training data while holding the NLP rate at 8%.The embodied/general allocation is swept over p ∈{0.30, 0.50, 0.70, 0.90}, with each side’s internal proportions preserved.

3 Data

MolmoAct2 addresses gaps in public robotics data by combining a new diverse bimanual dataset with curated SO-100/101 and filtered DROID corpora. The resulting mixture emphasizes scale, quality, embodiment diversity, and realistic variation across tasks, objects, scenes, and environments.

  • Data gaps: Public corpora lacked high-quality bimanual manipulation data for the deployment platform and often had limited scene, object, and spatial variation.These gaps motivate MolmoAct2’s targeted data collection and curation strategy.
  • Training mixture: MolmoAct2’s training mixture combines a new BimanualYAM dataset with curated SO-100/101 data and filtered DROID data.These three sources are designed to complement one another across scale, quality, and embodiment coverage.
  • MolmoAct2-BimanualYAM: The open-source BimanualYAM dataset emphasizes repeatability and diversity across household, factory, and coffee-shop manipulation behaviors.It is collected on a custom bimanual YAM setup and is designed for both scale and quality.
  • MolmoAct2-SO100/101: 1,222 public LeRobot datasets from 377 users yield 38,059 episodes, 19.8M frames, and approximately 184 hours of SO-100/101 interaction data.The curated corpus spans both embodiments, multiple camera configurations, varied manipulation tasks, and diverse real-world environments.
  • MolmoAct2-DROID: DROID is filtered using supplementary annotations from its accompanying HuggingFace repository to improve the quality of MolmoAct2’s Franka training data.The source dataset covers in-the-wild manipulation across diverse real-world deployment scenarios with a unified Franka setup.

4 MolmoAct2

MolmoAct2 adapts a pretrained vision-language backbone into a discrete autoregressive robot policy, then conditions a continuous flow-matching action expert on visual-language representations at every layer. Its training formulation preserves the Molmo2 token interface while representing heterogeneous robot trajectories with an open, compact action tokenizer.

  • Pre-training: MolmoAct2-Pretrain extends the Molmo2 token interface with state tokens for robot configuration and action tokens for future one-second motion.Images and video frames remain ViT-encoded, connector-projected, and passed to the LLM with text.
  • FAST Tokenizer: A 2048-token action vocabulary compactly represents continuous, embodiment-specific trajectories after frequency-domain transformation, coefficient quantization, and byte-pair encoding.Each trajectory covers one second, while control frequency determines the number of actions in its chunk.
  • FAST Tokenizer: One million action sequences across five embodiments train MolmoAct2-FAST Tokenizer from a transparent action mixture.The tokenizer maps one-second trajectories into discrete tokens using a frequency-domain transform, coefficient quantization, and byte-pair encoding with a 2048-token action vocabulary.
  • Post-training architecture: The post-training action expert cross-attends to corresponding VLM-layer keys and values, providing per-layer KV conditioning instead of relying on a single final hidden-state projection.Lightweight learned projections map VLM attention states into the expert’s cross-attention width.

5 MolmoAct2-Think

MolmoAct2-Think extends MolmoAct2 with lightweight depth-token reasoning and adaptive temporal updates, predicting only scene regions whose RGB evidence changes while conditioning action generation on the resulting depth state. It preserves the existing action interface and trains the depth-aware behavior through action, depth, and joint depth-and-action objectives.

  • Motivation: MolmoAct2-Think makes spatial structure explicit before acting by adding depth-token prediction to MolmoAct2.The motivation is that object distance, free space, occlusion, and surface layout affect actions but are only indirectly supervised by imitation.
  • Depth representation: Each depth map becomes 100 autoregressive tokens from a 128-value codebook arranged on a 10 × 10 spatial grid.This lightweight representation uses ordinary next-token prediction while exposing an interpretable intermediate depth estimate.
  • Adaptive depth reasoning: MolmoAct2-Think reuses cached depth codes for static regions and predicts only cells whose RGB evidence changes between control steps.This adaptive design targets temporal redundancy in scene-level depth grids rather than regenerating every depth code at every step.
  • Training: During post-training, robot data uniformly samples action prediction, depth prediction, and depth-and-action prediction tasks.In the joint task, the model predicts 100 depth-buffer tokens before discrete actions, while the action expert uses the predicted depth state for continuous control.
  • Inference: At inference, unchanged depth cells are replayed from the previous buffer, changed cells are decoded, and the completed depth prefix conditions continuous action generation.The action expert receives VLM keys and values for the prompt plus filled depth prefix, so adaptive depth changes only intermediate-prefix production.

6 Experiments

MolmoAct2 is evaluated across embodied reasoning, zero-shot deployment, fine-tuning, robustness, and trajectory quality in simulation and real-world settings. It achieves leading results across these evaluations, while Molmo2-ER and MolmoAct2-Think provide additional gains in reasoning and task performance.

  • Embodied reasoning: 63.8% overall average: Molmo2-ER outperforms all baseline VLMs on 9 of 13 embodied-reasoning benchmarks, exceeding Gemini-ER 1.5 Thinking by 2.5 points.Molmo2-ER also improves over its base model Molmo2 by 17%.
  • Zero-shot deployment: +10.6% on MolmoBot and +3.2% on MolmoSpaces: MolmoAct2-DROID outperforms all baselines across both simulation benchmarks.π0.5-DROID is the runner-up on each benchmark, and trial counts are sufficient for statistical significance.
  • Fine-tuned performance: 97.2% average success on LIBERO: MolmoAct2 leads all compared methods, reaches 100% on LIBERO-Object, and improves over MolmoAct-7B-D by 10.6%.It also achieves 44.3% on RoboEval and 50.1% across real-world deployment tasks, outperforming all baselines on 7 of 8 tasks.
  • Adaptive-depth reasoning: 98.1% versus 97.2%: MolmoAct2-Think improves average LIBERO success by +0.9% across 2,000 rollouts.Its largest gain is +2.2% on the most challenging suite, while it matches the baseline on the saturated 100% suite.
  • Robustness: 50.7% average success across perturbation types: MolmoAct2 leads all evaluated models by approximately 10.8% over OpenVLA-OFT.Its lowest absolute score is 26.25% on Spatial Variance, indicating remaining room for fine-grained spatial generalization.
  • Trajectory quality: 4.70s completion time on Stack Two Blocks: MolmoAct2 is faster than π0.5 at 5.87s and Diffusion at 7.27s, while operating near the best normalized stability values.The evaluation measures efficiency, behavioral stability, and outcome quality beyond task success on RoboEval.

K Spatial Object Goal Long Average

MolmoAct2’s full-model fine-tuning with discrete and continuous action co-training achieves the best average, while caching and CUDA Graphs substantially improve continuous control rates. The discrete action path remains slower because of VLM decoding overhead.

  • Fine-tuning design: Full-model adaptation with discrete and continuous action co-training provides the best average, while action-expert-only tuning drops the average to 93.05%.LoRA remains strong on Spatial but loses 2.8 points on Long relative to full fine-tuning.
  • Depth-aware fine-tuning: Removing depth-token noise and the learned per-layer depth gate reduces the average from 98.10% to 97.65%, mostly through a 1.8-point Goal drop.The baseline combines 10% depth-token noise, a learned per-layer depth gate, and uniformly mixed action-only and depth-and-action examples.
  • Inference latency: 12.71 Hz: CUDA Graph replay raises MolmoAct2-Think’s control rate, a 1.58× speedup over its original path.Caching alone improves MolmoAct2-Think from 8.04 Hz to 9.72 Hz.
  • Inference latency: 14.17 Hz: the discrete MolmoAct2 action path runs 3.94× slower than its corresponding continuous path because of VLM decoding overhead.MolmoAct2-Think runs at 6.82 Hz on the same optimized graph, 1.86× slower than its continuous path.

7 Related work

Related work has shifted toward generalist robot manipulation policies and embodied reasoning, while VLA research continues to explore interfaces between vision-language models and action generators. Existing reasoning methods often incur substantial token and latency costs, motivating adaptive-depth approaches.

  • Generalist robot manipulation policies: Robotic manipulation research has shifted from task-specific designs toward generalist policies operating across diverse scenarios and embodiments.
  • Embodied reasoning for robotic manipulation: Embodied-reasoning methods use visual or spatial representations, future image predictions, latent reasoning, and 2D/3D point trajectories to support robotic manipulation.
  • Embodied reasoning for robotic manipulation: Existing embodied-reasoning methods commonly consume many tokens before action generation, severely degrading inference latency.
  • Bridging VLM and action expert: VLA architectures use either fully discretized actions via vector quantization or continuous generative action experts trained with diffusion or flow matching.

8 Conclusion … A.3 Adaptive-Depth Extension

MolmoAct2 is presented as a fully open action-reasoning family for heterogeneous robots, combining geometrically grounded control with adaptive-depth reasoning and an architecture spanning discrete VLM outputs and continuous action prediction.

  • 8 Conclusion: MolmoAct2 is a fully open action-reasoning model family designed for real-world deployment across heterogeneous robot platforms.The conclusion emphasizes performant, geometrically grounded behavior across diverse manipulation tasks.
  • 8 Conclusion: MolmoAct2-Think adds adaptive-depth reasoning to provide interpretable, depth-aware control with efficient inference.The thinking variant retains the MolmoAct2 backbone and action expert while adding an autoregressive depth-token interface.
  • Appendix: The appendix documents model, training, evaluation, dataset, and limitations details in sections A–E.These sections cover model details, training details, evaluation details, dataset details, and limitations with potential solutions.
  • A Model Details: MolmoAct2 uses two architectural stages: autoregressive pretraining adapts Molmo2-ER with robot interfaces, followed by post-training with a continuous action expert.Robot-specific state, setup/control, and action-output tokens allow trajectories to use the same next-token objective as vision-language data.
  • A.1 MolmoAct2 Backbone: The backbone combines SigLIP2 visual encoding, a Molmo2 connector, autoregressive language modeling, robot descriptors, 256 state tokens, and 2048 action tokens.Setup and control descriptors make embodiment and action semantics explicit, allowing one tokenizer and backbone to support different robots.
  • A.2 Continuous Action Expert: The continuous action expert predicts normalized action chunks with flow matching through 36 transformer blocks and per-layer KV conditioning from the VLM.The released architecture maps VLM attention states into the expert’s cross-attention width, while the released checkpoints use N = 10 inference steps.
  • A.3 Adaptive-Depth Extension: MolmoAct2-Think serializes a 10 × 10 depth-token grid with 128 learned depth-code values before continuous action prediction.The LIBERO depth fine-tuned model additionally applies a learned gate only to depth-token keys and values, initialized with bias −4.

B Training Details … B.3 GPU Cluster

MolmoAct2 uses a packed, mixed-precision PyTorch training pipeline with standardized robot prompting and augmentation, an open-data FAST tokenizer supporting heterogeneous embodiments, and the Jupiter GPU cluster for distributed training.

  • B.1 Implementation: MolmoAct2 trains with PyTorch FSDP2, scaled dot-product attention, torch.compile, static padded shapes, and custom masks for packed multimodal robot-conditioning sequences.bfloat16 is used for most operations, while layer normalization and RoPE remain in full precision.
  • B.1 Implementation: Token losses are normalized by the cross-device average loss-token count, preventing short robot-action targets from being overweighted relative to longer language or video examples.Gradients are computed on local minibatches and averaged across devices.
  • B.1 Implementation: Dynamic packing selects examples from pools of M = 48 to maximize utilization, with Molmo2 budgets of Tmax = 16384 and Imax = 128 or long-context budgets of Tmax = 36864 and Imax = 384.MolmoAct2 uses the same solver with stage-specific sequence budgets.
  • B.1 Implementation: Robot training applies stochastic image augmentation to images and videos, while evaluation and inference disable augmentation.The recipe includes a 5% random crop, resizing, 5-degree rotation, color jitter, and Gaussian blur with probability 0.2.
  • B.1 Implementation: Standard MolmoAct2 uses action prompts, whereas MolmoAct2-Think supports action, depth, and depth-then-action output styles through shared chat formatting and output triggers.The assistant-side triggers are <action_output>, <depth_output>, and <depth_output><action_output>.
  • B.2 MolmoAct2-FAST Tokenizer: The MolmoAct2-FAST Tokenizer is an open-data DCT-BPE implementation whose transparent training mix addresses the data opacity of prior open-weight releases.It follows the core frequency-domain compression logic of the FAST framework.
  • B.2 MolmoAct2-FAST Tokenizer: A standardized 32-dimensional action vector accommodates single-arm and bimanual morphologies, while training spans absolute-joint and delta-end-effector action dialects.The VLM associates each representation with the embodiment specified in the task prompt or visual context.
  • B.3 GPU Cluster: MolmoAct2 was trained on Jupiter, an Ai2 GPU cluster in Austin operated through Beaker and Cirrascale Cloud Services.Jupiter provides 1,024 NVIDIA H100 GPUs across 128 servers, with high-throughput WEKA storage and RDMA over InfiniBand interconnects.

C Evaluation Details … D Data Details

The evaluation details span joint multi-task RoboEval training and testing, real-world DROID, Bimanual YAM, and SO-100/101 evaluations under controlled spatial or camera configurations. The supplied passages provide no substantive details for the listed D Data Details section.

  • C.1 RoboEval: MolmoAct2-Post is fine-tuned jointly on all eight RoboEval tasks for 10,000 steps with 8 GPUs and a global batch size of 64.Demonstrations from every task and variation are pooled into one training distribution.
  • D Data Details: The supplied passages contain no substantive evidence describing the listed D Data Details section.No passage in the input is labeled with subsection D or provides dataset-specific details beyond the evaluation procedures above.
  • C.1 RoboEval: RoboEval evaluates eight bimanual manipulation tasks across 3–5 structured variations, using Static, Position, Orientation, and combined Position+Rotation configurations.The tasks include Cube Handover, Lift Pot, Lift Tray, Pack Box, Pick Single Book From Table, Rotate Valve, Stack Single Book Shelf, and Stack Two Blocks.
  • C.2 Real-world Zero-shot DROID: DROID evaluation uses a kitchen setup with five tasks and three spatial variants, while keeping camera positions fixed across MolmoAct2, π0, and MolmoBot.The evaluation results are reported in Table 18, with sample trajectories shown in Figure 9.
  • C.3 Real-world Bimanual YAM: Bimanual YAM trains MolmoAct2-Post as eight separate single-task policies and applies the identical protocol to four baselines, with all policies trained to convergence.Each task is collected and evaluated under the same three spatial variants.
  • C.4 Real-world SO-100/101 zero-shot: SO-100 evaluation is zero-shot from a fixed, pre-initialized camera viewpoint, comparing MolmoAct2 with DePi and SmolVLA without additional policy training.Episodes use partial-credit scoring; the supplied table specifies milestone scores of 0.25, 0.5, and 1.0 across 15 episodes per task.
  • C.4 Real-world SO-100/101 zero-shot: Table 17 reports mean ± 95% confidence intervals for performance metrics across methods on the combined variation, alongside human demonstration statistics from successful teleoperated demonstrations.Metrics include Success, TP, BAVD, BGVD, CPL, CT, ECC, JPL, and MCJ.

D.1 MolmoAct2-SO100/101 Dataset Statistics

The MolmoAct2-SO100/101 training mixture starts with 1,660 dataset entries and applies TOPReward filtering to produce a final set of 1,222 datasets, whose statistics are summarized across training, robot variants, and metadata.

  • Dataset construction: 1,660 entries were collected, and TOPReward filtering removed 438 low-quality entries, yielding 1,222 final datasets.All subsequent statistics are computed only over this final filtered set.
  • Dataset statistics: The final filtered set is summarized through training-partition statistics, SO-10x robot-type counts, and metadata distributions.The metadata include FPS, camera configuration, and action/state dimensions.

D.2 Language Annotation Pipeline Implementation Details

The language annotation pipeline prompts Qwen3.5-27B to generate imperative robot-arm instructions from multi-view demonstration videos, using controlled inference and temporally sampled frames. It increases the number of unique instructions across MolmoAct2’s robotics training datasets.

  • Inference settings: Inference uses temperature 0.1 and a maximum length of 1024 tokens.
  • Input sampling: For each non-frozen camera, the pipeline samples 12 equally temporally spaced frames alongside the text prompt.The instruction-length word_limit is randomly sampled from a right-tailed power distribution between 5 and 25.
  • Annotation diversity: The pipeline increases the number of unique instructions for all robotics datasets in MolmoAct2’s training mix.The resulting changes are reported in Table 24.

E Limitations and Potential Solutions

MolmoAct2 has deployment limitations around fixed-horizon, open-loop action chunks and embodiment-specific zero-shot use. Independently generated chunks can create boundary discontinuities, while out-of-the-box deployment is currently limited to three robot platforms with large-scale training data.

  • Action chunks and the absence of real-time re-chunking: MolmoAct2 predicts fixed-horizon action chunks and executes them open-loop before re-querying the policy.Chunk lengths are 30 steps at 30 Hz for YAM/SO-100/101, 15 at 15 Hz for DROID, and 10 at 10 Hz for LIBERO.
  • Action chunks and the absence of real-time re-chunking: Independent chunk denoising lacks continuity loss, allowing visible velocity or acceleration discontinuities at chunk seams.The effect is especially apparent when the VLM context shifts between policy queries.
  • Embodiment-specific zero-shot deployment: Zero-shot deployment is currently tied to three embodiments with large-scale training data: bimanual YAM, SO-100/101, and DROID Franka.Each corresponding model can be deployed out-of-the-box on its respective robot setup.
Loading 2605.02881v2…