PaperlayerView exact revision on arXiv

Source-linked AI summary

MolmoAct2: Action Reasoning Models for Real-world Deployment

Haoquan Fang, Jiafei Duan, Donovan Clay, Sam Wang, Shuo Liu, Weikai Huang, Xiang Fan, Wei-Chuan Tsai, Shirui Chen, Yi Ru Wang, Shanli Xing, Jaemin Cho, Jae Sung Park, Ainaz Eftekhar, Peter Sushko, Karen Farley, Angad Wadhwa, Cole Harrison, Winson Han, Ying-Chun Lee, Eli VanderBilt, Rose Hendrix, Suveen Ellawela, Lucas Ngoo, Joyce Chai, Zhongzheng Ren, Ali Farhadi, Dieter Fox, Ranjay Krishna

arXiv:2605.02881v2cs.RO

TL;DR

Open VLAs lack structured spatial reasoning, efficient inference, and sufficiently open, reliable training resources for real-world deployment. MolmoAct2 addresses these gaps with an embodied-reasoning backbone, open multi-embodiment data and tokenizer, redesigned VLA architecture, and adaptive reasoning, outperforming strong baselines across seven simulation and real-world benchmarks.

  • Problem

    Open robot foundation models often lack structured spatial representations, incur heavyweight reasoning costs, and remain difficult to adapt because of limited openness.

    Source: §1 · p. 2

  • Method

    MolmoAct2 combines a spatially and embodied-trained VLM, new open action datasets, an open multi-embodiment tokenizer, a redesigned VLA architecture, and adaptive depth reasoning.

    Source: §1 · p. 3

  • Results

    Across 7 simulation and real-world environment benchmarks, MolmoAct2 with MolmoAct2-Think outperforms every strong baseline, while Molmo2-ER surpasses GPT-5 and Gemini Robotics models.

    Source: §1 · pp. 3–4

  • Takeaways & Limitations

    The results support assessing MolmoAct2 as an open VLA for real-world deployment, including out-of-the-box use of fine-tuned checkpoints on their respective embodiments.

    Source: §1 · p. 3

    The post-training recipe does not use K = 8 because of GPU memory constraints.

    Source: §4.2.2 · p. 13

Abstract

from arXiv · show

Vision-Language-Action (VLA) models aim to provide a single generalist controller for robots, but today's systems fall short on the criteria that matter for real-world deployment. Frontier models are closed, open-weight alternatives are tied to expensive hardware, reasoning-augmented policies pay prohibitive latency for their grounding, and fine-tuned success rates remain below the threshold for dependable use. We present MolmoAct2, a fully open action reasoning model built for practical deployment, advancing its predecessor along five axes. We introduce MolmoER, a VLM backbone specialized for spatial and embodied reasoning, trained on a 3.3M-sample corpus with a specialize-then-rehearse recipe. We release three new datasets spanning low-to-medium cost platforms, including MolmoAct2-BimanualYAM, 720 hours of teleoperated bimanual trajectories that constitute the largest open bimanual dataset to date, together with quality-filtered Franka (DROID) and SO100/101 subsets. We provide OpenFAST, an open-weight, open-data action tokenizer trained on millions of trajectories across five embodiments. We redesign the architecture to graft a flow-matching continuous-action expert onto a discrete-token VLM via per-layer KV-cache conditioning. Finally, we propose MolmoThink, an adaptive-depth reasoning variant that re-predicts depth tokens only for scene regions that change between timesteps, retaining geometric grounding at a fraction of prior latency. In the most extensive empirical study of any open VLA to date, spanning 7 simulation and real-world benchmarks, MolmoAct2 outperforms strong baselines including Pi-05, while MolmoER surpasses GPT-5 and Gemini Robotics ER-1.5 across 13 embodied-reasoning benchmarks. We release model weights, training code, and complete training data. Project page: https://allenai.org/blog/molmoact2

1 Introduction

MolmoAct2 is presented as a fully open action-reasoning model for practical real-world deployment, addressing limitations in spatial grounding, inference latency, adaptability, and dependable performance. It advances MolmoAct across five axes: embodied reasoning, datasets, action tokenization, VLA architecture, and adaptive inference.

Motivation

  • Existing robot foundation models lack structured spatial representations, incur prohibitive reasoning latency, and remain closed or difficult to adapt, limiting dependable real-world deployment.Reasoning can improve action quality and interpretability, but generating hundreds of tokens, predicted frames, or world-model rollouts before each action impedes closed-loop control.Source: §1 · p. 2

Openness and deployment

  • MolmoAct2 releases model weights, training code, and the complete training dataset, aiming to support deployment across multiple embodiments rather than only academic evaluation.The model is described as fully open, deployable out of the box on multiple embodiments, performant, and capable of fast, interpretable reasoning.Source: §1 · pp. 2, 4

Backbone and datasets

  • Molmo2-ER is a spatial- and embodied-reasoning VLM trained on a 3.3M-sample corpus with a specialize-then-rehearse recipe, while three datasets expand open multi-embodiment action training.The datasets include 720 hours of teleoperated bimanual YAM trajectories, quality-filtered SO100/101 and Franka DROID subsets, and re-annotated instructions for the filtered datasets.Source: §1 · p. 3

Action tokenization

  • MolmoAct2-FAST Tokenizer is an open-weight, open-data tokenizer trained on millions of trajectories across five embodiments, compressing one second of 32-D continuous actions into a compact discrete sequence.It addresses closed, embodiment-specific, or insufficiently documented tokenizers for reproducible discrete-action VLAs.Source: §1 · p. 3

VLA architecture

  • MolmoAct2 connects a discrete-token VLM to smooth continuous control through a flow-matching DiT-style action expert conditioned on corresponding VLM-layer keys and values.Per-layer KV-cache conditioning exposes the VLM attention state while preserving a similar compute profile.Source: §1 · p. 3

Adaptive reasoning

  • MolmoAct2-Think predicts tokens only for scene regions that change between timesteps, reducing latency with static-scene redundancy while retaining geometric grounding and improving interpretability.The adaptive-depth approach reduces latency proportional to the static scene fraction, and ablations report further gains over MolmoAct2.Source: §1 · pp. 3–4

Empirical results

  • Across 7 simulation and real-world benchmarks, MolmoAct2 with MolmoAct2-Think outperforms every strong baseline, while Molmo2-ER surpasses GPT-5 and Gemini Robotics ER-1.5 on 13 embodied-reasoning benchmarks.MolmoAct2-DROID and MolmoAct2-SO100/101 deploy out of the box on their respective embodiments and significantly surpass π0.5; evaluation also includes LIBERO, RoboEval, and 8 real-world YAM tasks.Source: §1 · pp. 3–4

2 Molmo2-ER

Molmo2-ER is a specialized embodied-reasoning VLM trained on a 3.3M-sample corpus with a two-stage specialize-then-rehearse recipe. It achieves 63.8% average accuracy, outperforming open-weight and leading closed-source baselines on 9 of 13 benchmarks.

2 Molmo2-ER

  • 63.8% overall average accuracy lets Molmo2-ER outperform every open-weight baseline and leading closed-source models on 9 of 13 embodied-reasoning benchmarks.It improves 17 points over Molmo2, including comparisons with Gemini Robot-ER 1.5 Thinking and GPT-5.Source: §2 · p. 4

2.1 Training data

  • Molmo2-ER is trained on approximately 3.3M samples spanning six pillars: embodied QA, pointing, detection, video reasoning, multi-image correspondence, and abstract reasoning.The corpus combines simulator ground truth, 3D-annotated scans, template-generated QA, and limited LLM-generated chain-of-thought across diverse spatial phenomena.Source: §2.1 · p. 4
  • The training data deliberately emphasizes pixel-accurate pointing, while covering temporal reasoning, long-horizon planning, affordances, future prediction, cross-view correspondence, and object-centric frame-of-reference reasoning.Key sources include RoboPoint’s 700K pointing targets, SIMS-VSI’s 203K temporal questions, RoboVQA’s 200K videos, SenseNova-SI’s 500K subset, and GRiD-3D’s 100K examples.Source: §2.1 · p. 5

2.2 Specialize-then-rehearse training recipe

  • A specialize-then-rehearse recipe first fine-tunes Molmo2 for 20K steps on embodied data, then jointly refines it for 1.5K steps with original multimodal data.Stage 2 uses p = 0.5 embodied-data mixing for the best Pareto trade-off between embodied-reasoning and general benchmarks, with sequence length increasing from 4,200 to 16,384.Source: §2.2 · p. 5

3 Data

MolmoAct2 assembles a deployment-oriented training mixture combining new bimanual data, filtered community and Franka corpora, academic robotics datasets, multimodal data, and re-annotated language instructions. The mixture addresses gaps in quality, embodiment coverage, task diversity, and realistic scene variation.

3 Data

  • MolmoAct2 combines BimanualYAM, curated SO-100/101 and DROID data, academic robotics datasets, multimodal data, and embodied-reasoning data.The mixture is designed to provide scale, quality, embodiment breadth, and broad visual-linguistic competence.Source: §3 · p. 6, Figure 2 · p. 7

3.1 MolmoAct2-BimanualYAM Dataset

  • Over 720 hours across 34.5k demonstrations and more than 28 tasks make BimanualYAM the core open bimanual dataset for varied real-world manipulation.Tasks span household, factory, and coffee-shop settings, with variation in scenes, objects, and placements; collection protocols limit failure retries and no-op segments.Source: §3.1 · p. 6

3.2 MolmoAct2-SO100/101 Dataset

  • 38,059 episodes from 1,222 datasets contributed by 377 users provide approximately 184 hours and 19.8M frames of SO-100/101 interaction data.Filtering preserves diversity across embodiments, camera configurations, tasks, environments, objects, and annotations.Source: §3.2 · pp. 6–7

3.3 MolmoAct2-DROID Dataset

  • 74,604 valid DROID episodes containing 17,758,044 frames are filtered to successful, instructed, non-paused trajectories.The pipeline uses extended language annotations, contiguous non-idle segments of at least one second, and subsequent language re-annotation.Source: §3.3 · p. 7

3.4 Language annotation pipeline

  • Re-annotating robotics demonstrations with Qwen3.5-27B increases unique labels from 71,121 (22%) to 146,485 (46%).The VLM uses sampled frames, the original instruction, and a randomly chosen approximate target length to improve instruction accuracy and diversity.Source: §3.4 · p. 8

3.5 Academic robotics datasets

  • A targeted Open X-Embodiment subset, including BC-Z, BridgeData V2, and RT-1, plus MolmoAct, broadens embodiments, camera layouts, task families, and control conventions.These academic datasets are complementary sources, while YAM, SO-100/101, and DROID provide most robot training examples.Source: §3.5 · p. 8

4 MolmoAct2

MolmoAct2 uses a staged pipeline that converts Molmo2-ER into a discrete robot policy, adds a continuously controlled flow-matching expert, and then adapts the shared architecture to individual embodiments. Its open tokenizer, per-layer KV conditioning, joint supervision, and inference optimizations target practical deployment across heterogeneous robots.

4.1 Pre-training

  • MolmoAct2 first adapts Molmo2-ER into a discrete autoregressive robot policy by extending the VLM sequence with state and future-action token streams.The model preserves the existing vision-language token interface while adding robot-specific state and action representations.Source: §4.1 · p. 9, §4 · p. 9

4.1.2 Pre-training recipe

  • Pre-training mixes multimodal data with robot trajectories under one next-token objective, using discrete action tokens, separately discretized state tokens, packed sequences, and no continuous action head.Robot data receives 90% of sampling, while multimodal data receives 10%, preserving general vision-language capability during robot-policy learning.Source: §4.1 · p. 9, §4.1.2 · pp. 10–11

4.1.1 MolmoAct2-FAST Tokenizer

  • The MolmoAct2-FAST Tokenizer encodes one-second, standardized trajectories into compact tokens using frequency transforms, quantization, and byte-pair encoding over a 2048-token vocabulary.It is trained transparently on one million action sequences spanning five embodiments and multiple control modes.Source: §4.1.1 · p. 10, Table 2 · p. 10

4.2 Post-training

  • Post-training converts the discrete policy into continuous control by attaching a DiT-style flow-matching action expert while retaining the pretrained VLM backbone.The VLM supplies visual-language and robot-state grounding, while the expert predicts continuous action trajectories from noisy inputs.Source: §4.2 · p. 12, §4.2.1 · p. 12

4.2.1 Action expert and per-layer KV conditioning

  • Per-layer KV conditioning lets every action-expert block cross-attend to projected keys and values from the corresponding VLM layer instead of using only a final hidden state.The VLM-to-expert conditioning path is modular, and its keys and values are detached during post-training so flow loss updates the expert and adapters without backpropagating through the VLM.Source: §4.2.1 · pp. 12–13

4.2.2 Post-training recipe

  • Post-training co-trains discrete autoregressive and continuous flow objectives, masks target action tokens from expert conditioning, and uses four flow samples per robot example.Padding masks support heterogeneous horizons and action widths, while knowledge insulation isolates the VLM from continuous-action gradients.Source: §4.2.2 · pp. 13–14

4.3.1 Embodiment-specific fine-tuning

  • Embodiment-specific fine-tuning preserves the shared architecture and joint objectives while adapting data, camera handling, action horizons, and control representations to each robot or benchmark.Fine-tuning increases flow samples from four to eight, permits flow gradients into the VLM, and uses robot-only data.Source: §4.3.1 · pp. 14–15

4.3.2 Inference optimization

  • Inference optimization caches invariant action-expert intermediates across flow steps and captures the fixed-shape generation loop with CUDA Graphs to reduce repeated execution overhead.Only the noisy action state and flow time change within an action chunk; context-dependent cross-attention states and position-dependent terms are reused.Source: §4.3.2 · p. 15

5 MolmoAct2-Think

MolmoAct2-Think adds compact depth-token reasoning before action generation and makes that reasoning adaptive by recomputing only scene regions whose RGB evidence changes. It trains and deploys this depth-conditioned policy with cached depth buffers, depth-aware objectives, and optimized decoding.

5 MolmoAct2-Think

  • MolmoAct2-Think predicts a compact discrete depth representation before actions, conditioning the action expert through per-layer KV conditioning.Depth maps use a 10 × 10 grid with 128 learned depth-code values per position, represented as autoregressive tokens.Source: §5 · p. 16
  • Adaptive depth reuses cached codes for static regions and predicts only cells whose RGB evidence changes, making reasoning cost scale with scene change.The policy maintains a full depth buffer and update mask across timesteps rather than regenerating all 100 depth codes.Source: §5 · p. 16, §5.1 · p. 16

5.1 Adaptive depth perception data

  • Depth data attaches to each policy observation stream, using monocular depth estimation and VQ-VAE quantization into 10 × 10 grids of indices from 0 to 127.Single-view datasets use their canonical camera, while multi-view datasets use the first policy view with available depth annotations.Source: §5.1 · p. 16

5.2 Training

  • Post-training uniformly mixes action, depth, and depth-and-action tasks, with predicted depth tokens conditioning continuous action generation through the action expert.Fine-tuning removes pure depth prediction, injects noise into 10% of teacher-forced depth-code inputs, and adds a learned per-layer gate for depth KV conditioning.Source: §5.2 · p. 17

5.3 Adaptive depth inference

  • At inference, the model predicts a full depth sequence only when needed, otherwise replaying unchanged cached spans and decoding changed cells before generating the action.The adaptive scheduler remains eager, while static KV caches and CUDA Graph stages reduce launch overhead during repeated one-token decoding.Source: §5.3 · pp. 17–18

6 Experiments

Across embodied-reasoning, zero-shot deployment, fine-tuning, robustness, trajectory quality, and ablation studies, MolmoAct2 achieves strong results across simulation and real-world evaluations. The experiments also show benefits from Molmo2-ER, adaptive-depth reasoning, and the model’s architectural choices.

6.1 Molmo2-ER evaluation

  • Molmo2-ER achieves a 63.8% average across 13 embodied-reasoning benchmarks, surpassing Gemini-ER 1.5 Thinking by 2.5 points and winning 9 benchmarks.It also improves over Molmo2 by 17%.Source: §6.1 · p. 20

6.6 Quality of MolmoAct2 trajectories for real-world deployment

  • 4.70s completion time and 1.04 joint path length on Stack Two Blocks demonstrate MolmoAct2’s shorter, more efficient trajectories than π0.5.Across RoboEval, it also operates near the best normalized values on Cartesian and joint-space stability metrics.Source: Figure 6 · p. 21, §6.6 · p. 24

6.2 Out-of-the-box deployment

  • MolmoAct2 achieves state-of-the-art zero-shot performance across simulation and real-world embodiments, including 87.1% on DROID and 56.7% on SO-100/101.It exceeds π0.5-DROID by +10.6% on MolmoBot and +3.2% on MolmoSpaces, while surpassing the real-world DROID runner-up by 38.7%.Source: §6.2 · pp. 20–21

6.3 Effective fine-tuning

  • MolmoAct2 leads adaptation benchmarks with 97.2% on LIBERO, 44.3% on RoboEval, and 50.1% average success across real-world tasks.It outperforms all baselines on 7 of 8 real-world deployment tasks and exceeds the runner-up by 15%.Source: §6.3 · p. 22, Table 8 · p. 22

6.4 Performance of MolmoAct2-Think

  • MolmoAct2-Think reaches 98.1% versus 97.2% for MolmoAct2 on LIBERO, a +0.9% gain concentrated on the hardest suite.It improves on three of four suites and matches the fourth, which is saturated at 100%.Source: Table 8 · p. 22, §6.4 · p. 23

6.5 Robustness of MolmoAct2 to out-of-distribution shifts

  • MolmoAct2 is most robust under perturbations, achieving 50.69% average success and a 10.80% absolute advantage over the next-best model.It leads every perturbation category, but scores 26.25% on Spatial Variance and has its narrowest margin on Distractor.Source: Table 9 · p. 24, §6.5 · p. 24

6.7 Systematic analysis

  • On LIBERO, Molmo2-ER improves success by 6.0 points, per-layer KV conditioning reaches 95.9%, and K = 8 flow samples reaches 95.90%.These results support the embodied-reasoning backbone, standard per-layer KV conditioning, and denser flow supervision.Source: §6.7 · pp. 24–25, Table 10 · p. 25, Table 11 · p. 25

K Spatial Object Goal Long Average

MolmoAct2’s full-model fine-tuning with discrete and continuous action co-training gives the best LIBERO average, while depth regularization and mixed training improve MolmoAct2-Think. Inference optimizations raise continuous control rates substantially, making that path the default deployment option.

K Spatial Object Goal Long Average

  • Full-model adaptation with discrete and continuous action co-training produces the best average LIBERO outcome, while action-expert-only tuning drops the average to 93.05%.LoRA remains strong on Spatial but loses 2.8 points on Long relative to full fine-tuning; removing discrete co-training shifts performance across suites, and knowledge insulation is slightly worse.Source: K Spatial Object Goal Long Average · pp. 25–26
  • Removing depth-token noise and the learned per-layer depth gate reduces the MolmoAct2-Think average from 98.10% to 97.65%, while also removing mixed training lowers it to 97.50%.The first change mainly causes a 1.8-point drop on Goal; mixed training preserves an action-only path alongside depth-and-action examples.Source: K Spatial Object Goal Long Average · p. 26, Table 14 · p. 26

6.8 Inference speed

  • 55.79 Hz for MolmoAct2 and 12.71 Hz for MolmoAct2-Think after CUDA Graph replay, corresponding to 2.42× and 1.58× speedups over the original paths.Measurements use horizon 10 on a single H100; caching alone reaches 27.39 Hz and 9.72 Hz, respectively.Source: §6.8 · p. 26, Figure 8 · p. 27
  • The continuous path is the default because the discrete path is 3.94× slower for MolmoAct2 and 1.86× slower for MolmoAct2-Think.The discrete path runs at 14.17 Hz and 6.82 Hz, respectively, because VLM decoding adds substantial overhead.Source: §6.8 · p. 27

7 Related work

Related work spans generalist vision-language-action policies, embodied reasoning, and interfaces between vision-language models and action experts. MolmoAct2 extends these directions through open reproducibility, adaptive depth reasoning, and layer-wise conditioning for continuous action generation.

Generalist robot manipulation policies

  • Generalist VLA policies transfer web-scale multimodal knowledge to diverse robotics data, but frontier systems remain difficult to reproduce because training data and code are closed.MolmoAct (Lee et al., 2025) is highlighted as an open exception, releasing code, data, and checkpoints with state-of-the-art manipulation performance.Source: §7 · p. 27

Embodied reasoning for robotic manipulation

  • Embodied reasoning replaces text-only chain-of-thought with visual or spatial representations, but heavy token use before action generation harms inference latency.MolmoAct2-Think computes new depth tokens only for novel visual signals, particularly benefiting third-person viewpoints.Source: §7 · pp. 27–28

Bridging VLM and action expert

  • VLA systems either discretize actions or pair VLMs with continuous diffusion or flow-matching experts, typically conditioning action generation only on final-layer hidden states.These approaches include vector-quantized actions (Brohan et al., 2022; Kim et al., 2024; Pertsch et al., 2025) and continuous experts (Black et al., 2024).Source: §7 · p. 28
  • MolmoAct2 uses per-layer KV conditioning to give a continuous flow-matching action expert dense access to hierarchical visual-semantic features.The VLM is first pretrained on discrete MolmoAct2-FAST Tokenizer action tokens, then the expert is post-trained with layer-wise conditioning; adaptive depth tokens provide explicit 3D grounding without bottlenecking latency.Source: §7 · p. 28

8 Conclusion … A.3 Adaptive-Depth Extension

MolmoAct2 is presented as a fully open action-reasoning model family for heterogeneous real-world robots, with geometrically grounded behavior and strong evaluation results. The paper and appendix detail its backbone, continuous action expert, adaptive-depth variant, released datasets, and reproducibility resources.

8 Conclusion

  • MolmoAct2 delivers performant, geometrically grounded behavior across diverse manipulation tasks, outperforming strong VLA baselines, fine-tuning efficiently, and transferring across three low-to-medium-cost embodiments.The authors release model weights, training code, and data, including the largest open bimanual manipulation dataset to date.Source: §8 · p. 28

Appendix

  • The appendix documents model, training, evaluation, dataset, and limitations sections, supporting reproducible deployment and further research.It explicitly lists sections for model details, training details, evaluation details, dataset details, and limitations with potential solutions.Source: Appendix · p. 35

A Model Details

  • MolmoAct2 uses a two-stage architecture: robot-aware autoregressive pretraining followed by a continuous DiT-style expert trained with flow matching and per-layer KV conditioning.MolmoAct2-Think retains this architecture and adds an autoregressive depth-token prefix before action prediction.Source: A Model Details · p. 35

A.1 MolmoAct2 Backbone

  • The Molmo2-ER backbone processes resized multi-camera crops, video sampled at up to 2 FPS and eight frames, robot state, setup/control descriptors, and discrete state and action tokens.Setup and control descriptors encode embodiment and action semantics, allowing one tokenizer and backbone to support different robots without a shared physical coordinate convention.Source: §A.1 · pp. 35–36

A.2 Continuous Action Expert

  • The continuous action expert predicts velocity over normalized action chunks using bidirectional self-attention, VLM cross-attention, time-conditioned DiT modulation, and rotary position embeddings.The released checkpoints use H = 30 for post-training, H = 10 for LIBERO fine-tuning, and N = 10 Euler inference steps; per-layer KV conditioning maps VLM states into the expert.Source: §A.2 · pp. 36–37

A.3 Adaptive-Depth Extension

  • MolmoAct2-Think adds a 10 × 10 autoregressive depth-token grid with 128 learned depth values before continuous action prediction, enabling depth-aware conditioning.Depth-and-action responses begin with both depth and action markers, and generated depth tokens are included in the action expert context.Source: §A.3 · p. 38
  • The LIBERO depth model learns a layer-specific gate that selectively modulates depth-token keys and values while initially preserving the standard action-conditioning path.The gate is initialized with bias −4 and is computed from non-depth context for each expert layer.Source: §A.3 · p. 38

B Training Details … B.3 GPU Cluster

MolmoAct2 training combines distributed PyTorch infrastructure, multimodal packing and prompting procedures, an open-data FAST tokenizer for heterogeneous embodiments, and a large-scale GPU cluster. The implementation includes precision-aware optimization, dynamic data handling, and dedicated hardware for distributed training.

B.1 Implementation

  • MolmoAct2 uses PyTorch FSDP2, scaled dot-product attention with custom masks, and torch.compile with static ViT and LLM shapes for efficient training.Custom masks support packed multimodal sequences and robot conditioning, while static padding and sequence budgets enable compiled-graph reuse.Source: §B.1 · p. 39
  • Training uses bfloat16 for most operations, full precision for layer normalization and RoPE, cross-device token-loss normalization, and within-batch dataset mixing.The normalization prevents short target spans from being over-weighted across mixtures containing robot-action, language, and video examples; fewer than 0.1% of examples are truncated.Source: §B.1 · p. 39
  • Dynamic packing selects examples from pools of M = 48 using a solver that balances text tokens and image crops under stage-specific sequence budgets.Molmo2 uses Tmax = 16384 and Imax = 128, while long-context training uses Tmax = 36864 and Imax = 384; MolmoAct2 applies stage-specific budgets.Source: §B.1 · p. 39
  • Robot prompts support action, depth, and depth-then-action styles, with output triggers selecting the desired target during training and inference.MolmoAct2-Think uses all three styles, and depth-then-action prediction lets the continuous action expert condition on predicted depth tokens.Source: §B.1 · p. 40

B.2 MolmoAct2-FAST Tokenizer

  • The MolmoAct2-FAST Tokenizer is an open-data FAST implementation using DCT-BPE and a transparent training mix that addresses prior data opacity.It standardizes actions into 32-dimensional vectors for single-arm and bimanual morphologies and supports heterogeneous action dialects without requiring a unified coordinate frame.Source: §B.2 · p. 41
  • The tokenizer represents single-arm and bimanual actions in a standardized 32-dimensional format, associating heterogeneous action representations with the embodiment specified in prompts or visual context.Training includes absolute joint positions and delta end-effector velocities rather than converting all data to one coordinate frame.Source: §B.2 · p. 41

B.3 GPU Cluster

  • MolmoAct2 was trained on Jupiter, an Ai2 GPU cluster providing 1,024 NVIDIA H100 GPUs across 128 servers with high-bandwidth storage and interconnects.Jupiter includes 1PB of NVMe SSD, 5PB of HDD, an 800Gbps local network, and RDMA over InfiniBand for cross-node communication.Source: §B.3 · pp. 41–42

C Evaluation Details … D Data Details

The paper evaluates MolmoAct2 across simulated and real-world bimanual settings, using multi-task or single-task training protocols and task-specific perturbations. Evaluation combines success rates with diagnostics for efficiency, safety, coordination, and task progression, including partial-credit scoring where applicable.

C.1 RoboEval

  • RoboEval trains MolmoAct2-Post jointly on eight bimanual tasks for 10,000 steps with eight GPUs and a global batch size of 64.Demonstrations from every task and variation are pooled into one multi-task training distribution.Source: §C.1 · p. 43
  • RoboEval evaluates eight tasks across structured scene perturbations and reports mean success with standard error from rollouts sampled from each task’s initial-state distribution.The protocol includes Static, Position, Orientation, and combined Position+Rotation conditions, plus task-specific variants such as Vertical, Drag, and obstacle-occluded settings.Source: §C.1 · p. 43
  • The RoboEval evaluation records trajectory efficiency, collision and slip safety, bimanual coordination, and stagewise task progression to identify failure points and execution quality.Diagnostics include path length, jerk, collisions, slip counts, end-effector height discrepancy, inter-arm velocity divergence, and semantic subphase completion flags.Source: §C.1 · p. 43

C.2 Real-world Zero-shot DROID

  • The zero-shot DROID evaluation uses five kitchen tasks with three spatial variants, fixed camera positions, and comparisons among MolmoAct2, π0, and MolmoBot.Sample trajectories are shown in Figure 9, while Table 18 reports the evaluation results.Source: §C.2 · p. 43, Figure 9 · p. 45, Table 18 · p. 45

C.3 Real-world Bimanual YAM

  • Bimanual YAM evaluation trains MolmoAct2-Post as eight separate single-task policies and applies the same converged training protocol to four baselines across matched spatial variants.Cortex AI conducted setup and evaluation, with each task collected and tested under three spatial variants.Source: §C.3 · p. 43, Figure 10 · p. 46, Figure 11 · p. 46

C.4 Real-world SO-100/101 zero-shot

  • SO-100 zero-shot evaluation compares MolmoAct2 with DePi and SmolVLA from a fixed camera without additional policy training, using partial-credit episode scores.The benchmark evaluates three milestone-based partial scores and reports task results in Table 20; sample trajectories appear in Figure 12.Source: §C.4 · pp. 43, 45, Table 20 · p. 47, Figure 12 · p. 47

D.1 MolmoAct2-SO100/101 Dataset Statistics

The MolmoAct2-SO100/101 training mixture is filtered from 1,660 collected entries to 1,222 final datasets using TOPReward, with all reported statistics computed on this filtered set.

Dataset construction

  • 1,222 final datasets remain after TOPReward filters 438 low-quality entries from 1,660 collected entries.All subsequent statistics are computed only over the final filtered set.Source: §D.1 · p. 48

Dataset statistics

  • The final SO-100/101 partition is characterized through summary statistics, robot-type breakdowns, and metadata distributions covering FPS, camera configuration, and action/state dimensions.These summaries are reported in Tables 21–23 for the filtered training partition.Source: §D.1 · p. 48, Table 21 · p. 48, Table 22 · p. 49

Language and trajectories

  • Language annotation changes the observed action-verb distribution, while sample trajectories illustrate the SO10x dataset partition.Figure 13 compares verb distributions before and after annotation, and Figure 14 shows representative trajectories.Source: §D.1 · p. 48, Figure 13 · p. 48, Figure 14 · p. 49

D.2 Language Annotation Pipeline Implementation Details

The language annotation pipeline uses Qwen3.5-27B to generate imperative robot-arm instructions from multi-camera demonstrations, with randomized length targets and temporally sampled frames. Applying the pipeline increases instruction diversity across all robotics datasets in MolmoAct2’s training mix.

D.2 Language Annotation Pipeline Implementation Details

  • Inference uses temperature 0.1 and a maximum length of 1024 tokens.Source: §D.2 · p. 49
  • The pipeline prompts Qwen3.5-27B to describe every robot-arm action imperatively, emphasizing scene-state changes, object colors and positions, pushing, and verified grasps.Grasps are reported only when the wrist camera shows a closed gripper holding an object.Source: §D.2 · p. 49
  • For each non-frozen camera, the pipeline samples 12 equally spaced frames and randomly sets the instruction length from 5 to 25 words using a right-tailed power distribution.Source: §D.2 · p. 49
  • The annotation pipeline increases the number of unique instructions across all robotics datasets in MolmoAct2’s training mix.Source: §D.2 · p. 49, Table 24 · p. 50

E Limitations and Potential Solutions

MolmoAct2’s deployment is limited by fixed open-loop action chunks and embodiment-specific zero-shot coverage. The authors propose improving responsiveness through re-chunking and broadening zero-shot transfer by scaling data collection and fine-tuning on target robots.

Action chunks and the absence of real-time re-chunking

  • Fixed-horizon chunks execute open-loop, causing seam discontinuities and preventing within-chunk reactions to perturbations, contact events, or tracking errors.Chunks span 30 steps at 30 Hz for YAM/SO-100/101, 15 at 15 Hz for DROID, and 10 at 10 Hz for LIBERO; the reported 55.79 Hz is amortized throughput rather than closed-loop reactivity.Source: E Limitations and Potential Solutions · p. 51

Embodiment-specific zero-shot deployment

  • Zero-shot deployment is currently limited to the YAM, SO-100/101, and DROID embodiments represented by large-scale training data.Other bimanual platforms, dexterous hands, mobile manipulators, and humanoids require target-embodiment demonstrations for effective fine-tuning; broader coverage depends on scaling data collection.Source: E Limitations and Potential Solutions · p. 51

Related papers

Loading 2605.02881v2…