MolmoAct: Action Reasoning Models that can Reason in Space

• MolmoAct autoregressively predicts depth-aware perception tokens, visual trajectory traces for mid-level plans, and precise low-level action tokens.The three chains can be decoded as a depth map, image-plane trajectory overlay, and executed robot actions, making behavior explainable and steerable.Source: Figure 1 · p. 2, §1 · p. 2
• Existing VLAs remain brittle and opaque, struggling to transfer across tasks, scenes, and embodiments or explain action choices.The introduction situates this limitation amid efforts to scale robotics datasets and models (Duan et al., 2022; Xu et al., 2024b; Firoozi et al., 2025; Black et al.; Kim et al., 2024; Team et al., 2025; NVIDIA et al., 2025; Yang et al., 2025b; Liu et al., 2025; Pumacay et al., 2024).Source: §1 · p. 2
• Because embodied interaction data is costly, ambiguous, and hard to scale, MolmoAct uses structured intermediate representations to support reasoning, abstraction, and control.The motivation is that embodied reasoning transforms perception into coherent, adaptable action, while robots must reason rather than merely map images and instructions to control.Source: §1 · pp. 1–2
• MolmoAct is fully open, with released model weights, training code, and all components of its action reasoning dataset.The authors present it as a blueprint for agents that transform perception into purposeful action through reasoning.Source: §1 · pp. 2–3

2 MolmoAct

• MolmoAct is a fully open-source action reasoning model built on Molmo and trained on selected Open X-Embodiment datasets plus the in-house MolmoAct Dataset.The cited OXE sources are BC-Z, BridgeData V2, and RT-1.Source: §2 · p. 3

2.1 Vision Language Model

• MolmoAct-7B-O uses an OpenCLIP/OLMo2-7B backbone, while MolmoAct-7B-D uses SigLIP2/Qwen2.5-7B.Molmo uses a ViT visual encoder, a two-layer MLP connector, and a decoder-only LLM backbone.Source: §2.1 · p. 3

2.2 Vision-Language-Action Model

• 9,216 GPU hours enable MolmoAct pre-training, over a 5x reduction from GR00T N1.5’s 50,000 GPU hours.Actions are quantile-normalized and discretized into 256 bins; adjacent bins map to adjacent byte-level BPE symbols to preserve action-bin similarity in token embeddings.Source: §2.2 · p. 4

2.3 Action Reasoning Model

• MolmoAct generates depth perception tokens, a visual reasoning trace, and then action tokens, grounding actions in inferred depth and planned motion.A specialist depth estimator trained on Depth Anything V2 maps depth into VQ-VAE codes, which MolmoAct learns to predict autoregressively from RGB observations.Source: §2.3 · pp. 5–6
• The model predicts a 2D end-effector trajectory alongside action commands, using waypoints to support precise localization and action prediction.The visual reasoning trace is a polyline of one to five points sampled across the future episode horizon.Source: §2.3 · p. 6

2.4 Action Steerability via Visual Reasoning Trace

• Users can steer behavior by drawing an unambiguous, editable end-effector path on the camera image, avoiding reliance on ambiguous language prompts.At each timestep, the model conditions on the image, instruction, and overlaid trace to generate actions in closed loop.Source: §2.4 · p. 7

3.1 Action Reasoning Data

• 10 million depth maps train a VQVAE whose 128-dimensional codebook encodes each 320×320 image as 100 depth tokens.DepthAnything-v2 supplies depth maps for RGB observations; the VQVAE is trained for 20 epochs.Source: §3.1 · p. 7
• Molmo predicts gripper coordinates per frame, which are linked into future-motion traces of 1–5 points, with separate left- and right-gripper traces for bimanual robots.Prompts ask Molmo to point to the gripper; traces include the current and final locations and up to three uniformly spaced intermediate points.Source: §3.1 · pp. 7–8
• Three auxiliary datasets supervise depth-token prediction, trace prediction, and trajectory-conditioned actions using a trace-overlaid image to enable steerability.For trajectory-conditioned action data, the model receives the instruction and current image overlaid with the ground-truth visual reasoning trace.Source: §3.1 · p. 8
• Robot episodes become action-reasoning training sequences by aligning depth-perception tokens, visual reasoning traces, and ground-truth actions at each timestep.The format extends conventional robot action datasets with spatial labels conditioned on language and observations.Source: §3.1 · pp. 7–8

3.2 MolmoAct Dataset

• 10,689 trajectories from 93 tasks form the MolmoAct Dataset: 7,730 home trajectories across 73 tasks and 2,959 tabletop trajectories across 20 atomic tasks.A single-arm Franka collected the data over two months with five full-time operators; trajectories average 112 timesteps. Home chores were decomposed into subtasks, while tabletop tasks were decomposed into motion primitives.Source: §3.2 · p. 8

3.3 Multimodal Web Data

• Multimodal web data adds academic visual-question-answering, document, chart, and diagram datasets, PixMo for fine-grained understanding and pointing, and LVIS bounding-box-center prediction for language grounding.PixMo supports fine-grained understanding and pointing; LVIS (Gupta et al., 2019) trains the model to ground categories by predicting instance bounding-box centers.Source: §3.3 · p. 9

4 Training Recipe

• The recipe progresses from broad action-reasoning pre-training through MolmoAct Dataset mid-training to task- and embodiment-specific post-training.The stages are designed to develop the model before adapting it to downstream settings.Source: §4 · p. 9

4.1 Pre-training

• 26.3M samples make up pre-training’s mixture of OXE-derived action reasoning, auxiliary robot data, and multimodal web data.The mixture includes 10.5M OXE samples converted to action reasoning data, auxiliary depth, trace, and trajectory-conditioned action data, and 2M multimodal web samples. Training uses 256 H100s for 100k gradient steps with batch size 512, taking around 9,728 GPU hours.Source: §4.1 · p. 9

4.2 Mid-training

• Mid-training uses 1M MolmoAct Dataset action-reasoning samples and 1M trajectory-conditioned action samples to improve performance and action steering.Three camera views are converted into two paired-view examples; depth tokens and reasoning traces come only from side views, while the wrist view provides additional information. Training runs for 50k steps with batch size 128 on 128 H100 GPUs, using around 2,304 GPU hours.Source: §4.2 · p. 10

4.3 Post-training

• Post-training adapts MolmoAct using 30–50 tele-operated demonstrations per task, action chunks of length N = 8, and parameter-efficient LoRA fine-tuning.LoRA rank and alpha are fixed at 32 and 16; batch size is 128 for simulation benchmarks and 64 for real-world tasks. Demonstrations may pair front- or side-view images with wrist views, including multiple wrist views in some bimanual setups.Source: §4.3 · p. 10

5.1 MolmoAct After Pre-training

• 70.5% success rate on SimplerEnv visual matching was achieved zero-shot, outperforming GR00T N1.5, π0, π0-FAST, and Magma.Fine-tuning on the same RT-1 subset of OXE raised performance to 71.6%, exceeding Magma by 3.2%.Source: §5.1 · p. 12

5.2 Fast Adaptation of MolmoAct in Post-training

• 86.6% average success rate on LIBERO was the highest among compared methods, with a 6.3% advantage over ThinkAct on LIBERO-Long; task progression exceeded π0-FAST by 10% on single-arm and 22.7% on bimanual tasks.The real-world comparison covers six tasks across single-arm and bimanual Franka setups, with 25 trials per task.Source: Figure 5 · p. 11, §5.2 · p. 12

5.3 Effectiveness of MolmoAct in Out-of-Distribution Generalization

• 72.1% on SimplerEnv variant-aggregation tasks exceeded all baselines and RT-2-X by 7.8%; real-world task progression improved by 23.3% on average over π0-FAST.The gap between variant-aggregation and visual-matching performance was less than 1%.Source: Figure 6 · p. 13, §5.3 · p. 14

5.4 Effect of the MolmoAct Dataset on MolmoAct Performance

• 5.5% average margin across three real-world tasks separated MolmoAct-7B-D from its counterpart without MolmoAct Dataset mid-training.Without mid-training, MolmoAct-7B-D-Pretrain exceeded π0-FAST by 14.8% and OpenVLA by 10.9%.Source: Figure 6 · p. 13, §5.4 · p. 14

5.5 Instruction Following of MolmoAct

• 109 Elo points over SpatialVLA and win rates of 58% against SpatialVLA and 81% against OpenVLA marked MolmoAct-7B-D-Pretrain’s simulation instruction-following results.For visual trace generation, MolmoAct also outperformed all baselines with higher Elo scores and non-overlapping 95% confidence intervals.Source: Figure 7 · p. 14, §5.5 · p. 15, Figure 8 · p. 15

5.6 Steerability of MolmoAct

• MolmoAct-7B-D achieved a 75% success rate with visual-trace steering, outperforming open-ended natural-language steering by 33%; under open-ended language steering, it surpassed π0-FAST by 29%.Source: §5.6 · p. 16

6.1 Generalist robot manipulation policies

• Generalist robot policies have shifted toward broad datasets and VLA models, which vary in action heads and hierarchical designs but often rely on extensive robotics-specific data.MolmoAct instead uses spatial reasoning and claims competitive or superior performance with a fraction of the data required by existing methods.Source: §6.1 · p. 16

6.2 Robot reasoning and planning with language

• Prior work integrates language-model reasoning into robot policies or separates semantic prediction and planning from low-level control; MolmoAct adds language- and sketch-steerable action prediction.RT-Trajectory and inference-time policy steering (Wang et al., 2024b) are tied to particular architectures or training regimes; HAMSTER (Li et al., 2025) uses 2D trajectories and a fixed-task low-level policy. MolmoAct’s steering generalizes to novel spatial configurations, unseen objects, and ambiguous instructions.Source: §6.2 · pp. 16–17

6.3 Embodied reasoning for robotic manipulation

• Embodied-reasoning approaches use subgoals, visual frames, video-derived language, or spatial representations, while MolmoAct grounds each reasoning step in the scene and 3D space.ECoT (Zawalski et al., 2024), CoT-VLA (Zhao et al., 2025), RAD (Clark et al., 2025), and ThinkAct (Huang et al., 2025) use varied reasoning signals; Emma-X (Sun et al., 2024) centers on gripper positions, while Yang et al. (2025a) studies depth-aware trajectory traces but evaluates diffusion policy on a small data region. MolmoAct’s steps can be visualized on the image plane and in 3D, supporting explainability and action prediction.Source: §6.3 · p. 17

7 Conclusion

• MolmoAct integrates perception, planning, and control through depth-aware perception tokens, editable spatial reasoning traces, and action prediction, enabling explainable and steerable behavior.The conclusion also reports robust out-of-distribution generalization, adaptation through lightweight fine-tuning, and release of model weights, code, and data.Source: §7 · p. 17

Appendix; A.6 Full Hyperparameters

• The appendix covers model, training, action vocabulary, evaluation, data, dataset examples, and limitations; its hyperparameter table notes LoRA adapters on all model linear layers.Source: Appendix · p. 24, §A.6 · p. 25

A Model Details; A.1 Backbone Overview

• MolmoAct combines a preprocessor for multi-scale cropping and optional image padding, a ViT image encoder, a vision–language connector, and an LLM that autoregressively processes vision and text tokens.MolmoAct-7B-D combines ViT-SO400M/14 384px SigLIP2 with Qwen2.5-7B.Source: A Model Details · p. 24, §A.1 · p. 24

A.2 Image Encoding and Cropping

• For high-resolution images, MolmoAct tiles overlapping square crops alongside a resized full-image crop, using neighboring context while excluding overlapping features from the final tokens.For CLIP, it resizes and pads images to preserve aspect ratio; SigLIP2 instead uses square resizing without padding.Source: §A.2 · pp. 24–25

A.3 Vision–language Connector

• The connector concatenates features from two ViT layers, attention-pools each 2 × 2 patch window, then maps pooled features into the LLM embedding space.The described layer selection and attention pooling outperform single-layer features and naive concatenation, respectively.Source: §A.3 · p. 25

A.4 Arranging Vision Tokens

• Vision tokens are serialized from the low-resolution image first, then high-resolution crops in row-major order, with start, end, and row-end markers.Source: §A.4 · p. 25

A.5 Multi-image Inputs

• For multi-image inputs, MolmoAct processes each image into vision tokens, prefixes them with image-index text tokens, and concatenates the sequences.Source: §A.5 · p. 25

B.1 Implementation

• MolmoAct training uses PyTorch FSDP and SDPA, with AMP bfloat16 for most operations and fp32 for layer normalization and RoPE.Gradients are averaged across devices. The implementation follows Molmo’s training approach.Source: §B.1 · p. 26
• Global-average loss-token normalization avoids device-local bias when up-weighted examples have fewer loss tokens and land on devices with smaller token counts.The passage notes this matters especially when the global batch is much larger than any single-device batch.Source: §B.1 · p. 26
• The depth-token setup unfreezes tokenizer embeddings and the language-model head, replacing 130 padding tokens with depth tokens for both 7B variants.MolmoAct-7B-D uses Qwen2.5-7B; MolmoAct-7B-O uses Olmo2-7B and first pads its tokenizer and head to a multiple of 512.Source: §B.1 · p. 26

B.2 GPU Cluster

• Table 4 summarizes training hyperparameters across MolmoAct-7B-D and -7B-O schemes; MolmoAct-7B-D-Pretrain ran 150K steps but performed better at 100K.Post-training duration varies by task and scenario because training continues until convergence, assessed using training loss and evaluation performance.Source: §B.1 · p. 26, Table 4 · p. 28

C Action Tokenization

• The action-token vocabulary maps discrete bin indices 0–255 to token strings, with Tables 5 and 6 covering the two index ranges.The text also explains that the displayed Unicode escape sequence decodes into the actual concatenated characters.Source: C Action Tokenization · p. 29, Table 5 · p. 29, Table 6 · p. 30

D.1 Evaluation on SimplerEnv (Google Robot)

• SimplerEnv tests out-of-the-box performance on a simulated Google Robot using front-view images, language instructions, and delta end-effector pose actions.Its evaluation has Visual Matching and Variant Aggregation components; the image is resized from 640 × 480 to 320 × 240 for this setup.Source: §D.1 · p. 31

D.2 Evaluation on LIBERO

• LIBERO evaluation covers Spatial, Object, Goal, and Long suites, each with 500 expert demonstrations across 10 tasks, using filtered training data and action chunks of K = 8.The evaluation executes each full chunk before replanning; the benchmark uses a Franka Panda simulation with front- and wrist-view images, language instructions, and delta end-effector pose actions.Source: §D.2 · p. 31
• LIBERO post-training uses MolmoAct-7B-D with learning rate 5e-4, LoRA rank 32, alpha 16, dropout 0, and no LoRA bias; Goal training ran 50K steps but performed better at 40K.These settings are specified in Table 7.Source: Table 7 · p. 31

Evaluation protocol

• MolmoAct was benchmarked against OpenVLA and π0-FAST after training to convergence, with baseline models using their official implementations and default configurations.The single-arm Franka was mounted on a movable platform, while the bimanual setup was fixed to a tabletop.Source: §D.3 · p. 32
• Evaluation pre-marked task-object locations to match the fine-tuning demonstration distribution and used wrist- and front-facing RealSense cameras.The cameras were an Intel RealSense D405 for the wrist view and a D435 for the front-facing view.Source: §D.3 · p. 32

Single-arm tasks

• The single-arm suite covers putting a bowl in a sink, wiping a table, and bussing a table, with progression scored through task-specific completion milestones.Milestones include grasping and placing the bowl, grasping and completing a wipe, and grasping, moving, and dropping a can into a bin.Source: §D.3 · p. 32

Bimanual tasks

• The bimanual suite covers setting a table, lifting a tray, and folding a towel, with progression scored through task-specific completion milestones.The milestones include placing a banana and pouring tea, lifting a tray with both arms, and folding a towel.Source: §D.3 · p. 32

D.4 Evaluation on Generalization in Real-world

• MolmoAct and the other models were trained on all six combinations of three objects and two plate colors in a multi-task setting.The objects were a green can, red cup, and banana; the target plates were yellow and blue.Source: §D.4 · pp. 32–33
• Generalization was evaluated on three placement tasks under four perturbations and one in-distribution condition.Perturbations changed the language, object positions, added distractors, or replaced the target objects with novel objects.Source: §D.4 · p. 33
• Task progression scores assign 0.25 for approaching the correct object, 0.5 for picking it up, 0.75 for approaching the correct plate, and 1 for placing it correctly.Source: §D.4 · p. 33

D.5 Evaluation on the Effect of MolmoAct Dataset for MolmoAct Mid-training

• The dataset mid-training evaluation compared MolmoAct with its no-dataset variant, OpenVLA, and π0-FAST on three real-world tasks, using 50 demonstrations and 10 trials per task.The tasks were close_lid, rotate_pot, and pour_tea; object locations were pre-marked for repeatability.Source: §D.5 · p. 33
• Task progression was scored at staged completion milestones for closing a lid, rotating a pot, and pouring tea into a cup.Milestone scores were close_lid: move lid toward closing direction (0.5), close lid (1); rotate_pot: reach handle (0.3), rotate 45 degrees (0.6), complete 90-degree rotation (1); pour_tea: grasp teapot (0.5), move it above cup (0.8), pour tea (1).Source: §D.5 · p. 33

D.6 Evaluation of MolmoAct on Instruction-following

• Instruction-following was tested in five customized SimplerEnv scenes, with open-ended prompts supplied by participants for each scene.The prompts were filtered before evaluation.Source: §D.6 · p. 33
• The evaluation used 29 prompts to generate 200-step rollouts from MolmoAct, OpenVLA, and SpatialVLA, which 100 participants rated in an arena-style interface.Source: §D.6 · p. 33

D.7 Evaluation of MolmoAct on Action Steerability

• The steerability task was pick_up_bowl, with progression scored for choosing the correct bowl direction (0.5), grasping the correct bowl (0.85), and lifting it (1).A web interface let users modify the instruction or sketch five points on the image to steer the visual trace at test time.Source: §D.7 · p. 34
• Steerability training used 200 demonstrations split evenly across clean- and dirty-bowl targets, with half of each target set exploring alternative paths conditioned on language.The authors intended this path diversity to help the model learn how visual traces relate to physical actions.Source: §D.7 · p. 34
• At test time, 10 participants provided open-ended language instructions, with variations restricted to verbs, nouns, or adjectives.Source: §D.7 · p. 34

E.1 MolmoAct Dataset

• For each task, MolmoAct ranks external views by scene clarity and robot occlusion, labeling them primary and secondary; home data is recorded at 15 Hz and tabletop data at 20 Hz.Home camera configurations may vary by task, while tabletop configurations are consistent; tabletop data also includes intrinsic and extrinsic calibration matrices for both external cameras, and task details appear in Tables 24 and 25.Source: §E.1 · p. 34

F Data Examples

• Figures 13–14 show Action Reasoning Data and Auxiliary Visual Reasoning Traces.Source: F Data Examples · p. 34
• Figures 15–18 show Auxiliary Depth Perception Tokens, Trajectory-conditioned Action Data, Multimodal Web Data, and MolmoAct Dataset examples from home environments.Source: F Data Examples · p. 34
• Figures 19–22 show MolmoAct Dataset tabletop examples and post-training demonstrations for single-arm Franka, bimanual Franka, and Rainbow.Source: F Data Examples · pp. 34–35

Camera Occlusion of End-effector

• Occluding the end-effector in the front-camera view can degrade visual-trace prediction and overall performance; a wide-field-of-view camera and SLAM-generated traces could enable temporal reasoning.Although post-training can process multiple camera views, spatial reasoning primarily relies on the front camera, which typically provides a full end-effector view.Source: G Limitations and Potential Solutions · p. 35

Robustness of Steerability via Visual Traces

• Robust steerability requires precise, diverse 2D traces and abundant, high-quality post-training data spanning action compositions.Bounding-box detectors can collapse points toward box centers and transfer poorly across embodiments, whereas VLM-based annotations such as Molmo and RoboPoint (Yuan et al., 2024) yield accurate, non-degenerate traces; varied task-completing robot motions can broaden action-composition coverage.Source: G Limitations and Potential Solutions · p. 35
• Because 2D trace steering does not use spatial reasoning, it can follow in-plane paths yet produce imprecise out-of-plane motion and fail on more complicated tasks.The authors hypothesize that conditioning on or reusing predicted depth-perception tokens to lift traces into 3D could mitigate this limitation; they leave this for future exploration.Source: G Limitations and Potential Solutions · p. 35

Speed of Action Reasoning Model prediction

• A mismatch between action-reasoning inference frequency and data-collection control frequency may reflect communication latency and the time needed to predict more reasoning tokens.Potential directions include reducing inference time through VLM optimization or developing smaller models for efficient edge or local execution.Source: G Limitations and Potential Solutions · p. 35

Precision in Depth Perception Token

• The fixed set of 100 depth-perception tokens may be insufficient for fine-grained manipulation, for which higher-resolution depth estimation could improve spatial reasoning and performance.The paper follows Bigverdi et al. (2025) in using 100 tokens and suggests increasing their number as a potential improvement.Source: G Limitations and Potential Solutions · p. 35