PaperlayerView exact revision on arXiv

Source-linked AI summary

IronMind: Scaling Humanoid Dexterous Manipulation via Camera-Space Ego-Centric Pretraining

Huimin Pan, Yufan Ren, Kunpeng Song, Siyang Wang, Xiwen Zhang, Xiaoyun Hu, Zhuoxu Duan, Hanrui Zheng, Jialeng Ni, Nathan Zhao, Sibo Ma, Zhenxuan Fan, Zhongyang Che, Danny Bao, Jiacheng Wei, Jerry Bai, Xiaoyu Yue, Xiaoyang Guo, Chenyi Chen

arXiv:2609.39403v1cs.RO

TL;DR

Dexterous humanoid manipulation is data-limited, and egocentric video introduces embodiment and data-quality challenges. IronMind pretrains on human and robot data using camera-space actions, and its 10,000-hour model achieves 55.0% success across six out-of-distribution tasks after post-training.

  • Problem

    Dexterous manipulation lacks a web-scale robot data corpus, while egocentric human video has embodiment and quality gaps that complicate its use for humanoid pretraining.

    Source: §1 · p. 1, §3.1.1 · p. 5

  • Method

    IronMind pretrains a VLA policy on egocentric human video and heterogeneous robot data using shared camera-space actions, semantic dimension alignment, and curated video.

    Source: §1 · p. 2

  • Results

    55.0% success across six out-of-distribution tasks for the 10,000-hour model after post-training, versus 26.7% for the torso-frame baseline.

    Source: §6 · p. 15

  • Takeaways & Limitations

    The reported findings support camera-space pretraining on large-scale egocentric human data as a scalable foundation for humanoid dexterous manipulation.

    Source: §1 · p. 3

    The scaling study covers one architecture and data mixture; robot evaluation covers one embodiment, six OOD tasks, and 10 trials per task.

    Source: §5 · p. 15

Abstract

from arXiv · show

Egocentric human video offers a scalable data source for dexterous manipulation, yet using it to train humanoid robots presents two challenges: (1) an embodiment gap, as human hands differ structurally from robot end-effectors and low-cost egocentric recordings lack the torso kinematics required by conventional retargeting; and (2) heterogeneous data quality, including noisy hand-pose tracking and weakly aligned text annotations. We introduce IronMind, a vision-language-action (VLA) model that uses egocentric human video and heterogeneous robot data to pretrain policies for humanoid dexterous manipulation. To bridge the embodiment gap, IronMind bypasses explicit body-retargeting by using a camera-space action representation, the native reference space of egocentric video, and semantically aligning robot and human action dimensions. Across total pretraining budgets from 250 to 10,000 hours, validation loss decreases approximately log-linearly with data scale. Larger pretraining budgets also improve out-of-distribution real-robot manipulation after post-training: across six challenging tasks with unseen objects, affordances, and reasoning prompts, the 10,000-hour model achieves a 55.0% success rate, compared with at most 11.7% for every pretraining budget up to 5,000 hours and 5.0% without pretraining. At the same pretraining budget, the camera-space action representation also outperforms the torso-frame baseline. Together, these findings support pretraining with a camera-space action representation on large-scale human egocentric data as a scalable foundation for humanoid robot manipulation.

1 Introduction

Dexterous robot manipulation remains data-limited, while egocentric human video offers scale but brings embodiment and data-quality gaps. IronMind uses camera-space actions and curated human–robot data, with larger pretraining budgets associated with stronger real-robot results.

1 Introduction

  • Dexterous manipulation remains data-limited because real-robot demonstrations are slow and expensive, while egocentric video has embodiment and quality gaps.Human and robot hands differ in structure; egocentric data can have noisy hand tracking, weakly aligned prompts, and non-manipulation segments.Source: §1 · p. 1
  • IronMind avoids body retargeting by expressing human and calibrated robot trajectories in camera space, semantically aligning action dimensions, and curating video with filtering, re-annotation, and per-frame quality weighting.The pretraining mixture includes over 10,000 hours of egocentric human video and heterogeneous robot data.Source: §1 · pp. 2–3
  • Its Mixture-of-Transformers policy uses separate vision-language and flow-matching action experts linked by shared attention, with optional future-frame world-prior supervision removed after pretraining.The optional priors use learnable query tokens to condition a frozen video-generation model.Source: §1 · p. 2
  • 55.0% success across six OOD tasks for the 10,000-hour model, versus at most 11.7% for budgets up to 5,000 hours; at equal budget, camera-space scores 55.0% vs. 26.7% for torso-frame.The comparison is after post-training on a real humanoid robot.Source: §1 · p. 3

2 Related Work

Related work spans robot VLA architectures, human-to-robot action representations, video-based world priors, and robot-learning scaling and evaluation. IronMind combines camera-space human–robot actions with training-only world priors and an evaluation suite spanning open- and closed-loop settings.

2 Related Work

  • Robot VLA work spans discrete text-token actions [Brohan et al., 2023; Kim et al., 2024], learned action tokenization [Pertsch et al., 2025], and continuous diffusion or flow heads [Chi et al., 2023; Liu et al., 2025; Black et al., 2025].IronMind focuses on scalable data representations and pretraining for humanoid manipulation from egocentric video.Source: §2 · p. 3
  • Egocentric transfer uses latent, camera-space, retargeted, translated, or joint action representations; IronMind keeps human and robot data in camera space and semantically aligns dimensions [Yuan et al., 2026a].Robot trajectories are transformed using measurable camera calibration parameters, without body retargeting or a learned latent bottleneck.Source: §2 · pp. 3–4
  • World-model approaches include action-dynamics modeling, future-frame context, and latent distillation; IronMind uses a frozen video generator for training-time priors and leaves it inactive at inference [Ma et al., 2026a].The priors are combined with semantic and geometric feature distillation within a Mixture-of-Transformers backbone.Source: §2 · p. 4
  • Robot scaling studies examine dataset size, composition, and quality, while evaluation often remains fragmented across offline metrics or a limited set of robot tasks; IronMind combines open-loop metrics, held-out-scene rollouts, and standardized robot testing.The evaluation suite is organized around a capability taxonomy and reports successes and failure modes.Source: §2 · p. 4

3 IronMind: The Framework

IronMind combines a camera-space action interface and curated human and robot data with a two-expert VLA architecture, optional multimodal priors, and pretraining followed by target-robot adaptation.

3.1.1 Camera-space action representation

  • Camera-space trajectories put human and robot data in a shared action interface, with robot motions calibrated into head-camera space and action dimensions aligned by meaning.This avoids body retargeting despite differing hand and end-effector structures.Source: §3 · p. 4, §3.1.1 · p. 5

3.1.2 Data curation pipeline

  • Per-frame quality scores determine sampling weights, prioritizing frames with smooth motion and clear manipulation intent while retaining neighboring context.The score uses visibility and tracking gates plus camera stability, hand translation, and segment-boundary pause measures.Source: Figure 3 · p. 6, §3.1.2 · p. 7
  • Curation filters invalid observations, segments recordings into single-intent clips with refined captions, and rescales human motion speed toward robot teleoperation.These stages address tracking failures, annotation mismatches, and source-dependent speed differences.Source: §3.1.2 · pp. 6–7, Figure 3 · p. 6

3.2 Model Architecture

  • IronMind maps egocentric images and open-vocabulary instructions to smooth metric camera-space trajectories using a VLA policy with optional multimodal priors.The architecture separates high-level vision-language understanding from low-level trajectory regression.Source: §3 · p. 4, §3.2 · p. 7

3.2.1 Dual-Expert Mixture-of-Transformers Architecture

  • The Mixture-of-Transformers uses separate understanding and action experts connected by shared attention, with flow matching generating continuous action trajectories.The understanding expert retains pretrained language capabilities; the action expert predicts trajectory chunks from state and noisy actions.Source: §3.2 · p. 7, §3.2.1 · p. 8

3.2.2 Multimodal World Priors Supervision

  • Optional latent-query priors supervise geometry, semantics, and future dynamics during pretraining, while auxiliary prediction branches are removed at inference.The learned query tokens remain available to the action expert through shared attention.Source: §3.2.2 · p. 8

3.3 Training Paradigm

  • Pretraining uses egocentric human video plus heterogeneous non-target robot data, with action and optional depth, semantic-feature, and dynamics objectives.Across scaling experiments, architecture, optimization, and robot-data mixture stay fixed as human-video experience increases.Source: §3.3 · p. 9
  • Post-training adapts both experts using curated IRON-R01 teleoperation while preserving the camera-space action interface through extrinsic calibration.IRON-R01 demonstrations are introduced only during this stage.Source: §3.3 · p. 9

4 Experiments

The experiments examine pretraining scale, optional world-model priors, camera-space versus torso-frame actions, and out-of-distribution real-robot performance. They use a tiered evaluation framework and include open-loop capability visualizations on unseen scenes.

4 Experiments

  • The evaluation examines policy capability scaling up to 10,000 pretraining hours, optional multimodal priors, action representation, and OOD real-robot success.The action comparison is between camera-space and torso-frame baselines.Source: §4 · p. 9
  • Evaluation spans camera-space open-loop metrics, offline policy-in-the-loop rollouts, and real-robot execution to trade interaction fidelity against throughput.The framework addresses the time, safety, and throughput constraints of direct humanoid evaluation.Source: §4 · p. 9
  • On unseen RGB scenes and text instructions, open-loop visualizations show predicted hand trajectories adjusting grasp aperture and wrist orientation to target geometry.The case studies probe semantic grounding, affordance, and fine-grained manipulation.Source: §4 · p. 9, Figure 6 · p. 10

4.1 Implementation Details and Training Recipe

The training recipe uses AdamW with a global batch size of 1024 and a cosine learning-rate schedule; pretraining excludes target-robot demonstrations, while post-training uses IRON-R01 teleoperation.

4.1 Implementation Details and Training Recipe

  • All IronMind models use AdamW with a global batch size of 1024 and a cosine schedule decaying to 10% of peak learning rate.Source: §4.1 · p. 10
  • Reported pretraining budgets combine egocentric human video and heterogeneous robot data, with no IRON-R01 demonstrations included.Source: §4.1 · p. 10
  • Post-training uses approximately 40,000 filtered teleoperated episodes collected on IRON-R01.Source: §4.1 · p. 10

4.2 Evaluation Protocols

Evaluation proceeds from single-chunk open-loop scoring to repeated policy-in-the-loop rollouts and closed-loop robot execution, with each tier exposing different sources of error. Open-loop tests cover reaching, grasping, individual capabilities, and general-action preferences.

4.2 Evaluation Protocols

  • The three evaluation tiers separate single-chunk prediction quality, sequential error accumulation, and real-world controller, camera-motion, and contact effects.Using the same approaching metrics in the first two tiers enables direct comparison before task-level robot success.Source: §4.2 · p. 10

4.2.1 Open-Loop Evaluation

  • Open-loop tests score one action chunk from reconstructed human-hand starts, measuring approach distance, progress, alignment, and grasp-closure error.The approaching set contains 970 cases; the grasping set contains 120 events from 10 scenes.Source: §4.2.1 · pp. 10–11
  • Case studies probe semantic grounding, affordance, and fine-grained manipulation, while a blind user study compares general-action predictions from 250 h and 10,000 h checkpoints.The study covers 20 held-out trials, with five participants judging each trial.Source: §4.2.1 · p. 11

4.2.2 Policy-in-the-Loop Rollout

  • Policy-in-the-loop rollouts chain three relative chunks across the same 970 approaching cases, rendering each predicted hand into the next observation while holding camera pose and scene geometry fixed.The protocol reports dp10, progress, and θalign across all cases.Source: §4.2.2 · p. 11

4.2.3 Closed-Loop Real-Robot Evaluation

  • Real-robot evaluation runs closed loop at 10 Hz on an IRON-R01 unit distinct from the unit used to collect post-training data.The setting includes contact dynamics and sensor feedback.Source: §4.2.3 · p. 11

4.3 Pretraining Experience Scaling

Across 250–10,000 hours of pretraining, validation loss scales approximately log-linearly without saturation. Larger budgets also improve open-loop behavior, policy-in-the-loop rollouts, and human preference.

4.3 Pretraining Experience Scaling

  • Validation loss follows an approximately log-linear scaling law with a slope of −0.058 per decade of pretraining hours, with no saturation through 10,000 hours.The reported scaling spans budgets from 250 to 10,000 hours.Source: §4.3 · p. 11, Figure 7 · p. 12
  • Scaling pretraining data from 250 h to 10,000 h reduces dp10 from 37.8 to 25.8 cm (31.7%), increases approach progress from 14.7% to 44.9%, and lowers eFA from 19.97 to 15.66 mm.Source: §4.3 · p. 11
  • In policy-in-the-loop rollouts, 250 h to 10,000 h raises progress from 21.4% to 49.3%, lowers alignment from 46.1° to 21.6°, and reduces dp10 from 32.0 to 14.6 cm.Raters preferred the 10,000 h checkpoint in 47 of 70 decided votes, a 67.1% win rate; the study recorded 30 ties among 100 votes.Source: §4.3 · p. 12, Table 3 · p. 12

4.4 World Priors Ablation

The ablation compares world-prior configurations on open-loop approaching, with the full semantic, depth, and dynamics configuration best across the reported metrics. Future prediction, horizon, and dynamics-target choices also affect the reported outcomes.

4.4 World Priors Ablation

  • The full three-prior configuration reduces dp10 from 27.6 to 22.6 cm, increases progress from 40.9% to 53.4%, and improves alignment from 34.8° to 26.5° versus no priors.Removing any branch increases dp10 by 1.5–3.1 cm, with the largest change when depth is removed.Source: §4.4 · p. 13, Table 4 · p. 13
  • Replacing future prediction with current-frame reconstruction largely reverses the gain, yielding 27.3 cm dp10 and 42.7% progress.The branches predict future-frame dynamics, 3D motion flow, semantic features, and geometry targets over a 0.4 s horizon.Source: Figure 8 · p. 12, §4.4 · p. 13
  • The approximately 0.4 s default horizon outperforms 0.2 s and 0.8 s, while the frozen video generator has a modest advantage over the 3D motion-flow head at this budget.The authors suggest sparse flow labels may partly explain the dynamics-target difference.Source: §4.4 · p. 13

4.5 Real-Robot Experiments

On six real-robot OOD tasks, success remains low through 5,000 hours of pretraining and rises substantially at 10,000 hours. The camera-space representation also outperforms a torso-frame baseline under the reported ablation setup.

4.5 Real-Robot Experiments

  • The six-task OOD evaluation tests unseen object instances and appearances, grasp affordances, and relational or attribute-based instructions on IRON-R01.Success requires full task completion; milestone progress credits completed stages over 10 trials per setting.Source: §4.5 · p. 13, Figure 9 · p. 14
  • At 10,000 h, real-robot success is 40–90% per OOD task, versus an average below 12% for every budget up to 5,000 h.The increase is strongest on Semantic and Reasoning tasks and weaker on Affordance tasks; these scale experiments omit world-prior supervision.Source: §4.5 · pp. 13–14, Figure 10 · p. 14
  • IronMind with camera-space actions and no world-prior supervision obtains the highest overall success and progress against InternVLA-A1.5 under this evaluation setup.InternVLA-A1.5 is post-trained on the same data; its action space is re-initialized, whereas IronMind shares a unified action space across pretraining and post-training.Source: §4.5 · p. 15
  • With the same 10,000-hour dataset and consistent action representations across pretraining and post-training, the camera-space variant performs better than the torso-frame baseline.World-prior supervision improves some open-loop metrics, but its real-robot variant only modestly exceeds InternVLA-A1.5 and remains below IronMind without world priors.Source: §4.5 · p. 15

5 Limitations

IronMind’s evidence is bounded by its data and evaluation scope. The authors identify remaining robot-specific requirements and several untested settings.

5 Limitations

  • The corpus is concentrated in homes, kitchens, and workshops, limiting extreme scene diversity.The camera-space representation also still requires camera extrinsic calibration and post-training on real-robot data.Source: §5 · p. 15
  • The scaling study covers one architecture and data mixture, while robot evaluation uses one embodiment, six OOD tasks, and 10 trials per task.Longer-horizon and broader contact-rich tasks remain untested.Source: §5 · p. 15
  • How world-prior supervision gains transfer from open-loop evaluation to closed-loop deployment remains for future study.Source: §5 · p. 15
  • Comparisons beyond InternVLA-A1.5, including DreamZero, remain future work.Source: §5 · p. 15

6 Conclusion

IronMind combines camera-space action pretraining on egocentric human and heterogeneous robot data with a dual-expert policy for humanoid dexterous manipulation. Its 10,000-hour model achieves 55.0% success across six OOD tasks and outperforms the torso-frame baseline.

6 Conclusion

  • 55.0% success across six out-of-distribution tasks, versus 26.7% for the torso-frame baseline, for the post-trained 10,000-hour model.Source: §6 · p. 15
  • Validation loss scales approximately log-linearly from 250 to 10,000 pretraining hours.Source: §6 · p. 15
  • IronMind uses a unified camera-space action representation to connect human observation and robot execution without explicit embodiment retargeting.Its curated data engine supports a dual-expert Mixture-of-Transformers policy with flow-matching action generation and optional multimodal world priors.Source: §6 · p. 15
  • Evaluation spans open-loop prediction, policy-in-the-loop rollouts, and closed-loop robot execution.Source: §6 · p. 15
  • The authors identify scaling data and embodiments, longer-horizon tasks, reinforcement learning, and human intervention as future directions.Source: §6 · p. 15

Related papers

Loading 2609.39403v1…