PaperlayerView exact revision on arXiv

Source-linked AI summary

What 30,000 Hours of Ego-centric Video Does Not Teach

Jiahua Dong, Anurag Bagchi, Yash Jangir, Muhammad Zubair Irshad, Sergey Zakharov, Martial Hebert, Homanga Bharadhwaj, Yu-Xiong Wang, Vitor Campagnolo Guizilini, Pavel Tokmakov

arXiv:2610.12464v1cs.CV

TL;DR

World models need to predict how objects respond to actions, but the effect of scaling ego-centric video on interaction fidelity remains insufficiently measured. The paper directly evaluates agent and object fidelity across scaled training data and introduces supervision focused on manipulated-object dynamics. Agent fidelity approaches its limit much earlier than object fidelity, and the findings transfer to humanoid modeling.

  • Problem

    World models depend on predicting object responses to actions, but prior downstream evaluations compound perception, motion, object dynamics, and policy behavior, leaving interaction fidelity itself insufficiently measured across scale.

    Source: §1 · p. 2

  • Method

    The paper trains Cosmos 3 on nested ego-centric human-manipulation datasets and separately evaluates agent motion and its effects on objects on an out-of-distribution benchmark.

    Source: §1 · p. 2

  • Results

    Agent fidelity approaches its limit much earlier than object fidelity, which converges far below it even with the proposed supervision scheme.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Scaling ego-centric video improves both agent and object fidelity, but collection alone is not expected to close the interaction gap; gains also transfer to humanoids.

    Source: §1 · p. 3, §7 · p. 10

    The data ladder couples data volume with optimization budget, so it establishes fidelity trends under joint scaling rather than either factor’s effect alone.

    Source: §7 · p. 10

Abstract

from arXiv · show

World models offer a promising alternative to physics-based simulators, yet remain far from practical deployment. We ask how far scaling ego-centric human video takes them, using a dataset of 30,000 hours spanning over 1,000 scene types and 14,000 contributors. Rather than relying on opaque downstream metrics, we directly evaluate agent and object-interaction fidelity on a challenging out-of-distribution benchmark. Increasing training data by 100x improves both, but unevenly: the agent is modeled well, while object fidelity remains far lower and improves slowly. We show that the agent gains need not come from data, and a careful visual conditioning design saturates fidelity with a fraction of it, which lets us measure object fidelity on its own and discover its saturation point. We then introduce a supervision scheme that shifts capacity from scene appearance toward object dynamics, improving object fidelity though a substantial gap remains. Finally, our conclusions transfer to downstream humanoid modeling. Overall, our results suggest that scaling ego-centric data brings agent modeling close to its limit while leaving its effects on the world far behind, and that closing this gap will depend on how models are trained, not only on how much data they see.

1 INTRODUCTION

The paper tests whether scaling ego-centric video improves interaction fidelity, which matters because world models need to predict how objects respond to actions. Across controlled data scaling, agent fidelity approaches its limit much sooner than object fidelity; a supervision intervention improves object fidelity, and the findings transfer to humanoid modeling.

1 INTRODUCTION

  • The study addresses the unresolved claim that pixel-level objectives underweight objects and contacts because they allocate capacity by pixel mass and predictability rather than causal relevance.Source: §1 · p. 2
  • The study directly measures agent motion and its effects on the world separately, using interaction-fidelity metrics on a new out-of-distribution benchmark.It trains Cosmos 3 on nested subsets from 300 to 30,000 hours and repeats the sweep across three model variants and two action-conditioning designs.Source: §1 · p. 2
  • Agent fidelity approaches its limit much earlier than object fidelity, whose convergence remains far lower even with the proposed supervision scheme.The paper also reports that careful conditioning reaches nearly the same agent-fidelity limit with 1k hours that standard conditioning reaches with 30k hours.Source: §1 · p. 2
  • The paper presents a controlled study of interaction fidelity as a function of ego-centric data scale, training Cosmos 3 on up to 30,000 hours of human–object interaction.Source: §1 · p. 2
  • Object-centric adaptive noise scheduling shifts supervision from overall scene appearance toward manipulated-object dynamics and raises the level to which object fidelity converges.It is the only tested intervention to raise that convergence level, though a substantial gap remains.Source: §1 · pp. 2–3
  • The gains transfer to downstream humanoid modeling, where the design contributes more than ego-centric pre-training alone, but world interactions remain a bottleneck.Source: §1 · p. 3

2 RELATED WORK

Related work has expanded world modeling from small robot datasets to large-scale ego-centric human video, while raising questions about whether current objectives and evaluations measure interaction dynamics well. This paper builds on direct structural evaluation and tests the concern that pixel prediction can underweight interaction-relevant objects.

2 RELATED WORK

  • Robot world models progressed from fixed, small-scale setups to diffusion and latent-transformer approaches with larger interaction datasets, but teleoperation remains costly and datasets cover only a few hundred scenes.Source: §2 · p. 3
  • Ego-centric human video is abundant and diverse, prompting large-scale pretraining and reports that more human video improves world models.Examples include DreamDojo at 44k hours and Being-H0.7 at 200k hours.Source: §2 · p. 3
  • Aggregate policy and action metrics compound perception, dynamics, and control, while video-generation metrics reward perceptual quality rather than accurate dynamics.The paper adopts segmentation-based structural-agreement metrics and reports agent and interacted-object fidelity separately.Source: §2 · p. 3
  • The pixel-prediction critique is that capacity may flow to dominant, predictable background detail instead of small, occluded objects where interactions occur.Source: §2 · p. 3
  • Prior object-centric representations and auxiliary targets sacrifice scalability or fail to generalize to ego-centric video, while PhysisForcing shows no improvement at scale in this setting.The paper proposes an approach that does not rely on PhysisForcing’s fixed-camera, largely unoccluded setting and raises the level object fidelity converges to.Source: §2 · p. 4

3 EXPERIMENT PROTOCOL

The protocol scales Cosmos 3 world models across model variants and nested training-data budgets, then evaluates their predictions on an out-of-distribution interaction benchmark. It details the model’s action and skeleton conditioning, diverse training corpus, and region-focused fidelity metrics.

3.1 WORLD MODEL DESIGN

  • Cosmos 3 predicts future frames from an initial RGB observation, text, and agent motion, using 4B, 16B, and 64B model variants.The 4B Edge variant differs in pretraining and design; Nano and Super share initialization, isolating scale in their comparison.Source: §3.1 · p. 4
  • The action space includes camera and wrist poses, hand joints, and shoulder and elbow positions, encoded as clean action tokens during denoising.Skeleton conditioning additionally projects the hands and shoulder–elbow–wrist chains onto the image plane, encoding depth with color and shading.Source: §3.1 · pp. 4–5

3.2 TRAINING

  • The 30,000-hour corpus contains 1.15 million first-person clips from 14,019 participants, spanning 116 environment categories, 1,106 scene categories, and 32,051 tasks.Training uses balanced, nested checkpoints at 300, 1k, 3k, 10k, and 30k hours, each extending the same progressive data prefix.Source: §3.2 · p. 5

3.3 EVALUATION

  • Models are tested on 150 out-of-distribution clips from a separate campaign, with different platforms, operators, and sites and no shared participants or locations.The evaluation clips contain active manipulation, segmentable objects, and accurately annotated body joints.Source: §3.3 · p. 6
  • Evaluation measures hand and manipulated-object structural consistency separately, with LPIPS as a perceptual control, on 17-frame windows containing one observed and 16 predicted frames.The 150-clip set has 11,328 instance masks; tracker agreement estimates attainable SCS maxima of 0.93 for agents and 0.90 for objects.Source: §3.3 · p. 6

4 HOW FAR DOES EGO-CENTRIC DATA TAKE US?

Scaling ego-centric training data improves both agent and object fidelity, but the agent saturates while object fidelity remains much lower. Skeleton conditioning reaches the agent’s limit with far less data, revealing a saturating but lower object ceiling.

4 HOW FAR DOES EGO-CENTRIC DATA TAKE US?

  • 0.12 SCS from 300 to 3k hours but 0.06 from 3k to 30k: agent fidelity decelerates, with a fitted asymptote of 0.811.Object fidelity shows no comparable deceleration without skeleton conditioning, leaving its limit undetermined over the observed data range.Source: §4 · p. 7
  • 300 hours of skeleton-conditioned Nano matches the baseline agent fidelity at roughly 15k hours, while both designs converge within about 0.01 SCS.The 300-hour conditioned model gains just 0.06 SCS over the remaining two orders of magnitude, helping isolate object scaling.Source: §4 · p. 7
  • 0.09 SCS: skeleton-conditioned object fidelity improves by this amount across two orders of magnitude and fits an asymptote of 0.565, with 93% realized at 30k hours.The model reaches 86% of the measurable agent range but 63% of the measurable object range.Source: §4 · p. 7
  • 0.034 SCS per fourfold model-size increase raises the object ceiling, but closing the remaining 0.205 to the agent ceiling would require roughly six more quadruplings.That estimate is a lower bound assuming the gain per quadrupling does not diminish; training compute rises in proportion to model size.Source: §4 · pp. 7–8
  • Across the data ladder, object fidelity with dynamic-region supervision exceeds the skeleton baseline at every budget and converges to a higher limit.Source: Figure 5 · p. 8

5 CAN THE OBJECT CEILING BE MOVED?

Object-centric adaptive noise scheduling raises the object-fidelity ceiling by shifting supervision toward manipulated-object dynamics, but the gain remains partial and object fidelity stays far below agent fidelity.

Existing interaction-focused supervision

  • PhysisForcing objectives do not improve over the skeleton-conditioned baseline at 10k hours because their static-camera and largely unoccluded-object assumptions do not hold in ego-centric manipulation.The camera moves with the agent, and the hand occludes the object at contact.Source: §5 · p. 8

Localizing dynamic regions

  • Dynamic regions are localized by tracking points through video and combining their 3D displacement and reliability with a soft prior around projected hands.The resulting scores are expanded over neighboring latent cells to form a dynamic region map.Source: §5 · p. 8

Shifting the supervision

  • Within selected regions, noise is raised so denoising must rely on context and dynamics rather than local appearance, while supervision still targets the same clean latent.The noise increment is scaled by λ; the method retains σ as the model’s noise-level input.Source: §5 · p. 8

Results

  • Raising noise in dynamic regions improves object fidelity, whereas reweighting alone does not meaningfully improve fidelity.Randomly placed regions of the same extent remove nearly all the gain; reweighting adds a small further improvement at no extra cost, and LPIPS remains unchanged.Source: §5 · pp. 8–9
  • At 30k hours, object interaction fidelity reaches 0.546 against the skeleton baseline’s 0.527, a 3.7% improvement, while remaining far below agent fidelity.The gain is roughly constant across budgets, raising the ceiling rather than the rate of approach.Source: §5 · p. 9

6 TRANSFER TO HUMANOIDS

Ego-centric pre-training transfers to humanoid modeling, and the paper’s conditioning and supervision design adds further gains. In this simulated setting, object fidelity is high in absolute terms, but prediction errors still grow more for objects than for hands over the horizon.

Setup

  • The fine-tuning comparison uses 11 simulated bimanual tasks, conditions on a start frame and humanoid state trajectory, and predicts the next 16 frames.The three variants isolate ego-centric pre-training and the paper’s design; evaluation uses EgoVLA policy rollouts.Source: §6 · p. 9

Metrics

  • The evaluation reports separate hand and object SCS and compares rendered-rollout success judgments with the simulator’s verdict over the critical 17-frame window.Restricting evaluation to that window isolates world-model fidelity from long-rollout drift.Source: §6 · p. 9

Results

  • Ego-centric human pre-training improves every measure, especially hand SCS (+0.07), while the conditioning and supervision design adds more than twice as much on top; gains favor the agent over manipulated objects (+0.17 versus +0.06).Object fidelity is high in absolute terms for every variant (0.70–0.79).Source: §6 · p. 9
  • Hand SCS for OURS falls from 0.89 at frame 2 to 0.85 at frame 16, while object SCS falls more than twice as far, from 0.82 to 0.71.Success agreement follows the same ordering with smaller margins, but is less diagnostic because it does not localize the error.Source: §6 · p. 9

7 DISCUSSION AND LIMITATIONS

More ego-centric data improves both agent and object fidelity, but collection alone does not close the interaction gap. The paper also describes its collection protocol, corpus diversity, and limits on what its scaling results establish.

7 DISCUSSION AND LIMITATIONS

  • Agent fidelity approaches its limit before 30k hours, while the fidelity of the agent’s effects on the world converges far below it, even with the supervision scheme.More data improved both axes at every measured scale, and pre-training transferred to humanoids.Source: §7 · p. 10
  • Jointly scaling data volume and optimization budget means the study establishes the direction of fidelity under joint scaling, not either factor’s effect alone.The smaller budgets are further from convergence, making the reported ceilings optimistic; generalization is also limited by a single collection protocol, a small evaluation set, and the focus on generative models.Source: §7 · p. 10

A.1 OVERVIEW AND MODALITIES

  • The training corpus comprises continuous head-mounted-camera recordings of everyday human manipulation, with aligned video, camera calibration, hand and body pose, and language annotations.Each clip averages 94.3 seconds; the corpus contains 1,146,100 clips totaling 30,012 hours.Source: §A.1 · p. 14

A.2 SCALE AND DIVERSITY

  • The corpus spans 116 environments, 1,106 scenes, 32,051 tasks, and 14,019 participants, with coverage distributed across many categories rather than concentrated in a few.The most common environment, home, accounts for 8.3% of clips; 34 environment categories are needed to cover 90%.Source: §A.2 · p. 14

A.3 CATEGORY DISTRIBUTIONS

  • Examples span food production, retail, and craft/repair workshops; tasks are dominated by fine bimanual manipulation, including ironing and folding clothes, shaping and rolling dough, cleaning, and packaging.Figure 6 reports mean and median clip lengths of 94.3 s and 94.5 s; 99.9% of clips are ≤123 s, while a sparse 0.04% tail extends to a 240 s cap and is omitted. Figure 7 presents the top 30 of 116 environments and top 30 of 1,106 scenes as percentages of all clips.Source: §A.3 · p. 15, Figure 6 · p. 15, Figure 7 · p. 15

A.4 BALANCED DATA SCALING LADDER

The scaling ladder uses a fixed, balanced episode order and nested data budgets. Its reported exposure in hours measures corpus consumption rather than the duration of sampled video windows.

Balanced ordering

  • The episode order groups shared environment and scene labels, orders episodes deterministically within each group, and interleaves groups according to their duration shares in the reference pool.The resulting balance is approximate because episode durations vary and group data are finite.Source: §A.4 · pp. 15–16

Balanced data scaling ladder

  • Task composition spans 874 distinct actions and 9,397 distinct objects across 32,051 action–object tasks.The figure groups the most frequent actions and objects while aggregating the remainder into “other” categories.Source: Figure 8 · p. 16

Nested budgets

  • Larger budgets include all samples from smaller budgets, and each configuration processes each sample once, so data exposure and optimizer updates rise together.The budgets are 300, 1k, 3k, 10k, and 30k hours.Source: §A.4 · p. 16

Exposure in hours

  • Reported hours measure the fraction of the fixed manifest consumed, normalized to its nominal 30,000-hour size, rather than summing sampled 17-frame window durations.Budgets are aligned to whole optimizer steps and hour labels are rounded.Source: §A.4 · p. 16

B.1 HUMAN SAMPLES

Human samples show improved agent motion alongside persistent object-interaction failures. In humanoid rollouts, accurate agent modeling does not guarantee that objects respond faithfully to actions.

B.1 HUMAN SAMPLES

  • Contact modeling, physical-property inference, object permanence, and non-rigid dynamics remain failure modes despite improvements in agent fidelity.Examples include a spoon that is not lifted, a rigid board that stretches, a disappearing cucumber, and incorrect paper-folding dynamics.Source: §B.1 · p. 17

B.2 HUMANOID SAMPLES

  • A correctly modeled agent can still produce incorrect object responses in humanoid rollouts, including a box that does not turn, a laptop that stays closed, and a can that fails to drop.The examples compare skeleton-conditioned predictions with simulator rollouts.Source: §B.2 · p. 17

C.1 SKELETON DROPOUT ABLATION

Skeleton dropout does not improve object interaction at a matched training budget, while perceptual quality shows similar saturation to interaction fidelity. Dynamic-region supervision targets moving regions near the hands, shifting training emphasis toward manipulated-object dynamics.

C.1 SKELETON DROPOUT ABLATION

  • At 10k training hours, skeleton dropout has no meaningful effect on agent fidelity and marginally lowers object fidelity.Both variants use full skeleton conditioning at evaluation and standard denoising supervision.Source: §C.1 · p. 17, Table 5 · p. 20
  • The human and humanoid examples show persistent interaction errors, including incorrect object configurations and objects that fail to respond to actions.Examples include a spoon that is not lifted, a box that does not turn, a laptop that stays closed, and a can that remains near the bin edge.Source: Figure 9 · p. 18, Figure 10 · p. 19

C.2 PERCEPTUAL SCALING

  • LPIPS shows similar saturation behavior to the main paper’s interaction-fidelity metrics as training exposure increases.The reported LPIPS (AlexNet) averages over 16 predicted validation frames and excludes the observed frame.Source: §C.2 · p. 20, Figure 11 · p. 20

D DYNAMIC-REGION SUPERVISION DETAILS

  • Dynamic-region supervision uses 3D motion, tracking reliability, and a soft hand-proximity prior to target regions for increased noise or loss reweighting.The hand prior is spatial, not a contact label; the random-region baseline instead uses a content-independent rectangle.Source: D DYNAMIC-REGION SUPERVISION DETAILS · pp. 20–22, Figure 12 · p. 21

E HUMANOID EVALUATION DETAILS

The humanoid evaluation fine-tunes three variants on demonstrations from the same 11 tasks and assesses mask-based fidelity and agreement with simulator success judgments. Its agreement protocol uses short decision windows, excludes tasks without a suitable decisive moment, and rectifies evaluation images to match the training camera model.

E HUMANOID EVALUATION DETAILS

  • The humanoid state space is 138-dimensional, so fine-tuning reinitializes the action projection while leaving the rest of the model unchanged.All three variants use ground-truth demonstrations from the same 11 tasks; OURS retains skeleton conditioning and dynamic-region supervision.Source: E HUMANOID EVALUATION DETAILS · p. 22
  • Hand and object SCS use manually verified masks, while success agreement compares verdicts on rendered 17-frame outcome windows against the simulator.Hand masks are tracked from SAM2 prompts and object masks use SAM3 with per-task text prompts.Source: E HUMANOID EVALUATION DETAILS · pp. 22–23
  • Variants are reported at their best checkpoint within capped windows because they converge at different rates, making a shared early iteration misleading.Checkpoint selection on reported clips is optimistic in absolute terms, though applied identically across variants.Source: E HUMANOID EVALUATION DETAILS · p. 23, Table 6 · p. 23
  • Tasks without a decisive outcome in a 17-frame window are excluded from success-agreement evaluation.Seven of 11 tasks have a decisive moment; other outcomes take longer or have overly lenient simulator success conditions.Source: E HUMANOID EVALUATION DETAILS · p. 23, Table 8 · p. 24

F EVALUATION DETAILS

  • Evaluation frames are reprojected from the stereo rig’s double-sphere fisheye into a 908×512 pinhole view consistent with training data.The correction isolates content and appearance shift from lens-model mismatch.Source: F EVALUATION DETAILS · p. 24, Figure 13 · p. 24

G IMPLEMENTATION DETAILS

The implementation predicts 16 future frames from one observation and uses a progressive training pass through the data manifest. Action coordinates are normalized using fixed percentile statistics, and evaluation uses EMA checkpoints with fixed sampling settings.

G IMPLEMENTATION DETAILS

  • Training uses 17-frame RGB sequences at 10 fps and 512 × 512 resolution, with one observed frame followed by 16 predicted frames spanning 1.6 seconds.Each configuration follows one progressive pass through the training manifest, so optimization budget scales with data exposure.Source: G IMPLEMENTATION DETAILS · p. 24
  • Action normalization maps per-coordinate 1st and 99th percentile intervals to [−1, 1] without clipping, using separate statistics for absolute states and frame differences.The statistics are fixed throughout training and evaluation.Source: G IMPLEMENTATION DETAILS · p. 24
  • Evaluation uses EMA checkpoints on 150 held-out clips, UniPC with 35 sampling steps, and fixed classifier-free guidance scale 5.Agent and object SCS and LPIPS average over future frames 1–16; checkpoints are reported at prescribed budgets without validation-based selection.Source: G IMPLEMENTATION DETAILS · p. 24
Loading 2610.12464v1…