PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoForesight: Effectively Scaling Policy Learning with Noisy Egocentric Actions

Chubin Zhang, Changyuan Wang, Yirui Wang, Ao Li, Shilin Ma, Yue Su, Cai Zhou, Xiuwei Xu, Tianle Zhang, Yansong Tang

arXiv:2610.31845v1cs.ROcs.CV

TL;DR

Egocentric videos expand policy-learning coverage but their reconstructed and retargeted actions can be noisy, while robot demonstrations are costly to collect. EgoForesight combines reliability-aware action learning with future-outcome representation distillation; its scaling experiments report stronger gains from added pretraining data and strong downstream performance in simulation and on real hardware.

  • Problem

    Egocentric videos broaden task and environment coverage, but tracking, reconstruction, and retargeting can make their action labels unreliable, while robot demonstrations are costly to collect.

    Source: §1 · p. 2

  • Method

    EgoForesight jointly learns future visual representations and action chunks, using source reliability to regulate action supervision and an EMA teacher with observed futures to distill outcome-informed action features.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Scaling experiments and component ablations support the combined approach, with improved returns from added pretraining data and strong downstream performance in simulation and on real hardware.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The results suggest that effective scaling depends on supervision quality as well as data volume.

    Source: §5 · p. 10

    Whether broader experience enables compositional generalization to unseen combinations of objects, skills, and scenes remains an open question at larger scales.

    Source: §5 · p. 10

Abstract

from arXiv · show

Scaling policy learning requires broad experience and reliable supervision. Robot demonstrations provide high-quality trajectory supervision but are costly to collect. Egocentric videos broaden task and environment coverage, yet motion reconstruction and retargeting introduce noisy action labels. To address this challenge, we introduce EgoForesight, a vision-language-action framework that learns jointly from both sources by reducing the influence of unreliable action labels and exploiting demonstrated outcomes as complementary supervision. A joint vision-action flow couples future visual representation prediction with action generation, unifying reliability-aware action learning with outcome-informed representation transfer. Specifically, reliability-conditioned corruption allocates action supervision across noise levels according to source reliability, while reliability-weighted regression regulates each source's contribution. Complementing this treatment of noisy labels, privileged-future representation distillation draws supervision from demonstrated outcomes. An exponential-moving-average teacher observes the current scene and clean future to encode action-relevant structure from the resulting scene change. Its deeper representations supervise an earlier student layer receiving a corrupted future, providing outcome-informed structure beyond noisy action annotations. Together, these mechanisms support effective scaling with noisy egocentric actions. Over the tested range, EgoForesight exhibits steeper scaling slopes than the baseline, reducing real-world open-loop action MSE by 20.5% and improving RoboCasa few-shot success by 8.1 percentage points at 30,000 hours of pretraining data. After post-training, it also achieves strong absolute performance in both simulated and real-world dexterous manipulation.

1 Introduction

EgoForesight addresses noisy action supervision in egocentric videos by jointly learning from action labels and demonstrated outcomes. It combines reliability-aware action learning with future-representation distillation, and its experiments report stronger gains from added pretraining data under matched budgets.

1 Introduction

  • The supervision sources occupy different roles: egocentric videos broaden task and environment coverage, while robot demonstrations provide high-quality trajectories at substantial collection cost.The paper situates these sources within a data pyramid that also includes vision-language data for semantic coverage.Source: §1 · p. 1
  • Egocentric action labels are noisy because tracking and hand-pose reconstruction can err under occlusion or rapid motion, while retargeting creates geometric mismatches between human and robot hands.Appearance and dynamics further complicate transfer.Source: §1 · p. 2
  • EgoForesight couples future-visual-representation and action-chunk experts with modality-specific flow paths to reduce unreliable-label influence and learn from demonstrated outcomes.Separate corruption times regulate noisy action supervision and enable outcome-informed representation transfer within the same model.Source: §1 · p. 2
  • Reliability-conditioned corruption shifts less reliable actions toward stronger corruption, while reliability-weighted regression controls each source’s contribution to learning.Together, these mechanisms support learning from diverse egocentric data without assuming uniformly reliable labels.Source: §1 · p. 2
  • An EMA teacher conditions on the current observation and clean future, then transfers deeper action representations to an earlier student layer receiving a corrupted future.The transfer supplies outcome-informed structure beyond noisy action annotations.Source: §1 · p. 2
  • Under matched data and optimization budgets, EgoForesight translates added pretraining data into stronger gains in action prediction and closed-loop control.Ablations support complementary roles for reliability conditioning and future distillation, as well as establishing action correspondences on higher-quality data before expanding coverage.Source: §1 · p. 3

2 Related Work

Related work traces embodied policy learning across model scale, action interfaces, conditioning information, and future prediction. EgoForesight builds on joint future-action modeling and compact future representations as sources of guidance for action generation.

2 Related Work

  • Embodied policies have expanded from end-to-end image-to-control learning toward generalist models sharing knowledge across tasks and embodiments, with tokenized, diffusion, and flow-based action interfaces.Pretrained-VLM transfer aims to retain semantic competence while grounding spatial and task knowledge in robot actions.Source: §2 · p. 3
  • Future prediction progressed from video-based planning to joint future-action models, including approaches using separate modality noise times and compact future representations to guide or couple action generation.LDA-1B uses future DINO features and allocates policy, dynamics, and forecasting objectives according to data quality.Source: §2 · p. 3
  • Policy conditioning has broadened beyond language and vision to include visual subgoals and execution metadata, while unified future-action models use anticipated effects as guidance.The section also points to data-pyramid, generative-representation, and noisy-data discussions in Appendix G.Source: §2 · p. 3

3 EgoForesight

EgoForesight jointly models future visual features and action chunks, using reliability to regulate noisy action supervision and demonstrated futures to inform action representations. Its training combines these mechanisms with staged data selection and flow, distillation, and language-model objectives.

3.1 Problem Formulation

  • The joint future-action model predicts future visual features and action chunks from context, while observed futures are available only during training.Its coupled experts use joint bidirectional attention, while the VLM conditioning stream cannot read generated futures.Source: §3.1 · p. 4

3.2 Reliability-Conditioned Corruption

  • Reliability-conditioned corruption gives less reliable actions stronger corruption while keeping future vision comparatively informative; reliability-weighted action loss also regulates each source’s contribution.The source reliability prior expresses relative trust without requiring calibrated error estimates or a known annotation-noise distribution.Source: §3.2 · p. 5

3.3 Privileged-Future Representation Distillation

  • An EMA teacher using the observed future transfers outcome-informed action features to a student with a corrupted future, aligning a deeper teacher feature with an earlier student layer.The teacher shares the student’s action mask, target, and Gaussian noise, while reusing detached contextual features.Source: §3.3 · pp. 5–6
  • Training first establishes visual-action correspondences on higher-quality robot demonstrations and curated human trajectories, then retains these sources while adding a larger collection of action-annotated human videos.The first-stage checkpoint initializes both student and EMA teacher; the second stage enables both proposed components.Source: §3.3 · p. 6

3.4 Training Objective

  • The training objective combines joint-flow and distillation losses, with autoregressive cross-entropy retained during continual adaptation to interleaved vision-language data.Stage one uses unit action weights and no distillation; stage two enables both proposed components, and the teacher is updated only by EMA.Source: §3.4 · p. 6

4 Experiments

The experiments evaluate RoboCasa and real-world dexterous manipulation, then test pretraining components, data scaling and allocation, and post-training choices. EgoForesight improves on the reported benchmarks and shows stronger gains as pretraining data increases within the tested range.

4.1 Model Pretraining

  • Pretraining uses two stages: S1 draws on four sources, and S2 retains them while adding 20,000 hours of action-annotated egocentric data.The 4B model trains for 100k updates in S1 and 200k in S2, with a global batch size of 2,048 in both stages.Source: §4.1 · pp. 6–7, Figure 3 · p. 7

4.2 Downstream Evaluation

  • Downstream evaluation covers 24 RoboCasa tasks and four real-world tasks, using 100 demonstrations per task in the reported few-shot and real-world protocols.RoboCasa tasks are evaluated over 50 episodes; real-world tasks use 20 trials per task on a Galaxea R1 Pro with Sharpa Wave hands.Source: §4.2 · p. 7

4.3 Overall Performance

  • 66.0% RoboCasa few-shot success exceeds robot-only DIAL by 7.7 points and DIAL with EgoDex by 4.9; real-world means reach 63.8% success and 84.8 progress.With full RoboCasa data, success is 73.9%, exceeding DIAL by 3.7 points; real-world evaluation uses four tasks and 20 trials per task.Source: §4.3 · pp. 7–8, Table 1 · p. 8, Figure 4 · p. 8

4.4.1 Pretraining

  • At one tenth pretraining scale, reliability and privileged-future distillation jointly raise RoboCasa average few-shot success to 58.6%, above either component alone.For articulated tasks, reliability alone raises success from 47.7% to 58.0%, and adding the teacher raises it to 63.3%.Source: §4.4.1 · pp. 8–9, Table 2 · p. 9
  • At 30k hours, with matched subsets and optimization budgets, EgoForesight’s real-world open-loop MSE is 20.5% lower than the baseline’s best-checkpoint MSE, and RoboCasa closed-loop success is 8.1 points higher.S1→S2 replays S1 data during S2; its average success is 58.6%, exceeding reverse ordering by 1.0 point and joint training by 4.3 points.Source: §4.4.1 · p. 9

4.4.2 Post-training

  • Privileged-future distillation improves average success during post-training, and a vision-leading denoising path yields the best full-data average of 73.9%.That path reaches 80.0% Art. success while retaining 71.9% on PnP.Source: §4.4.2 · p. 9, Table 3 · p. 10

5 Conclusion

EgoForesight combines reliability-aware action supervision with privileged-future representation distillation; scaling experiments and ablations support improved returns from added pretraining data and strong downstream performance in simulation and on real hardware.

5 Conclusion

  • Scaling experiments and ablations support improved returns from added pretraining data and strong downstream performance in simulation and on real hardware.The paper concludes that effective scaling depends on supervision quality as well as data volume.Source: §5 · p. 10
  • The authors propose extending outcome-based supervision to videos without action annotations and identify compositional generalization to unseen object, skill, and scene combinations as an open question.They suggest self-supervised objectives for learning interaction structure without reconstructed controls.Source: §5 · p. 10

A Reliability-conditioned sampling

  • For ordinary examples, lower source reliability raises action corruption while leaving the vision-time distribution unchanged.The sampling rule draws τv uniformly and sets τa = τv + (1 − τv)g, with g drawn from a reliability-dependent Beta distribution.Source: A Reliability-conditioned sampling · p. 19
  • Boundary examples set vision time to zero, preserve action-flow and representation supervision, and mask vision-flow loss, calibrating action prediction on observed futures.The boundary action distribution covers action times used by the privileged teacher without requiring an identical density.Source: A Reliability-conditioned sampling · p. 19

A.1 Population interpretation of action corruption

  • Under the stated Gaussian annotation-noise assumptions, action-field discrepancy vanishes as action corruption approaches one, motivating stronger corruption for less reliable sources.The bound provides a conditional-flow interpretation for the setting; source weights separately regulate each source's contribution to the shared objective.Source: §A.1 · p. 20

B Masked objectives and teacher implementation

The appendices specify masking and teacher implementation, data filtering, model and input formats, and a shared canonical action interface. Together, these details describe how heterogeneous visual and action sources are prepared for training.

B Masked objectives and teacher implementation

  • Valid-coordinate and valid-token masks exclude padding from action losses, attention, and representation alignment, while loss reductions differ across vision, action, and distillation objectives.Vision loss excludes boundary examples; distillation averages over valid action tokens without reliability weighting and includes boundary examples. The EMA teacher reuses detached context, visual, and state features, runs without gradients, and keeps FP32 parameters.Source: B Masked objectives and teacher… · pp. 20–21

C.1 Data and quality filtering

  • Human-video filtering retains intervals with adequate visual quality, visible hands, and acceptable global motion, while leaving pose-estimation and retargeting noise for training to handle.S1 combines real and simulated robot demonstrations with curated human trajectories; S2 adds a larger internal egocentric collection, and unavailable action coordinates are masked.Source: §C.1 · pp. 21–22, Figure 6 · p. 21

C.2 Model architecture

  • A 4B Qwen3.5 VLM conditions a 16-block coupled vision–action module, whose vision and action streams exchange information through joint attention.The vision expert predicts future features and the action expert predicts 30-step chunks, conditioned on the current observation, instruction, state, and source/control metadata, without observation history.Source: §C.2 · p. 23

C.3 Input format

  • Inputs use a composite of a primary view above two wrist-view tiles, followed by task instructions and available metadata in the VLM message.Views preserve aspect ratio with centered black padding; unavailable wrist tiles remain black, and numeric state values enter through a separate state encoder.Source: Figure 7 · p. 22, §C.3 · p. 23

C.4 Canonical action space

  • All sources use a 130-dimensional normalized action interface with source-specific availability masks; EEF targets are expressed relative to the current state and camera frame.Camera-motion compensation transports targets into the current camera frame, so consistent pose estimates remove apparent EEF motion caused solely by camera movement; valid coordinates are normalized to [−1, 1].Source: §C.4 · pp. 23–24

C.5 Tracking and retargeting errors

Egocentric action labels carry errors from tracking, hand reconstruction, and human-to-robot retargeting. The retargeting implementation converts tracked hand data into robot joint targets using a constrained geometric fit and temporal filtering.

C.5 Tracking and retargeting errors

  • Camera-tracking and hand-reconstruction errors depend on the capture and annotation pipeline, with occlusion and rapid motion degrading hand tracking even in calibrated multi-camera data.Monocular video adds camera-motion and metric-scale challenges, while EgoDex reports imperfect annotations under heavy occlusion and rapid motion.Source: §C.5 · p. 24
  • Retargeting adds morphology-induced mismatch even with accurate human poses, and matching fingertips alone need not preserve contact geometry.Human and robot hands differ in palm size, finger proportions, and joint structure; this error is distinct from upstream tracking errors and the remaining visual and dynamical domain gap.Source: §C.5 · p. 24

C.6 Retargeting implementation

  • The implementation converts EgoDex wrist and fingertip tracks into 22 absolute Sharpa Wave joint angles, processing each hand independently and resetting optimizer and filter states at episode boundaries.Tracking-validity and solver-failure flags, fitting costs, pinch weights, and keypoint errors are retained as diagnostics, but do not define the training action-validity mask.Source: §C.6 · pp. 25–26
  • Retargeting fits finger positions and distal directions with joint limits and a pinch term, then filters accepted solutions and clips per-frame joint changes.The fit uses selected wrist and finger keypoints; it enforces neither object-contact nor collision constraints, and its diagnostic RMSE measures residual geometric mismatch.Source: §C.6 · pp. 25–26

C.7 Training details

Pretraining uses a two-stage schedule, expanding from higher-quality demonstrations to a larger mixture with both proposed mechanisms enabled. The section also specifies the RoboCasa evaluation and few-shot post-training setup.

C.7 Training details

  • S1 trains for 100k updates without a privileged teacher, while S2 trains for 200k updates on the expanded mixture with both mechanisms enabled.Both stages use batch size 2,048 and 5k warmup steps; total two-stage pretraining uses 22k NVIDIA H200 GPU-hours.Source: §C.7 · p. 26, Table 6 · p. 26

D.1 RoboCasa GR1 Tabletop

  • RoboCasa evaluation follows DIAL’s 24-task protocol using humanoid tabletop trajectories, with PnP tasks and articulated tasks that additionally require closing the destination fixture.The model uses the pretraining composite format with blank wrist-camera regions, while the GR1 controller executes 29 joint commands.Source: §D.1 · pp. 26–27
  • Few-shot post-training uses one tenth of the full robot demonstrations across all 24 tasks, retains privileged-future distillation during training, and evaluates the EMA teacher with 20 flow integration steps.Privileged future observations are used only during training; each prediction generates and executes 16 action steps.Source: §D.1 · p. 27

D.2 Real-world dexterous manipulation

The real-world evaluation uses a teleoperated Galaxea R1 Pro with Sharpa Wave hands across four tabletop tasks. Methods are compared after post-training on a shared per-task demonstration budget, with task success and progress measured separately.

D.2 Real-world dexterous manipulation

  • The platform records head and wrist RGB views, joint states, end-effector poses, and hand-joint targets using a Galaxea R1 Pro with two Sharpa Wave hands.Teleoperation uses MANUS gloves and VIVE trackers; the recorded views are stored at 1280 × 720 and 10 Hz.Source: §D.2 · p. 27, Figure 9 · p. 29
  • The four tasks are bottle opening, handover, pick and place, and syringe transfer, with complete-goal success distinguished from milestone-based progress scores.Progress scores are averaged over trials and scaled to 0–100; syringe transfer accumulates credit for completed steps.Source: §D.2 · pp. 27–28
  • Each method is post-trained on 100 demonstrations per task and evaluated over 20 trials per task; π0.5's action interface is replaced with newly initialized 58-dimensional layers.That action-space adaptation changes π0.5's pretrained interface and should be considered when interpreting its performance.Source: §D.2 · pp. 28–29

D.3 Additional ablations

D.3 evaluates reliability sensitivity and scaling under controlled pretraining and downstream budgets; the appendix also presents qualitative future-prediction and IDM outputs with explicit limits on their interpretation.

D.3 Additional ablations

  • The reduced-pretraining recipe uses approximately 1k hours of S1 data and 2k hours of additional S2 egocentric data, retaining S1 replay during S2.It uses 50k S1 updates and 100k S2 updates at batch size 256 before RoboCasa few-shot adaptation.Source: §D.3 · p. 29
  • Scaling compares matched baseline and EgoForesight runs on nested pretraining subsets, with fixed downstream adaptation budgets and the baseline disabling both proposed mechanisms.The nonzero scales retain matched source proportions and S1 replay during S2; the zero-data control skips both pretraining stages.Source: §D.3 · p. 29
  • Figure 10 presents real-world task demonstrations as rows ordered from each task’s initial configuration to completion.Each row progresses left to right.Source: Figure 10 · p. 30
  • The scaling evaluation fits log10(MSE) or success percentage against log10(hours) at 3k, 10k, and 30k hours, excluding the zero-hour control.Real-world MSE uses held-out teleoperated actions excluded from adaptation; the baseline full-scale value is its lowest observed MSE, 0.039, rather than its 0.042 endpoint.Source: §D.3 · p. 30
  • All tested human-data reliability values below one raise average RoboCasa success from 52.0% to 54.0–54.3%, with gains concentrated on Art. and little change from loss downweighting at qd = 0.2.Art. rises from 47.7% to 54.9–58.0%, while PnP remains near 53%; at qd = 0.2, unit loss weights give 54.2% versus 54.3% average success.Source: §D.3 · p. 30, Table 9 · p. 31

E Qualitative future prediction and IDM outputs

  • Figure 11 qualitatively shows current-only future prediction and IDM outputs conditioned on current and ground-truth future observations, but its reconstructions are lossy illustrations rather than pixel-accurate futures or held-out evaluations.Pooled ground-truth and predicted features use the same decoder; the displayed IDM outputs are relatively smooth against ground truth.Source: E Qualitative future prediction and… · pp. 30–31, Figure 11 · p. 31

F Limitations

The analysis identifies assumptions and scope boundaries in the noise model, reliability estimates, supervision, and tested generalization.

F Limitations

  • The appendix analysis assumes conditionally independent, zero-mean Gaussian annotation noise; systematic retargeting bias can still shift conditional action means at maximal corruption.Source weighting alone does not correct these shifts.Source: F Limitations · p. 31
  • Dataset-level reliability misses within-source variation, and the method depends on action annotations rather than learning self-supervised from unannotated video.The authors identify example-, timestep-, or action-coordinate-level reliability estimates as future directions.Source: F Limitations · p. 31
  • The tested scales do not verify compositional generalization to unseen object–skill–scene combinations, while predicted futures and EMA teachers may provide biased or action-irrelevant supervision at added computational cost.Source: F Limitations · p. 31

G.1 Embodied Data Pyramid

The related-work discussion situates learning from diverse embodied data alongside representation guidance and noisy-data methods; the remaining statements address AI use, ethics, and reproducibility.

G.1 Embodied Data Pyramid

  • Human videos broaden interaction coverage, while robot demonstrations link observations to executable controls; prior approaches infer latent actions or transfer action distributions across embodiments.EgoMimic and EgoScale convert tracked human hand motion into robot supervision, while X-Diffusion transfers action distributions across embodiments (Zheng et al., 2026b; Pace et al., 2025).Source: §G.1 · pp. 31–32

G.2 Representation Learning for Generation

  • Representation guidance ranges from pretrained clean-image features to generator-internal targets, including EMA-teacher alignment of deeper features to earlier student features.Self-Flow also evaluates this internal approach on joint video–action prediction.Source: §G.2 · p. 32

G.3 Learning from Noisy Data

  • Prior noisy-data methods adjust example contributions or use noise- and timestep-dependent data, and this work jointly adapts action corruption and loss weighting to source reliability.Source: §G.3 · p. 32

AI use statement

  • Generative AI tools assisted with manuscript polishing, teaser and pipeline-icon creation, coding, and literature search and review; AI-generated visuals exclude dataset example images.The authors state that they remain responsible for manuscript, code, references, and reported-result accuracy.Source: AI use statement · pp. 32–33

Ethics statement

  • Internal egocentric pretraining data cannot be publicly released under applicable access and disclosure policies, and reported results establish no safety beyond evaluated tasks and conditions.The authors call for task-specific safety assessment, safeguards, and human oversight for deployment.Source: Ethics statement · p. 33

Reproducibility statement

  • Full pretraining reproduction is limited by restricted internal egocentric data, while post-training is specified for reproduction from pretrained checkpoints.The authors say they will open-source the implementation and release model checkpoints.Source: Reproducibility statement · p. 33

Related papers

Loading 2610.31845v1…