PaperlayerView exact revision on arXiv

Source-linked AI summary

World Action Learning via Interaction-Centric Spectral Latent Guidance

Zhiming Liu, Yikun Miao, Ying Chen, Hongrui Yin, Fangqi Zhu, Xiaoyi Pang, Quanxin Shou, Zhengyang Yan, Haodong Wang, Song Guo

arXiv:2610.03607v1cs.RO

TL;DR

Robot demonstrations are costly to scale, and egocentric videos offer abundant interaction experience but contain observer motion and differ from robot behavior in temporal dynamics. WING separates observer motion from interaction dynamics and uses shared low-frequency latent structure to guide robot action generation. It reports strong results across simulation benchmarks and real-world manipulation tasks.

  • Problem

    Robot demonstrations are costly to scale, while egocentric latent actions can mix interaction dynamics with observer motion and human–robot executions differ in temporal dynamics.

    Source: §1 · pp. 1–2

  • Method

    WING distills interaction-centric latent actions from egocentric video and uses low-frequency spectral components as guidance for robot action generation.

    Source: §1 · pp. 2–3

  • Results

    99.2% average success on LIBERO, 93.80% on RoboTwin 2.0, and 57.7% on RoboCasa–GR1; WING also performs strongly across four real-world bimanual tasks under diverse generalization settings.

    Source: §1 · p. 3

  • Takeaways & Limitations

    WING’s results suggest that robot learning from human experience benefits from identifying interaction structures consistent across observers and embodiments.

    Source: §6 · p. 10

    WING-LAM construction relies on external motion cues and an image-space camera–interaction decomposition whose single global affine warp can be less accurate under strong parallax, large depth variation, or pronounced 3D camera motion.

    Source: §5 · p. 10

Abstract

from arXiv · show

Learning general-purpose robot policies requires large-scale real-world interaction data, yet collecting robot demonstrations remains expensive and difficult to scale. Egocentric videos offer abundant human interaction experience with task-relevant semantics for robotic manipulation, but direct transfer is challenging for two reasons: latent actions inferred from frame reconstruction can be dominated by nuisance variation such as ego-camera motion, and human and robot behaviors often exhibit different temporal dynamics. We propose WING (World Action Learning via INteraction-Centric Spectral Latent Guidance), a framework for transferring interaction knowledge from egocentric videos to robot policies. WING first separates observer-induced motion from hand-object interaction and distills the interaction-centric component into latent actions. It then exploits the observation that cross-embodiment task semantics are concentrated in slowly varying temporal structures, identifying shared low-frequency components between egocentric latent actions and robot behaviors in the spectral domain and using them to guide action generation. WING achieves average success rates of 99.20% on LIBERO, 93.80% on RoboTwin 2.0, and 57.7% on RoboCasa-GR1, and also performs strongly across four real-world manipulation tasks under diverse generalization settings. These results show that interaction-centric spectral guidance provides an effective and scalable way to transfer physical interaction knowledge from human egocentric video to robot control. Project page: https://mikuz12.github.io/wing/

1 Introduction

Egocentric video offers abundant interaction experience, but transferring it to robots is difficult because learned latent actions can mix interaction dynamics with observer motion, while human and robot executions differ temporally. WING separates these sources and uses shared low-frequency structure to guide robot actions.

Motivation

  • Egocentric videos offer abundant human-object interaction experience, while collecting robot teleoperation data is costly and difficult to scale.First-person video observations resemble those available to robots.Source: §1 · p. 1

Challenge

  • Reconstruction-based latent actions may capture visually predictive variation instead of interaction-relevant dynamics, motivating the question of how to extract transferable interaction knowledge from egocentric video.Explicit hand-pose supervision can also be tightly coupled to human morphology and kinematics.Source: §1 · pp. 1–2
  • Human head and body motion can entangle with, or dominate, latent representations even though transferable information primarily lies in hand-object interactions.Robot head cameras are typically stable, unlike human-captured egocentric viewpoints.Source: §1 · p. 2

Approach

  • WING distills interaction-centric latent actions and uses low-frequency spectral structure, which shows stronger correspondence across semantically matched human–robot interactions, to guide action generation.Its approach combines interaction-centric latent learning with spectral latent guidance.Source: §1 · pp. 2–3

Results

  • 99.2% average success on LIBERO, 93.80% on RoboTwin 2.0, and 57.7% on RoboCasa–GR1; WING also maintains strong performance across four real-world bimanual tasks and diverse generalization settings.The reported simulation results outperform strong baselines including π0.5 and LingBot-VA.Source: §1 · p. 3

2 Related Work

Prior work uses egocentric human video for robot learning, but ego-centric transitions can entangle manipulation structure with observer-induced camera motion. WING targets this gap by separating observer motion from interaction dynamics and extracting shared low-frequency structure.

Prior work

  • VLA and world-action models still rely heavily on robot data, while prior egocentric-video approaches span representation learning, behavior transfer, and latent-action modeling.Existing approaches model full visual transitions, reduce action-irrelevant variation, or use camera motion as an active-perception cue.Source: §2 · p. 3

Gap and approach

  • Ego-centric transitions entangle interaction dynamics with observer-induced camera motion, complicating identification of transferable manipulation structure.WING addresses this gap by disentangling observer motion and extracting low-frequency temporal structure shared more consistently across human and robot executions.Source: §2 · p. 3

3 Methods

WING transfers interaction structure from egocentric video to robot policies through interaction-centric latent actions and low-frequency spectral guidance. Its training pipeline learns video representations and guidance before adapting them to robot demonstrations.

3.1 Overview

  • WING maps language, visual observations, and robot state to executable action chunks, addressing observer-motion entanglement and human–robot temporal mismatch with two components.WING-LAM encodes egocentric transitions while suppressing task-irrelevant motion; spectral guidance retains slowly varying latent structure more consistent across embodiments.Source: §3.1 · pp. 3–4

3.2 Learning Interaction-Centric Latent Actions

  • WING-LAM separates global observer motion from localized interaction motion, distilling interaction dynamics into latent actions that are less sensitive to observer variation.The design uses motion-aware supervision and camera interventions, then distills the teacher’s interaction representation to an RGB-only student; consistency under camera-like warps is encouraged during student training.Source: §3.2 · pp. 4–5

3.3 Spectral Latent Action Guidance

  • WING retains the first K low-frequency DCT components of latent trajectories because these components show stronger correspondence across semantically matched human and robot executions.The truncation applies to latent guidance, not robot actions.Source: §3.3 · p. 5
  • A predictor estimates spectral guidance from current context, and projected, normalized frequency tokens condition the action model through a gated residual update.Future observations form training targets; the guidance predictor uses current observation and instruction, with proprioception when available.Source: §3.3 · pp. 5–6

3.4 Training Strategy

  • WAM training first pretrains on egocentric video for future-frame generation and spectral-guidance prediction, then transfers the visual backbone and predictor for joint robot policy learning.The WING-LAM encoder remains frozen during robot training, while robot demonstrations supervise executable actions.Source: §3.4 · p. 6

4 Experiments

The experiments examine transferable structure in egocentric latent actions, WING’s robot-policy performance, component ablations, and egocentric pretraining across simulation and real-world tasks.

4.1 Experimental Setup

  • The evaluation compares latent-motion readout, camera perturbation, action classification, and frequency-band similarity, alongside policy results on three simulation benchmarks and a real-world dual-arm platform.Policy evaluations report simulation success rates and real-world task completion and progress.Source: §4.1 · p. 6

4.2 (Q1): Characterizing Egocentric Latent Structure For Robot Learning.

  • Low-frequency components separate semantically matched from mismatched human–robot pairs more strongly than full trajectories or high-frequency components.WING-LAM also retains more interaction-motion and action-classification information while reducing camera-motion information relative to the compared baselines.Source: §4.2 · p. 7, Figure 3 · p. 7

4.3 (Q2): Robot Policy Performance

  • 99.20% on LIBERO, 93.80% on RoboTwin 2.0, and 57.7% on RoboCasa–GR1; WING outperforms the strongest baselines on all three.The evaluations span long-horizon tasks and randomized scenes.Source: Table 1 · p. 7, §4.3 · p. 7, Table 2 · p. 8
  • 75.0% average success under the standard setting; across settings WING surpasses FastWAM and GR00T-N1.7 in success and progress, and matches π0.5 success under generalization with slightly higher progress.Figure 4 covers four real-world manipulation tasks.Source: §4.3 · p. 7, Figure 4 · p. 8

4.4 (Q3): Ablation Studies on WING-LAM and Spectral Guidance.

  • Removing either WING-LAM or DCT guidance lowers average performance on LIBERO and real-world evaluations; DCT gains are attributed to retaining slowly varying interaction dynamics.The ablations are evaluated in a controlled setting without ego-pretraining of the WAM.Source: Table 3 · p. 8, §4.4 · p. 8
  • K = 4 consistently performs best; K = 2 discards useful dynamics, while the full spectrum adds high-frequency components without improving downstream control.The tested values are K ∈ {2, 4, 8}, with K = 8 corresponding to the full spectrum.Source: Figure 5 · p. 9, §4.4 · p. 9

4.5 (Q4): Analysis of Ego Pre-training Strategies.

  • Under a shared 200h pretraining budget, spectral-guidance pretraining beats video-only, while replacing WING-LAM or predicting latent actions directly degrades performance in most settings.The analyses also report stronger downstream performance as robot supervision becomes more limited.Source: §4.5 · p. 9, Figure 5 · p. 9

5 Limitations and Future Work

WING’s teacher construction and training pipeline have stated modeling and computational constraints, and the paper identifies directions for addressing them.

5 Limitations and Future Work

  • The teacher relies on external motion cues and image-space camera–interaction decomposition, approximating observer motion with a single global affine warp.Under strong parallax, large depth variation, or pronounced 3D camera motion, this approximation may become less accurate and introduce noise into interaction supervision.Source: §5 · p. 10
  • Latent-action pretraining and spectral guidance add training cost and pipeline complexity.Source: §5 · p. 10
  • The paper proposes more self-contained, depth-aware motion modeling and more efficient guidance mechanisms as future directions.Source: §5 · p. 10

6 Conclusion

WING transfers interaction knowledge from egocentric videos to robot learning by suppressing observer-induced variation and using cross-embodiment-consistent low-frequency dynamics. Simulation and real-world experiments show strong performance and generalization across diverse tasks.

6 Conclusion

  • WING learns interaction-centric latent knowledge by suppressing observer-induced variation, then uses low-frequency dynamics that are more consistent across embodiments as spectral guidance.Source: §6 · p. 10
  • Extensive simulation and real-world experiments demonstrate effective transfer and strong performance and generalization across diverse manipulation tasks.Source: §6 · p. 10
  • The results suggest that scaling robot learning with human experience requires identifying interaction structures consistent across observers and embodiments, rather than modeling all observed dynamics.Source: §6 · p. 10

A.1 Observer Variation in a Reconstruction Bottleneck

The appendix analyzes observer variation in reconstruction bottlenecks, latent-action sensitivity to observer motion, DCT frequency ordering, and WING’s relationship to world-action models.

A.1 Observer Variation in a Reconstruction Bottleneck

  • A reconstruction bottleneck selects directions by total transition variance, so observer variation remains whenever a selected direction has positive observer variance.Source: §A.1 · p. 15

A.2 Occurrence and Sensitivity of Observer Variation

  • Expected squared latent change separates observer-motion occurrence probability from mean squared encoder response conditional on motion; comparisons require matched perturbations and latent scale.Source: §A.2 · p. 16

A.3 Information Preservation and Frequency-Ordered Coordinates

  • The DCT preserves full-trajectory information and orders modes by temporal variation, with higher indices representing faster changes; this alone does not establish cross-embodiment correspondence.Source: §A.3 · pp. 16–17

B.1 Vision-Language-Action and World-Action Models

  • WING extracts interaction priors from egocentric videos into compact latent guidance that complements a pretrained world-action model without directly mapping human actions to robot controls.Source: §B.1 · p. 17

B.2 Learning from Egocentric Human Videos for Robot Manipulation

Prior approaches leave a gap in transferring interaction dynamics from egocentric videos: explicit human motion cues require alignment to robot actions, while latent actions may retain observer motion. WING separates these signals and uses shared low-frequency dynamics to guide robot actions.

B.2 Learning from Egocentric Human Videos for Robot Manipulation

  • Earlier motion-cue methods require human motion to be aligned with robot actions, while existing latent-action debiasing methods do not explicitly separate observer and interaction motion as supervision channels.Source: §B.2 · p. 18
  • WING-LAM routes dominant observer-induced image motion away from local interaction residuals, then distills the interaction representation into an RGB-only latent-action encoder.Source: §B.2 · p. 18
  • WING identifies low-frequency latent dynamics with stronger correspondence between semantically matched human and robot interactions and uses them to guide robot action generation.Source: §B.2 · p. 18

C.1 WING-LAM Architecture and Training Details

WING-LAM separates observer motion from local interaction motion, then distills the interaction representation into an RGB-only latent action. WING uses these latents to train spectral guidance for robot policies, and the section also describes representation evaluations for motion information, camera sensitivity, and action semantics.

C.1.1 Offline Motion Supervision Construction

  • WING-LAM estimates global observer motion from background tracks, then identifies nearby tracks with large camera-compensated residuals as interaction motion.Reliable tracks are weighted using tracking visibility and confidence; the affine warp is fitted with deterministic RANSAC.Source: §C.1.1 · pp. 18–19

C.1.2 Motion-Routed Teacher Architecture and Objectives

  • A motion-routed teacher separately predicts observer warp and interaction residuals, with supervision for each component and their joint reconstruction of tracked endpoints.Background tracks receive only the global warp, while interaction tracks also receive decoded residual motion.Source: §C.1.2 · p. 19

C.1.3 Camera Intervention and Representation Regularization

  • Camera interventions preserve interaction dynamics while perturbing global camera motion, and latent consistency plus variance and covariance regularization reduce leakage and redundancy.The teacher objective combines motion supervision with consistency and representation regularizers.Source: §C.1.3 · p. 20

C.1.4 Interaction Distillation

  • A frozen teacher’s interaction representation supervises an RGB-only student, which is then frozen and used offline to create WING latent-action targets.The student receives frame pairs without point tracks, masks, or motion parameters; privileged components are discarded after distillation.Source: §C.1.4 · p. 20

C.2.1 Pre-training Data and Preprocessing.

  • Egocentric pretraining uses about 200 hours from Ego4D, EgoDex, and EgoVerse, with latent-action trajectories sampled every 0.2 seconds across a 1.6-second window.The trajectories contain eight 32-dimensional transitions; hand-pose pretraining uses supplementary annotations from EgoDex and EgoVerse.Source: §C.2.1 · p. 21

C.2.2 Guidance Predictor Training

  • The guidance predictor learns to estimate the four lowest temporal DCT components of an eight-step latent-action trajectory from the current image and language instruction.Future frames construct training targets only; they are not input to the predictor.Source: §C.2.2 · p. 21

C.3.1 Training Framework

  • During robot training, predicted low-frequency coefficients condition action prediction through guidance tokens and gated cross-attention, alongside the policy objective.The predictor is jointly optimized with the robot policy using visual observation, language, and proprioceptive state.Source: §C.3.1 · p. 22

D.1 Latent-Action Representation Evaluation Details

  • Frozen latent encoders are evaluated for decoded camera and interaction motion, response to matched camera perturbations, and action-category classification across four video datasets.The perturbation protocol compares dynamic inter-frame transforms with matched static controls, and semantic evaluation follows LARYBench.Source: §D.1 · p. 22, §D.1.1 · p. 22, §D.1.2 · p. 22

D.2 Pair Construction for Spectral Similarity Analysis

The evaluation constructs human–robot clip pairs that match or mismatch in dominant interaction semantics, rather than requiring frame or trajectory alignment. It compares their temporal frequency representations using a shared, egocentric-trained latent-action encoder.

D.2 Pair Construction for Spectral Similarity Analysis

  • Matched pairs share dominant interaction dynamics despite potentially differing in objects, scenes, viewpoints, operators, and end-effector trajectories.Task labels filter candidate episodes first; human annotators then select semantically similar clips within those pools.Source: §D.2 · p. 24
  • Mismatched pairs come from the same task-conditioned pools but have inconsistent dominant motion semantics, separating action-level correspondence from task distribution.Each source episode appears at most once in each set; the final set has 100 matched and 100 mismatched pairs across 25 tasks.Source: §D.2 · p. 24
  • WING-LAM encodes both clip types without robot-data adaptation, testing whether its egocentric-trained interaction space generalizes across embodiments.The evaluation uses the same encoder that constructs WING’s spectral guidance.Source: §D.2 · p. 24
  • Both domains are sampled every 0.2 seconds into eight-step, 1.6-second trajectories, then compared using DCT-II cosine similarity for low-, high-, and full-frequency representations.Reported similarities are averaged separately over the matched and mismatched pairs.Source: §D.2 · p. 24

D.3.1 Benchmark Protocols and Metrics

The section specifies multi-task simulation benchmarks and four real-world manipulation tasks, including evaluation protocols, comparison methods, and controlled generalization shifts. It also reports representation evidence that WING-LAM retains interaction information while reducing camera-motion information and preserves action semantics.

D.3.1 Benchmark Protocols and Metrics

  • LIBERO averages task success across four suites, RoboTwin 2.0 averages clean and randomized results across 50 tasks, and RoboCasa–GR1 averages success across 24 tasks.The protocols use 2,000 LIBERO rollouts, 100 rollouts per RoboTwin task and scene condition, and 50 rollouts per RoboCasa task.Source: §D.3.1 · pp. 24–25

D.3.2 Baselines

  • Simulation comparisons span direct vision-language-action policies, latent-action policies, and world-action models; real-world comparisons include GR00T-N1.7, π0.5, and FastWAM.These groups cover direct action prediction, intermediate latent actions, and predictive world representations.Source: §D.3.2 · p. 25

D.4.1 Tasks and Progress Metrics

  • Real-world evaluation covers Pack Objects, Stack Cups, Battery Insertion, and Long-Horizon Battery Assembly, with 20 trials per task and binary success plus milestone-based progress scores.Success requires the final condition to hold for at least three seconds; autonomous policy safety terminations count as failures.Source: §D.4.1 · p. 26

D.4.2 Generalization Conditions

  • Generalization tests vary one factor at a time—appearance, background, geometry, or spatial layout—while holding other factors fixed and excluding shifted conditions from task-specific demonstrations.Battery geometry is excluded because changing insertion dimensions would alter task mechanics.Source: §D.4.2 · pp. 26–27

E.1.1 Camera–Interaction Readout Results

  • Across EgoDex, EgoVerse, and Ego4D, WING-LAM’s interaction representation retains interaction readout while suppressing observer-motion readout, and the student inherits this routing without privileged inference inputs.The EgoDex camera target is recorded 6D head pose, so reduced readout from the interaction latent transfers at least partially beyond the affine training proxy; routing is robust across Ridge, MLP, and RFF probes.Source: §E.1.1 · p. 28, Table 6 · p. 30

E.1.2 Detailed LARYBench Results

  • WING-LAM achieves the highest average action-classification accuracy across Ego4D, EPIC-KITCHENS, HoloAssist, and EgoDex, while decoded trajectories qualitatively concentrate around hands, manipulated objects, and interaction regions.The qualitative examples show relatively few background responses even when viewpoint changes are noticeable.Source: §E.1.2 · p. 28, §E.1.3 · pp. 28–29, Table 7 · p. 30

E.2.1 Cross-Embodiment Spectral Similarity Analysis

Low-frequency latent components show the strongest separation between semantically matched and mismatched human–robot interactions, and action semantics are more clearly organized in low-frequency latents. The section also reports a domain bandwidth difference and ablations on latency, denoising, and egocentric pretraining.

E.2.1 Cross-Embodiment Spectral Similarity Analysis

  • 0.51 matched–mismatched gap (95% bootstrap CI: [0.48, 0.53]) for low-frequency latents exceeds the full-spectrum gap of 0.26 ([0.22, 0.27]) and high-frequency gap of 0.05 ([0.03, 0.10]).The ordering also appears with DreamDojo; excluding q0, q1:q3 similarity is 0.42 for matched versus 0.08 for mismatched pairs, while adding q4:q7 weakens separation.Source: §E.2.1 · p. 31

E.2.2 Spectral Bandwidth Gap Between Egocentric and Robot Data

  • Egocentric trajectories have broader frequency distributions than robot trajectories, whose spectral energy is more concentrated at low frequencies.The analysis uses DreamDojo for a more domain-balanced bandwidth comparison, and the qualitative trend also appears with WING-LAM.Source: §E.2.2 · p. 31, Figure 9 · p. 31

E.2.3 Action Semantics Across Frequency Bands

  • 0.55 ± 0.02 NMI for low-frequency latents exceeds full-spectrum at 0.31 ± 0.01 and high-frequency at 0.17 ± 0.05 on Ego4D action-label clustering.The analysis applies temporal DCT to eight-step trajectories and evaluates cluster agreement with official labels.Source: Figure 10 · p. 32, §E.2.3 · p. 33

E.3.1 Latency Measurement

  • WING’s inference latency exceeds lightweight baselines such as GR00T and π0.5, though it is described as computationally competitive with generative policies of comparable complexity.The authors attribute the added latency to dense world-modeling computation; latency is measured on one RTX 4090 with action chunk size 32 and 10 denoising steps.Source: §E.3.1 · p. 33, Table 8 · p. 33

E.3.2 Ablation on Denoising Steps

  • 80.46% average RoboTwin success with one denoising step rises to 93.20% at two steps; 12 steps reaches 94.56% Clean and 93.04% Randomized.Performance remains relatively stable at larger step counts; 10 steps achieves 93.55% success rate.Source: §E.3.2 · p. 33, Table 9 · p. 33

E.3.3 Egocentric Pretraining Efficiency

  • Longer egocentric pretraining consistently yields lower validation loss and faster convergence when initializing robot policy training on Pack Objects.Checkpoints from 15k, 30k, 45k, and 60k pretraining steps use the same 200-hour video corpus.Source: §E.3.3 · p. 34

E.3.4 Downstream Data Efficiency

Egocentric pretraining improves downstream success on LIBERO and real-world manipulation, with larger gains as robot training data decreases. The appendices provide per-task benchmark breakdowns, full real-world generalization metrics, and qualitative examples across four manipulation tasks.

E.3.4 Downstream Data Efficiency

  • Pretraining improves LIBERO and real-world success rates, with the largest gain in the 25% real-world data setting.The comparison evaluates models with and without pretraining at 100%, 50%, and 25% of downstream robot training data.Source: Figure 12 · p. 34, §E.3.4 · p. 34

E.4.1 RoboCasa–GR1 Per-Task Results

  • Table 10 reports per-task success rates for all 24 RoboCasa–GR1 tabletop tasks, including a FastWAM reproduction.Published baseline entries are transcribed from the corresponding method papers.Source: Table 10 · p. 35, §E.4.1 · p. 35

E.4.2 RoboTwin 2.0 Per-Task Results

  • Table 11 gives clean and randomized success rates for all 50 RoboTwin 2.0 tasks, evaluated over 100 trials per task and setting.Source: §E.4.2 · p. 35, Table 11 · p. 36

E.5 Full Real-World Generalization Results

  • The real-world generalization results report success rate and average task score across visual and spatial settings.Aggregate metrics use unrounded underlying values, with rounding applied only to the final reported number.Source: §E.5 · p. 35, Table 12 · p. 37

E.6 Case Study

Related papers

Loading 2610.03607v1…