Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videos

• Hands create occlusion and a visual mismatch with robot end-effectors, yet their poses and motions can signal affordances and dense rewards.Source: §1 · p. 1
• AEF separates the segmented hand from an environment image inpainted with VIDM, preserving agent information while allowing the two representations to be manipulated independently.Source: §1 · pp. 1–2
• VIDM combines a large-scale pretrained diffusion prior with attention-based lookup of earlier video frames, outperforming DLFormer by a large margin and running 8× faster at test time.Source: §1 · p. 2
• Across five benchmark tasks spanning perception and robot learning, selectively choosing and modifying AEF representations improves performance over existing approaches.Source: §1 · p. 2

• Prior work uses human-object interaction videos to predict interaction regions, grasps, and post-grasp trajectories, and to learn reward functions and policy features [19] [49] [19] [94] [2] [42] [1] [8] [68] [50] [59] [89].Source: §2 · pp. 2–3
• Approaches to handling the hand include masking, inpainting, pixel conversion, and embodiment-invariant representations; this work instead lets downstream tasks choose how to use agent and environment information.Source: §2 · p. 3
• Latent diffusion compresses images into an autoencoder latent space for diffusion, and this work builds on that approach [62] [81].Source: §2 · p. 3
• Prior video inpainting uses temporal consistency, optical flow, and other methods; unlike approaches that inpaint entire videos, this work uses prior frames as context to inpaint a specific frame [36] [41] [88].Source: §2 · p. 3

• The agent representation shows the agent, while the environment representation shows the scene without it; together they form the AEF representation.Source: §3 · p. 4

Using the Factorization

• Tasks can add the unoccluded environment to the original image for perception, use it to predict agent properties, or abstract agent appearance for reward learning.Source: §3 · p. 4

• For egocentric video, VISOR segments hands; for robot depictions, DeepLabV3 is fine-tuned on manually annotated end-effector frames.Source: §3 · p. 4
• Because masking the agent leaves artifacts and the scene may be occluded, the environment image is inpainted using the current frame and earlier frames that may reveal the hidden content.Source: §3 · p. 4

4 Video Inpainting via Diffusion Models (VIDM)

• The model processes 256 × 256 images using 64 × 64 latent features, stacking context frames and allowing attention across frames.Its U-Net accepts a (2d + 1, h + 1, 64, 64) tensor, where h is the number of additional context frames.Source: §4 · p. 4
• VIDM outperforms prior inpainting methods using only a strong generative prior or only past frames on held-out Epic-Kitchens clips.The supplied passage gives no numerical results.Source: Table 1 · p. 5
• VIDM combines a pretrained generative prior with attention to use information from past frames, allowing inpainting from earlier scene appearances without explicit tracking.The model uses latent diffusion, and attention learns where to draw information from previous frames.Source: §4 · pp. 4–5

5.1 Reconstruction Quality Evaluation

• VIDM outperforms all inpainting baselines, runs 8× faster than DLFormer, and uses only three prior frames; examples show temporal background retrieval and wok-handle reconstruction.Evaluation uses held-out Epic-Kitchens clips with hand-shaped masks and ground-truth pixels; DLFormer sees every test-clip frame.Source: §5.1 · pp. 5–6, Figure 4 · p. 6

5.2 Application 1: Object Detection

• The full formulation improves object-detection average recall by 24%–51% relative to raw images, while naive masking hurts and single-frame inpainting gains are inconsistent.The evaluation uses a COCO-trained Mask RCNN on active objects in VISOR; Table 2 reports average recall.Source: §5.2 · pp. 6–7, Table 2 · p. 6

5.3 Application 2: 3D Reconstruction of Hand-Held Objects

• Adding ground-truth inpainted object views improves reconstruction over raw images: F1 at 5mm rises from 0.41 to 0.48, F1 at 10mm from 0.62 to 0.68, and chamfer distance from 1.23 to 1.06.Because the added environment view is ground truth, this evaluation isolates the factorization’s impact rather than testing generated inpainting.Source: §5.3 · p. 7

5.4 Application 3: Affordance Prediction

• VIDM-inpainted images improve performance over masked images on both Region of Interaction and Grasps Afforded by Objects tasks.The models are trained with 1,000 images, using the training scheme, metrics, and test data from the benchmark.Source: §5.4 · pp. 7–8

5.5 Application 4: Learning Reward Functions from Videos

• AEF correlates best with ground-truth task progress on robotic end-effector trajectories; its agent-agnostic green-dot encoding retains hand-to-scene location information.The reward predictor is trained on Epic-Kitchens clips for drawer, cabinet, and refrigerator tasks; the evaluation uses Spearman’s rank correlation.Source: §5.5 · p. 8, Table 4 · p. 8

5.6 Application 5: Real-World Policy Learning using Learned Rewards

• In real-world drawer-opening policy learning, rewards from AEF outperform those from raw images or environment-only inputs as CEM iterations proceed.The policy is learned on a Stretch RE2 robot; success rate is measured across CEM iterations.Source: §5.6 · pp. 8–9, Figure 5 · p. 9

6 Discussion

• VIDM outperforms methods using only a pre-trained diffusion prior or past-frame information, as assessed by qualitative metrics and visualizations.The cues are a large-scale pre-trained inpainting diffusion model and the ability to use content from previous frames.Source: §6 · p. 9
• Across five benchmarks, the factored representation improves object detection and 3D reconstruction, aids affordance prediction, and supports better cross-embodiment reward transfer.Reward transfer gains appear in offline evaluations and online learning on a physical platform.Source: §6 · p. 9
• Inpainted images can be reused with off-the-shelf models, while foundation-model pre-training may make learning under occlusion more data-efficient.The passage also notes that separate downstream applications otherwise pay an implicit training cost for interpreting masked images.Source: §6 · p. 9

7 Limitations and Broader Impact

• At 13.6 seconds per inference, the method is infeasible for real-time robotic applications.The authors identify this as a limitation shared with diffusion-model work generally.Source: §7 · p. 10
• The method removes hands to recover the environment; inpainting environment occlusions of the agent remains an open direction.The authors suggest exploring similar inpainting for agent occlusion.Source: §7 · p. 10
• VIDM's possible use as a general-purpose video inpainting model brings uncertain societal implications associated with generative AI.Source: §7 · p. 10

S1 VIDM Training Details

• At 64 × 64 resolution, VIDM omits self-attention because attending over 16,384 tokens is computationally intractable on the available hardware.Its largest attention layer is at 32 × 32 across four images, totaling 4,096 tokens.Source: S1 VIDM Training Details · p. 15
• VIDM uses residual CNN blocks with SiLU nonlinearities, GroupNorm, and a U-Net feature dimension rising from 256 to 1,024.Upsampling and downsampling use residual CNN blocks with internal nearest-neighbor 2× upsampling or average-pooling 2× downsampling.Source: S1 VIDM Training Details · p. 15
• Training targets came from Ego4D and VISOR; VISOR participants P37, P35, P29, P05, and P07 were held out for reconstruction-quality and object-detection evaluation.No Ego4D data was held out because Ego4D was not used for evaluation.Source: S1 VIDM Training Details · p. 15

S2.1 Detection

• Detection evaluation uses off-the-shelf Mask R-CNN R_101_FPN_3x trained on COCO and classes shared with VISOR annotations.The passage enumerates the overlapping evaluation classes, including common kitchen objects and appliances.Source: S2.1 Detection · p. 15

S2.2 Affordance Prediction

• Affordance experiments use 1K sampled images per task from EPIC-ROI and GAO, with sampling strategies differing between the two datasets.EPIC-ROI samples inversely to video length; GAO samples randomly.Source: S2.2 Affordance Prediction · p. 16
• For affordance prediction, ACP's architecture takes in hand-removed inpainted images without a hand mask, and separate models are trained for EPIC-ROI and GAO.Training follows ACP's loss and hyperparameters and reports mean and standard deviation over three seeds per task.Source: S2.2 Affordance Prediction · p. 16

S2.3 3D Reconstruction of Hand-held Objects

• Hand-held-object reconstruction uses ObMan's 134K training, 7K validation, and 6.2K test images, combining features from hand-removed object images with features from the original inputs.The model follows Ye et al.'s architecture, conditioning DeepSDF on hand articulation and image features.Source: S2.3 3D Reconstruction of Hand-held… · p. 16

• Figure S1’s training batch shows four network input images and a target image for reconstruction.Source: Figure S1 · p. 18
• The batch visualization shows masked regions on the target and pixels where reconstruction loss is propagated.White pixels receive loss; gray pixels do not.Source: Figure S1 · p. 18
• Hands masked in the target image receive no reconstruction loss.Source: Figure S1 · p. 18
• Figure S2 provides additional visualizations of predictions from the method and baselines.Source: S3 Visualizations · p. 17, Figure S2 · p. 19