PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoPhys: Learning Generalizable Physics Models of Deformable Objects from Egocentric Video

Hyunjin Kim, Ri-Zhao Qiu, Guangqi Jiang, Xiaolong Wang

arXiv:2606.16202v1cs.CVcs.AIcs.RO

TL;DR

Constructing physically faithful deformable-object twins from ordinary egocentric video remains difficult because existing approaches lack explicit physics or require controlled capture and scene-specific optimization. EgoPhys learns a reusable physics prior from egocentric RGB video, and matches or surpasses per-scene optimization baselines across reconstruction, future prediction, and unseen-object generalization while supporting robot planning.

  • Problem

    Existing methods either lack explicit physics or require controlled capture and scene-specific optimization, leaving open whether deformable physical twins can be built from egocentric RGB video.

  • Method

    EgoPhys distills dense per-object spring stiffness fields into a reusable state-conditioned codebook for constructing deformable digital twins from single egocentric RGB videos.

  • Results

    EgoPhys matches or surpasses per-scene optimization baselines on reconstruction, future prediction, and unseen-object generalization across object types, viewpoints, and occlusion patterns.

  • Takeaways & Limitations

    Egocentric RGB-only physical twins can support downstream deformable-object planning, as demonstrated by trajectories transferred to a physical xArm6 robot.

  • Takeaways & Limitations

    The modest-scale dataset and proof-of-concept robot evaluation do not cover the full diversity of deformable materials, long-horizon interactions, or complex contact-rich skills.

Abstract

from arXiv · show

Humans naturally understand object physics through everyday interactions, but faithfully predicting complex deformable dynamics, such as elastic materials and fabrics, remains a major challenge for computer vision and robotics. We present EgoPhys, a framework that constructs deformable physical digital twins from egocentric RGB-only video using generalizable priors. EgoPhys overcomes the limitations of existing methods to enable controllable deformable digital twin generation from egocentric videos by distilling per-object inverse-physics solutions into a compact codebook, enabling prediction of dense spring stiffness fields for unseen objects without per-spring test-time optimization. Trained with generalizable priors from diverse egocentric interactions, EgoPhys outperforms baselines in reconstruction, future prediction, and zero-shot generalization. To support training and evaluation, we curate an egocentric interaction dataset covering diverse deformable objects, scenes, and manipulation styles. We deploy EgoPhys on a real xArm6 robot, demonstrating that a digital twin initialized from a single egocentric human play video can serve as an internal world representation to aid in deformable-object planning, highlighting egocentric RGB observations as a scalable path toward real-to-sim pipelines.

1 Introduction

EgoPhys addresses the challenge of simulating deformable-object dynamics by constructing physical digital twins from a single egocentric RGB video. It combines a reusable, state-conditioned material codebook with an egocentric interaction dataset and robot validation to support generalizable reconstruction, prediction, and deployment.

  • Existing video-generation methods produce plausible dynamics without explicit physics, while inverse-physics methods typically require scene-specific optimization and controlled third-person capture.These limitations hinder interpretability and action-conditioned simulation under novel interactions.
  • EgoPhys constructs deformable physical digital twins from a single egocentric RGB video without depth sensing or calibrated multi-view capture.The framework is presented as the first such deformable real-to-sim approach.
  • EgoPhys distills dense per-object spring stiffness solutions into a compact, state-conditioned material codebook that predicts stiffness fields for unseen objects without per-spring test-time optimization.The pipeline builds coherent 4D point clouds, obtains a coarse spring graph and global physical parameters, then predicts dense spring stiffnesses.
  • EgoPhys learns a transferable stiffness prior that improves over coarse per-object initialization and generalizes across held-out objects, viewpoints, and occlusion patterns while preserving high simulation fidelity.Compared with dense per-scene optimization, it replaces object-specific stiffness refinement with a compact reusable representation.
  • The work introduces an egocentric deformable-object interaction dataset, evaluates reconstruction, future prediction, and zero-shot object generalization, and demonstrates deployment with a physical xArm6 robot.The robot experiment uses the learned digital twin for sim-to-real validation.

2 Related Work

Prior deformable-object digital-twin methods combine reconstruction with physics simulation but often depend on controlled capture, pre-scanned geometry, or per-scene optimization. Related simulation benchmarks have narrowed the visual gap, yet remain limited in their coverage of deformable objects and dynamics.

  • Earlier methods estimate physical parameters for simulatable deformable-object twins, often assuming pre-scanned geometry, clean point clouds, or controlled capture.
  • Recent approaches jointly recover geometry and material properties from video using learned 3D representations, including SDF, NeRF, or Gaussian Splatting, with differentiable simulation.
  • PhysTwin and PhysWorld reconstruct appearance and physically simulatable dynamics from sparse interaction videos but require controlled capture and per-scene optimization, while MatPhys predicts spring-mass parameters from single-view videos using material priors and a learned codebook.
  • Simulation-based policy evaluation can correlate strongly with real outcomes and photorealistic Gaussian-Splatting stacks narrow the visual gap, but SplatSim and GSWorld remain limited to rigid or articulated objects and states.

3 Method

EgoPhys reconstructs temporally coherent 4D point clouds from a single egocentric RGB video, fits coarse spring-mass simulators through inverse physics, and distills dense stiffness fields into a shared material codebook. The codebook predicts state-dependent spring stiffnesses during rollout, replacing expensive per-spring test-time refinement.

  • 3.1 Egocentric 4D Reconstruction: EgoPhys reconstructs 4D point clouds from wearable-camera RGB video despite camera motion, partial visibility, hand occlusion, missing depth, and absent calibrated multi-view capture.The pipeline extracts masks and dense 2D trajectories, lifts tracked pixels with per-frame VGGT world-point maps, filters by confidence and depth, and completes heavily occluded geometry with TRELLIS.
  • 3.2 Inverse Physics: Each object is modeled as a spring-mass graph whose explicit-Euler simulator is fitted by minimizing geometry and motion discrepancies.CMA-ES estimates graph construction and coarse physical parameters from rollout reconstruction and motion error.
  • 3.3 Codebook-Based Physics Prior: The codebook predicts dense stiffness corrections from static graph features and dynamic deformation features after coarse CMA initialization, avoiding per-spring gradient refinement.Dynamic features include strain, strain rate, orientation, and height; static features encode graph geometry, endpoint degrees, spring types, and normalized 3D shape cues.
  • 3.3 Codebook-Based Physics Prior: Spring stiffness is parameterized as a learned log-stiffness offset relative to CMA-estimated global stiffness, preserving the coarse fit while allowing variation across springs and deformation states.The formulation is log ke,t = log ¯k + ∆e,t, with bounded stiffness conversion and log-space clipping for numerical stability.
  • 3.3 Codebook-Based Physics Prior: Two prototype banks model asymmetric tension and compression responses, while rollout and distillation losses train shared assignments to match dense inverse-physics stiffness targets.Regularizers constrain prototype offsets, encourage diverse prototype usage, and promote decisive assignments.

4 Experiments

Experiments evaluate EgoPhys on egocentric deformable-object reconstruction, future prediction, zero-shot transfer, physics-prior ablations, and robot manipulation planning. Across these settings, EgoPhys improves physical and rendering outcomes, generalizes to held-out objects and interactions, and supports zero-shot planning on a real xArm6 robot.

  • Dataset: The dataset contains 19 seven-second egocentric interaction sequences covering plush toys, towels, cloth, and bags with lifting, pulling, pushing, and folding motions.Each sequence uses a 7:3 temporal train/test split.
  • Reconstruction, Resimulation, and Future Prediction: EgoPhys outperforms adapted PhysTwin and Spring-Gaus on observed-window reconstruction, resimulation, and future prediction, with strongest gains on physical metrics.Qualitative results show better deformation matching, while baselines diverge and become unstable under large egocentric deformations.
  • Generalization to Unseen Objects and Interactions: In zero-shot evaluation on held-out object-interaction sequences, EgoPhys outperforms adapted PhysTwin across physical and rendering metrics while predicting dense spring stiffnesses without per-spring refinement.The held-out set includes unseen object categories and novel interaction patterns.
  • Ablation of the Learned Physics Prior: A learned prototype codebook improves held-out rollout accuracy over the coarse CMA-ES anchor, while a direct MLP substantially worsens both CD and TE.The comparison uses the same input features and backbone, indicating that the prototype bottleneck acts as a regularizer.
  • Ablation of the Learned Physics Prior: Dynamic conditioning lowers test TE versus a static K = 4 codebook with nearly identical CD, while increasing K from 4 to 8 or 16 yields only marginal changes.K = 8 gives the lowest test TE and K = 16 gives the lowest test CD; all dynamic variants remain close.
  • Robot Manipulation Planning: On a physical xArm6 robot, MPPI trajectories planned from a single egocentric video reached target configurations and reduced object-configuration error without ground-truth parameters or instance-specific re-optimization.Trials covered lifting and pulling across three plush toys, with deformation patterns consistent between simulation and execution.

5 Conclusion

EgoPhys constructs deformable physical digital twins from a single egocentric RGB-only video. Its codebook transfers across object types, viewpoints, and occlusion patterns, while planned trajectories transfer to a physical xArm6 robot.

  • EgoPhys constructs deformable physical digital twins from a single egocentric RGB-only video.
  • The codebook-based representation transfers effectively across object types, viewpoints, and occlusion patterns.
  • EgoPhys matches or surpasses per-scene optimization baselines on reconstruction, future prediction, and unseen object generalization.
  • Trajectories planned inside EgoPhys digital twins transfer to a physical xArm6 robot.

6 Limitations

EgoPhys is evaluated on a modest-scale egocentric dataset, so its learned codebook is an initial reusable physics prior rather than a universal material model. The dataset does not cover the full diversity of real-world deformable materials, long-horizon interactions, or complex contact-rich skills.

  • Dataset scope: EgoPhys is evaluated on a modest-scale egocentric dataset, limiting the universality of its learned physics codebook.The codebook should be viewed as an initial reusable physics prior rather than a universal material model.
  • Dataset scope: The dataset spans multiple object categories, manipulation styles, viewpoints, and occlusion patterns, but not the full diversity of real-world deformable materials.Its coverage remains narrower than the diversity encountered in real-world deformable-object settings.
  • Dataset scope: The evaluation does not cover long-horizon interactions or complex contact-rich skills.These interaction regimes remain outside the dataset’s reported coverage.

A Additional Ablations … B.2 Per-Object Physics Initialization

The appendix evaluates codebook-based physics inference against dense per-sequence refinement and specifies the egocentric reconstruction and object-specific simulation procedures used by EgoPhys. These procedures include calibrated frame processing, tracked 3D point filtering, geometry completion, coarse CMA-ES initialization, object-dependent springs, strain handling, and ground-contact force projection.

  • A.1 Codebook inference vs. dense per-sequence refinement.: EgoPhys predicts dense spring stiffness fields from a shared codebook without per-spring test-time gradient updates, unlike PhysTwin’s per-sequence refinement.The comparison uses the same coarse CMA-ES physical anchor and evaluates held-out object-interaction sequences.
  • A.1 Codebook inference vs. dense per-sequence refinement.: The ablation compares codebook inference with variants that refine spring stiffnesses and contact parameters before future rollout.Runtime is reported for post-anchor refinement and rollout per sequence, with 100- and 200-step budgets matching default cloth-like and real-object PhysTwin settings.
  • B.1 Egocentric 4D Reconstruction: RGB frames are factory-calibrated, undistorted to a linear pinhole model, rotated upright, and center-cropped to a square S×S canvas with S = 518.The crop is intended to maximize object visibility.
  • B.1 Egocentric 4D Reconstruction: Grounded-SAM2 initializes object and manipulator masks, SAM2 propagates them, and CoTracker3 tracks up to 5000 queried pixels while retaining mask-consistent tracks.The query pixels come from the union of first-frame object and hand masks.
  • B.1 Egocentric 4D Reconstruction: VGGT supplies per-frame world-point and confidence maps, retaining points with ρt[ut] ≥τc = 0.5 and predicted depth 0.2 < z < 1.5 m.When views leave large gaps, TRELLIS optionally completes geometry, while observed tracked points are prioritized over completed samples.
  • B.2 Per-Object Physics Initialization: CMA-ES optimizes bounded normalized coarse physical parameters by rolling out the spring-mass simulator on the training window.For flat objects, parameters include uniform spring stiffness, node and controller neighborhoods, collision terms, damping, and strain limits.
  • B.2 Per-Object Physics Initialization: Bending springs are disabled for plush toys but enabled for cloth and bags, while self-collision is enabled for cloth-like configurations.Enabled bending springs use longer-range object connections with reduced stiffness and increased damping.
  • B.2 Per-Object Physics Initialization: Controller springs use radius-based hand-object connections, with Kmin = 5 neighbors and fallback cutoff dmax = max(2.5 rc, 0.15 m) for flat objects.For plush objects, stricter radius search avoids spurious long-range springs, and fallback is used only when no controller springs are created.

B.3 Codebook Training · B.4 3D Gaussian Splatting · B.5 Baseline Adaptation and Fairness

The method trains a compact, dynamic stiffness codebook from sequential object data, reconstructs scenes with hybrid 3D Gaussian Splatting, and compares adapted physics baselines under matched egocentric observations and rendering protocols.

  • B.3 Codebook Training: The default prior is a dynamic, sign-aware codebook with K = 4 prototypes and softmax temperature τ = 0.7.Predictions use delta mode, representing log-stiffness offsets around CMA-estimated global stiffness before clipping to the simulator’s allowed range.
  • B.3 Codebook Training: A lightweight MLP encoder uses hidden width 64 with separate tension/compression linear heads.Static inputs include spring and shape properties, while dynamic inputs include stretch, strain, orientation, height, and control signals.
  • B.3 Codebook Training: Training proceeds sequentially over 8 codebook-training sequences using a shared checkpoint carried between objects.Adam uses learning rate 10^-3, with 100 iterations for cloth-like configurations and 200 for the standard real-object configuration.
  • B.4 3D Gaussian Splatting: After physics fitting, 3DGS is trained per sequence using Gaussians initialized from both static background and dynamic object frame-0 point clouds.Per-camera exposure compensation addresses Aria camera gain control.
  • B.4 3D Gaussian Splatting: Dynamic rollouts are rendered on a white background, with human hand regions masked before computing PSNR, SSIM, and LPIPS.
  • B.5 Baseline Adaptation and Fairness: All methods receive identical egocentric-video-derived 4D point clouds, object masks, control trajectories, temporal split, and rendering protocol.This isolates physical modeling and stiffness-estimation strategies from upstream reconstruction quality.
  • B.5 Baseline Adaptation and Fairness: PhysTwin is adapted as a per-scene optimization baseline, while Spring-Gaus is adapted as a spring-mass 3DGS baseline.Because both were designed for stronger observation settings, comparisons represent the closest adaptable alternatives under egocentric capture rather than native benchmarks.

C Additional Qualitative Results

EgoPhys qualitatively outperforms baselines across reconstruction, resimulation, future prediction, unseen-object generalization, and sim-to-real transfer. It preserves plausible deformation and task-relevant contact dynamics across diverse objects and interactions.

  • Qualitative Comparisons: Additional qualitative comparisons evaluate EgoPhys against PhysTwin across object categories and interaction types.These comparisons are presented as supplementary qualitative results.
  • Generalization to Unseen Objects: EgoPhys demonstrates qualitative generalization on lion and green monster plush toys, both held out from training.These results evaluate generalization to unseen objects.
  • Sim-to-Real Transfer: For the Doraemon plush toy pulling task, real-robot deformation patterns are consistent with EgoPhys’ predicted trajectory, preserving task-relevant contact and deformation modes without exact visual alignment.The result compares simulated rollouts with physical robot execution.
  • Reconstruction, Resimulation, and Future Prediction: On oven mitt lifting, EgoPhys accurately reconstructs and resimulates observed deformation while closely following human-hand motion in future prediction; baselines remain static.The comparison covers reconstruction, resimulation, and future prediction.

D Dataset Details

The dataset uses short, high-resolution egocentric videos of users manipulating diverse deformable objects under varied capture conditions. It supports temporal training/evaluation and zero-shot testing on disjoint sequences, with automatically generated and manually verified segmentation masks.

  • Capture setup: Videos are captured at 30 fps and 1408 × 1408 resolution, with each 7-second sequence containing 210 frames of tabletop interaction with one deformable object.The capture setup uses a Meta Project Aria Gen 1 wearable camera and spans multiple backgrounds and lighting conditions.
  • Object categories: The dataset covers plush toys, towels, an oven mitt, and a soft brown bag spanning varied stiffnesses, geometries, aspect ratios, and surface textures.Plush categories include alien, green monster, fox, lion, Santa, Doraemon, teddy bear, and a large plush toy.
  • Train/test split: Each sequence uses a 7:3 temporal split, assigning the first 70% of frames to training and the last 30% to evaluation.Zero-shot evaluation instead separates data at the object-interaction-sequence level.
  • Train/test split: Zero-shot codebook evaluation trains on 8 sequences and evaluates on 11 disjoint sequences containing unseen object instances and interaction modes.The held-out set spans plush toys, towels, and cloth-like objects.
  • Ground-truth annotations: Object and hand segmentation masks are generated automatically with Grounded-SAM2 and manually verified.This annotation process provides verified masks for the captured interactions.

E Real-Robot Experiment Details

The real-robot experiments provide a proof of concept for using EgoPhys as an MPPI forward model to plan open-loop deformable-object manipulation on a physical xArm6. They evaluate whether an egocentric-video-derived digital twin can reduce configuration error without real-world fine-tuning or instance-specific physical-parameter re-optimization.

  • Real-robot planning: EgoPhys serves as the forward model inside an MPPI planner, with planned waypoints transferred to a physical xArm6 robot without real-world fine-tuning.The experiments test open-loop plans derived from an egocentric-video digital twin.
  • Evaluation scope: The trials assess whether an egocentric-video-derived digital twin can reduce configuration error during real-robot manipulation.The experiments are explicitly described as a proof of concept rather than a statistically powered manipulation benchmark.
  • Unseen-object generalization: Qualitative unseen-object results show better agreement with observations and accurate future-state prediction for lifting and pushing plush-toy tasks.The visualized tasks involve lifting a green monster plush toy and pushing a lion plush toy.
Loading 2606.16202v1…