PaperlayerView exact revision on arXiv

Source-linked AI summary

Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild

Yehonathan Litman, Xiaoxuan Ma, Manan Shah, Nicolas Ugrinovic, Kris Kitani, Fernando De la Torre, Shubham Tulsiani

arXiv:2606.23688v1cs.CV

TL;DR

Recovering complete, coherent 4D geometry, appearance, and deformation from monocular in-the-wild video remains difficult because observations are sparse and occluded. Lift4D harmonizes single-view 3D and diffusion priors through test-time optimization, achieving state-of-the-art reconstruction quality and better motion accuracy on challenging sequences.

  • Problem

    Monocular in-the-wild videos provide limited supervision for recovering complete 360° geometry, appearance, and deformation of generic dynamic objects under occlusion.

  • Method

    Lift4D uses causal latent conditioning and curriculum-based test-time optimization to combine geometry and appearance priors in a deformable 4D Gaussian representation.

  • Results

    Lift4D achieves state-of-the-art 4D reconstruction quality, with better perceptual and semantic fidelity on benchmarks and substantially better motion accuracy on challenging in-the-wild videos.

  • Takeaways & Limitations

    Lift4D provides generalizable complete 4D reconstructions for deforming, occluded objects from monocular video without multi-view data or category-specific templates.

  • Takeaways & Limitations

    Performance depends on the quality of initial SAM3D predictions, and cascading errors can propagate without oversight.

Abstract

from arXiv · show

Reconstructing dynamic non-rigid objects from monocular video requires integrating visual cues from direct observations with data-driven priors over geometry and appearance. Prior approaches either learn to directly predict 4D representations from visual input or initialize a 3D representation that is subsequently deformed and refined based on video evidence. However, the former are constrained by the scarcity of 4D training data, while the latter leverage priors only for the initial reconstruction and rely solely on video supervision thereafter; neither handles complex in-the-wild scenarios with large deformations and occlusions well. We present Lift4D, a test-time optimization framework that addresses both limitations. First, we adapt an existing single-view 3D reconstruction model to yield temporally consistent per-frame predictions via causal latent conditioning, providing a coherent initialization for a deformable 3D Gaussian Splatting representation. We then ``sculpt'' this representation to match the input video through an occlusion-aware optimization that faithfully recovers visible surface details while completing unobserved regions using a view-conditioned diffusion prior. We demonstrate that Lift4D clearly improves over prior 4D reconstruction methods, particularly on challenging in-the-wild sequences with severe occlusions and non-rigid motion.

1. Introduction

Lift4D targets complete 4D reconstruction of generic, deformable objects from monocular in-the-wild video despite occlusions and single-view ambiguity. It combines temporally coherent single-view 3D priors with occlusion-aware supervision and diffusion priors, achieving state-of-the-art quality across synthetic and challenging real-world videos.

  • Motivation: The problem is difficult because in-the-wild objects are unconstrained, deform substantially, and become occluded, while single-view geometric cues cannot determine complete 4D structure.These conditions motivate the use of data-driven priors for reconstruction.
  • Method: Lift4D adapts single-view 3D reconstruction into temporally coherent per-frame 4D priors using causal latent conditioning across video frames.Independent frame reconstruction produces temporally inconsistent geometry; latent information is propagated across frames to improve coherence.
  • Method: Occlusion-aware rendering supervision uses depth cues and color matching to create clean references, while generic image diffusion priors complete plausible unseen appearances.The approach addresses incomplete supervision from occluded and unobserved regions.
  • Scope: Lift4D reconstructs generic dynamic objects from monocular videos with rapid motion, large deformations, and severe occlusions without multi-view data or category-specific templates.The method is evaluated on synthetic benchmarks and challenging in-the-wild videos.
  • Results: Lift4D achieves state-of-the-art 4D reconstruction quality on synthetic benchmarks and challenging in-the-wild videos.The supplied passage states that it outperforms existing methods, but does not provide numerical results or identify the comparison baselines.

2. Related Works

Prior work reconstructs dynamic 4D scenes through video-based Gaussian deformation, monocular novel-view synthesis, feedforward generative prediction, or test-time optimization guided by generative priors. These approaches are limited by extreme viewpoint extrapolation and the scarcity of diverse multi-view video and 4D training data.

  • Dynamic Reconstruction and Tracking from Videos: Dynamic reconstruction methods extend 3D Gaussian Splatting [23] with deformation fields or canonical representations, while monocular methods address the harder single-view setting [25].Shape of Motion jointly optimizes a canonical 3DGS and per-frame deformation.
  • Dynamic Reconstruction and Tracking from Videos: GEN3C and CogNVS [6] use explicit 3D conditioning or reconstruct–inpaint–finetune pipelines for monocular dynamic novel-view synthesis, but degrade under extreme extrapolation.Large unseen regions must be hallucinated, and diverse multi-view video training data are scarce.
  • Feedforward Generative 4D Reconstruction: Feedforward methods directly predict complete 4D representations in one pass, including L4GM for sub-second Gaussian reconstruction and ActionMesh for temporally coherent animated meshes.L4GM trains on synthetic multi-view videos of animated assets, while ActionMesh extends 3D latent diffusion with a temporal axis and trains on animated assets [11] [12].
  • Prior-aided 4D Reconstruction: Prior-aided 4D reconstruction uses test-time optimization with large-scale 2D or 3D generative priors because 4D training data and multi-view video are scarce.Some methods keep priors in the loop through score distillation or diffusion-based video supervision [8] [9], while others use priors only to initialize canonical geometry.

3. Methodology

Lift4D constructs a deformable 4D Gaussian representation through curriculum-based test-time optimization, assigning single-view 3D and view-conditioned diffusion priors to geometry, appearance, and unobserved regions. Causal latent propagation provides temporally consistent per-frame reconstructions, which are distilled into a canonical representation and refined with occlusion-aware video supervision.

  • Method overview: Lift4D factorizes each object into canonical 3D Gaussians and deformation parameters, optimized with complementary 2D and 3D priors for in-the-wild videos.The framework uses detailed geometry from image-to-3D models and appearance completion from view-conditioned diffusion priors.
  • 3.1 Causal Latent Propagation: Causal latent propagation adapts an off-the-shelf flow-matching image-to-3D model into a temporally consistent 4D prior without retraining.Each frame reuses the previous frame’s denoised latent as a noise prior; t0 controls the tradeoff between temporal consistency and per-frame fidelity.
  • 3.2 Deformable 3D Optimization: The temporally consistent but independent per-frame Gaussian sets are distilled into one canonical representation whose sparse control-node deformations explain every frame.A deformation MLP predicts time-varying SE(3) transformations, while Chamfer and multi-view losses align geometry, appearance, and depth across frames.
  • 3.3 Appearance Reconstruction: Appearance optimization freezes the learned deformation and uses denser control nodes to prevent sparse photometric supervision from corrupting geometry.This addresses partial observation and scene occlusions, which make direct pixel supervision incomplete or unreliable.
  • 3.3 Appearance Reconstruction: Occlusion-aware rendering keeps trustworthy video pixels and replaces detected occlusions with color-corrected per-frame reconstructions before appearance supervision.Scene depth, rendered object depth, and object masks identify pixels where surrounding content lies in front of the subject.
  • 3.3 Appearance Reconstruction: A view-conditioned diffusion prior regularizes non-visible surfaces, complementing rendering losses that supervise observed pixels with score distillation for plausible unobserved content.Conditioning on the occlusion-completed reference rather than the raw image improves novel-view quality.

4. Experiments

Lift4D is evaluated against diffusion- and feedforward-based 4D reconstruction baselines on synthetic and in-the-wild monocular videos, using novel-view quality, realism, coherence, and semantic metrics. It outperforms baselines in reconstruction quality and tracking, while ablations show that removing geometric, regularization, or image-prior components degrades structure, coherence, deformation stability, or unobserved appearance.

  • Experimental setup: Lift4D is compared with diffusion, feedforward-transformer, and 2D- or 3D-prior test-time optimization baselines using novel-view metrics tailored to synthetic and in-the-wild videos.Synthetic evaluation uses LPIPS, FVD, and CLIP; in-the-wild evaluation uses image/text CLIP scores and EPE.
  • Synthetic results: On synthetic Consistent4D videos, Lift4D produces more faithful, topologically accurate unobserved geometry and appearance than baselines, improving novel-view semantic and structural quality.The dataset contains seven input videos and four ground-truth novel-view videos per object, each with 32 frames.
  • In-the-wild results: Lift4D greatly outperforms baselines on 4D reconstruction quality and tracking across in-the-wild Pexels and selected DAVIS sequences.The comparison covers diverse deformable, rigid, and occluded objects under varied lighting and background conditions.
  • Ablations: Removing initialized geometry, tracking, or velocity-motion losses, and image-prior distillation, reduces reconstruction quality in the ablations.The ablations evaluate physical and image information on Consistent4D, with quality drops reported in the quantitative comparison.
  • Ablations: Independent batch-wise latent initialization worsens deformation quality and causes frame-to-frame geometry jitter, while excluding regularization causes overfitting and similar deformation instability.These findings support temporally consistent latent conditioning and deformation regularization during optimization.
  • Ablations: Without image-prior distillation through LSDS, novel-view quality drops and unobserved regions become flat and blurry because optimization relies on coarse initial appearance.The image prior is therefore essential for filling details in regions unobserved by the camera.

5. Conclusion

Lift4D is a test-time optimization framework that recovers complete 4D dynamic objects from monocular video by harmonizing image-to-3D reconstructions as priors for 4D inference. It supports deforming and occluded scenes through causal latent conditioning and image/3D priors, but remains dependent on initial SAM3D quality and cascading hyperparameters.

  • Lift4D recovers complete 4D dynamic objects from monocular video by using image-to-3D reconstructions as priors for 4D inference.
  • Causal latent conditioning enforces temporal consistency, while image and 3D priors refine unobserved regions into a coherent reconstruction under deformation and occlusion.
  • Performance depends on initial SAM3D prediction quality and cascade control, allowing errors to propagate without oversight.Improving the geometry-estimation backbone could extend generalization to more complex interactions such as human grasping.

A. Supplementary

The supplementary material expands quantitative and qualitative evaluation on in-the-wild data, adds baseline comparisons, details implementation and metrics, and discusses limitations and failure cases.

  • Expanded quantitative and qualitative results evaluate the method on a larger in-the-wild dataset and include comparisons with additional baselines.
  • A.1 Implementation Details: Section A.1 provides additional implementation details for the pipeline.
  • A.2 Evaluation Metrics: Section A.2 defines how CLIP and EPE metrics are used for evaluation in the paper and expanded experiments.
  • A.3 Limitations and Failure Cases: Section A.3 discusses limitations and failure cases.

A.1. Deformable representation.

Lift4D represents deformation with sparse surface control nodes carrying time-varying rigid transformations, which are blended to deform canonical Gaussians. Motion regularization preserves local rigidity and smooth node trajectories, stabilizing optimization under noisy per-frame inputs.

  • Deformable representation: Sparse surface control nodes [17] carry time-varying SE(3) transformations that deform canonical Gaussians through linear blend skinning.Each Gaussian is influenced by its k nearest control nodes through blend weights.
  • Deformable representation: The regularizer combines total-variation and As-Rigid-As-Possible terms with control-node position smoothness to prevent per-frame noise overfitting.These priors preserve local rigidity and penalize abrupt control-node motion.
  • Deformable representation: Together, the motion priors stabilize optimization under noisy per-frame inputs and allow appearance refinement without distorting geometry.The stabilized representation supports later optimization stages.

A.1.1. Causal Reconstruction.

Causal reconstruction uses a default consistency strength of t0 = 0.2, balancing preservation of the previous frame’s structure with per-frame deformation. The first frame serves as the reference, and object-to-camera transform layout tokens are initialized from N(0, I).

  • The default consistency strength is t0 = 0.2, balancing previous-frame structure preservation against per-frame deformation.
  • The reference frame I⋆ is set to the first video frame.
  • Per-frame object-to-camera transform layout tokens are initialized from N(0, I).

A.1.2. 4D Optimization.

The 4D optimization initializes a sparsely controlled canonical Gaussian surface, adaptively refines it, and uses Stable Zero123 as a view-conditioned diffusion prior for LSDS.

  • The canonical Gaussian surface uses 1,024 farthest-point-sampled control nodes, k = 4 nearest nodes per Gaussian for linear blend skinning, plus adaptive densification and pruning.
  • Stable Zero123 provides the view-conditioned image diffusion prior for LSDS, with diffusion timesteps sampled from [0.2, 0.5] and guidance scale 3.0.

A.2. Metric Details … A.3.2. Balancing for Consistency & Fidelity.

Lift4D is evaluated using semantic novel-view fidelity and input-view motion fidelity, while its limitations include sensitivity to initialization and a consistency–fidelity tradeoff controlled by conditioning. It also reconstructs rich geometry and texture on simpler synthetic objects.

  • A.2.1. CLIP Score.: Novel-view semantic fidelity is measured by averaging CLIP embedding cosine similarity between rendered orbit views and corresponding input frames across frames and viewpoints.The orbit views use three uniformly spaced viewpoints at 90°, 180°, and 270° from the input views.
  • A.2.1. CLIP Score.: Novel rendered views are additionally evaluated with a text alignment score across all views.
  • A.2.2. End-Point Error (EPE).: Input-view motion fidelity is measured by comparing CoTracker3 point trajectories with trajectories obtained by tracking reconstructed geometry deformations, using EPE.CoTracker3 uses a grid size of 20, producing approximately 400 tracks per video; frame-0 points are matched to the nearest vertex or Gaussian geometry.
  • A.3.1. Dependence on Initial 3D Reconstructions.: Because the pipeline is cascaded, poor initial SAM3D geometry or layout predictions can propagate errors into canonical representations and occlusion-aware reconstruction.This failure is especially typical for high-frame-rate videos and thin objects, where jumps in per-frame transforms challenge optimization.
  • A.3.2. Balancing for Consistency & Fidelity.: The conditioning timestep t0 balances cross-frame consistency against per-frame fidelity: high values can suppress legitimate deformation, whereas low values may permit geometric flickering.The default is t0 = 0.2; some sequences may benefit from tuning, and rigid objects benefit from a higher t0.
  • A.3.2. Balancing for Consistency & Fidelity.: On simpler synthetic objects such as Consistent4D [18], Lift4D robustly reconstructs richer, more complete geometry and texture than baselines.The baselines recover simpler geometry and texture or produce incorrect deformations.
Loading 2606.23688v1…