PaperlayerView exact revision on arXiv

Source-linked AI summary

UniFusion: Sparse-View 4D Reconstruction via Unified Spatio-temporal Depth Alignment

Yongzhe Lyu, Shaofei Wang, Yixin Chen, Siyuan Huang

arXiv:2609.05888v1cs.CV

TL;DR

Sparse-view 4D reconstruction is difficult because limited cross-view overlap and temporal variation make monocular depth predictions inconsistent. UniFusion jointly aligns depths across views and time with spatio-temporal neural fields, then uses them to initialize and supervise Gaussian-splatting reconstruction. On Ego-Exo4D and EgoHuman, the paper reports better synthesis and geometry, while noting that reconstruction quality depends on the accuracy of its depth and SfM priors.

  • Problem

    Sparse-view videos have limited cross-view overlap, while monocular depth predictions are inconsistent across views and time, complicating 4D reconstruction.

    Source: §1 · pp. 1–2

  • Method

    UniFusion jointly aligns depth across views and time with spatio-temporal neural fields, then uses aligned depths to initialize and supervise Gaussian-splatting reconstruction.

    Source: §1 · p. 3, §3.1 · p. 5, §5 · p. 14

  • Results

    On Ego-Exo4D and EgoHuman, aligned depths improve novel-view synthesis by PSNR +4 ∼6dB and reduce geometric error (AbsRel) by 14–40%.

    Source: §1 · p. 3, §5 · p. 14

  • Takeaways & Limitations

    The results support the conclusion that accurately aligned geometric priors matter more than downstream architectural complexity for sparse-view 4D reconstruction.

    Source: §1 · p. 3, §5 · p. 14

    Reconstruction quality is bounded by the accuracy of the monocular depth foundation models and sparse SfM points used for initialization.

    Source: §5 · p. 14

Abstract

from arXiv · show

In this paper, we address the challenging problem of 4D reconstruction from sparse-view videos. This setup usually relies on monocular depth estimation to provide priors for the reconstruction model. A key challenge arises from limited cross-view overlap and temporal variation, making monocular depth predictions inconsistent across views and time. Existing methods align spatial and temporal dimensions in separate stages, requiring foreground segmentation masks while failing to leverage temporal cues for cross-view alignment. Contrary to these methods, we propose a unified spatial-temporal depth alignment framework that jointly resolves cross-view and cross-time inconsistencies without distinguishing foreground/background. Our method represents depth maps across views and time as a set of spatio-temporal neural fields. This representation not only yields fast convergence, but also captures spatio-temporal correlation among depth maps implicitly, without dependence on external segmentation/tracking models. We also propose a multi-view depth-order loss while leveraging the classic scale-and-shift-invariant loss to further improve the final depth quality. The aligned depths initialize and supervise Gaussian splatting models for 4D reconstruction. Experiments on Ego-Exo4D and EgoHuman demonstrate that our improved depth alignment substantially benefits dynamic Gaussian-splatting-based reconstruction methods for novel-time/view synthesis and geometry accuracy/consistency.

1 Introduction

Sparse-view 4D reconstruction uses few cameras with limited cross-view correspondences, while monocular depth predictions vary across views and time. UniFusion jointly aligns those depths and uses them to initialize and supervise Gaussian-splatting reconstruction.

1 Introduction

  • Figure 1 depicts aligned depths as multi-view-consistent and temporally coherent, then uses them for reconstruction that remains reasonable about 45° from training views.The depths serve as both initialization and supervision.Source: Figure 1 · p. 2
  • Sparse-view cameras may have minimal overlap, and independent monocular predictions have unknown per-view scale and shift and are inconsistent across views and time.Scene motion is entangled with depth-prediction errors, making these inconsistencies difficult to disentangle.Source: §1 · p. 2
  • A single framework jointly aligns monocular depth across views and time using spatio-temporal neural fields, rather than separate spatial and temporal stages.The formulation handles regions uniformly without foreground segmentation or external tracking models.Source: §1 · p. 3
  • UniFusion’s aligned depths also let straightforward Gaussian-splatting frameworks outperform existing complex multi-stage processing pipelines.The paper reports consistent improvements across different Gaussian-splatting variants.Source: §1 · p. 3

2 Related Work

Prior work provides monocular depth priors, multi-view geometry, and dynamic-scene reconstruction methods, but leaves joint depth alignment across views and time unresolved. UniFusion extends sparse-view chart alignment to the spatio-temporal setting.

Monocular depth estimation

  • Monocular depth models such as MiDaS, Omnidata, ZoeDepth [3], Metric3D v2, Marigold, Depth Anything, and MoGe improve predictions, but process images independently.Video depth methods DepthCrafter and Geo4D pursue temporal consistency without multi-view geometric consistency.Source: §2 · pp. 3–4

Multi-view geometry and depth alignment

  • DUSt3R, MASt3R, and VGGT provide multi-view-consistent point maps, while MAtCha aligns monocular depths and point maps for static sparse-view geometry.The cited passage characterizes learned point maps as relatively coarse compared with dedicated monocular depth predictions; MAtCha does not address dynamic reconstruction.Source: §2 · p. 4

Dynamic scene reconstruction

  • Dynamic-scene work spans deformation or temporal conditioning in neural radiance fields and efficient representations such as HexPlane, K-Planes, Tensor4D, and ResFields.Gaussian-splatting approaches model dynamics through deformation or temporally extended primitives.Source: §2 · p. 4
  • Sparse-view dynamic reconstruction remains under-explored: SplatFields focuses on foreground objects, while MonoFusion uses separate spatial and temporal alignment stages.UniFusion combines unified spatio-temporal alignment with deformable Gaussian splatting, and the paper reports outperforming more complex multi-stage approaches.Source: §2 · p. 5

3 Method

UniFusion jointly aligns monocular depths across views and time, then uses the aligned depths to initialize and supervise 4D Gaussian splatting. Its neural fields share spatial representations across frames, while temporal conditioning and geometric losses address depth variation and cross-view consistency.

3 Method

  • The method first aligns monocular depth maps across views and time, then uses the aligned depths to initialize and supervise 4D Gaussian-splatting reconstruction.It targets novel-view and novel-time synthesis.Source: §3 · p. 5, §3.3 · p. 9

3.1 Spatio-Temporal Depth Alignment

  • Within each view, frames share spatial and depth encodings, and a deformation network predicts depth offsets from the combined encoding.The encodings are shared across time steps; the predicted offset is applied along the camera ray.Source: §3.1 · pp. 6–7

Temporal conditioning via residual weights.

  • Per-frame residual weights extend the deformation network across time, allowing static-region residuals to vanish and dynamic regions to receive frame-specific adjustments without segmentation masks.The residuals use a compact vector-matrix factorization; neural-field smoothness also reduces temporal jitter in aligned depths.Source: Temporal conditioning via residual… · p. 7

3.2 Optimization Objectives

  • The multi-view depth-order loss penalizes reprojected points that fall in front of another view’s recorded surface, using depth maps as geometric supervision without monocular priors or explicit correspondences.It reprojects deformed depths between view pairs at the same time step.Source: §3.2 · p. 8
  • The scale-and-shift-invariant loss compares normalized depth maps to capture per-view shape similarity independently of scale and shift.Normalization subtracts the median and divides by mean absolute deviation; the loss is the L1 distance between normalized maps.Source: §3.2 · p. 8
  • Additional alignment losses fit sparse SfM observations, preserve monocular structure, and enforce cross-view matching, alongside spatial-encoding and depth-encoding regularization.The total objective combines these terms with the depth-order and SSI losses.Source: §3.2 · pp. 8–9

3.3 4D Gaussian Splatting Reconstruction

  • Aligned depths initialize and supervise Gaussian-splatting reconstruction approaches, improving the quality of a variety of baselines.Training follows each reconstruction method’s depth-supervision schedule when available.Source: §3.3 · p. 9

4 Experiments

Experiments compare aligned-depth supervision across two Gaussian-splatting backends on Ego-Exo4D and EgoHuman, then ablate alignment components. Results show gains in synthesis and geometry, with improvements across both backends.

4.1 Experimental Setup

  • Evaluation uses four roughly 90°-separated cameras for Ego-Exo4D and four exocentric training cameras for two EgoHuman sequences, with intermediate cameras held out.Both datasets use ground-truth camera poses; evaluation reports image-quality metrics, with AbsRel also reported for Ego-Exo4D.Source: §4.1 · p. 9

4.2 Comparisons with State-of-the-Art

  • Def-2DGS + Ours beats the state-of-the-art method by PSNR +4.25dB and AbsRel −0.055 on Ego-Exo4D.Table 1 reports full-image and dynamic-region metrics.Source: §4.2 · pp. 10–11, Table 1 · p. 11
  • Aligned depths yield reasonable 30°–45° novel views and improve geometry, removing ghosting in held-out training-camera frames.Qualitative comparisons also report sharper, more geometrically coherent results, especially in dynamic regions.Source: Figure 3 · p. 10, §4.2 · pp. 11–12, Figure 4 · p. 12
  • On EgoHuman, novel-view synthesis improves +6.26dB over MonoFusion and +3.88dB over deformable 2DGS.Table 2 evaluates two EgoHuman sequences.Source: Table 2 · p. 11, §4.2 · p. 12

4.3 Ablation Studies

  • Per-frame alignment lacks temporal modeling, and a naive time-conditioned MLP severely degrades quality; temporal residual weights provide the largest improvement.The ablation trends are consistent across MonoFusion and deformable 2DGS.Source: §4.3 · pp. 12–13
  • Aligning 100 frames from four views takes one hour, versus about half a day for per-frame MAtCha alignment, while achieving improved quality.Source: §4.3 · pp. 12–13
  • Depth-order regularization and the scale-and-shift-invariant loss each add further improvement.The ablations add these losses successively to the alignment model.Source: §4.3 · pp. 12–13

5 Conclusion

UniFusion jointly aligns depth across views and time using spatio-temporal neural fields, then uses the aligned depths to initialize and supervise Gaussian-splatting reconstruction. Experiments report improved geometry and novel-view synthesis, while prior accuracy and fast, complex motion remain constraints.

5 Conclusion

  • Spatio-temporal neural fields with temporal residual weight conditioning jointly align depth across views and time, and aligned depths initialize and supervise Gaussian-splatting reconstruction.The unified optimization eliminates separate alignment stages, foreground segmentation masks, and pixel-tracking models.Source: §5 · pp. 13–14, Figure 6 · p. 14
  • Across Ego-Exo4D and EgoHuman, alignment reduces AbsRel by 14–40% and improves novel-view PSNR by 4–6 dB, about 10× faster than naive per-frame alignment.Source: §5 · p. 14
  • Reconstruction quality is bounded by monocular-depth and sparse-SfM prior accuracy, and basic deformable 2DGS can fail with fast, complex motion.Point tracking could still help in that motion setting.Source: §5 · p. 14, Figure 6 · p. 14

A Implementation Details

The implementation evaluates two dynamic Gaussian-splatting backends: Deformable 2DGS and MonoFusion. Deformable 2DGS deforms a canonical Gaussian scene using spatio-temporal features, while MonoFusion blends learned rigid-motion bases.

A.1 Gaussian Splatting Backends

  • Deformable 2DGS and MonoFusion are the two backends used for 4D reconstruction.Deformable 2DGS uses a canonical scene and deformation network; MonoFusion uses a canonical Gaussian field and a deformation model.Source: §A.1 · pp. 15–16
  • Deformable 2DGS uses 2D Gaussians for the canonical scene to support depth, normal, and curvature supervision.Its Gaussian covariance is factorized into rotation and per-axis scaling to keep it positive semidefinite.Source: §A.1 · p. 15
  • A multi-resolution 4D K-Planes representation and linear projection encode spatio-temporal features for deformation, with separate decoders predicting position, rotation, and scale changes.The deformed Gaussian retains its opacity and color.Source: §A.1 · pp. 15–16
  • MonoFusion models motion as weighted combinations of learnable rigid basis trajectories, with per-point weights initialized by K-means clustering semantic features.The weighted rigid transformation maps each Gaussian from canonical space to the target time.Source: §A.1 · p. 16
  • MonoFusion aligns depth scale using DUSt3R as a metric reference and estimates a global scale from median depth ratios over SAM2-identified static background regions.Scaling, opacity, and color remain invariant over time in this implementation.Source: §A.1 · p. 16

A.2 Evaluation Details

The evaluation details describe a reimplemented dynamic-region protocol and its padding sensitivity, alongside fixed depth-alignment loss weights and optimization duration.

A.2 Evaluation Details

  • Because MonoFusion's dynamic-region protocol is vaguely described and its official code lacks evaluation scripts, the authors reimplement the foreground-object evaluation pipeline.They mask foregrounds, crop their bounding boxes, and pad the crops with a white background because foreground-only PSNR did not reproduce MonoFusion's reported metrics.Source: §A.2 · pp. 16–17
  • Padding ratio substantially shifts dynamic-region photometric metrics; the authors select 1.4× because it most closely matches MonoFusion's reported baseline values.The sensitivity analysis is summarized in Table 4.Source: §A.2 · p. 17, Table 4 · p. 17

A.3 Losses

  • Depth alignment uses λs = 4, λm = 5, λo = 5, and λssi = 1 for 10,000 optimization iterations in all experiments.Gaussian-splatting backends follow their respective official training configurations.Source: §A.3 · p. 17

B Qualitative Analysis of Individual Components

The qualitative ablation examines temporal encoding and depth-order loss by unprojecting aligned depth maps into 3D.

B Qualitative Analysis of Individual Components

  • Temporal encoding consistently improves depth quality across time in the qualitative ablation.Figure 7 uses depth unprojections from one view to demonstrate the effect.Source: B Qualitative Analysis of Individual… · p. 17, Figure 7 · p. 18
  • Depth-order loss refines multi-view geometry and removes floaters in the qualitative ablation.Figure 8 uses depth unprojections from two opposed views to highlight multi-view consistency.Source: B Qualitative Analysis of Individual… · p. 17
  • Figure 7 visualizes temporal encoding by unprojecting depth maps from one view.The figure is presented as a qualitative analysis of temporal encoding.Source: Figure 7 · p. 18

C Direct Evaluation of Depth Alignment Quality

Direct alignment metrics show improved depth consistency before rendering optimization, and further experiments report gains on an additional scene and across ablations.

C Direct Evaluation of Depth Alignment Quality

  • On Ego-Exo4D, the method improves all three direct alignment metrics over MAtCha and MonoFusion: cross-view reprojection error, flow-warped depth error, and depth-order violation rate.These backend-independent measures assess cross-view correspondence, temporal stability, and ordinal geometry; Table 5 reports the comparison.Source: C Direct Evaluation of Depth Alignment… · p. 18, Table 5 · p. 19

D Additional Scenes

  • On a Harmony4D hugging scene with close human interaction, occlusions, and non-rigid motion, the method significantly improves reconstruction quality.The experiment uses four input cameras; results are shown in Table 6.Source: D Additional Scenes · p. 19, Table 6 · p. 20

E.1 Ablation on Novel View Synthesis

  • On EgoHuman, the ablated alignment components progressively improve novel-view synthesis across two Gaussian-splatting backends.The progression is reported in Table 7.Source: §E.1 · p. 20, Table 7 · p. 20

E.2 Ablations on Residual Rank and Lorder

  • Increasing residual rank improves performance through R = 8, then performance saturates and slightly degrades, indicating a capacity–overfitting trade-off.The comparison evaluates R ∈ {4, 8, 16} on Ego-Exo4D with Def-2DGS plus the full model.Source: §E.2 · p. 20
  • The hinge and symmetric L1 depth-order formulations perform comparably, with the hinge loss yielding slightly better PSNR.The comparison studies robustness to the penalty design on Ego-Exo4D with Def-2DGS plus the full model.Source: §E.2 · p. 20

E.3 Ablation on Different Camera Baselines

Under 3- and 6-camera EgoHuman settings, the method improves reconstruction quality; additional novel-view results show improved geometry quality.

E.3 Ablation on Different Camera Baselines

  • The evaluation uses 3 and 6 input cameras, with cameras between selected input views held out as novel views and Def-2DGS as the backend.The camera settings roughly correspond to 120° and 60° spacing, respectively.Source: §E.3 · pp. 20–21
  • Separate Ego-Exo4D ablations examine residual rank R and L_order formulation with Def-2DGS + Ours.Source: Table 8 · p. 21, Table 9 · p. 21
  • The method consistently improves reconstruction quality under both settings and remains effective across different camera baselines, including the more challenging 3-camera setting with wider view spacing.Source: §E.3 · p. 21

F More Qualitative Results

  • Additional novel-view results show that the method robustly improves reconstructed-scene geometry quality.These qualitative results are presented in Fig. 9.Source: F More Qualitative Results · p. 21

G CMU Panoptic Dataset

MonoFusion could not be reproduced on CMU Panoptic because DUSt3R preprocessing failed for four input views. VGGT depth priors produced vague renderings and were suboptimal for robust scene reconstruction.

G CMU Panoptic Dataset

  • DUSt3R preprocessing failed for MonoFusion on CMU Panoptic with 4 input views, even with known camera parameters.The authors hypothesize that the capture dome’s pure white background may contribute; similar issues were reported on the official repository without a solution at submission.Source: G CMU Panoptic Dataset · p. 21

H VGGT Priors

  • VGGT depth priors were suboptimal for robust scene reconstruction, producing vague renderings when used to initialize and supervise Deformable 2DGS.Fig. 10 illustrates this evaluation on EgoHuman.Source: H VGGT Priors · p. 22, Figure 10 · p. 22

Related papers

Loading 2609.05888v1…