PaperlayerView exact revision on arXiv

Source-linked AI summary

DUET4D: Self-Corrective Coupled Guidance for Sparse-View 4D Reconstruction

Beizhen Zhao, Nanjie Yao, Boyi Fu, Qingsong Yan, Haofan Ren, Kaiyong Zhao, Hao Wang

arXiv:2610.35002v1cs.CVcs.CG

TL;DR

Sparse-view 4D reconstruction is under-constrained, and diffusion-based supervision is either computationally impractical or temporally inconsistent. DUET4D couples Gaussian-guided restoration with selective primitive updates; across three datasets, it improves novel-view rendering quality while maintaining efficiency and a compact representation.

  • Problem

    Sparse-view 4D reconstruction is under-constrained, while video diffusion is impractical for iterative optimization and independent single-view predictions can be inconsistent and spatially drifted.

    Source: §1 · p. 1

  • Method

    DUET4D uses Gaussian-derived alignments to steer a frozen single-view diffusion model and confidence-gated Sequential Fidelity Resampling to selectively update Gaussian primitives.

    Source: §1 · p. 2

  • Results

    Across three widely-used datasets, DUET4D improves novel-view rendering quality while maintaining high rendering efficiency and a compact Gaussian representation.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The evolving Gaussian representation and generative prior progressively refine each other through temporally coherent restoration and selective densification and pruning.

    Source: §1 · p. 2, §5 · p. 7

    When geometry is severely corrupted or motion is extremely fast with long-term occlusion, Gaussian evidence may be insufficient, and unseen content cannot be fully recovered.

    Source: §4.4 · p. 7

Abstract

from arXiv · show

Sparse-view 4D reconstruction is highly under-constrained when only a few cameras are available. Generative priors can recover missing appearances, but video models are computationally impractical for iterative reconstruction, whereas single-view models produce temporally inconsistent pseudo-observations. Moreover, directly absorbing uncertain generative outputs into Gaussian optimization often creates redundant primitives and floating artifacts. To address these challenges, we propose DUET4D, a self-corrective coupled guidance framework for sparse-view 4D Gaussian splatting. The core of our approach is a self-corrective framework where the 4D representation guides generative prior for spatio-temporal consistent supervisions. We first construct sequential alignments from current Gaussian evidence to steer the latent restoration. Then we use Sequential Fidelity Resampling to lift generative supervisions into Gaussian space to selectively distinguish repeatedly supported primitives. The corrected representation then provides more reliable Gaussian evidence for the next restoration cycle. Extensive experiments demonstrate that DUET4D significantly improves rendering quality and rendering efficiency.

1 Introduction

Sparse-view 4D reconstruction is under-constrained, while available generative priors either cost too much to query iteratively or produce inconsistent supervision. DUET4D couples Gaussian-guided restoration with selective representation updates in a self-corrective cycle.

1 Introduction

  • Sparse-view reconstruction degrades when few cameras leave scene regions unobserved and dynamic objects only partially constrained, producing distorted artifacts, floating Gaussians, and inconsistent motion.Existing texture regularization and Gaussian decaying remain insufficient under large viewpoint drifts.Source: §1 · p. 1
  • Video diffusion models are computationally impractical for iterative optimization, while independent single-view predictions lack cross-view and temporal awareness and can drift spatially.Blindly densifying drifted supervision can create redundant primitives and floating Gaussians.Source: §1 · p. 1
  • DUET4D uses current 4D Gaussians as spatiotemporal correspondence carriers to align and steer a frozen single-view diffusion model toward temporally coherent pseudo-observations.This avoids training or invoking an additional video model.Source: §1 · p. 2
  • Sequential Fidelity Resampling uses confidence-gated pseudo-supervisions and real observations to distinguish supported deficits from inconsistencies, then updates individual 3D primitives.The corrected representation supplies cleaner correspondence evidence for the next restoration round, suppressing redundant Gaussians in a closed cycle.Source: §1 · p. 2

2 Related Work

Prior sparse-view dynamic reconstruction methods use specialized priors or diffusion-generated observations, but sparse cameras and the cost or inconsistency of video diffusion remain challenges. DUET4D instead couples a 4D representation with generative guidance.

Sparse-view Dynamic Reconstruction

  • Sparse-view 4D methods degrade under few cameras; prior approaches use human-category priors such as SMPL [Peng et al. 2021a,b; Weng et al. 2022; Hu, Hu, and Liu 2024] or monocular depth and tracking cues [Lei et al. 2025; Wang et al. 2025].Recent methods still report unstable appearance under large viewpoint changes [Shi et al. 2026; Zhou et al. 2026].Source: §2 · p. 2

Diffusion Models for Novel-View Synthesis

  • Video diffusion methods synthesize novel-view dynamic observations, but expensive multi-frame denoising can introduce cross-view or temporal inconsistencies that harm 4D optimization.Diffuman4D and SparseCam4D take 40 hours for a single dynamic scene [Jin et al. 2025; Pan et al. 2026].Source: §2 · p. 2
  • DUET4D is presented as a self-corrective coupled pipeline in which the 4D representation guides generative priors for spatiotemporally consistent supervision.This is proposed to address the trade-off between consistency and efficiency in diffusion-based sparse dynamic reconstruction.Source: §2 · p. 2

3 Methodology

DUET4D uses Gaussian-derived evidence to guide single-view restoration and selectively lift supported supervision into the 4D representation. Motion regularization then improves the evidence used in the next restoration cycle.

3.1 Sequential Anchor Alignment

  • Gaussian-derived correspondences align bounded neighboring restoration residuals into a confidence-masked spatiotemporal anchor, with invalid or unsupported projections suppressed.Residuals are aggregated independently of graph traversal order, avoiding recursive error propagation within a round.Source: §3.1 · pp. 2–3

3.2 Anchor-Driven Latent Steering

  • A frozen SDM uses parallel base and anchor-guided paths, with confidence-weighted latent guidance and RGB fusion to retain supported corrections while reverting uncertain regions.The guided prediction emphasizes the anchor-guided component orthogonal to the base prediction, scaled by their discrepancy and guidance strength.Source: §3.2 · pp. 3–4

3.3 Sequential Fidelity Resampling

  • Sequential Fidelity Resampling lifts confidence-gated pseudo-view deficits into Gaussian space and uses real-view fidelity evidence to prune primitives lacking persistent support.It distinguishes repeatedly supported reconstruction deficits from primitives incompatible with sparse real observations rather than treating every image-space residual as missing geometry.Source: §3.3 · p. 4

3.4 Smoothness Control Regularization

  • After resampling, spatial-grid and temporal spline priors regularize active Gaussian motion, reducing local trajectory jitter while preserving non-rigid displacement.The corrected, motion-regularized representation supplies cleaner correspondence evidence for the next restoration cycle.Source: §3.4 · p. 4

4 Experiments

DUET4D is evaluated on three sparse-view 4D reconstruction benchmarks against dynamic Gaussian and sparse-view methods. The experiments report stronger reconstruction quality, rendering efficiency, and representation compactness, while ablations examine its components and the limitations of its Gaussian-derived correspondence evidence.

4.1 Experiment Setup

  • Evaluation uses Neu3d, Nvidia Dynamic, and Google Immersive Dataset, with PSNR, SSIM, LPIPS, temporal-PSNR, Gaussian count, and rendering FPS.Baselines include dynamic Gaussian reconstruction methods and recent sparse-view approaches; unavailable official sparse-view results are reproduced under the same input cameras and protocol.Source: §4.1 · p. 5

4.2 Evaluation

  • DUET4D achieves the highest average PSNR across evaluated datasets and outperforms SparseCam4D in reconstruction fidelity while maintaining competitive SSIM and LPIPS.The comparison with SparseCam4D uses an identical preprocessing pipeline.Source: §4.2 · p. 6
  • Up to 49× rendering acceleration and about 20% of baseline Gaussian primitives on Neural 3D demonstrate DUET4D’s efficiency and compactness.Sequential Fidelity Resampling densifies reliably under-reconstructed regions and prunes persistently inconsistent or redundant Gaussians.Source: §4.2 · p. 6

4.3 Ablation Study

  • Sequential Anchor Alignment improves temporal PSNR and rendering quality by aligning restoration residuals across neighboring views and frames without overriding current geometry or view-dependent content.These aligned residuals provide a more reliable temporal condition for latent guidance.Source: §4.3 · p. 6
  • Sequential Fidelity Resampling improves compactness and rendering efficiency by adding capacity to reliably under-reconstructed regions and removing weakly supported or redundant Gaussians.The ablation attributes this selection to confidence-gated multi-view evidence.Source: §4.3 · p. 6
  • Smoothness Control Regularization raises rendering quality, reduces Gaussian count from 0.04M to 0.03M, and leaves rendering speed nearly unchanged.The passage attributes this to spatial and temporal smoothness constraints preventing the deformation field from fitting local noise.Source: §4.3 · p. 7
  • Anchor-Driven Latent Steering yields the best overall performance, improving temporal coherence while preserving local details in pseudo-supervision.The ablation contrasts latent-space steering with directly blending restored RGB images.Source: §4.3 · p. 7
  • On fixed-view sequences, the full model reduces both long-term drift and short-term flicker in static regions; severe initial geometry corruption, extreme motion, or long-term occlusion can leave anchors insufficient.In those regions, confidence-aware fallback limits error propagation but cannot fully recover content that is never observed.Source: §4.3 · p. 7, §4.4 · p. 7

5 Conclusion

DUET4D uses an evolving Gaussian representation to guide temporally coherent restoration and selectively convert confidence-filtered discrepancies into densification and pruning, enabling the generative prior and representation to refine each other.

5 Conclusion

  • DUET4D couples a frozen single-view diffusion model with an evolving Gaussian representation to guide temporally coherent restoration and selectively convert confidence-filtered discrepancies into densification and pruning.The paper describes these components as progressively refining each other.Source: §5 · p. 7
  • Across three widely used datasets, DUET4D improves novel-view rendering quality while maintaining high rendering efficiency and a compact Gaussian representation.Source: §5 · p. 7
  • The framework addresses spatiotemporal inconsistency in generative priors as a motivation for improving sparse-view 4D Gaussian reconstruction quality.Source: §5 · p. 7

Related papers

Loading 2610.35002v1…