PaperlayerView exact revision on arXiv

Source-linked AI summary

GaussVid: Sparse-View Gaussian Splatting with 3D-Aware Video Diffusion Priors

Xinhui Liu, Can Wang, Wei Jiang, Wei Wang, Dong Xu

arXiv:2608.21849v1cs.CV

TL;DR

Sparse-view 3DGS produces artifacts because limited observations leave geometry ambiguous, while video priors lack explicit multi-camera awareness. GaussVid fine-tunes a boundary-anchored video restoration model using paired data, camera conditioning, and progressive training. The method reports stronger fidelity and multi-view consistency than existing approaches while remaining competitive perceptually.

  • Problem

    Sparse-view 3DGS lacks sufficient multi-view constraints, causing ambiguous geometry, floaters, blurred structures, and poor generalization.

  • Method

    GaussVid formulates restoration as frame-conditioned reconstruction, trains on paired data, injects camera-aware geometric conditioning, and uses progressive training.

  • Results

    GaussVid significantly outperforms existing approaches and achieves strong visual fidelity and cross-view consistency in sparse-view restoration.

  • Takeaways & Limitations

    Explicit camera poses provide a reliable geometric signal for restoring artifact-corrupted sparse 3DGS renderings across viewpoints.

Abstract

from arXiv · show

3D Gaussian Splatting (3DGS) has achieved remarkable success in novel view synthesis; however, reconstructions under sparse views often exhibit noticeable artifacts. While recent video diffusion models provide strong spatio-temporal priors for 3DGS restoration, directly fine-tuning them for restoration is suboptimal, as they lack awareness of the underlying multi-camera geometry, resulting in multi-view inconsistencies. In this work, we propose a novel 3D-aware video restoration framework designed to enhance the quality of sparse 3DGS reconstruction. Specifically, we construct a large-scale 3DGS video dataset to enable specialized fine-tuning. To bridge the gap between 2D video generation and 3D multi-view constraints, we introduce a camera-conditioned geometric prior. By using the first and last frames as boundary anchors and encoding the corresponding camera relationships, we explicitly inject spatial structure into the video generation pipeline. This boundary-anchored, camera-aware prior guides the network toward geometrically grounded restoration that remains coherent across viewpoints. Extensive experiments show that, among video-prior restoration methods, our approach attains the best pixel- and structure-level fidelity (PSNR/SSIM) and improves multi-view consistency, while remaining competitive in perceptual quality (LPIPS).

I. INTRODUCTION

Sparse-view 3DGS suffers from ambiguous geometry and artifacts because limited observations provide insufficient multi-view constraints. GaussVid addresses this with a paired-data, camera-aware video restoration framework and progressive training.

  • Motivation: Sparse-view 3DGS often produces floaters, blurred structures, and poor generalization because insufficient multi-view constraints make geometry ambiguous.Regularization-based methods can remain noise-sensitive and provide limited improvements in sparse-view novel view synthesis.
  • Approach: GaussVid formulates sparse 3DGS restoration as frame-conditioned reconstruction, using first and last frames as anchors and restoring intermediate frames.The formulation treats corrupted intermediate renderings as low-quality sources for reconstruction.
  • Approach: A large-scale paired dataset with diverse camera trajectories enables supervised fine-tuning for artifact-corrupted and clean video restoration.The dataset is designed specifically to support the proposed reconstruction formulation.
  • Approach: A camera injection module adds 3D awareness to a first-to-last frame conditioned video diffusion model, improving visual fidelity and cross-view consistency.The method uses camera information as a geometric condition rather than relying only on video priors.
  • Training: Progressive training stabilizes fine-tuning and enhances restoration performance.The strategy is presented as an additional component of the restoration framework.
  • Results: Extensive experiments show that GaussVid significantly outperforms existing approaches and can enhance sparse 3DGS methods.Restored frames can also be fed back to refine 3DGS itself.

II. RELATED WORK

Prior sparse-view methods use handcrafted regularization, 2D diffusion, or video diffusion, but they remain limited by noise sensitivity, instability, or insufficient 3D awareness. GaussVid combines video priors with explicit camera-aware restoration.

  • Regularization: Regularization methods impose geometric, sparsity, opacity, perceptual, and cross-view constraints, yet remain sensitive to noise under sparse-view settings.These methods provide limited improvements and can leave geometric ambiguities and rendered-view artifacts.
  • 2D Diffusion Priors: 2D diffusion priors guide renderings toward realistic and semantically consistent appearances using methods such as SDS and progressive dataset updating.Their supervision is image-level rather than explicitly multi-view geometric.
  • Video Diffusion Priors: Video diffusion models provide temporal consistency and spatial correlations across frames, offering stronger supervision for sparse-view 3DGS.Their temporal priors can improve visual continuity compared with per-frame generation.
  • Video Diffusion Priors: Video-prior methods without explicit 3D awareness may produce geometrically inconsistent supervision across viewpoints.4DSloMo is described as improving visual continuity while lacking explicit alignment with the underlying 3D representation.
  • GaussVid: GaussVid restores intermediate frames from sparse 3DGS using anchor frames, camera poses, and a 3D-aware video restoration model.The formulation targets temporal coherence, cross-view consistency, and geometric fidelity.
  • GaussVid: Its camera-conditioning mechanism encodes camera ray fields into geometric tokens and injects them into the video model, while progressive training exposes increasingly difficult samples.The method combines a Plücker Token Encoder, Dual-Stream Camera Injection, and curriculum-based training.

B. GaussVid Dataset Construction

GaussVid constructs paired training and evaluation clips by simulating sparse-view 3DGS artifacts and sampling clean–noisy–clean sequences with camera annotations. View sparsity controls artifact severity and curriculum difficulty.

  • Dataset Construction: The dataset uses approximately 5,000 DL3DV scenes and samples one 5-frame clip per scene for paired restoration training.The construction begins because no existing dataset provides paired corrupted and clean videos with camera poses.
  • Artifact Simulation: Sparse input views are reconstructed with early-terminated 3DGS optimization to simulate realistic artifact-corrupted renderings.The optimization is intentionally terminated at 8,000 iterations, with remaining frames reserved as held-out targets.
  • Artifact Simulation: Simulated renderings are paired with original clean frames and corresponding camera poses.This produces supervision for restoring corrupted views while retaining camera information.
  • Artifact Simulation: Larger sparsity intervals yield sparser views and more pronounced floaters, blur, distortions, and structural inconsistencies.The interval n directly controls artifact severity.
  • Anchor-Based Clip Sampling: Each sample follows a clean–noisy–clean strategy with clean first and last anchors and three noisy intermediate frames.The clips are sampled at 560×1024 resolution, with intermediate targets supplied as high-quality frames.
  • Dataset Statistics: The dataset contains 5,000 training clips and 500 evaluation clips annotated with camera poses and sparsity interval n.The sparsity interval serves as a controllable difficulty measure for curriculum learning.

C. 3D-Aware Video Restoration Model

GaussVid extends a first-to-last frame-conditioned video model for sparse 3DGS restoration by adding explicit camera geometry. It uses Plücker-based camera tokens, separate camera-video processing, and temporally aligned representations to support 3D-consistent restoration.

  • Base Video Model: Wan2.1-VACE decomposes sparse 3DGS inputs into inactive and reactive components, using intermediate frames as reactive content and first/last anchors as inactive content.Both components are encoded by a frozen VAE and converted into layer-wise hints for designated DiT blocks.
  • Camera Injection: The camera-injection module keeps geometric tokens separate from noisy video features, while zero-initialized projection and γ=0.1 preserve pretrained DiT behavior at training start.This design provides explicit 3D conditioning without relying on noisy geometry estimated from sparse views.
  • Camera Representation: Given camera poses, GaussVid maps each pixel to a Plücker ray containing direction and moment features, normalized to the first camera frame.Direction depends on rotation and camera intrinsics, while moment depends on camera center and translation.
  • Temporal Alignment: Camera features are temporally packed and encoded with the same kernel and stride as video inputs so camera tokens align with video latents.This mirrors Wan2.1-VACE’s temporal packing and 3D-VAE encoding process.
  • Camera Representation: The Plücker Token Encoder disentangles moment and direction streams before fusing them through cross-gating and projecting them into the DiT hidden dimension.Cross-gating uses lightweight MLPs, sigmoid activation, and element-wise multiplication.

2) Dual-Stream Camera Injection:

Dual-Stream Camera Injection integrates camera geometry into DiT blocks through separately modulated video and camera streams. Asymmetric attention lets video tokens receive geometric context while preventing noisy video features from modifying camera tokens.

  • Motivation: Naïve token addition or concatenation entangles camera geometry with noisy video representations and permits artifact information to corrupt camera tokens.GaussVid instead treats camera features as a separate token stream.
  • Stream Modulation: Video and camera tokens undergo separate LayerNorm and adaptive modulation, with video modulation using timestep t and camera modulation using a fixed zero timestep.The two streams therefore receive distinct scale and bias parameters.
  • Asymmetric Joint Attention: The asymmetric attention mask allows video tokens to attend to all tokens, while camera tokens attend only to other camera tokens.This directional interaction transfers geometric context to video latents while shielding camera tokens from noisy video features.

D. Training Strategy

GaussVid trains a restoration model with flow matching and progressive supervision, using corrupted renderings as perturbed latent inputs and gradually introducing harder samples.

  • Progressive Latent Reconstruction: The flow-matching model interpolates Gaussian noise with the corrupted-rendering latent and predicts a velocity field toward the clean latent.The conditioning combines anchor frames, corrupted renderings, and geometric camera tokens, while training is restricted to t ∈[0.0, 0.48].
  • Progressive Latent Reconstruction: An auxiliary L1 loss constrains an approximate clean-latent reconstruction derived from the predicted velocity.The reconstruction loss is most reliable at low noise levels, motivating progressive weighting during training.
  • Difficulty-Aware Curriculum Learning: The difficulty-aware curriculum ranks samples by subsampling interval, with n = 6 representing mild artifacts and n = 26 severe degradation and structural inconsistency.Training all difficulty levels simultaneously can let noisy high-difficulty gradients overwhelm early reconstruction learning.
  • Difficulty-Aware Curriculum Learning: The curriculum progressively exposes harder samples by excluding those above a progress-dependent threshold and softly weighting the remaining samples.The schedule uses a warm-up fraction and maintains exposure to the full data range through soft weighting.
  • Total Objective: The final objective combines flow-matching and latent-reconstruction losses under curriculum weighting to stabilize fine-tuning.The staged supervision delays high-difficulty gradients until the model develops reconstruction capacity on simpler instances.

A. Experimental Setup

Experiments measure in-distribution performance on DL3DV and cross-dataset generalization on Zip-NeRF, using controlled sparse-view settings and fixed implementation choices.

  • Evaluation Benchmarks: DL3DV evaluation uses 500 held-out clips from disjoint scenes, rendered at intervals n ∈6, 8, 10, ..., 24, 26 and 560×1024 resolution.The benchmark spans indoor and outdoor environments.
  • Evaluation Benchmarks: Zip-NeRF evaluation tests four large-scale indoor scenes with Octree-GS as the base reconstruction model.This setting evaluates generalization to unseen scenes and reconstruction methods.
  • Implementation: Training uses 5-frame clips at 560 × 1024 resolution, AdamW with a 1×10−4 learning rate, LoRA adapters, and a trainable Pl¨ucker Token Encoder.The VAE encoder and decoder remain frozen.

B. Comparison with Existing Methods

Across within-dataset and cross-dataset comparisons, GaussVid provides strong fidelity and consistency while addressing limitations of per-frame, temporal-only, and noisy-geometry restoration approaches.

  • Within-Dataset Evaluation: On DL3DV, GaussVid consistently outperforms optimization-based sparse 3DGS methods and achieves the best results in most restoration comparisons.The comparison applies restoration methods as post-processing across the same sparse 3DGS baselines.
  • Within-Dataset Evaluation: DiFix3D achieves the best LPIPS but shifts geometry and hallucinates content, lowering PSNR/SSIM and breaking multi-view consistency.Its perceptual advantage reflects sharp single-frame textures rather than geometrically faithful reconstruction.
  • Within-Dataset Evaluation: 4DSloMo improves PSNR through temporal coherence, whereas its lack of explicit geometric guidance limits gains in SSIM and LPIPS.GSFIXER uses VGGT-derived geometry, but sparse-view geometry remains noisy and ambiguous.
  • Qualitative Comparison: GaussVid removes floaters and structural distortions while recovering sharp boundaries, preserving fine details, and producing results closest to ground truth qualitatively.The qualitative comparison includes corrupted 3DGS, CoR-GS, DiFix3D, GSFIXER, GaussVid, and ground truth.
  • Cross-Dataset Evaluation: On Zip-NeRF with Octree-GS, GaussVid outperforms other methods in PSNR across sparsity levels n ∈6, 9, 12.The evaluation uses indoor scenes with capture setups and distributions differing from DL3DV.

C. Ablation Studies

Ablations show that dual clean anchors, uniform and difficulty-diverse sampling, and decomposed camera conditioning each improve sparse-view restoration, with camera-aware injection producing the strongest gains.

  • Effect of Data Construction Pipeline: 22.74 dB with dual anchors exceeds the non-anchor variant by +0.84 dB, showing that clean boundary frames guide intermediate-frame restoration.Using only the first anchor reaches 22.11 dB, while removing all anchors reaches 21.90 dB.
  • Effect of Data Construction Pipeline: Qualitatively, GaussVid produces the cleanest Zip-NeRF results across sparsity levels n=6, 9, 12, while alternatives retain misalignment, structural errors, or local artifacts.The comparison highlights improved fine structures and fewer artifacts relative to DiFix3D, GSFIXER, and 4DSloMo.
  • Effect of Data Construction Pipeline: 22.74 dB from uniform sampling outperforms random sampling at 22.35 dB by 0.39 dB through more consistent temporal coverage.The comparison replaces uniform sampling of three intermediate corrupted frames with random selection.
  • Effect of Data Construction Pipeline: 22.74 dB from the mixed difficulty range outperforms narrow-only training at 22.19 dB and wide-only training at 22.47 dB.The mixed range spans n ∈{6, 8, . . . , 26}, covering mild and severe artifacts.
  • Effect of Camera Conditioning Injection: Decomposed cross-gate injection achieves the best results for all metrics, improving over no camera conditioning by +1.47 dB.It separates Pl¨ucker moment and direction streams and injects them through noise-invariant asymmetric attention.

3) Effect of Training Strategy:

Training and adaptation ablations isolate the contributions of curriculum learning, progressive latent reconstruction, and camera-aware conditioning. Full GaussVid substantially outperforms zero-shot and vanilla LoRA adaptation under a fixed backbone.

  • Effect of Training Strategy: Difficulty-aware curriculum learning raises clip difficulty from easy to hard, while progressive latent reconstruction shifts supervision toward perceptual quality.The two components are evaluated incrementally against a shuffle-trained baseline with identical data and training steps.
  • Effect of Training Strategy: Zero-shot adaptation reaches 20.56 dB, below the 3DGS baseline at 20.67 dB, while vanilla LoRA provides only a marginal +0.44 dB gain.These results indicate that direct backbone use or naive adaptation is insufficient for the restoration task.
  • Effect of Training Strategy: Full GaussVid delivers +2.07 dB over the baseline and +1.63 dB over vanilla LoRA, with consistent SSIM and LPIPS gains.All variants share the same backbone, so the gains are attributed to the proposed method rather than backbone differences.
  • Effect of Training Strategy: GaussVid combines paired artifact data, decomposed camera-aware conditioning, difficulty-aware curriculum learning, and progressive latent reconstruction loss.The conclusion reports state-of-the-art quantitative and qualitative performance across DL3DV and Zip-NeRF, including generalization to diverse baselines and unseen scenes.
Loading 2608.21849v1…