PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking 3D Noise: Learning 3D-Aware Video Priors via Optimization-Free Morphological Perturbations

Onat Şahin, Mohammad Altillawi, George Eskandar, Carlos Carbone, Ziyuan Liu

arXiv:2609.03657v1cs.CV

TL;DR

Sparse-view 3D representations produce inconsistent geometry and artifacts, while existing augmentations do not preserve multi-view consistency. The paper introduces optimization-free morphological perturbations for 3DGS, reducing mean depth error by 12.5% and improving robotics policy success by up to 8.0%.

  • Problem

    Sparse-view NeRF and 3DGS reconstructions suffer inconsistent geometry and rendering artifacts, while existing augmentations lack multi-view consistency.

  • Method

    3D Morphological Perturbations directly perturb 3DGS primitives through scale, rotation, and pruning to create spatially consistent training pairs without per-scene optimization.

  • Results

    12.5% lower mean depth error and up to 8.0% higher downstream robotics policy success are reported after scaling through ControlNet to a 14B video model.

  • Takeaways & Limitations

    The approach improves 3D-aware generative training while making corrupted-scene dataset curation more efficient.

Abstract

from arXiv · show

3D scene representations like NeRF and 3D Gaussian Splatting (3DGS) suffer severe artifacts in sparse-view settings. Recent generative 3D artifact fixers attempt to address this, but rely on paired corrupted and clean renders requiring costly, per-scene reconstructions across varying view configurations. While 2D image augmentations act as instant regularizers, no explicit equivalents exist for 3D representations to preserve spatial consistency across views, an essential property for 3D-aware training. We propose 3D Morphological Perturbations as an optimization-free regularizer that preserves spatial consistency. Leveraging explicit 3DGS, we treat each Gaussian as a fundamental building block - analogous to a 2D pixel - and apply perturbations across its morphological parameter space via scale, rotation, and pruning. Our method eliminates per-scene 3DGS optimization loops from dataset curation while enabling models to learn stronger geometric priors than sparse-view baselines in diagnostic ablations conducted on a lightweight video diffusion sandbox. Scaled to a 14B-parameter video model via ControlNet, our approach maintains visual fidelity while reducing mean depth error by 12.5% over state-of-the-art image-to-image 3D artifact refiners, ultimately boosting downstream robotics policy success rates by up to 8.0% across 3 of 4 manipulation tasks.

1. Introduction

Sparse-view NeRF and 3DGS reconstructions produce inconsistent geometry and artifacts, while existing data curation relies on costly per-scene optimization. The paper introduces optimization-free 3D Gaussian perturbations that preserve spatial consistency and improve refinement results.

  • Motivation: 3DGS and NeRF struggle with unobserved views, causing inconsistent geometry, structural distortions, and rendering artifacts in sparse-view settings.
  • Motivation: Existing pipelines require repeated specialized 3DGS optimizations for sparse-view or masked reconstructions, making dataset curation computationally prohibitive to scale.
  • Method: 3D Gaussian Perturbations directly noise-corrupt Gaussian primitives to generate spatially consistent corrupted-clean render pairs without per-scene optimization.The perturbations extend across Gaussian parameters, with morphological perturbations combining scale, orientation, and pruning.
  • Results: 12.5% mean depth error reduction over image-based refiners and up to 8.0% higher robotics policy success across 3 of 4 manipulation tasks demonstrate the method’s practical impact.The approach scales through ControlNet to a 14B video model while preserving visual fidelity.
  • Method: The method uses 3D Morphological Perturbations as a regularizer and diagnostic ablations to identify stronger geometric priors than sparse-view reconstructed pair sets.

2. Related Works

Prior work addresses sparse-view 3D reconstruction through per-scene regularization, feed-forward primitive prediction, and generative refinement. This paper extends noise injection from Gaussian positions to the broader 3DGS parameter space for optimization-free 3D-aware augmentation.

  • Sparse-View 3DGS & Generative Refinement: Sparse-view artifact correction combines per-scene regularization, feed-forward 3D primitive regression, and 2D/3D generative diffusion refinement.
  • Data Perturbations as Implicit Regularization: Stochastic noise injection serves as implicit regularization across input, feature, image, and 3D vision settings.
  • Data Perturbations as Implicit Regularization: Unlike prior methods that perturb only Gaussian positions, this approach injects noise across scale, orientation, color, and opacity for 3D-aware video diffusion fine-tuning.3DGS is sensitive to parameter shifts, so small perturbations can strongly degrade representations.

3. Methodology

The method injects stochastic perturbations directly into 3DGS primitives, producing optimization-free, multi-view-consistent scene degradations for regularization. It combines spatial, radiometric, and morphological operations, with morphology targeting geometry and structural density through scale, rotation, and pruning.

  • 3.1. 3D Gaussian Perturbations as Regularization: 3D primitive perturbations induce a manifold-constrained regularizer that targets geometry-consistent surface variations instead of random 2D pixel noise.Projecting model sensitivity through the rasterizer Jacobian encourages recovery of structural artifacts, while pruning requires reconstruction of missing geometry.
  • 3.2. Primitive Perturbations: The method separates perturbations into spatial, radiometric, and morphological categories targeting primitive positions, appearance, geometry, and density.Spatial noise affects centers, radiometric noise affects color and opacity, and morphological noise changes scale, rotation, and primitive retention.
  • 3.2.3. 3D Morphological Perturbations: Morphological perturbations alter primitive extents and density while preserving global alignment, modeling structural degradation and volume loss.They include scale perturbation, quaternion-based rotation perturbation, and random pruning.
  • 3.2.3. 3D Morphological Perturbations: Pruning with p_prune = 0.5 halves expected scene density, forcing reconstruction of complete surfaces from sparse primitive coverage.Each primitive receives a binary retention mask, with E[M′] = 0.5M under the stated setting.
  • 3.2.3. 3D Morphological Perturbations: Scale and rotation noise simulate shape and orientation errors while maintaining valid primitives and local geometry.Scale jitter is set to 1%, while rotation noise introduces approximately 1.15° angular jitter after quaternion renormalization.
  • 3.1. 3D Gaussian Perturbations as Regularization: Perturbing primitives before rasterization yields trajectory renders with exact multi-view epipolar consistency without optimization.The method applies Tη to 3DGS primitives and renders the perturbed scene along the camera path.

4. Experiments

Experiments progress from diagnostic perturbation ablations to a 14B-parameter ControlNet refinement model and downstream robotics evaluation. Morphological perturbations provide strong geometric fidelity, stable representations, and improved refinement and policy performance.

  • 4.1. Diagnostic Ablation: Perturbation Categories: Morphological perturbations achieve high geometric fidelity and appearance closest to ground truth, outperforming alternative perturbation strategies under extreme sparsity.Naive perturbations fail to reconstruct coherent geometry or color, while spatial and radiometric perturbations trade off geometry and appearance.
  • 4.1. Diagnostic Ablation: Perturbation Categories: CKA analysis shows morphological perturbations produce the cleanest, least fluctuating feature trajectory among isolated 3D perturbations.Naive perturbations leave activations stagnant, whereas 2D blur causes excessive activation drift.
  • 4.2. Scaling to Video Foundation Models for Trajectory Render Refinement: The scaled model uses Wan2.2 with a dilated ControlNet trained on 9,866 perturbed-clean 49-frame video pairs from three datasets.Evaluation covers Low, Medium, and High difficulty regimes on 49-frame trajectories rendered at 832 × 480.
  • 4.2. Scaling to Video Foundation Models for Trajectory Render Refinement: Wan + Ctrl (Morph.) achieves top performance across nearly all geometric metrics in Medium and High difficulty regimes, with depth RMSE values of 16.18, 30.65, and 29.53.The morphologically perturbed dataset guides artifact refinement and inpainting toward details that blend naturally with the original scene while preserving multi-view coherence.
  • 4.3. Downstream Robotics Imitation Learning: The downstream refinement improves DREMA policy success by up to 8.0% across three of four manipulation tasks.DREMA+ref. reaches 59.2% versus 51.2% on Close Jar, 24.4% versus 23.6% on Lift, and 38.0% versus 34.4% on Pick Cup.

5. Conclusion

The paper concludes that optimization-free morphological perturbations make 3D-aware generative training more efficient while improving geometric refinement and downstream robotics performance.

  • 5. Conclusion: Morphological perturbations bypass repeated per-scene reconstructions, reduce mean depth error by 12.5% over image-based refiners, and boost robotics policy success by up to 8.0%.The perturbations modify scale, rotation, and pruning in explicit 3DGS representations and are scaled to a 14B-parameter video model through ControlNet.

A. Diagnostic Ablation: Perturbation Categories

This diagnostic ablation section extends the lightweight video diffusion experiments with additional architectural, protocol, and qualitative details.

  • A. Diagnostic Ablation: Perturbation Categories: The supplementary section documents architectural modifications, optimization parameters, evaluation protocols, and qualitative comparisons for the AnimateDiff-based video diffusion model.Extended comparisons cover extreme, high, and moderate sparsity settings S80, S40, and S20.

A.1. AnimateDiff + Stable Diffusion + LoRA Implementation Details for Video-to-Video Refinement

The video-to-video refinement model injects perturbed 3DGS conditioning into an expanded Stable Diffusion and AnimateDiff backbone while restricting most adaptation to LoRA parameters.

  • A.1. AnimateDiff + Stable Diffusion + LoRA Implementation Details for Video-to-Video Refinement: The UNet input expands from four to eight channels so zcond directly carries noisy 3D Gaussian rendering latents for spatial-temporal conditioning.The conditioning tensor concatenates zt and zcond, anchoring scene geometry and spatial alignment to 3D Gaussian renderings.
  • A.1. AnimateDiff + Stable Diffusion + LoRA Implementation Details for Video-to-Video Refinement: The added conditioning weights are initialized at 0.1 times the base weights to prevent activation magnitude explosions while enabling immediate gradient flow.The original four channels retain Wbase, while the four new channels receive 0.1 × Wbase.
  • A.1. AnimateDiff + Stable Diffusion + LoRA Implementation Details for Video-to-Video Refinement: The 2D UNet and temporal motion modules remain frozen, with updates restricted mainly to LoRA matrices and the newly added input convolution.The input convolution is optimized with an elevated learning rate to adapt spatial alignment and conditioning.
  • A.1. AnimateDiff + Stable Diffusion + LoRA Implementation Details for Video-to-Video Refinement: Training uses AdamW with cosine learning-rate scheduling, 5% linear warmup, and bfloat16 automatic mixed precision.Validation spans temporal stride regimes S20, S40, and S80.

A.2. Extended Qualitative Results

Extended ablations show that 3D Morphological Perturbations remain the most consistent across sparsity regimes, while the refinement model uses a frozen 14B backbone with a trainable ControlNet.

  • Extended Qualitative Results: 3D Morphological Perturbations remain the most consistent across S80, S40, and S20 sparsity regimes.Higher input density generally improves quality for all models, whereas 2D Render Blur and 3D Naive Perturbations show the least improvement.
  • Model Details: The refinement model uses Wan2.2’s 14B text-to-video backbone with 49-frame, 832 × 480 video sequences.Training and testing videos are formatted consistently before refinement.
  • Model Details: The ControlNet receives noisy trajectory videos while the frozen base model receives text captions and latent inputs.ControlNet activation offsets are injected additively into corresponding frozen backbone blocks.
  • Model Details: Training predicts the noise residual needed to reconstruct clean latents using a standard noise-prediction objective.Gradients update only ControlNet parameters, while the Wan2.2 base model remains frozen.
  • Model Details: Inference initializes the latent video representation with random noise and performs 50 iterative denoising steps using the frozen ControlNet.The scheduler updates the latent representation at each timestep.

B.2. Extended Qualitative Results

Across sparse-view difficulty levels, Wan + ControlNet (Morph.) preserves visual quality, sharp details, and multi-view coherence better than single-frame and video baselines.

  • Extended Qualitative Results: Single-frame Fixer outputs are smooth but lose scene structure, while Difix degrades rapidly as multi-view consistency errors increase in harder regimes.Reconstructing 3DGS from these outputs causes severe geometric degradation because both methods lack cross-frame temporal awareness.
  • Extended Qualitative Results: Wan + ControlNet (Morph.) maintains stable visual quality, removes rendering artifacts, and preserves sharp details and multi-view coherence across all difficulty levels.The extended comparisons cover High, Medium, and Low sparse-view settings with corresponding depth renders.
  • Extended Qualitative Results: Wan+Tile fails to preserve multi-view geometry and hallucinates details that diverge from the ground truth.

C. Downstream Robotics Imitation Learning

The downstream robotics evaluation uses refined 3DGS scenes to generate synthetic manipulation episodes, adding appearance diversity while preserving temporal and multi-view geometry.

  • Downstream Robotics Imitation Learning: The refinement pipeline adds photorealistic lighting, realistic color shifts, and enhanced surface textures while preserving multi-view and temporal 3D consistency.The additional results are shown for synthetic manipulation episodes generated by DREMA and DREMA+ref.
  • Downstream Robotics Imitation Learning: Appearance diversity encourages PerAct to rely on underlying scene geometry during evaluation.The refinement can occasionally introduce noisy surfaces, such as on the table in the Pick Cup task.
  • Downstream Robotics Imitation Learning: The extended scene comparisons cover extreme, high, moderate, and multiple sparse-view difficulty settings using re-optimized 3DGS representations.
  • Downstream Robotics Imitation Learning: The qualitative robotics comparisons include trajectory keyframes and close-ups of target objects across synthetic RLBench task episodes.
Loading 2609.03657v1…