PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Alias-Free 4D Gaussian Representations with Motion-Aware Filtering

Ankit Dhiman, Kunal A Kathare, Pranav Vignesh, Lokesh R Boregowda, Venkatesh Babu Radhakrishnan

arXiv:2608.21828v1cs.CV

TL;DR

Dynamic 4D representations still exhibit aliasing when novel views diverge from training viewpoints, and static filters do not account for scene motion. The paper introduces a motion-aware 3D smoothing filter that estimates time- and focal-length-to-depth-dependent sampling intervals, outperforming baseline representations at higher resolutions while remaining representation-independent.

  • Problem

    4D representations for dynamic-scene novel-view synthesis suffer aliasing artifacts from divergent viewpoints, while static anti-aliasing filters fail to account for motion.

  • Method

    The method uses Parzen Window kernel density estimation to learn each Gaussian primitive’s joint distribution over discrete time and focal-length-to-depth ratio, then selects a time-dependent sampling interval.

  • Results

    The method outperforms baseline 4D representations when trained at lower resolution and evaluated at higher resolutions, including a 2.11 dB improvement over SARO-GS on the Plenoptic dataset.

  • Takeaways & Limitations

    The filter supports alias-free zoom-in and zoom-out rendering and can be applied to different 4D representations.

  • Takeaways & Limitations

    Storing a discrete joint density matrix for every Gaussian primitive has memory complexity O(N × D × T), which can become significant for large scenes.

Abstract

from arXiv · show

Novel-view synthesis of dynamic scenes, crucial for AR/VR applications, remains a challenging problem. Recent methods adapt representations like 3D Gaussian Splatting (3DGS) and Neural Radiance Fields (NeRF) for dynamic scenes by incorporating time as the fourth dimension (4D representations). These 4D representations still suffer from aliasing artifacts, especially when generating novel views from divergent viewpoints (zoom-in/zoom-out operations). While using 3D smoothing filters like those proposed in Mip-Splatting might seem like a possible solution, they fail to account for local motion and also exhibit aliasing. To address this, we propose a motion-aware 3D smoothing filter specifically designed for 4D representations. Our approach adapts the filter strength based on local motion information, effectively mitigating aliasing without compromising rendering quality. This is achieved by estimating the joint density function of time and focal-to-depth ratio using a non-parametric estimation method. During inference, we sample from this joint distribution to determine the appropriate smoothing filter. This flexible strategy can be integrated with various 4D representations. Our evaluations on standard datasets demonstrate superior performance compared to state-of-the-art methods.

1 Introduction

Dynamic-scene novel-view synthesis remains difficult because 4D representations produce aliasing from divergent viewpoints, while static filtering ignores motion. The paper proposes a motion-aware filter based on joint time and focal-length-to-depth statistics, with representation-independent integration.

  • 1 Introduction: 4D dynamic-scene representations suffer aliasing artifacts when rendered from viewpoints divergent from training views.The problem is especially visible during zoom-in and zoom-out operations.
  • 1 Introduction: Static anti-aliasing filters such as Mip-Splatting remain ineffective for dynamic scenes because they fail to account for temporal motion.Motion leads to incorrect filter estimation and persistent artifacts.
  • 1 Introduction: A motion-aware smoothing filter determines the maximum sampling interval according to local temporal changes in the scene.The strategy is designed to enable accurate rendering of dynamic scenes.
  • 1 Introduction: The method learns a joint distribution of discrete time and focal-length-to-depth ratio using Parzen Window kernel density estimation.The distribution is learned from accumulated data points to avoid noisy timestamp-wise sampling-interval estimates.
  • 1 Introduction: The modifications are invariant to the underlying 4D representation and can be plugged into other representations.The paper also demonstrates efficacy on challenging real-world datasets.

2 Related Work

Prior work spans implicit, grid-based, point-based, and Gaussian representations for radiance fields and dynamic scenes. Existing anti-aliasing approaches either target static scenes, operate in screen space, or fail to account for motion in 4D representations.

  • 2 Related Work: NeRF uses volumetric rendering, while later methods accelerate radiance-field processing with hash grids, voxel grids, point-based methods, and depth supervision.These approaches address the computational cost of NeRF-based rendering and training.
  • 2 Related Work: 3DGS represents scenes with 3D Gaussians projected into 2D splats and rasterized with alpha blending for high-quality real-time rendering.Subsequent work has targeted rendering quality, speed, and memory optimization.
  • 2 Related Work: Dynamic-scene methods extend NeRF and 3DGS using deformation fields, canonical representations, spatial-temporal decompositions, or directly optimized 4D Gaussians.These approaches model temporal changes through different representation choices.
  • 2 Related Work: Pre-filtering and EWA splatting address aliasing in earlier radiance-field or screen-space settings, whereas Mip-Splatting designs a 3D filter during optimization for static scenes.Mip-Splatting is reported to fail for dynamic scenes, motivating a motion-dependent 3D band-limited filter.

3 Preliminaries

The preliminaries introduce 3D Gaussian Splatting, its dynamic 4D extensions, and Mip-Splatting’s filtering strategy. They then frame motion-aware sampling-rate estimation as a per-primitive procedure for selecting filters over time.

  • 3.1 3D Gaussian Splatting: 3DGS represents scene geometry with 3D Gaussian primitives, projects them into 2D splats, depth-sorts them, and rasterizes them with alpha blending.Each primitive includes position, anisotropic covariance, opacity, and view-dependent color represented by spherical harmonics.
  • 3.2 Dynamic Representations using 3DGS: Dynamic 4D representations parameterize Gaussian primitives over space and time, with SARO-GS adding time-dependent residual features before rasterization.SARO-GS uses 4D location, covariance, color, and opacity parameters together with a lightweight residual-field MLP.
  • 3.3 Anti-aliasing filter for 3DGS: The motion-aware sampling procedure takes timestamped focal-length-to-depth ratios, masks empty observations, normalizes ratios, and estimates a joint density on time and ratio grids.At inference, it selects the nearest time grid point, finds the conditional-density mode, and rescales that mode to obtain the sampling rate.
  • 3.3 Anti-aliasing filter for 3DGS: Sampling intervals larger than twice the world-space sampling interval can introduce aliasing, motivating filtering based on the estimated sampling rate.The sampling-rate formulation uses focal length f and camera-to-primitive distance d through their ratio f/d.
  • 3.3 Anti-aliasing filter for 3DGS: Mip-Splatting estimates each primitive’s depth-dependent sampling rate from focal length, camera visibility, and distance, then applies a 3D Gaussian low-pass filter.The filtered Gaussian is obtained by convolution, with filter size controlled by s and a primitive-specific sampling-rate term remaining fixed after training.

4 Proposed Method

The method addresses aliasing in dynamic 4D representations by adapting smoothing to time-varying sampling requirements caused by motion. It estimates motion-aware sampling rates and applies filters that preserve detail across multiple evaluation scales.

  • Aliasing in Dynamic Representations: 4D representations exhibit disappearing primitives during zoom-in and noticeable aliasing during zoom-out when viewpoints diverge from training views.These failures occur in both dynamic and static regions for several evaluated methods.
  • Limitations of Static Anti-Aliasing Filters: Static anti-aliasing filters fail in dynamic scenes because they do not account for temporal motion and changing primitive depth.Dynamic regions have substantially varying f/d values over time, making a single static smoothing filter inadequate.
  • Necessity of Time-Varying Sampling Rates: Each timestamp requires a separate maximal sampling rate because motion changes a primitive’s focal-length-to-depth ratio f/d across spatio-temporal projections.The filter adapts per primitive and timestep while satisfying the Nyquist criterion.
  • Motion-Aware Smoothing Filter: The method estimates the joint density P(X, T) of normalized f/d and time using Parzen window density estimation, then recovers sampling rates for filtering.Per-primitive f/d values are normalized by their maximum, estimated over density grids, and rescaled during inference.
  • Evaluation: On the Synthetic Dataset, the method outperforms baselines at the 2× and 4× evaluation scales after single-scale training.The methods are trained at 1× resolution and evaluated at 1×, 2×, and 4×.
  • Motion-Aware Smoothing Filter: The recovered sampling rate is applied to a deformed Gaussian primitive so its highest frequency component does not exceed half its maximal sampling rate.This design encodes the stated anti-aliasing constraint for each primitive.

5 Experiments

Experiments show that the motion-aware filter preserves rendering quality across scale changes, improves temporal consistency, and generalizes beyond its base representation. It performs especially strongly when models are trained at lower resolution and evaluated at higher or lower resolutions.

  • Qualitative Results: The method preserves high-frequency details in qualitative comparisons where baselines show blur, aliasing, or artifacts.This pattern appears on D-NeRF, HyperNeRF, and scenes such as “hellwarrior,” “lego,” “printer head,” and “yellow sticker.”
  • Multi-Scale Testing: The method achieves comparable quality at the training scale while substantially improving fidelity during multi-scale evaluation.On Plenoptic, it significantly outperforms baselines at 2× and 4× resolution, including a 2.11 dB gain over SARO-GS.
  • Temporal Consistency: The method produces the most coherent renderings across consecutive frames according to warped temporal metrics.Temporal consistency is evaluated with warped error, W-LPIPS, and W-SSIM.
  • D-NeRF Evaluation: 5.06 dB over Deformable3DGS, 5.21 dB over SARO-GS, 3.83 dB over Grid4D, 6.78 dB over 4DGS, and 7.53 dB over 4DRotorGS on D-NeRF.Baseline quality declined substantially at higher resolutions; Grid4D’s PSNR dropped 10.24 dB from 1× to 4×.
  • Static-Filter Comparison: The motion-aware filter substantially outperforms SARO-GS plus a static 3D filter, which causes blur and temporal inconsistencies around moving subjects.The reported comparison gives 31.40 PSNR, 0.941 SSIM, and 0.117 LPIPS for the method versus 24.92, 0.840, and 0.270 for the baseline.
  • Integration with Other 4D Representations: The filter transfers to SpeeDe3DGS, improving PSNR by 4.45 dB while using fewer primitives and adding about two minutes of training.The reported per-primitive density estimate adds 17.7 MB of memory.
  • Zoom-Out Evaluation: At sub-1× render resolutions, the filtered SpeeDe3DGS remains nearly scale-consistent while the base representation degrades sharply.The base PSNR falls from 35.0 to 28.1 dB at 1/4×, whereas the filtered result stays around 35.9 dB across scales.

6 Limitations

The main limitation is memory usage from storing a joint density matrix for every Gaussian primitive. This overhead grows with scene size and grid resolution, motivating memory-efficient alternatives.

  • Memory Footprint: Memory scales as O(N × D × T) because each Gaussian stores a discrete density matrix over focal-to-depth ratio and time.The overhead can become significant for large scenes and finer grids.
  • Potential Mitigations: Future reductions include storing only a time-varying sampling-rate curve, predicting rates with a shared network, or clustering primitives with similar motion.The first option reduces storage from O(NDT) to O(NT).

7 Conclusion and Future Work

The paper presents a motion-aware smoothing filter for alias-free 4D rendering and reports improved high-resolution rendering across representations and datasets. It also identifies rapid, complex motion as a future evaluation direction.

  • Conclusion: The proposed filter estimates a motion-dependent sampling interval from a joint distribution and selects the maximum-probability value at each query timestamp.Experiments report better performance than baseline 4D representations under lower-resolution training and higher-resolution evaluation.
  • Future Work: The strategy is flexible across 4D representations, while future work targets dynamic scenes with rapid, complex motion.The stated scope includes both per-scene and feed-forward methods.

Supplementary Material

The method separates static and dynamic Gaussian primitives using image masks generated with SAM and manually provided seed points. This enables separate analysis of their sampling behavior.

  • Dynamic-Region Identification: SAM-based static and dynamic masks classify projected Gaussian primitives according to whether they fall in static or dynamic image regions.The classified primitives are then analyzed separately.

A.2 Temporal Stability Metrics

The paper evaluates temporal stability using flow-based warping errors and perceptual metrics, alongside SSIM, LPIPS, and distributional analysis. It also studies temporal-grid and filter-update settings.

  • Metrics: The paper uses PSNR, SSIM, and LPIPS to measure perceptual similarity alongside temporal-stability metrics.
  • Metrics: Temporal stability is measured with flow warping error between consecutive frames.The warped frame is compared with the subsequent frame after optical-flow alignment.
  • Metrics: Warped LPIPS and warped SSIM quantify perceptual and structural similarity after frame warping.Both metrics compare consecutive frames using the warped-frame construction.
  • Metrics: RAFT computes optical flow for all temporal-stability metrics.
  • Distributional analysis: The evaluation includes the Weibull probability density function with shape parameter k and scale parameter λ.The supplied passage identifies the distribution and its two parameters.

A.4 Ablation Studies

The ablation study finds that temporal smoothing is best configured with a temporal grid size of T = 100 and filter recomputation every 100 training iterations.

  • Temporal grid size: T = 100 gives the best performance across all tested scales.T = 25 under-resolves temporal variation of f/d, whereas T = 150 produces noisy per-bin density estimates.
  • Filter update interval: The filter update interval is also evaluated as a key hyperparameter governing temporal smoothing.The passage introduces update frequency as the second ablated setting.
  • Experimental setup: The ablation uses SARO-GS on D-NeRF with single-scale training and multi-scale testing.

B.1 Per-Scene Zoom-Out Results

Zoom-out evaluation shows that the proposed filter remains stable as render resolution decreases, with gains concentrated around silhouettes and high-frequency edges rather than broad visual reconstruction changes.

  • Per-scene zoom-out results: At reduced render resolutions, the base degrades while our method stays stable, especially on thin-structure scenes such as mutant and trex.Both methods are on par at the training resolution.
  • Quantitative results: Up to +7.8 dB PSNR improvement is reported at 1/4× zoom-out.SSIM changes from 0.951 to 0.985 and LPIPS from 0.029 to 0.014 in the summarized zoom-out setting.
  • Error localization: At 1/4×, 3.5% of pixels account for 90% of the base error, while high-error pixels decrease from 1.35% to 0.03%.The high-error threshold is squared error > 0.05.
  • Error localization: The silhouette boundary band contributes about 70% of total error reduction while covering about 7% of pixels.The far background contributes essentially nothing; interior gains are concentrated on high-frequency texture edges.
  • Interpretation: The filter primarily anti-aliases silhouettes and interior texture edges rather than broadly improving reconstruction quality.This explains why large PSNR gains can remain perceptually subtle.

B.2 Per-Scene Results

Per-scene evaluations on Plenoptic Video, synthetic, and temporal-stability settings show comparable training-scale performance and stronger results across multiple evaluation scales. Qualitative and temporal plots provide additional evidence across scenes.

  • Plenoptic Video: Per-scene tables report single-scale training at 1× and multi-scale evaluation at 1×, 2×, and 4× across six Plenoptic scenes.The reported scenes include cook spinach, coffee martini, cut roasted beef, flame salmon, flame steak, and sear steak.
  • Temporal stability: Temporal stability is evaluated across multiple scales for sear steak, flame steak, and flame salmon scenes.The cook spinach temporal plot shows consistently lower warping error than baseline methods.
  • Qualitative results: Additional qualitative comparisons highlight dynamic and static regions in coffee martini and flame steak scenes, with further synthetic-dataset results.
Loading 2608.21828v1…