Source-linked AI summary
SpectralSplats: Robust Differentiable Tracking via Spectral Moment Supervision
Avigail Cohen Rimon, Amir Mann, Mirela Ben Chen, Or Litany
TL;DR
Model-based 3DGS tracking can fail when localized photometric losses receive no informative gradient under severe misalignment. SpectralSplats uses global Spectral Moments and first-principles Frequency Annealing to recover alignment, and reports robust performance across deformation models and realistic misalignment experiments.
Problem
3DGS tracking remains fragile because spatial photometric objectives depend on overlap, and large displacements can eliminate the basin leading to the correct state.
Method
SpectralSplats shifts supervision to global complex sinusoidal moments and progressively activates frequency bands through Frequency Annealing.
Results
SpectralSplats consistently improves robustness to initial spatial misalignment across MLP and sparse-control-point deformation models, outperforming pixel supervision in the reported GART experiment.
Takeaways & Limitations
The framework serves as a model-agnostic replacement for localized spatial losses, enabling recovery from extreme initialization errors without manual alignment.
Takeaways & Limitations
The current formulation assumes a pre-initialized canonical asset and therefore remains limited to model-based tracking.
Abstract
from arXiv · showhide
3D Gaussian Splatting (3DGS) enables real-time, photorealistic novel view synthesis, making it a highly attractive representation for model-based video tracking. However, leveraging the differentiability of the 3DGS renderer "in the wild" remains notoriously fragile. A fundamental bottleneck lies in the compact, local support of the Gaussian primitives. Standard photometric objectives implicitly rely on spatial overlap; if severe camera misalignment places the rendered object outside the target's local footprint, gradients strictly vanish, leaving the optimizer stranded. We introduce SpectralSplats, a robust tracking framework that resolves this "vanishing gradient" problem by shifting the optimization objective from the spatial to the frequency domain. By supervising the rendered image via a set of global complex sinusoidal features (Spectral Moments), we construct a global basin of attraction, ensuring that a valid, directional gradient toward the target exists across the entire image domain, even when pixel overlap is completely nonexistent. To harness this global basin without introducing periodic local minima associated with high frequencies, we derive a principled Frequency Annealing schedule from first principles, gracefully transitioning the optimizer from global convexity to precise spatial alignment. We demonstrate that SpectralSplats acts as a seamless, drop-in replacement for spatial losses across diverse deformation parameterizations (from MLPs to sparse control points), successfully recovering complex deformations even from severely misaligned initializations where standard appearance-based tracking catastrophically fails.
1 Introduction
Model-based video tracking with differentiable 3DGS is fragile because localized photometric supervision requires spatial overlap. SpectralSplats replaces this dependency with global spectral supervision and frequency annealing to recover alignment from severe misinitialization.
- 3DGS supports real-time, photorealistic rendering and model-based video tracking, but estimating continuous deformations from visual observations remains fragile.
- Photometric tracking depends on local overlap between rendered Gaussian footprints and target structures, so large displacements can produce flat or corrupted gradients.
- Spectral Moment supervision projects images onto global sinusoidal features, turning spatial displacement into phase shifts that provide non-zero directional gradients without pixel overlap.
- Frequency Annealing begins with low frequencies for long-range attraction and progressively activates higher frequencies for precise structural alignment.
- SpectralSplats guides both neural-MLP and sparse-control-point deformation models from extreme initial displacements toward accurate poses without modifying the deformation models.
- The framework combines Spectral Moment Loss, principled Frequency Annealing, and initialization-robust tracking across synthetic and real-world settings.
2 Related Work
Prior dynamic-scene methods use canonical representations, structured deformation models, and frequency-management strategies to improve reconstruction and optimization. These choices do not resolve failures caused by spatially disjoint renderings, which SpectralSplats addresses with complementary global supervision.
- Dynamic 3DGS commonly pairs a static canonical Gaussian set with a time-varying deformation model trained from video or in stages.
- Coordinate-based MLPs, structured grids, and other structural priors promote smoothness, coherence, and efficient deformation modeling in under-constrained tracking.
- These deformation choices still assume photometric gradients remain informative and therefore do not solve failures when rendered objects are spatially disjoint from targets.
- Category-specific articulated priors can perform strongly when assumptions hold but remain brittle to initialization errors.
- Recent frequency-management methods improve reconstruction fidelity, while high-frequency components can still complicate geometric optimization through rugged loss landscapes.
3 Method
SpectralSplats formulates the locality failure of photometric tracking and replaces direct pixel comparisons with global image moments. Frequency Annealing then balances global convergence against high-frequency precision.
- The method formalizes 3DGS tracking from a canonical asset through deformation parameters, rasterization, and alignment of the rendered image with the target.
- Vanishing Gradient Problem: With zero spatial overlap, localized photometric loss can become insensitive to translation, while unrelated scene content can supply corrupted gradients.
- Image Moments and Spectral Duality: Image moments multiply an image by a global field and integrate it, allowing the field gradient to provide directional information even for disjoint objects.
- Image Moments and Spectral Duality: The moment-matching objective compares rendered and target projections, with its gradient combining projection error and the rendered-moment derivative.
- Spectral Duality: A complete static spectral basis is equivalent to spatial L2 by Parseval’s theorem, but high frequencies can introduce phase wrapping and false local minima.
- Frequency Annealing: Frequency Annealing activates coarse frequencies first and progressively expands bandwidth, using global attraction before refining high-frequency alignment.
4 Experiments
Experiments evaluate SpectralSplats under increasing spatial misalignment on SC4D, GART dogs, and fast-motion basketball data. Across quantitative and qualitative comparisons, spectral supervision is more robust than pixel-only supervision in training and novel views, including zero-overlap conditions.
- Experimental settings: The evaluation covers controlled SC4D animations, real-world GART dog videos, and a fast-motion basketball sequence with increasing spatial misalignment.SC4D and GART use shifted initial 3DGS models, while the basketball experiment tests temporal motion and monocular supervision.
- SC4D results: As shift radius increases, pixel-based PSNR rapidly decreases while SpectralSplats remains considerably more stable in both training and novel views.The performance gap widens with misalignment, and robustness persists under multi-view supervision.
- SC4D results: At shift radius 0.5 on SC4D, SpectralSplats consistently improves PSNR, SSIM, and LPIPS across deformation parameterizations, spatial losses, and evaluation views.The comparison includes MLP and direct morph-field parameterizations and also tests replacing the pixel-loss phase with LPIPS.
- GART results: At shift radius 0.6 on GART, mean PSNR improves from 20.15 to 22.05, SSIM from 0.891 to 0.907, and LPIPS from 0.258 to 0.216 over pixel-only supervision.Qualitative results also show better pose recovery and sharper structure, while the pixel baseline exhibits blur and poor alignment.
- Fast-motion results: In the thrown-basketball sequence, pixel supervision fails in a zero-overlap regime, whereas SpectralSplats maintains accurate alignment despite imperfect dataset masks.The experiment increases difficulty through temporal subsampling and optimizes only foreground Gaussians before recombining foreground and background at rendering time.
5 Conclusion
The conclusion presents SpectralSplats as a model-agnostic solution to vanishing gradients in dynamic 3DGS tracking. Its current scope is limited by the requirement for a pre-initialized canonical asset.
- Conclusion: SpectralSplats replaces localized spatial losses with Spectral Moment supervision and frequency annealing to recover from extreme initialization misalignments without manual alignment or category-specific priors.The framework is described as model-agnostic and targeted at dynamic 3DGS tracking.
- Scope and limitations: The current formulation assumes a pre-initialized canonical asset, restricting SpectralSplats to model-based tracking.The authors identify extension to jointly optimizing canonical geometry and motion from uncalibrated video as a future direction.
A Derivation of the Phase-Wrapping Condition
The derivation shows that spectral supervision has a unique correct attraction basin when phase wrapping is avoided, motivating a linear frequency-annealing schedule and subsequent spatial refinement.
- Spectral-spatial relationship: For a displaced image, Fourier features convert spatial translation d into a frequency-dependent phase shift ω^T d.The spectral moment of the rendered image equals the target moment multiplied by exp(-jω^T d).
- Phase-wrapping condition: The condition |ω^T d| < π is necessary and sufficient for a unique correct basin at optimization step t.Beyond π, periodic false minima can trap optimization at incorrect alignments.
- Convergence: Within the phase-wrapping regime, the loss is approximately quadratic and gradient descent yields exponential displacement decay d_t = d_0γ^t.For a sufficiently small learning rate, the contraction factor γ lies in (0, 1).
- Frequency annealing: Because safe frequency magnitude grows exponentially as displacement decays, logarithmic frequency grids imply a linear growth of the active frequency index.This provides the first-principles justification for the linear annealing schedule α(t).
- Demonstration: The 2D demo shows Pixel MSE remaining near initialization under large translation and rotation, while spectral supervision produces coherent motion and converges to the target.The annealing visualization shows low frequencies active first, followed by gradual higher-frequency refinement.
- Optimization pipeline: The training objective uses spectrally annealed supervision before transitioning to spatial-domain supervision for high-frequency refinement.The spectral phase minimizes discrepancy between rendered and target signatures over the active frequency band.
C.2 Spatial Loss Ablation
The spatial-loss ablation finds that the full spatial-phase objective performs best, while SpectralSplats remains effective when the spatial loss is replaced by LPIPS.
- Ablation results: The full spatial-phase loss achieves the best PSNR in the GART Shiba example under the 0.6 shift setting.Its improvement is modest but consistent across the ablated loss variants.
- Ablation results: Replacing the pixel-loss phase with LPIPS still demonstrates the effectiveness of SpectralSplats regardless of the spatial loss choice.The reported evaluation includes PSNR for each configuration.
C.3 Annealing Schedule Ablation
The annealing schedule remains stable across a broad range of frequency-band counts, with K = 8 producing the best PSNR at shift radius 0.6. Spectral Moment supervision also preserves or improves performance when initialization is aligned.
- Annealing Schedule Ablation: 21.79, 22.02, 22.06, 21.28, and 21.35 PSNR are obtained for K = 4, 6, 8, 10, and 12, respectively, at shift radius 0.6.The best result is achieved at K = 8.
- Annealing Schedule Ablation: The annealing method is relatively stable across a broad range of frequency-band schedules.
- Aligned Initialization: Spectral Moment supervision consistently matches or outperforms pixel-only supervision across PSNR, SSIM, and LPIPS on training and novel views.
- Aligned Initialization: In aligned SC4D settings, the method does not degrade performance and improves results in most cases.This indicates robustness when pixel-only supervision does not catastrophically fail.
- Qualitative Comparison: With shift = 0.0, both methods recover the target pose, while the proposed method produces sharper details and cleaner structure.
D.3 Multi-View Training Analysis
Multi-view experiments compare one, two, and four supervision views on SC4D under increasing initial shifts. Pixel-only supervision degrades rapidly with misalignment, whereas the proposed method remains more stable and performs better across view configurations.
- View Configurations: One, two, and four supervision views use angles 0°, 0°/180°, and 0°/90°/180°/270°, respectively.Performance is evaluated on front and side views.
- Performance under Misalignment: Across all view configurations, pixel-only supervision degrades rapidly as the initial shift radius increases.
- Performance under Misalignment: The proposed method remains significantly more stable and achieves higher performance across the evaluated view configurations, including zero shift.
- Evaluation Protocol: PSNR, LPIPS, and SSIM are measured on front and side views, with the side view novel for the single- and two-view settings.
- Supervision Setup: The experiments use per-frame masks to isolate the asset before compositing rendered outputs over a uniform background.
- Initialization Context: The GART initialization differs from supervision in pose, outline, color, viewpoint, zoom, and lighting conditions.
E.2 Additional Results
Additional GART results show that spectral supervision is more stable than pixel supervision as spatial shifts increase. The experiments also examine the performance gap, LPIPS limitations, and the formulation of distance-transform supervision.
- GART Robustness: Across nearly all dogs, spectral optimization remains significantly more stable than pixel-based optimization as shift increases.
- Performance Gap: As shift radius increases, the mean improvement of the spectral method over MLP+Pixel steadily grows for PSNR, SSIM, and LPIPS.Positive values denote gains using MLP+Ours over MLP+Pixel, with metric-specific difference definitions.
- Performance Gap: Figure 14 plots per-dog PSNR, SSIM, and LPIPS curves alongside mean gains and per-dog minimum and maximum values.
- LPIPS Limitation: LPIPS can prioritize lighting-gap compensation over geometric consistency when reconstructed 3DGS colors differ from the video.This produces weaker geometric constraints and blurrier results.
- Distance Transform Loss: The distance-transform loss combines foreground and background distance maps with the rendered soft mask and its complement.
- Distance Transform Loss: Its first term pushes Gaussians away from background regions, while its second encourages dense foreground coverage.
F.2 Qualitative Results
Qualitative comparisons against global-loss baselines show that object merging can occur during optimization. SpectralSplats instead converges correctly while preserving object separation.
- Global-Loss Baselines: PoG and DT exhibit object merging during optimization on the teaser scene.Under DT, the center strawberry is absorbed into the banana model.
- Proposed Method: SpectralSplats converges correctly and preserves separation among the scene’s objects.
G Implementation Details
The experiments use a frequency-annealing schedule with fixed training settings across configurations, while adding negligible overhead over pixel-based loss.
- All experiments use 800 control points, 10K training iterations, λimage fixed at 5000, and arap start iter set to 1000.These settings are applied across the reported experiments.
- Table 6 describes the training hyperparameters and their roles.
- Table 7 reports hyperparameter settings for SC4D with MLP, SC4D with Direct Morph Field, and GART.
- Training a single sequence takes approximately 8–15 minutes on one NVIDIA L40 GPU, with overhead remaining within 0–6% across experiments.On SC4D, the reported comparison is 437s versus 443s for pixel-based loss.