Instant4D: 4D Gaussian Splatting in Minutes

1 Introduction

• Dynamic reconstruction from casually captured, uncalibrated video remains challenging because monocular moving objects, occlusion, deformation, and irregular camera paths complicate coherent modeling.Prior approaches often require hours of optimization to recover scene geometry and accurate motion.Source: §1 · pp. 1–2
• Deep visual SLAM estimates camera trajectories, refines monocular depth into video-consistent depth, and supplies point-cloud geometry for 4DGS optimization.The method uses a native 4D Gaussian primitive to capture motion without rigidly segmenting static and dynamic scene parts, addressing overfitting and premature disappearance in poorly observed regions with initialization and motion-aware covariance.Source: §1 · p. 2
• INSTANT4D reports average reconstruction times of 2 minutes on NVIDIA and 7.2 minutes on Dycheck, 30× speed-up, and 90% memory reduction.For a typical 5-second, 30 FPS video, optimization completes within 8 minutes.Source: §1 · p. 2
• Grid pruning reduces the number of Gaussians by 92%, preserves occlusion structures, and enables scalability to long video sequences.Source: §1 · p. 2
• 29% better performance than current state-of-the-art methods on the Dycheck dataset is reported for the simplified, isotropic, motion-aware 4DGS formulation.The paper also reports 30× reconstruction acceleration compared to recent works.Source: §1 · p. 2

2.1 Dynamic Novel View Synthesis (NVS)

• RoDynRF typically takes over 2 days to reconstruct a casual video, while NeRF-based dynamic methods also face slow rendering and many iteration demands.Source: §2.1 · p. 2
• Gaussian-based NVS includes deformation-based, trajectory-based, and 4D-Gaussian methods, with trade-offs in model complexity, preprocessing, and temporal robustness.Deformation methods add learned continuous deformation; trajectory methods require substantial preprocessing; 4D Gaussians can overfit or vanish prematurely in monocular settings when observations are brief or temporally underconstrained.Source: §2.1 · pp. 2–3
• The INSTANT4D pipeline uses Deep Visual SLAM and Unidepth for camera parameters and metric depth, refines depth for video consistency, then voxel-filters back-projected points before 4DGS.The caption states that scene reconstruction from 4D Gaussian initialization takes 2 minutes.Source: Figure 2 · p. 3

2.2 Visual SLAM and SfM

• Classical SfM and SLAM recover poses and sparse geometry through bundle adjustment, while learned systems such as DROID-SLAM, MegaSAM, DUSt3R, MonST3R, and CUT3R extend pose and depth recovery.Source: §2.2 · p. 3

3.1 Preliminary

• MegaSAM estimates camera poses and intrinsics, then refines them by minimizing weighted reprojection residuals; 4DGS uses temporal Fourier bases with spherical harmonics for view- and time-dependent appearance.The method also describes Levenberg–Marquardt updates using reprojection residuals and an adaptively predicted damping factor.Source: §3.1 · p. 4

3.2 Geometric Recovery

• MegaSAM poses and temporally consistent depth are used to back-project image pixels into world coordinates, yielding a dense colored point cloud.The voxel size is adapted to scene scale, particularly for outdoor scenes with unbounded regions.Source: §3.2 · p. 4
• SLAM motion probabilities yield thresholded masks, with endpoint pseudo-frames for sparse sampling; voxel pruning replaces occupied cells with centroids and discards weakly supported voxels.The method averages attributes such as color, timestamp, time scale, and motion probability within each voxel.Source: §3.2 · p. 5
• 92% less memory (10.7 GB to 0.83 GB), 4× faster training (181 s to 42 s), and rendering rises from 154 FPS to 981 FPS after grid pruning.Reported on the NVIDIA Dynamic Scene benchmark.Source: §3.2 · p. 5

3.3 Optimization

• A lightweight 4D Gaussian uses RGB appearance rather than high-order harmonics, cutting per-Gaussian parameters by over 60% and empirically lessening monocular overfitting while encoding continuous motion without explicit trajectories.The primitive is conditioned to a 3D Gaussian at each rendering timestamp.Source: §3.3 · p. 5
• The isotropic model fixes orientation and uses spatial and temporal scales; larger temporal scales for static regions help preserve them, while time-distant Gaussians are culled by opacity.The isotropic parameterization is described as improving numerical stability, reducing memory usage, and acting as an implicit regularizer.Source: §3.3 · pp. 5–6

4.1 Training and Inference Detail

• 96–131 seconds: Lite training takes 96 seconds on a 235-frame sequence and 131 seconds on a 379-frame sequence, with peak memory of 988 MB and 1,147 MB, respectively.For Full on the 379-frame sequence, geometric recovery takes about 5 minutes; inference exceeds 400 Hz and voxel pruning takes under 5 seconds per scene.Source: §4.1 · p. 7

4.2 Evaluation on NVIDIA & Dycheck Benchmarks

• INSTANT4D reports higher rendering quality with faster training than InstantSplat-style and standard 4DGS baselines, although some prior methods reach higher PSNR on the 12-frame NVIDIA dataset.The authors attribute the NVIDIA methods’ higher PSNR to additional regularization for limited data; their own method offers a quality–speed trade-off.Source: §4.2 · p. 8
• 24.52 dB: INSTANT4D Full exceeds RoDyGS by 7.15 dB and Deform3D by 1.89 dB on DyCheck while training in 7.2 minutes.The Lite variant reaches 23.02 dB in 0.03 h with a 1.1 GB footprint; the visual comparison reports better static-background and dynamic-object preservation than baseline 4DGS.Source: Figure 4 · p. 7, §4.2 · p. 8

4.3 Evaluation on in-the-wild video

• 2 minutes: Each of the 82-frame Bear and 68-frame Breakdance DAVIS sequences takes 2 minutes for SLAM calibration and 2 minutes for 4D reconstruction.The qualitative evaluation reports crisp object boundaries and temporally coherent appearance in renderings from novel viewpoints and timestamps.Source: Figure 5 · p. 8, §4.3 · p. 9
• Low texture limits pose accuracy: in Kite-surf, the ocean-dominated view leads to inaccurate visual-SLAM poses, and the surfer occasionally disappears in rendered output.The authors identify texture-robust pose initialization as future work.Source: §4.3 · p. 9

4.4 Ablation and Analysis

• Grid pruning reduces memory use and runtime while maintaining rendering quality, and the simple RGB representation offers a beneficial performance–speed trade-off.Table 2 evaluates spherical harmonics, grid filtering, and Gaussian densification across rendering quality, runtime, memory, and frame rate.Source: Table 2 · p. 7, §4.4 · p. 9
• 1.0 dB: Replacing per-Gaussian RGB coefficients with the time-varying 4D sphericylindrical harmonic basis increases parameters but decreases PSNR on DyCheck at 2× resolution.The authors conclude the simpler RGB representation is better suited to monocular, unconstrained video.Source: §4.4 · p. 9
• 1.25 dB: Isotropic Gaussians improve PSNR, while uniform temporal scaling reduces PSNR by 3.4 dB and produces motion blur.Uniform temporal scaling fails to distinguish static and dynamic regions; the reported ablation supports the role of motion-aware temporal scaling in visual fidelity and temporal stability.Source: §4.4 · p. 9

5 Discussion & Conclusion

• Memory grows linearly with sequence length because visual SLAM retains depth maps for every frame, constraining use on multi-minute scenes and continuous streams.The discussion proposes hierarchical memory management and online depth-map compression as future directions.Source: §5 · p. 9
• Depth estimation is less stable on highly reflective or transparent surfaces, which remain a challenging scene type.Source: §5 · p. 9
• The conclusion presents INSTANT4D as a fast 4D reconstruction system for casual, uncalibrated monocular video, using grid pruning, motion-aware Gaussian splatting, and an efficient representation.The stated future directions include arbitrary-length video, hierarchical SLAM representations, and efficient memory management.Source: §5 · pp. 9–10

A Additional Evaluation

• On the Bear and Rhino scenes, Instant4D preserves fine detail and temporal consistency, while RoDyGS exhibits blurring, artifacts, or lost structure.The comparisons highlight Bear fur and moving limbs, and Rhino skin texture and shading.Source: A Additional Evaluation · p. 14, Figure A1 · p. 14, Figure A2 · p. 14
• 6.22 dB over RoDynRF on Dycheck and 1.43 dB (PSNR) over InstantSplat on NVIDIA are the reported gains.The model reconstructs a scene in 0.03 h, compared with RoDynRF's more-than-24-hour process.Source: A Additional Evaluation · p. 15
• Instant4D matches or exceeds state-of-the-art accuracy on the NVIDIA Dynamic Scene and Dycheck iPhone datasets without auxiliary rigidity, point-track, or depth regularization terms.Source: A Additional Evaluation · p. 15
• Instant4D is 20× faster than RoDyGS with a 7.15 dB PSNR gain; against 4DGS, grid pruning yields 8× acceleration on NVIDIA while improving visual quality.Source: A Additional Evaluation · p. 15

B.1 Runtime Analysis

• 6 minutes are required for end-to-end training on an 82-frame, 854 × 480 video using a single A6000 GPU.Source: §B.1 · p. 15
• 247 FPS is the reported rendering speed after training.Source: §B.1 · p. 15
• Grid pruning takes less than 3 seconds, and the optimization process uses less than 4 GB memory.Source: §B.1 · p. 15

B.2 Additional Discussion on Motion Awareness

• The motion mask is formed by interpolating per-frame motion probabilities to image resolution and applying Otsu thresholding.The resulting binary segmentation is used to assign temporal scaling.Source: §B.2 · p. 15, Figure B3 · p. 16
• Adding a pseudo-frame removes camera-motion noise and reduces the dynamic-region area and computation overhead, though interpolation leaves the mask eroded by a few pixels.Source: §B.2 · p. 15
• The motion-mask method is fully automated without human annotation, unlike approaches using interactive segmentation, extra tracking, or flow-error maps to guide segmentation.Source: §B.2 · p. 16

C Visualization on the AIGC Video

• The reconstruction pipeline is applied to Sora-generated video for AI-generated content creation.Source: C Visualization on the AIGC Video · p. 16
• The example is a five-second video generated with prompts including Einstein riding a bear and a panda playing guitar.Source: C Visualization on the AIGC Video · p. 16
• Additional results are available on the project website.Source: C Visualization on the AIGC Video · p. 16

• The checklist answers affirm that the main claims are accurately reflected, the work follows the ethics code, and the research involves neither human subjects nor core-method LLM use.Source: NeurIPS Paper Checklist · pp. 17, 20, 22, 23
• The authors affirm that the paper discusses its limitations in Section 5.Source: NeurIPS Paper Checklist · p. 17
• The authors say they listed the details needed to reproduce experiments and will provide code if accepted.Source: NeurIPS Paper Checklist · p. 18
• The authors say datasets are available and that instructions for reproducing the main experiments will be provided.Source: NeurIPS Paper Checklist · p. 19
• The authors state that experimental implementation details appear in Section 4.1 and that runtime and memory information is reported there.Source: NeurIPS Paper Checklist · pp. 19–20