Source-linked AI summary
4DGS360: 360° Gaussian Reconstruction of Dynamic Objects from a Single Video
Jae Won Jang, Yeonjin Chang, Wonsik Shin, Juhwan Cho, Nojun Kwak
TL;DR
4D monocular reconstruction is ill-posed because each frame provides only one viewpoint, while existing datasets lack sufficient train–test view disparity for evaluating truly novel viewpoints. 4DGS360 uses 3D-native, anchor-guided initialization and introduces iPhone360; it achieves state-of-the-art performance under ordinary and extreme novel-view conditions.
Problem
4D reconstruction from monocular video is ill-posed because each frame offers only one viewpoint and lacks multi-view stereo cues.
Method
4DGS360 combines 3D-native initialization with AnchorTAP3D, an anchor-guided model integrating 2D and 3D tracking strengths for geometry-aware reconstruction.
Results
4DGS360 achieves state-of-the-art performance qualitatively and quantitatively under ordinary and extreme novel-view synthesis conditions.
Takeaways & Limitations
iPhone360 enables more realistic evaluation of dynamic-object reconstruction under extreme viewpoint disparity and diverse real-world motion.
Takeaways & Limitations
Performance still depends on pretrained models, and extreme-viewpoint background regions invisible in the input video cannot be synthesized.
Abstract
from arXiv · showhide
We introduce 4DGS360, a diffusion-free framework for 360$^{\circ}$ dynamic object reconstruction from casual monocular video. Existing methods often fail to reconstruct consistent 360$^{\circ}$ geometry, as their heavy reliance on 2D-native priors causes initial points to overfit to visible surface in each training view. 4DGS360 addresses this challenge through a advanced 3D-native initialization that mitigates the geometric ambiguity of occluded regions. Our proposed 3D tracker, AnchorTAP3D, produces reinforced 3D point trajectories by leveraging confident 2D track points as anchors, suppressing drift and providing reliable initialization that preserves geometry in occluded regions. This initialization, combined with optimization, yields coherent 360$^{\circ}$ 4D reconstructions. We further present iPhone360, a new benchmark where test cameras are placed up to 135$^{\circ}$ apart from training views, enabling 360$^{\circ}$ evaluation that existing datasets cannot provide. Experiments show that 4DGS360 achieves state-of-the-art performance on the iPhone360, iPhone, and DAVIS datasets, both qualitatively and quantitatively.
1 Introduction
4DGS360 targets the ill-posed problem of reconstructing complete 360° dynamic objects from casual monocular video. It combines 3D-native occlusion-aware initialization with the iPhone360 benchmark for extreme novel-view evaluation.
- Monocular 4D reconstruction is highly ill-posed because each frame provides only one viewpoint and no multi-view stereo cues.
- Existing methods fail on extreme novel viewpoints because 2D-native initialization can overfit visible surfaces and leave occluded geometry ambiguous.
- 4DGS360 introduces AnchorTAP3D, a 3D-native tracker that uses confident 2D track points as anchors to improve reliability and resolve occluded-region depth ambiguity.
- The resulting initialization enables optimization strategies such as ARAP regularization to operate on geometry in occluded regions.
- 4DGS360 achieves state-of-the-art performance qualitatively and quantitatively under ordinary and extreme novel-view synthesis conditions.
2 Related Work
Related work spans explicit and implicit dynamic scene representations, monocular tracking-based reconstruction, diffusion-based unseen-view synthesis, and benchmarks that largely lack extreme viewpoint disparity.
- Dynamic novel-view methods represent temporal deformation through implicit motion fields, Gaussian trajectories, learned bases, or hierarchical motion structures.
- Monocular reconstruction remains ill-posed; 2D cues recover image-plane motion but often fail on occluded geometry, while diffusion methods synthesize unseen views at substantial computational cost.
- 4DGS360 introduces occluded geometry-aware initialization to support consistent reconstruction beyond observed views and provide a stronger baseline for diffusion-based approaches.
- Existing dynamic-scene benchmarks mainly evaluate temporal interpolation or short-range novel views, with test cameras close to training cameras or unrealistic capture setups.
- 2D trackers use global matching, refinement, iterative visibility inference, or self-supervision, whereas 3D trackers infer trajectories directly in XYZ space using spatio-temporal geometric features.
3 Method
4DGS360 represents dynamic objects with canonical-space 3D Gaussians deformed hierarchically over time, then projects and renders them differentiably. Its AnchorTAP3D initialization uses confident 2D tracks as anchors to stabilize 3D trajectories and preserve occluded geometry for 360° reconstruction.
- Dynamic Gaussian representation: Dynamic objects are represented by canonical-space 3D Gaussians whose motion is modeled through hierarchical transformations over time.Each Gaussian is deformed using node motions interpolated from nearby leaf nodes and shared motion bases.
- Dynamic Gaussian representation: The hierarchical motion model combines nearby leaf-node transformations to capture both global motion and local deformation.Higher-level nodes model smooth global patterns, while deeper nodes refine fine-grained changes.
- AnchorTAP3D initialization: AnchorTAP3D improves 360° reconstruction by preserving occluded geometry that naive 2D or 3D tracking may fail to recover.The reported comparison finds more stable tracking and better occluded-region geometry than naive 3D tracking baselines.
- Differentiable rendering: At each time step, transformed Gaussians are projected into image space and rendered by depth-sorted alpha blending.Projection uses camera parameters, while each projected Gaussian contributes according to its opacity and view-dependent color.
- AnchorTAP3D initialization: Multiple anchors condition tracking within temporal windows, suppressing drift and producing stable initialization for occluded-region reconstruction.The method uses a fixed-length sliding window and reports better long-term tracking and occluded geometry than naive 2D or 3D initialization.
4 Experiments
Experiments evaluate 4DGS360 on iPhone360, iPhone, and DAVIS, emphasizing extreme-view reconstruction, perceptual quality, and occluded geometry. Across these settings, the method produces coherent novel-view results and improves over prior approaches.
- Datasets and evaluation: iPhone360 places synchronized test cameras 70°–135° from a single handheld training view, enabling realistic extreme-novel-view evaluation.The dataset contains six dynamic scenes captured with multiple iPhone devices.
- Datasets and evaluation: Bounding-box evaluation restricts comparisons to regions around the ground-truth dynamic object, reducing the influence of empty extreme-view backgrounds.The same region is applied to ground truth and foreground-only renders before computing metrics.
- iPhone360: 4DGS360 achieves consistently higher perceptual scores than baselines on iPhone360 and maintains coherent geometry from extreme viewpoints.The paper also demonstrates 360° object reconstruction from a single video.
- iPhone: Across five iPhone scenes and CLIP-I, CLIP-T, and LPIPS, 4DGS360 consistently improves over previous models and outperforms them on most scenes.Comparisons include NeRF-based and 3DGS-based approaches.
- Cross-dataset qualitative results: Qualitative results show higher-fidelity unseen-region reconstruction on iPhone and more reliable preservation of occluded geometry on DAVIS.The DAVIS scenes represent fast-moving objects in in-the-wild conditions without ground-truth camera parameters or depth maps.
- Ablations: AnchorTAP3D preserves occluded-region geometry better than unprojected 2D tracks and naive 3D tracking in walk-around and jelly ablations.Naive 3D tracking partially recovers geometry in walk-around but completely fails on the jelly occluded area, while the proposed method preserves overall shape in both scenes.
5 Limitations and Conclusion
The paper concludes that AnchorTAP3D enables consistent extreme novel-view synthesis from monocular video, while iPhone360 supports realistic evaluation under large viewpoint disparities. Limitations remain in pretrained-model dependence, invisible backgrounds, and pixel-level accuracy.
- Limitations: Performance still depends on pretrained-model capability, cannot synthesize backgrounds invisible in the input video, and struggles with PSNR and SSIM in large extrapolation settings.The authors leave improved pixel-level reconstruction accuracy and invisible-background synthesis for future work.
- Conclusion: AnchorTAP3D provides geometry-aware initialization that supports consistent 360° reconstruction of dynamic objects from monocular videos.The conclusion contrasts this with prior 2D-feature methods that suffer from 3D ambiguity.
- Conclusion: iPhone360 evaluates generalization under extreme viewpoint changes and diverse dynamic-object motion patterns in realistic conditions.The benchmark addresses limitations of existing datasets for 360° reconstruction evaluation.
Supplementary Material
The supplementary material expands implementation, evaluation, dataset, and ablation details for 4DGS360. It specifies AnchorTAP3D confidence handling, tracking windows, Gaussian initialization, optimization, and computational cost.
- Organization: The supplement includes additional details, results, ablations, evaluation information, dataset material, and an iPhone360 dataset overview.Sections cover AnchorTAP3D, initialization, optimization, evaluation, datasets, and ablation studies.
- AnchorTAP3D details: AnchorTAP3D combines BootsTAP 2D tracking with TAPIP3D 3D tracking and converts visibility and uncertainty logits into probabilities.Visibility and certainty are multiplied to form each track point’s confidence.
- AnchorTAP3D details: The tracker processes fixed windows of 16 frames for six iterations, replacing inferred points with high-confidence unprojected 2D tracks as anchors.Dynamic object masks exclude inferred points outside the object region.
- Initialization: Each selected 3D track initializes one Gaussian and stores its position across frames as that Gaussian’s temporal trajectory.Initial Gaussian color is taken from the point at its query time.
- Optimization: Optimization uses Adam with photometric, mask, depth, trajectory, and ARAP-related objectives, while canonical Gaussians and node-based motion are jointly optimized.The supplement also reports learning rates for Gaussian, motion-basis, and node parameters.
- Experimental settings: The iPhone360 experiments use fixed hyperparameters rather than per-scene tuning because no universally optimal values exist across scene characteristics.This setting reflects variation in scene complexity and tracking reliability.
- Computational cost: Reconstructing 140 images takes approximately 3 hours on an NVIDIA RTX A6000 D6 48GB GPU without substantial additional optimization overhead.The pipeline is described as computationally comparable to existing approaches.
B.1 Evaluation details
Extreme-view evaluation uses foreground-focused bounding boxes because unseen background regions create ambiguous comparisons. The authors also compare perceptual and pixel-wise metrics, noting that pixel-wise scores may not reflect visual quality.
- Evaluation protocol: Bounding boxes are computed from processed ground-truth masks and applied to ground truth and foreground renderings for reliable object-focused comparison.Direct full-image comparison is unreliable because of background ambiguity, while masked comparisons can contain excessive white regions.
- Metric interpretation: Per-scene CLIP-I scores better reflect substantial perceptual differences between HiMoR and 4DGS360 than pixel-wise metrics in extreme novel views.The reported image pairs and per-scene scores illustrate the correspondence between perceptual scores and visual results.
B.2 iPhone360 dataset
Qualitative results on iPhone360 show that 4DGS360 reconstructs coherent 360° geometry, while baselines exhibit deformations and motion artifacts in unseen regions.
- Qualitative comparison: 4DGS360 produces proper geometry across 360° bullet-time reconstructions in the jelly, jacket, and goat scenes.The supplementary results report reconstruction at every dynamic time step when the object appears in the video.
- Qualitative comparison: Baselines overfit the training video and exhibit abnormal geometric deformations in the jelly, jacket, and goat scenes.
- Qualitative comparison: Figure 11 presents qualitative comparisons with baseline models on the Block and Teddy scenes of the iPhone dataset.
- Qualitative comparison: In the walk-around scene, baseline reconstructions show visual artifacts, abrupt velocity changes, and disconnected lower-body motion.The artifacts arise when Gaussians pass from regions visible to training cameras through occluded areas.
B.3 iPhone dataset
The iPhone experiments evaluate reconstruction from moving-camera views and compare methods on pseudo test views and the Block and Teddy scenes.
- Evaluation setup: The Haru-sit scene uses a pseudo test view because the iPhone dataset lacks novel-view ground truth.The scene captures different object parts through a moving camera, enabling qualitative evaluation of regions far from the training view.
- Additional analysis: The threshold-ablation figure relates 2D tracking confidence distributions to final 4DGS360 renderings under varying τ values.
C.1 AnchorTAP3D threshold ablation
The threshold ablation examines how AnchorTAP3D’s confidence threshold interacts with scene-specific 2D tracking confidence, with quantitative results reported in an ablation table.
- Threshold setting: The general experiments use a fixed threshold of τ = 0.5, while Fig. 12 evaluates alternative threshold choices.
- Confidence distribution: The goat scene has mean 2D tracking confidence 0.357, compared with 0.593 for the jelly scene.These means indicate that the tracker finds the goat scene more challenging than the jelly scene.
- Quantitative evaluation: Table 5 reports quantitative results for the AnchorTAP3D ablation.
- Quantitative evaluation: Table 6 reports CLIP-I scores for the DIFIX3D+ ablation.
C.2 In-the-wild depth estimation ablation
The depth ablation tests reconstruction using MoGe-only depth instead of LiDAR-corrected estimates and evaluates the corresponding iPhone360 results.
- Depth correction: The standard iPhone360 results use LiDAR sensor values to correct estimated depth.
- Depth correction: MoGe-only depth preserves object reconstruction quality but produces arbitrary scale and lacks spatial alignment with ground truth.
- Quantitative evaluation: Table 6 reports quantitative CLIP-I results for DIFIX3D+ applied to HiMoR and 4DGS360 on the Jacket and Jelly scenes.
D iPhone360 Dataset Overview
iPhone360 comprises six real-world scenes captured with handheld moving cameras and designed to evaluate monocular dynamic reconstruction from extreme novel viewpoints.
- iPhone360 includes goat, jelly, walk-around, block2, jacket, and pull-up scenes with varied object and human motions.The scenes cover rigid-object movement, deformable objects, and human motion.
- Goat, jelly, and walk-around use one training camera and two test cameras, whereas block2, jacket, and pull-up use one training camera and one test camera.
- Training and test cameras differ by over 90° in four scenes, while block2 and pull-up have varying angular differences due to active training-camera motion.
- The handheld moving-camera captures reflect real-world capture scenarios and support extreme novel-view evaluation.The dataset is designed specifically for monocular dynamic reconstruction under extreme novel-view synthesis.