Source-linked AI summary
ATGS: Anchored Temporal Gaussian Splatting for Long Volumetric Video Representation
Jiahao Wu, Jie Liang, Die Hu, Jiayu Yang, Kaiqiang Xiong, Xiang Li, Xiaoyun Zheng, Chao Wang, Ronggang Wang
TL;DR
Volumetric-video reconstruction remains challenging for long sequences and complex motions because direct long-term Gaussian tracking is unstable. ATGS uses time-conditioned anchors, temporal windowing, and hierarchical features to localize and stabilize Gaussian generation, and reports state-of-the-art performance across long multi-view sequences. The method is nevertheless offline, depends on COLMAP initialization, and can show mild jitter where viewpoints are extremely sparse.
Problem
Existing volumetric-video methods struggle to jointly reconstruct long sequences and complex motions, producing temporal instability and artifacts.
Method
ATGS uses time-conditioned anchors, temporal windowing, and global, local-spatial, and local-temporal features to constrain temporally aware Gaussian generation.
Results
ATGS consistently outperforms existing approaches and achieves state-of-the-art performance on long-sequence volumetric videos with complex motions.
Takeaways & Limitations
The anchor-based representation stably reconstructs volumetric videos spanning thousands of frames while supporting large-scale motion.
Takeaways & Limitations
ATGS uses offline reconstruction, depends on COLMAP camera poses and sparse point clouds, and can exhibit mild jitter in extremely sparse-viewpoint regions.
Abstract
from arXiv · showhide
Volumetric video enables immersive free viewpoint rendering of dynamic real world scenes, yet existing methods struggle with long sequences and complex motions, often leading to temporal instability and visual artifacts. To address these challenges, we propose \ourname, a Gaussian splatting based framework for volumetric video reconstruction. Our key insight is that explicitly tracking long term complex motion with individual Gaussian primitives is inherently unstable. Instead, we organize Gaussians around time conditioned anchors that localize their spatial and temporal support, thereby reducing long range motion complexity. We further introduce a temporal windowing strategy to activate only anchors relevant to the queried time, which improves scalability and temporal coherence. In addition, to ensure spatial and temporal stability, we design a compact set of multi level anchor features that encode global features, local spatial features, and local temporal features, jointly constraining Gaussian generation. Extensive experiments demonstrate that \ourname \ consistently outperforms prior methods on long sequence volumetric videos with complex motions. Project page: https://github.com/WuJH2001/ATGS.
1 Introduction
ATGS addresses the open problem of reconstructing long volumetric-video sequences with complex motion by using time-conditioned anchors, temporal windowing, and hierarchical features to stabilize Gaussian generation. It is evaluated on diverse long-sequence datasets and reported to outperform existing approaches.
- Problem: Volumetric video reconstruction must jointly handle minute-level sequences and complex motions involving large inter-frame variations.Examples include fast multi-person movements from diverse viewpoints and dynamic lighting changes.
- Problem: Existing methods tend to target either long sequences or complex dynamics, while splitting long videos into short clips causes temporal inconsistencies and flickering artifacts.LongVolCap degrades under complex motion; FreeTimeGS and LocalDyGS use one- to two-second windows.
- Method: ATGS organizes Gaussian primitives around time-conditioned anchors to decompose long-term dynamics into temporally localized modeling tasks.This avoids directly tracking complex motion over long sequences, which can cause unstable optimization and accumulated errors.
- Method: A hierarchical anchor feature formulation combines global spatial, local spatial, and temporal features to progressively constrain Gaussian generation.The global feature supplies coarse context, while the hierarchy addresses local spatial consistency and temporal variation.
- Experiments: ATGS is evaluated on diverse long-sequence multi-view datasets, including N3DV, VRU, MeetRoom, and SelfCap.The evaluation includes several thousand frames on SelfCap.
- Results: Extensive experiments demonstrate consistent improvements over existing approaches for volumetric-video reconstruction with long temporal durations and complex motions.The contribution statement reports higher-quality reconstruction through improved spatial stability and temporal consistency.
2 Related Work
Prior volumetric-video methods use geometric priors, generative models, NeRF-based representations, or Gaussian splatting, but practical reconstruction remains difficult for complex structures, occlusions, long sequences, and complex motion.
- Volumetric video: Volumetric video represents dynamic scenes with depth and color as viewable three-dimensional models, enabling full 6-DoF free-viewpoint rendering.Multi-view camera setups overcome the planar limitations of conventional 2D video.
- Volumetric video: Early reconstruction methods rely on shape silhouettes, deformable templates, depth sensors, or generative models.These approaches can struggle with complex structures and severe occlusions or require expensive capture systems.
- Neural rendering: Dynamic NeRF methods model time through deformation fields, canonical-space warping, or structured planes for dynamic view synthesis.These approaches seek a balance between efficiency and expressiveness in temporal representations.
- Gaussian splatting: 3D Gaussian Splatting efficiently reconstructs high-fidelity scenes from posed images and has inspired extensions for dynamic settings.Dynamic directions include frame-wise training, deformation-based methods, and other temporal formulations.
- 4D Gaussian methods: Recent 4D Gaussian methods separately emphasize long-sequence reconstruction or complex-motion modeling, leaving their joint treatment challenging.LongVolCap has limited highly complex-motion capability, while FreeTimeGS and LocalDyGS target complex dynamics.
3 Method
ATGS represents long, complex volumetric-video dynamics with time-conditioned anchors, temporal windowing, and hierarchical spatial-temporal features that constrain Gaussian generation and stabilize rendering.
- 3.2 Time-conditioned Anchor Initialization: ATGS represents dynamic scenes with sparse anchors initialized from periodically sampled keyframes, associating Gaussian content with spatial and temporal locations.Each time-conditioned anchor stores position, spatial scale, a learnable anchor feature, and a keyframe-derived temporal index.
- 3.3 Temporal Windowing Strategy: Temporal windowing activates anchors near the queried time, making the inference anchor set evolve smoothly and reducing temporal jitter in long-sequence rendering.The default temporal window size is W = 7.
- 3.4 Spatial-Temporal Feature Representation: A shared spatial grid supplies locally consistent, time-invariant features that complement independently optimized anchor features and improve optimization stability.The spatial feature is obtained by trilinear interpolation at each anchor’s spatial location.
- 3.4 Spatial-Temporal Feature Representation: The temporal domain is partitioned into M segments with separate temporal structures, decoupling local temporal modeling from global sequence length.Each temporal structure models variation within its segment and is instantiated as a 4D hash grid.
- 3.4 Spatial-Temporal Feature Representation: Anchor, static, and dynamic features jointly constrain Gaussian generation: anchor and static features encode most scene content, while dynamic features model time-varying behavior.The temporal feature enables time-varying Gaussians, whereas the static feature encodes low-frequency spatial structure and improves stability.
- 3.5 Temporal Gaussian Decoding: Enriched anchor features are decoded into Temporal Gaussians whose parameters vary continuously with time, with compactness regularization encouraging localized primitives.Training combines image reconstruction losses with a volume-based penalty on Gaussian scales.
4 Experiments
ATGS is evaluated across datasets spanning different temporal lengths, motion complexities, and camera-view densities. Implementation settings vary with scene motion, while experiments cover quantitative comparisons and temporal-grid ablations.
- Implementation: Keyframe and temporal-grid counts are adjusted to scene motion, using K=15 and M=2 for small-motion scenes and K=250 and M=12 for VRU.The temporal window size W is fixed to 7 across scenes.
- Datasets: Experiments cover N3DV, VRU, and MeetRoom, including small-motion, large-motion, and sparse-view scenarios.N3DV includes sequences up to 1,200 frames, VRU includes large-scale motion and sequences up to 1,400 frames, and MeetRoom evaluates sparse-view settings.
- Evaluation protocol: The evaluation compares ATGS with representative dynamic-scene methods across reconstruction quality, training time, rendering time, and storage.The benchmarks include N3DV, VRU-basketball, and MeetRoom, with additional datasets described for broader evaluation.
LocalDyGS PSNR: 23.60 dB
ATGS reports stronger reconstruction quality across evaluated datasets while maintaining comparable efficiency and supporting long-sequence reconstruction. Qualitative and ablation studies examine detail preservation, temporal coverage, and feature and windowing choices.
- Quantitative evaluation: ATGS consistently achieves superior PSNR, SSIM, and LPIPS reconstruction quality across the evaluated datasets.The gains indicate improvements in both pixel accuracy and perceptual quality.
- Qualitative evaluation: Qualitative results preserve fine details and fast-moving content, while competing methods show blur, missing geometry, or missing body parts.The reported comparisons include human hands, rapidly moving athletes, and sparse-view MeetRoom scenes.
- Ablations: Increasing temporal-grid count improves rendering quality on the 250-frame VRU-GZ sequence.The evaluated settings are M=1, 4, 8, and 12.
- Ablations: Removing the temporal window degrades rendering quality, whereas performance remains relatively stable across different window sizes.This study evaluates window size W on VRU-Long.
- Ablations: Removing either the static feature f_s or anchor feature f_a degrades reconstruction quality, especially for large motion and fine geometric details.The ablation supports the role of both features in spatial-context encoding and feature aggregation stability.
5 Discussion and Limitations
ATGS remains subject to computational and observational limitations. It does not support real-time processing, and sparse viewpoints can produce mild peripheral temporal jitter and weak geometric constraints.
- Computational limitations: ATGS follows an offline reconstruction paradigm and does not support real-time processing.The authors identify real-time processing as an open challenge for volumetric video reconstruction.
- Observational limitations: Extremely sparse viewpoints can cause mild temporal jitter in weakly observed peripheral regions such as distant audiences in VRU.Insufficient multi-view observations under-constrain optimization and make temporal coherence difficult to enforce consistently.
- Observational limitations: The reported artifacts are restricted to peripheral, weakly observed areas outside the primary reconstruction and evaluation focus.The authors state that these artifacts have limited impact on main-subject quality and the work’s overall conclusions.
- Future directions: Stronger temporal regularization and additional geometric priors for sparsely observed regions remain directions for future research.These additions are proposed to address under-constrained regions more effectively.
- Dataset constraints: Dynamic datasets often provide only 10–40 views, weakening geometric constraints in large-scale scenes such as VRU.In VRU, cameras are concentrated around the stadium boundary, leaving the inner field weakly constrained; dynamic subjects also occupy few pixels.
6 Conclusion
ATGS targets long multi-view volumetric videos with large-scale motion using time-conditioned anchors and feature decomposition. The resulting representation reconstructs thousands of frames with state-of-the-art objective and perceptual quality.
- Conclusion: ATGS reconstructs long multi-view volumetric videos with large-scale motion, extending beyond small-motion, short-sequence settings.The framework is designed for sequences spanning thousands of frames.
- Conclusion: Time-conditioned anchors localize moving objects in space-time and associate them with corresponding timestamps.These anchors provide the basis for mapping dynamic scene content to Gaussian primitives.
- Conclusion: Anchor, static, and temporal features jointly provide fine-detail control, spatial stability, and temporal modeling.Feature fusion supports stable Gaussian generation across long dynamic sequences.
- Conclusion: ATGS reconstructs volumetric videos spanning thousands of frames while achieving state-of-the-art objective and perceptual quality.The conclusion attributes this capability to fusing the decomposed anchor features.