PaperlayerView exact revision on arXiv

Source-linked AI summary

Amortized Anchor Refinement for Deployable Continuous-Time 4D Gaussian Reconstruction

Jingong Chen, Qingwen Zhang, Sanghyeon Jun, Chulwoo Pack, Kyle Gao, Kwanghee Won

arXiv:2608.30218v1cs.CV

TL;DR

Continuous-time 4D reconstruction is difficult to deploy on standalone XR headsets because per-scene optimization is computationally costly and the deployment budget must also accommodate the temporal field. Amortized Anchor Refinement combines feed-forward anchoring, budget-constrained refinement, capacity-preserving optimization, and topologically constrained compression, achieving the required reconstruction quality within the target budget for consumer-GPU reconstruction and standalone-headset playback.

  • Problem

    Per-scene optimization for dynamic Gaussian reconstruction requires costly iterations, while standalone XR headsets have limited memory, compute, and streaming capacity.

  • Method

    A frozen feed-forward backbone predicts a geometric anchor, which is refined under a fixed budget, then compressed with capacity and persistent-homology floors while streaming fitted trajectories as continuous-time scene flow.

  • Results

    24.22 dB at a 2.12 GB fitting peak with 5.4× fewer Gaussians than a heavyweight optimizer, while meeting the target budget for consumer-GPU reconstruction and standalone-headset playback.

  • Takeaways & Limitations

    The approach supports continuous-time reconstruction within the target deployment budget without per-frame optimization or residual coding.

Abstract

from arXiv · show

Continuous-time 4D reconstruction remains impractical on standalone XR headsets. Per-scene optimization demands deployment-infeasible compute, and lower budgets cause collapse rather than degrade gradually. Feed-forward prediction is fast, but struggle to recover scene-specific detail. We present Amortized Anchor Refinement, which uses a frozen backbone to predict an initial Gaussian representation and a short optimization to specialize it under a fixed compute budget, with a capacity floor preserving representational density. A training-free stage then applies a persistent-homology constraint to prune unstable Gaussians while preserving topologically persistent structures, and streams the resulting trajectories directly as scene flow. On the Stage-Capture benchmark, Amortized Anchor Refinement achieves 24.31$\pm$2.22dB, while our deployment experiments demonstrate reconstruction within the target budget on a single consumer GPU and playback on a standalone XR headset.

1. Introduction

Continuous-time 4D reconstruction is difficult to deploy on standalone XR headsets because per-scene optimization exceeds their resources and reduced budgets can cause collapse. Amortized Anchor Refinement combines feed-forward initialization, budgeted refinement, capacity control, and topology-aware compression to support deployment.

  • Motivation: Dynamic Gaussian methods achieve high quality with scene-specific optimization, but their iteration costs exceed standalone XR headset budgets.Headsets also have limited memory and compute, while continuous-time fields must remain resident for arbitrary-time queries.
  • Motivation: 16.4 dB results when reduced optimization budgets leave unanchored Gaussians without consistent geometric updates.The densify-and-prune schedule removes Gaussians faster than valid geometry is established, causing collapse rather than gradual degradation.
  • Approach: Feed-forward prediction supplies an initial geometric solution, while subsequent optimization recovers scene-specific detail beyond the learned prior.This addresses complementary weaknesses: prediction avoids geometry search, and refinement specializes the representation.
  • Approach: Amortized Anchor Refinement uses a frozen feed-forward backbone, a capacity floor, and a training-free persistence floor before streaming Gaussian trajectories as scene flow.The pipeline preserves arbitrary-time queries without per-frame optimization or residual coding.
  • Results: 24.22 dB is reached at a 2.12 GB fitting peak with 5.4× fewer Gaussians than a heavyweight optimizer.The reported cascade runs on a single consumer GPU and targets playback on standalone XR hardware.

2. Related Work

Prior dynamic Gaussian methods either optimize each scene at workstation-scale cost or rely on feed-forward priors that limit scene-specific reconstruction. Amortized Anchor Refinement instead optimizes a compact anchored representation and applies topology-aware compression directly to the fitted trajectories.

  • Dynamic representations: Existing dynamic Gaussian methods represent motion with deformation fields or explicit Gaussian trajectories, but reconstruct scenes through lengthy per-scene optimization.These methods typically require tens of minutes, hundreds of thousands of Gaussians, and workstation-class GPUs.
  • Amortized reconstruction: Feed-forward dynamic reconstruction is fast but remains constrained by its learned geometric prior and can require task-specific pretraining.Pixel-wise Gaussian prediction limits reconstruction to the geometry learned during pretraining.
  • Compaction: Post-processing compression reduces already-optimized fields, whereas this approach maintains an anchored compact representation during refinement with an explicit capacity floor.Peak memory is dominated by the frozen backbone, whose activations are CPU-offloaded.
  • Streaming: Persistent homology is used as a pruning criterion to preserve topologically important structures while Gaussian trajectories are transmitted directly as continuous-time scene flow.This complements the refinement-stage capacity floor and avoids per-frame optimization or residual coding.

3. Method

Amortized Anchor Refinement combines a frozen feed-forward anchor with short, capacity-constrained per-scene refinement, then compresses the resulting continuous-time Gaussian field using persistent homology while preserving important structures.

  • Framework: The framework targets deployable continuous-time 4D reconstruction by proposing Amortized Anchor Refinement for standalone XR headset memory and compute budgets.The pipeline includes anchored initialization, refinement, and compression for dynamic geometry.
  • Preliminaries and Anchor Backbone: A frozen backbone predicts per-view depth and features in one forward pass, avoiding camera poses and task-specific fine-tuning.Its activations are CPU-offloaded layer by layer to bound peak memory.
  • Grounded Feed-forward Anchor: Grounded Feed-forward Anchor places predicted Gaussians along input-view rays with learnable residual offsets, forming a thin volumetric initialization.The prediction head emits K Gaussians along each ray, with K=2 by default.
  • Grounded Feed-forward Anchor: Mirror-symmetry initialization closes the occluded side of the subject without additional parameters or supervision, especially regularizing sparse-time optimization.The prior has little effect for long, densely supervised sequences but becomes critical for short clips.
  • Capacity Constrained Refinement: Capacity Constrained Refinement retains at least Nmin highest-opacity Gaussians after pruning, while removing distant or oversized primitives to suppress floaters and bound memory.Refinement adjusts the grounded initialization rather than searching for geometry, so short optimization suffices.
  • Persistent Homology Pruning and Compression: Persistent Homology Pruning and Compression computes persistence over density grids, restores Gaussians linked to missing persistent features, and transmits quantized trajectories directly as scene flow.The prune-verify-restore-reverify procedure maintains the target Gaussian budget and accepts restoration only when the retained persistent-feature fraction improves.

4. Experiments

Experiments show that Amortized Anchor Refinement improves held-timestamp reconstruction while meeting constrained deployment budgets. Ablations and streaming evaluations attribute these gains to anchoring, capacity preservation, and topology-aware pruning.

  • Comparison with State of the Art: Amortized Anchor Refinement achieves the best accuracy on every scene, outperforming Deformable-3DGS by an average of +5.6 dB and by up to +10.9 dB on walk.The comparison uses foreground-masked held-timestamp PSNR.
  • Deployment Efficiency: All competing 4D Gaussian methods require 6–9 GB during optimization, whereas Amortized Anchor Refinement peaks at 2.12 GB and renders at 689 fps.The frozen backbone is CPU-offloaded at FP32, leaving refinement at a 0.83 GB peak.
  • Per-scene Results: 24.31±2.22 dB is the mean held-timestamp PSNR across nine scenes, with two camera splits agreeing at 24.31 and 24.11 dB.Quality degrades gradually with scene complexity, and no scene requires its own hyperparameters.
  • Module Ablation: Anchoring and the capacity floor raise walk quality from 16.4 dB with from-scratch optimization to 24.22 dB while retaining 23.9×10^4 Gaussians.The anchor supplies initial geometry, while the floor prevents the pruning schedule from removing too much capacity.
  • Topology Floor: At 2×10^4 Gaussians, topology-aware restoration raises preserved ε-persistent features from 0.82 to 0.88 and decoded quality from 19.52 to 20.30 dB.The topology floor is inactive above 4×10^4 Gaussians, where no ε-persistent feature is lost.
  • Streaming to Edge Devices: Below the lossless pruning budget, the proposed packages are strongest and their margin widens as the budget tightens.The streaming evaluation compares decoded PSNR and the fraction of ε-persistent features preserved across nine scenes.

5. Conclusion

The experiments show that scratch optimization cannot meet the target budget, while feed-forward prediction alone leaves a reconstruction-quality gap. Amortized Anchor Refinement combines feed-forward initialization, budget-constrained refinement, and topologically constrained compression to achieve the required quality within budget.

  • Scratch optimization cannot satisfy the target deployment budget.
  • Feed-forward prediction alone does not eliminate the reconstruction-quality gap.
  • Amortized Anchor Refinement combines a frozen backbone, budget-constrained anchor refinement, and topologically constrained compression.
  • The approach achieves the required reconstruction quality within the target budget, enabling reconstruction on a single consumer GPU and playback on a standalone headset.

A. Choice of Feed-forward Backbone

The main experiments use a frozen feed-forward backbone to initialize the Gaussian field, with subsequent stages held fixed across five evaluated alternatives. VGGT-Ω is selected for CPU offloading compatibility rather than the highest score, while direct Gaussian predictors are excluded because their geometric assumptions mismatch the stationary-camera, moving-subject setting.

  • The grounded feed-forward anchor uses a frozen backbone to initialize the Gaussian field.Each backbone supplies a depth map per input view, while later stages remain fixed.
  • Five alternative backbones are evaluated to measure sensitivity to the backbone choice.
  • VGGT-Ω is used because CPU offloading fits the 2.12 GB deployment requirement, not because it achieves the highest score.
  • Direct Gaussian predictors are excluded because they assume a static scene with a moving camera, unlike the stationary-camera, moving-subject anchor configuration.
  • A fair comparison would require a different anchor protocol with one multi-view reconstruction per timestamp, which remains outside this ablation.

B. The Transmitted Package from Free Viewpoints

The transmitted package is evaluated from viewpoints outside the capture rig using a common WebGL client and deployment-budget decoding. The package is substantially smaller than the raw field, while topology-aware retention preserves thin structures that pruning alone removes.

  • Figure 6 renders the transmitted package from three viewpoints outside the capture rig.
  • At the deployment budget, the package is 22× smaller than the raw field and visually indistinguishable from it.
  • The WebGL client decodes the package in shader at 87 fps on a Meta Quest Pro.
  • Figure 7 compares three package files in one client build, keeping the renderer, decoder, camera, viewpoint, timestamp, and coder fixed while varying retained-Gaussian ranking.
  • The topology-aware package retains the feet and lower half of the carried case that the pruning baseline removes at the same deployment budget.

C. Degradation Along the Budget Ladder

Importance-only pruning and topology-aware pruning are indistinguishable at loose budgets, but diverge as the budget tightens. The topology floor allocates part of the budget to thin structures, turning abrupt component loss into degraded appearance.

  • The two pruning strategies are indistinguishable at loose budgets because importance ranking already retains every subject part.
  • As the budget tightens, importance-only pruning loses the face and carried-case top abruptly.
  • The topology floor spends part of the same budget on Gaussians carrying those components.
  • With the floor, component appearance degrades instead of the components being removed.

D. Reconstruction on All Nine Scenes

Across all nine scenes, the refined field reconstructs bodies and rigid carried objects faithfully at held cameras and timestamps. Continuous-time rendering also interpolates motion between captured frames and supports timestamps with no corresponding capture.

  • All nine scenes: The refined field reconstructs bodies and rigid carried objects faithfully across all nine scenes at a held camera and timestamp.The scenes include locomotion, two-person interaction, object manipulation, and loose clothing; no bottom-row pixel was supervised.
  • All nine scenes: The remaining errors concentrate on trailing edges of loose fabric, where the cubic trajectory cannot represent non-polynomial motion.
  • All nine scenes: Both baselines reconstruct static subject parts but lose the fastest-moving content across four additional held-camera, held-timestamp scenes.Examples include a disappearing toy, a thrown object smeared into the background, and motion involving two interacting subjects.
  • All nine scenes: Quarter-frame rendering produces consistent object advances without repeating poses, indicating interpolation between captured frames.Only the two endpoints were supervised; the midpoint was held out, while six other timestamps had no corresponding capture.

G. Residual Error Modes

Residual errors arise from the motion model and depth ambiguity rather than insufficient Gaussian count. The displayed budget ladder and nine-scene renderings frame these errors within the broader reconstruction results.

  • Residual error modes: Loose fabric loses its trailing edge because a cubic trajectory cannot follow a fold that changes direction within one interval.
  • Residual error modes: Contact between two subjects becomes a translucent mixture because the anchor supplies a single depth per ray.The mirrored back side of the nearer subject is placed inside the farther one.
  • Residual error modes: The budget-ladder comparison uses the same ranking, coder, and starting field across Gaussian budgets, with the leading column as reference.
  • Residual error modes: Fast-arc motion blurs along the object's path, extending the loose-fabric limitation to larger amplitude.
  • Residual error modes: All three residual modes are properties of the motion model, and keeping more Gaussians does not repair them.

H. Cross-Dataset Results on DNA-Rendering

On DNA-Rendering, the pipeline is evaluated on a different capture system using the same core recipe with a portrait-adapted anchor grid. It reaches higher mean PSNR than the two listed baselines while remaining comparable in scope to methods trained on the same sequences.

  • Protocol: The cross-dataset test uses an independent multi-view capture with different rig geometry, camera count, framing, and garments.
  • Protocol: The protocol preserves the same anchor, capacity floor, loss weights, step count, and densification-window rule, changing only the anchor ray-grid shape for portrait framing.
  • Results: 25.20 dB mean held-timestamp PSNR exceeds 23.29 dB for Deformable-3DGS and 24.45 dB for 4D-GS.The fields averaged 42.4×10^4 Gaussians and were fitted in 51 minutes; the method led on two of three sequences.
  • Results: Absolute values are not comparable with Table 3 because the subject occupies a different frame fraction and the masks come from a different segmentation system.
  • Results: The pipeline runs on an untuned capture and reconstructs it at quality comparable to methods trained on the same sequences.The anchor supplies usable geometry for an unseen rig and wardrobe.
Loading 2608.30218v1…