PaperlayerView exact revision on arXiv

Source-linked AI summary

STORM: Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenes

Jiawei Yang, Jiahui Huang, Yuxiao Chen, Yan Wang, Boyi Li, Yurong You, Apoorva Sharma, Maximilian Igl, Peter Karkus, Danfei Xu, Boris Ivanovic, Yue Wang, Marco Pavone

arXiv:2501.00602v1cs.CVcs.LG

TL;DR

Dynamic scene reconstruction commonly depends on dense observations, per-scene optimization, and motion supervision. STORM uses a self-supervised Transformer to aggregate and transform frame-level Gaussians, and reports fast reconstruction that surpasses per-scene and feed-forward competitors.

  • Problem

    Dynamic scenes are difficult to reconstruct from sparse, incomplete observations, while existing methods often rely on dense data, per-scene optimization, and motion supervision.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    STORM uses a Transformer to infer dynamic scene representations from sparse posed images, transforming and aggregating frame-level Gaussians with self-supervised reconstruction loss and motion tokens.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    0.2 seconds for a 2-second clip; STORM significantly surpasses per-scene optimization and feed-forward approaches in dynamic-scene reconstruction quality.

    Source: §1 · p. 2, §5 · p. 11

  • Takeaways & Limitations

    STORM’s reconstructions support applications including view synthesis, scene editing, and point tracking.

    Source: §5 · p. 11

    STORM typically processes three cameras and four timesteps at 8× downsampled resolution (~7k tokens); longer sequences increase computational costs, and the model requires camera parameters.

    Source: §5 · p. 11

Abstract

from arXiv · show

We present STORM, a spatio-temporal reconstruction model designed for reconstructing dynamic outdoor scenes from sparse observations. Existing dynamic reconstruction methods often rely on per-scene optimization, dense observations across space and time, and strong motion supervision, resulting in lengthy optimization times, limited generalization to novel views or scenes, and degenerated quality caused by noisy pseudo-labels for dynamics. To address these challenges, STORM leverages a data-driven Transformer architecture that directly infers dynamic 3D scene representations--parameterized by 3D Gaussians and their velocities--in a single forward pass. Our key design is to aggregate 3D Gaussians from all frames using self-supervised scene flows, transforming them to the target timestep to enable complete (i.e., "amodal") reconstructions from arbitrary viewpoints at any moment in time. As an emergent property, STORM automatically captures dynamic instances and generates high-quality masks using only reconstruction losses. Extensive experiments on public datasets show that STORM achieves precise dynamic scene reconstruction, surpassing state-of-the-art per-scene optimization methods (+4.3 to 6.6 PSNR) and existing feed-forward approaches (+2.1 to 4.7 PSNR) in dynamic regions. STORM reconstructs large-scale outdoor scenes in 200ms, supports real-time rendering, and outperforms competitors in scene flow estimation, improving 3D EPE by 0.422m and Acc5 by 28.02%. Beyond reconstruction, we showcase four additional applications of our model, illustrating the potential of self-supervised learning for broader dynamic scene understanding.

1 INTRODUCTION

STORM replaces dynamic-scene methods that depend on dense observations, per-scene optimization, and motion supervision with a self-supervised, feed-forward model. It aggregates and transforms frame-level Gaussians to reconstruct scenes across time, while motion tokens capture shared motion patterns.

1 INTRODUCTION

  • Existing dynamic reconstruction methods often require dense spatio-temporal observations, per-scene optimization, and motion supervision, with lengthy training and noisy pseudo-labels.These approaches also cannot benefit from data-driven advances used by generalizable static reconstruction methods.Source: §1 · p. 1
  • STORM transforms per-frame 3D Gaussian Splats to a target timestep and aggregates them into an amodal scene representation, using reconstruction loss to estimate temporal changes.Inaccurate dynamics produce poor aggregation and larger reconstruction errors.Source: §1 · p. 2
  • Learnable motion tokens capture shared motion primitives and regularize Gaussian velocity predictions, while enabling unsupervised dynamic-instance or motion-group segmentation.Gaussian motion is represented as a weighted combination of shared velocity bases.Source: §1 · p. 2
  • Auxiliary sky and affine tokens address sky modeling and exposure mismatches, while latent Gaussians and a latent decoder improve novel-view extrapolation and fine-grained human motion modeling.Source: §1 · p. 2
  • STORM reconstructs a 2-second clip in 0.2 seconds and significantly surpasses per-scene optimization and feed-forward methods in photorealism, geometry, and motion estimation quality.Experiments cover Waymo Open, NuScenes, and Argoverse2.Source: §1 · p. 2

2 RELATED WORK

Prior dynamic reconstruction methods commonly rely on per-scene optimization, dense observations, or explicit motion supervision, while feed-forward reconstruction has mainly targeted static scenes. STORM instead targets outdoor dynamic scenes with a feed-forward model that estimates motion without explicit supervision.

Dynamic scene reconstruction

  • NeRF- and 3DGS-based dynamic methods typically use per-scene optimization and depend on dense views or explicit motion supervision to address sparse-view ambiguity.Approaches include canonical-space deformations and point-level scene-flow or particle transformations.Source: §2 · pp. 2–3

Feed-forward reconstruction

  • Feed-forward reconstruction methods have generalized across scenes, but recent 3DGS approaches are primarily designed for static scenes and struggle with dynamics.Earlier generalizable NeRF methods often require extensive point sampling for rendering, limiting speed and detail.Source: §2 · p. 3

Reconstruction for outdoor urban scenes

  • Dynamic urban-scene methods often use costly box annotations or lack individual-object controllability, and remain per-scene approaches with lengthy training.No-box methods typically lack controllability over individual objects; noisy pseudo-labels can also degrade performance.Source: §2 · p. 3

3 SELF-SUPERVISED SPATIAL-TEMPORAL RECONSTRUCTION MODEL

STORM addresses sparse, incomplete spatiotemporal observations with a Transformer that predicts per-frame Gaussian scene states and motions, then transports and combines them into an amodal scene at target times.

3.1 STORM

  • STORM predicts per-frame 3D Gaussians and velocities, then transforms and aggregates them across timesteps into an amodal scene representation.The representation supports rendering from novel viewpoints and moments in time.Source: §3 · p. 3, §3.1 · pp. 4–5
  • A Transformer embeds image patches, camera-ray geometry, and time, while motion tokens decode shared velocity bases weighted by pixel-aligned motion keys.The weighted bases produce Gaussian velocities and support motion-group segmentation without explicit motion supervision.Source: §3.1 · pp. 4–5
  • Within a clip, typically around two seconds, Gaussian translation assumes constant velocity to balance model complexity and representational power.Source: §3.1 · p. 4

3.2 STORM IN THE WILD

  • For in-the-wild captures, a sky token predicts ray-conditioned sky color, while camera-specific affine tokens model exposure variation.These tokens address sky modeling and exposure mismatches between cameras.Source: §3.2 · pp. 5–6
  • Latent-STORM uses patch-aligned latent Gaussians and a decoder to reconstruct occluded regions and fine human motions, with a slight compromise in pixel sharpness.The decoder can recover subtle leg and arm movements from sparse observations.Source: §3.2 · p. 6

3.3 IMPLEMENTATION

  • Training samples four target timesteps within [t, t + 2s] and combines RGB, perceptual, and depth reconstruction losses with sky and velocity regularization.Source: §3.3 · p. 7

4 EXPERIMENTS

STORM is evaluated for sparse-view rendering and scene-flow estimation, then examined through ablations and applications to long-video reconstruction, tracking, segmentation, and editing. Across the reported comparisons, it performs strongly against per-scene and feed-forward baselines while supporting fast inference.

4.1 RENDERING

  • 5 dB PSNR and 0.346 SSIM gains in dynamic regions over the best per-scene method accompany 0.18-second inference; full-image PSNR gains are around 0.5 to 1 dB.Table 1 evaluates Waymo novel-view synthesis on full images and dynamic regions, comparing per-scene optimization and feed-forward methods.Source: §4.1 · pp. 7–8, Table 1 · p. 8
  • On NuScenes and Argoverse2, STORM achieves the best full-image PSNR and Depth RMSE among the compared generalizable methods.The reported result supports generalization across these additional datasets.Source: §4.1 · p. 8, Table 2 · p. 8

4.2 FLOW ESTIMATION

  • STORM consistently outperforms all compared methods across scene-flow metrics, including EPE3D, Acc5, and Acc10, despite using only camera images.The evaluation uses Waymo context frames; competing methods require LiDAR at test time, and methods that excel with dense 10Hz observations struggle with sparse 2Hz observations.Source: §4.2 · pp. 8–9, Table 3 · p. 8

4.3 ABLATION STUDY, QUALITATIVE RESULTS AND APPLICATION

  • Without velocity regularization, training collapses from gradient explosion; λreg=5e-3 performs best, and M=16 performs best for rendering and flow estimation.Training with four input timesteps also generalizes zero-shot to test-time inputs of one, two, six, or ten timesteps, though configuration-specific retraining achieves optimal results.Source: §4.3 · p. 9
  • STORM reconstructs 20-second videos iteratively, and merging clip-level Gaussians produces a feed-forward reconstruction of the full video in 0.5 seconds.The sequences contain 600 images from three cameras over 200 timesteps; merging can introduce artifacts in overlapping regions, and the Gaussians can initialize per-scene refinement.Source: §4.3 · p. 9, Figure 4 · p. 9, Figure 5 · p. 9
  • Chaining scene flows yields point trajectories, while motion-token assignments group Gaussians by moving pattern and enable editing without ground-truth 3D boxes.Figure 6 shows rendered camera images, scene flows, and motion assignments.Source: §4.3 · pp. 9–10, Figure 6 · p. 10
  • Latent-STORM supports scene edits, human-motion reconstruction, and synthesis of occluded regions, though inpainting can be slightly blurry.The examples include removing or cloning vehicles and adding or replacing pedestrians and vehicles; diffusion models are proposed as a possible future way to improve generation quality.Source: §4.3 · p. 10, Figure 7 · p. 10
  • Default STORM performs well for vehicle editing but has slightly reduced performance for human modeling.Latent Gaussians and a latent decoder are introduced to improve human-body modeling.Source: §4.3 · p. 10

5 CONCLUSION

STORM reconstructs dynamic outdoor scenes and estimates their dynamics from sparse observations without explicit motion supervision, with applications including view synthesis, editing, and point tracking. Its sequence length and camera-parameter requirements impose practical constraints.

5 CONCLUSION

  • STORM typically processes images downsampled by a factor of 8 from three cameras and four timesteps, yielding around 7k tokens.Handling up to 32,000 tokens in preliminary experiments enables higher resolution, longer temporal windows, or additional views, but increases training and inference costs.Source: §5 · p. 11
  • STORM requires camera intrinsics and extrinsics as inputs, which may require additional effort or preprocessing in domains where they are not readily available.The authors identify reduced-complexity Transformers, joint camera-parameter optimization, and geometric foundation models as future directions.Source: §5 · p. 11
  • STORM significantly surpasses existing per-scene optimization and feed-forward approaches in reconstructing dynamic outdoor scenes and estimating scene dynamics.The conclusion identifies view synthesis, scene editing, and point tracking as applications.Source: §5 · p. 11

A IMPLEMENTATION DETAILS

This section introduces the implementation details of STORM.

A IMPLEMENTATION DETAILS

A.1 STORM IMPLEMENTATION DETAILS

STORM predicts Gaussian parameters from image inputs, with camera parameters used to recover 3D coordinates. The implementation also describes sky and mask decoding, training settings, and trajectory estimation.

A.1 STORM IMPLEMENTATION DETAILS

  • Each Gaussian is parameterized by a 3D center and orientation, 3D scale, opacity, and 3D color; centers are recovered from camera rays and predicted depth.Depth is mapped between near 0.1 and far 400.0 in the experiments.Source: §A.1 · p. 17
  • Rotations use normalized 4D quaternions, scales are exponentiated and capped at 0.5, and opacity and color are mapped with sigmoid functions.Color is mapped back to [0, 1] when computing PSNR.Source: §A.1 · p. 17
  • The sky MLP predicts color from viewing directions, conditioned by a Transformer sky token through adaptive layer normalization.The modulated features are transformed to 3D color.Source: §A.1 · p. 17
  • The mask decoder uses convolutional upsampling layers, while the ViT feature maps and motion tokens are projected into 32-dimensional spaces.The decoder’s transposed convolutions output 512-, 256-, then 128-channel features.Source: §A.1 · p. 17
  • Training uses 100,000 iterations, a global batch size of 64, and AdamW with a cosine schedule and 5,000-iteration warmup.LPIPS loss is enabled after 5,000 iterations; gradient checkpointing reduces memory use, and the authors observe benefits from longer training and larger models.Source: §A.1 · pp. 17–18
  • Dynamic Gaussian trajectories are estimated by chaining per-frame scene flows and recursively connecting each Gaussian to its nearest transformed predecessor.Source: §A.1 · p. 18

A.2 LOSS FUNCTION

The loss combines image reconstruction terms with sky and velocity regularization. Its inputs include rendered outputs, observed images and depth, and predicted sky masks.

A.2 LOSS FUNCTION

  • The loss uses rendered images, depth and opacity maps, Gaussian velocities, observed images and depth maps, and predicted sky masks.Source: §A.2 · p. 18
  • Reconstruction loss combines RGB, depth, and LPIPS losses, while sky loss and velocity regularization use MSE losses that encourage sparsity.Source: §A.2 · p. 18
  • The loss weights are λlpips = 0.05, λsky = 0.1, and λreg = 5e-3 in all experiments.The LPIPS term uses a VGG-19-based implementation.Source: §A.2 · p. 18

A.3 BASELINE IMPLEMENTATIONS

The baseline implementations are adapted or reimplemented to match STORM’s sparse, short-sequence setting and to align training conditions across generalizable models.

A.3 BASELINE IMPLEMENTATIONS

  • Per-scene baselines include PVG [Chen et al., 2023], DeformableGS [Yang et al., 2024b], 3DGS [Kerbl et al., 2023], and EmerNeRF [Yang et al., 2024a].They use DriveStudio code for the first three and EmerNeRF’s official code with modifications; iterations were reduced from 20,000 to 5,000 for sparse 3-camera × 4-timestep observations.Source: §A.3 · p. 18
  • Generalizable baselines use aligned datasets, supervision, optimizers, and iteration counts, with parameter counts near STORM’s 100.60M.GS-LRM has 86.68M parameters and LGM has 103.29M; both were adjusted for sky depth prediction.Source: §A.3 · p. 18
  • Figure B.1 describes ablations of velocity regularization and motion-token count, measuring dynamic PSNR, EPE3D, and Acc5.The caption notes that omitting velocity regularization frequently causes gradient explosion and NaN loss.Source: Figure B.1 · p. 19
  • Figure B.2 describes zero-shot tests with varying input timesteps and training with different timestep counts.The caption frames the latter as an assessment of adaptability to changes in input timesteps.Source: Figure B.2 · p. 19

B.1 COMPARISON ON ADDITIONAL DATASETS

The additional-dataset evaluation uses NuScenes and Argoverse2 with three frontal cameras, with dataset-specific image processing and evaluation choices.

B.1 COMPARISON ON ADDITIONAL DATASETS

  • NuScenes evaluation uses three frontal cameras at 160 × 288 resolution and both sample key frames and sweep data.The dataset is split into 700 training, 150 validation, and 150 test scenes; evaluation uses the validation set with unchanged hyperparameters.Source: §B.1 · p. 19
  • Argoverse2 evaluation uses three frontal cameras resized uniformly to 192 × 256, without special handling for reversed camera aspect ratios.Performance is measured on full images without extracting dynamic instances.Source: §B.1 · p. 19
  • Argoverse2 depth RMSE is higher than on NuScenes and Waymo, consistent with its sensing range exceeding 200m versus approximately 80m.The passage attributes the difference as expected given the larger sensing range.Source: §B.1 · p. 19

B.2 ABLATION STUDY

The ablations examine input-timestep flexibility and two motion-design choices, finding benefits from more input timesteps and a specific regularization coefficient.

B.2 ABLATION STUDY

  • The ablations use Waymo; Fig. B.1 models train with batch size 32 for 100k iterations, while Fig. B.2-(b) models use batch size 64 for 100k iterations.Source: §B.2 · p. 19
  • 5 × 10−3 is the optimal velocity regularization coefficient in the controlled setting, giving the best dynamic PSNR and second-best flow estimation performance.Omitting regularization often causes gradient explosion and NaN loss.Source: §B.2 · p. 20
  • M = 16 motion tokens gives the best results, while STORM performs robustly across tested motion-token counts.The M = 0 comparison predicts pixel-aligned velocities directly from image embeddings while keeping learnable parameter counts nearly constant.Source: §B.2 · p. 20
  • 2 to 4 PSNR improvement on dynamic regions is reported when single-timestep-trained STORM predicts the future, compared with per-scene methods using 4 input timesteps.STORM generalizes to unseen test-time timestep configurations without retraining, peaks at 4 timesteps, and improves as training timesteps increase.Source: §B.2 · p. 20
Loading 2501.00602v1…