MoVieS: Motion-Aware 4D Dynamic View Synthesis in One Second

• Existing 3D tasks are commonly studied separately, while view synthesis and reconstruction often target static scenes and require costly per-scene optimization.Source: §1 · p. 1
• MoVieS represents dynamic scenes with deformable, renderable 3D Gaussian particles called dynamic splatter pixels.Each input pixel maps to a Gaussian primitive whose 3D location is determined by predicted depth.Source: §1 · p. 1
• MoVieS trains across static, dynamic, and point-tracking datasets and supports view synthesis, depth estimation, and 3D point tracking in one model.Source: §1 · p. 1
• MoVieS achieves competitive performance across diverse 4D perception tasks while running several orders of magnitude faster than prior methods.The framework also supports scene flow estimation and moving-object segmentation from sparse tracking supervision.Source: §1 · pp. 1–2
• A shared encoder and attention backbone feed depth, splatter, and time-conditioned motion heads that jointly model appearance, geometry, and motion.The motion head predicts movements toward target timestamps, while the splatter head predicts Gaussian appearance attributes for novel-view rendering.Source: §1 · p. 1, Figure 1 · p. 2

• Traditional reconstruction relies on dense multi-view supervision and per-scene optimization, while learning-based methods enable feed-forward prediction of several 3D properties.Source: §2.1 · p. 2
• MoVieS extends feed-forward 3D Gaussian reconstruction from static scenes to dynamic scenarios with moving objects.Source: §2.1 · p. 2
• Existing dynamic reconstruction extensions are often limited to two-frame inputs or sparse point-cloud outputs, while recurrent and diffusion-based approaches pursue broader video reconstruction.Source: §2.2 · p. 2
• Dynamic novel-view synthesis methods target rendering quality using NeRFs, 4D primitives, deformable fields, or explicit Gaussian motion, but many require scratch training, iterative optimization, or external supervision.Source: §2.2 · p. 2
• Feed-forward 4D view-synthesis methods cover specialized settings, but BTimer lacks frame-wise Gaussian relations and NutWorld lacks explicit motion supervision and uses an orthographic camera.Source: §2.2 · pp. 2–3

3.1. Dynamic Splatter Pixel

• Dynamic splatter pixels represent each video pixel as a canonical-space Gaussian primitive with time-dependent position and attribute deformations.Static Gaussian primitives are paired with deformation fields, establishing temporal correspondence for dynamic modeling and dense motion estimation.Source: §3.1 · p. 3

3.2. Prediction Heads

• MoVieS uses a pretrained geometric transformer with camera and timestamp conditioning, followed by parallel heads for depth, appearance, and motion prediction.Camera pose and intrinsics are embedded through Plücker embeddings and camera tokens, while query time conditions the motion head through adaptive layer normalization.Source: §3.2.1 · p. 3, §3.2.2 · pp. 3–4
• The model trains across eight diverse datasets by aligning objectives with available annotations, including static scenes, dynamic scenes, and point tracking.This heterogeneous-data strategy addresses the infeasibility of collecting synchronized multiview videos with dense depth and tracking labels at scale.Source: Table 1 · p. 4, §3.3.1 · p. 4

3.3.2. Objectives

• MoVieS combines depth, rendering, and motion losses in a multitask objective, using depth supervision, rendered-image reconstruction, and point-motion constraints.Motion supervision combines point-wise alignment with a distribution loss that preserves relative distances among tracked points.Source: §3.3.2 · p. 4
• The framework supports novel-view synthesis by rendering temporally deformed Gaussian primitives from monocular-video reconstruction.Figure 3 compares synthesized novel views while marking invisible regions according to each implementation.Source: Figure 3 · p. 5

Static Scene View Synthesis

• MoVieS maintains competitive static-scene performance, with predicted motion converging below 1e-3 on static inputs.This indicates that the model can implicitly distinguish static from dynamic regions.Source: §4.2.1 · p. 5

Dynamic Scene View Synthesis

• 0.93s per scene yields competitive or superior dynamic reconstruction performance while remaining orders of magnitude faster than prior approaches.MoVieS also generalizes more smoothly to sparse inputs and novel views than optimization-heavy alternatives.Source: §4.2.2 · p. 6

3D Point Tracking

• Directly estimating 3D point positions in a shared world coordinate produces consistently superior or competitive tracking across datasets.Compared with 2D trackers and SpatialTracker, MoVieS avoids geometry noise from separately pretrained monocular depth estimators.Source: §4.3 · pp. 6–7

Ablation and Analysis

• Camera tokens provide effective camera-aware conditioning, while combining them with Plücker embeddings gives the strongest conditioning.Plücker embeddings alone are comparable to using no camera information.Source: §4.4 · p. 7, Table 4 · p. 7
• Combining point-wise L1 and pixel-wise motion objectives produces sharper motion boundaries than either objective alone.The corresponding qualitative comparisons are shown in Figure 4.Source: §4.4 · p. 7, Figure 4 · p. 7
• Joint training of novel view synthesis and motion estimation is mutually reinforcing: removing motion supervision yields static predictions, while removing NVS produces blurry motion and harms rendering.The ablation therefore supports coupling both tasks through the unified architecture.Source: §4.4 · p. 7

Zero-shot Applications

• Scene flow and moving-object segmentation emerge zero-shot from predicted motion maps without task-specific fine-tuning or explicit supervision.Scene flow is obtained by transforming world-coordinate motion into target-camera coordinates, while segmentation thresholds motion magnitude.Source: §4.5 · p. 8

5. Conclusion

• MoVieS jointly models appearance, geometry, and motion from monocular videos using dynamic splatter pixels for temporally coherent 4D reconstruction.The model supports depth estimation, 3D point tracking, scene flow estimation, and moving-object segmentation.Source: §5 · p. 8

A.1. Dataset Details

• Most training datasets are synthetic and provide pixel-aligned depth, metric camera parameters, and, for some datasets, ground-truth 3D point trajectories.Stereo4D adds large-scale YouTube stereo diversity despite partially noisy annotations.Source: §A.1 · p. 13
• VGGT requires unit-scale 3D scenes with metric-aligned depths and camera extrinsics, but RealEstate10K supplies only relative COLMAP camera parameters.The method therefore needs to address RealEstate10K's unknown global scale before training.Source: §A.1 · p. 13

A.2. Curriculum Training

• MoVieS training frequently experiences abrupt loss fluctuations and missing gradients when combining heterogeneous datasets and sparse annotations.The sources differ in domain, camera field of view, and recording frame rate.Source: §A.2 · p. 13
• Static pretraining with depth and photometric losses is followed by dynamic data and motion supervision in a curriculum that stabilizes training.Removing static datasets makes the loss highly unstable, while later training varies frame count and aspect ratio.Source: §A.2 · p. 13

• Selected MoVieS components are initialized from VGGT and its pointmap head, while the splatter head and camera/time embeddings train from scratch.Source: §A.3 · p. 13
• AdamW with weight decay 0.05 and cosine learning-rate scheduling with linear warm-up is used across curriculum training stages.Source: §A.3 · p. 13
• λd = λr = λpt = 1 and λm = λdist = 10 are chosen to place multi-task loss terms in roughly similar numerical ranges.Source: §A.3 · p. 13

• MoVieS remains several orders of magnitude faster than optimization-based methods, but a noticeable reconstruction-quality gap persists.The gap is partly attributed to optimization-based approaches using multiple pretrained models for richer and more accurate input cues.Source: B. Limitations and Future Work · p. 13
• MoVieS depends on off-the-shelf camera-parameter estimation, limiting end-to-end optimization and motivating integrated pose estimation.The proposed integration could simplify the workflow, reduce error accumulation, and improve adaptability.Source: B. Limitations and Future Work · p. 13
• Long-video processing and high-resolution rendering remain challenging because computational cost and memory demand grow with scene complexity and resolution.More compact dynamic representations, novel architectures, or sparse encoding strategies are proposed directions.Source: B. Limitations and Future Work · p. 14
• MoVieS comparisons use open-source implementations including DepthSplat, Splatter-a-Video, Shape-of-Motion, MoSca, BootsTAPIR, CoTracker3, and SpatialTracker.Source: C. License Information · p. 14
• Training draws on eight datasets from diverse sources, including RealEstate10K, TartanAir, MatrixCity, PointOdyssey, DynamicReplica, Spring, VKITTI2, and Stereo4D.Source: C. License Information · p. 14, Table S.1 · p. 14
• MoVieS may benefit robotics simulation, AR/VR, autonomous driving, and digital twins, while raising risks of unauthorized content generation and privacy violations.Source: D. Broader Impact · p. 14

• Supplementary results visualize novel view synthesis, 3D point tracking, scene flow estimation, and moving object segmentation across Figures S.1–S.4.Source: E. More Visualization Results · p. 14, Figure S.1 · p. 15, Figure S.2 · p. 16
• Novel view synthesis and 3D point tracking use DAVIS and TAPVid-3D, which are excluded from training, while other visualizations use curated-dataset test sets.Source: E. More Visualization Results · p. 14
• Figure S.1 evaluates novel view synthesis on DAVIS, showing the reconstructed scene at a target time from a target viewpoint using MegaSaM camera poses.Source: Figure S.1 · p. 15
• Figure S.2 visualizes 3D point tracking by projecting 3D point movements relative to the first frame into image space.Source: Figure S.2 · p. 16
• Figure S.3 derives scene-flow visualizations by projecting estimated 3D motion maps into camera space as optical flow, with colors indicating 2D directions.Source: Figure S.3 · p. 17
• Figure S.4 highlights moving parts in light red by filtering estimated motion maps and treating high-motion regions as moving.Source: Figure S.4 · p. 18