STAG4D: Spatial-Temporal Anchored Generative 4D Gaussians

1 Introduction

• High-quality 4D generation remains difficult because prior methods can produce blurry rendering, spatial-temporal inconsistency, and slow generation.The paper notes that high-quality 4D content is relevant to applications including simulation, games and film, digital avatars, and spatial video.Source: §1 · p. 2
• STAG4D generates multi-view images across scene timestamps, fuses spatial and temporal attention to improve initialization consistency, then optimizes a 4D Gaussian point cloud with reference photometric and SDS losses.The fusion implicitly addresses texture degradation and geometric misalignment without adding explicit multi-view or temporal consistency losses during SDS optimization.Source: §1 · p. 2
• An adaptive densification strategy informed by Gaussian gradient distributions is designed to stabilize optimization and support high-quality, robust 4D generation from monocular video.The paper also describes the approach as tailored to the generation task.Source: §1 · pp. 2–3
• STAG4D reports 2x faster generation than previous video-to-4D approaches (e.g., [7]) and improved generation quality, rendering quality, and 4D consistency across text, image, and video inputs.The work also describes its generated 4D content as renderable in real time.Source: §1 · p. 3

2.1 3D Generation

• 3D generation work uses multi-view diffusion for viewpoint control and explores representations including meshes and 3D Gaussians; STAG4D introduces tailored 4D Gaussian optimization.The cited approaches include DreamGaussian [30] for 3D Gaussian generation and multi-view diffusion methods [16] [27] [28].Source: §2.1 · pp. 3–4

2.2 4D Reconstruction

• Dynamic NeRF methods face slow optimization and low reconstruction quality, while dynamic 3D Gaussian methods offer faster training and rendering; extending Gaussian reconstruction to 4D requires heuristic hyperparameter tuning.The reviewed approaches include per-frame 3DGS, hybrid explicit-point and implicit-field models, and an explicit per-point motion model.Source: §2.2 · p. 4

2.3 4D Generation

• MAV3D [29] and Consistent4D [7] established text- and video-to-4D settings; STAG4D uses anchored multi-view diffusion to optimize a deformable 3D Gaussian with SDS and reference losses.The pipeline can extend to text-to-4D by using an off-the-shelf text-to-video module.Source: §2.3 · p. 4, Figure 2 · p. 5

3.1 4D Representation

• A 4D scene is represented as a 3D Gaussian point cloud deformed over time by a hex-plane-based motion field.At each time, the deformation updates Gaussian positions, scales, and rotations.Source: 4D Gaussian Splatting · p. 6
• Because monocular inputs make a fixed gradient threshold suboptimal, densification selects the top λ% highest-gradient points at each operation.The adaptive threshold responds to the gradient distribution by filtering out points with relatively small gradients.Source: Adaptive Densification · p. 6

3.2 Temporal and Multi-view Consistent Diffusion

• Zero123++ supplies multi-view-consistent images to Zero123 SDS, addressing their respective fixed-camera and single-target-view constraints.Zero123++ provides multi-view consistency but uses fixed target camera locations; Zero123 can generate a target view at any relative camera location, one at a time.Source: Multi-view Consistent Diffusion · p. 7
• A training-free temporal attention module reuses first-frame attention during later denoising, mixing each later frame’s keys and values with the first frame’s using γ.The method also uses reference attention to condition on the input image.Source: Temporally Consistent Diffusion · p. 7

3.3 Training Objectives

• Optimization first supervises a fixed frame to create canonical static Gaussians, then trains the dynamic 4D Gaussians using all anchors and reference images.The multi-view SDS loss selects the generated image nearest the rendered camera view; the final objective adds reference reconstruction and foreground-mask losses.Source: §3.3 · p. 8

3.4 Text-to-4D Extension

• For text prompts, SDXL first generates a 2D image.Source: §3.4 · p. 9
• A video diffusion model such as SVD transforms the image into a video sequence with temporal coherence and motion.Source: §3.4 · p. 9
• The video sequence is lifted to a 4D scene, introducing an extra spatial dimension beyond conventional 2D or 3D media.Source: §3.4 · p. 9

4.1 Experiment Setup

• Video-to-4D quantitative evaluation uses Consistent4D's dataset of 7 dynamic objects and CLIP, LPIPS, FVD, and FID-VID metrics.CLIP and LPIPS are image-level measures; FVD assesses video-level quality and temporal coherence, while FID-VID measures temporal consistency.Source: §4.1 · p. 9

4.2 Implementation Details

• Training uses 1,000 canonical-space and 7,000 deformation steps; densification targets the top 2.5% gradients, with approximately 1-hour training and 150-FPS rendering.The reported training time is on an RTX 3090 GPU.Source: §4.2 · p. 10

4.3 Comparison with State-of-the-art Approaches

• In qualitative text-to-4D comparison, our method is best in text alignment and visual quality; on Text&Image-to-4D it yields smooth cross-view geometry and realistic texture.The comparison describes artifacts and distortions in side and back views of competing results.Source: Figure 5 · p. 11, §4.3 · p. 12

4.4 Ablation Study

• The final configuration with spatial-temporal anchor and adaptive densification achieves the best CLIP, LPIPS, FID-VID, and FVD scores.Table 2 evaluates baseline diffusion, spatial anchor, spatial-temporal anchor, and adaptive densification.Source: §4.4 · p. 12, Table 2 · p. 13
• Without spatial constraints the baseline deforms point clouds poorly; spatial-only anchoring still struggles on solid shapes, while both anchors yield visually consistent and accurate content.Figure 6 compares models without spatial anchoring, without temporal anchoring, and with both anchors.Source: Figure 6 · p. 12, §4.4 · p. 13
• Adaptive densification improves FID-VID and FVD; fixed thresholds can over-densify Bird or under-densify Duck, while gradient-adaptive thresholds yield robust, stable results.The threshold adapts to different cases based on accumulated Gaussian gradients.Source: §4.4 · p. 13
• Spatial-only attention does not ensure sequential consistency, cross-frame attention improves coherence but weakens reference alignment, while spatial-temporal attention leads on all four metrics.The compared metrics are CLIP, LPIPS, FID-VID, and FVD.Source: Table 3 · p. 13, §4.4 · p. 14

4.5 Limitations

• The method is constrained with complicated and fast motion, blurry input videos, and videos without well-segmented foregrounds or with multiple foreground objects.The paper attributes the motion constraint to current 4D Gaussian Splatting representation limitations and says input blurriness can reduce diffusion effectiveness and subsequent optimization quality.Source: §4.5 · p. 14

• The approach targets dynamic 3D generation from monocular videos by addressing 4D representation and spatial-temporal consistency.Source: §5 · p. 14
• It combines specially tailored 4D Gaussian splatting and an information fusion module for high-quality, robust 4D scene generation.Source: §5 · p. 14
• Experiments report faster generation and significant rendering-quality and temporal-consistency improvements over prior state-of-the-art methods, which the paper frames as a new benchmark.The conclusion names training speed, rendering quality, and 4D consistency as the benchmark dimensions.Source: §5 · p. 14

• Duck's suitable gradient threshold is approximately 40 (e3.7), versus approximately 150 (e5) for Bird; distributions retain a consistent shape over training but converge to disparate values.Source: §1 · p. 18
• These scenario-specific gradient scales motivate adaptive densification, which dynamically adjusts the threshold to select points with relatively higher accumulated gradients.Source: §1 · p. 18

2.1 Video-to-4D

• In the user study of best-performing Video-to-4D methods, our method received 71.4% of votes for visual quality and temporal consistency and 64.3% for alignment, versus Consistent4D’s 28.6%, 28.6%, and 35.7%, respectively.4DGen received no endorsements in the evaluated categories.Source: §2.1 · p. 19, Table 1 · p. 19

2.2 Text/Video-to-4D

• Across 14 Text&Image-to-4D scenarios assessed by 30 evaluators, our approach received the highest ratings for visual quality, temporal consistency, and text alignment.Source: §2.2 · p. 19, Table 2 · p. 19

3 Attention Mechanism for Normal Map

• Integrating the attention mechanism with Zero123++ enables coherent multiview images with corresponding normal maps and demonstrates the framework’s flexibility.The authors describe this as broadening the method’s scope to diverse 4D generation scenarios.Source: §3 · p. 19

4 More Results for Multiview Generation

• Additional multiview results demonstrate the robustness and effectiveness of the method for spatially and temporally consistent images.The results are shown in Figures 3 and 4.Source: §4 · p. 20

5 More Results for 4D Generation

• The method handles various objects and motions, and the authors say it applies widely to different cases.Additional 4D-generation results are shown in Figure 5.Source: §5 · p. 20

6 Discussion on the Potential Negative Impact

• Use of pre-trained diffusion models raises copyright controversy around generated results, an ethical issue the authors say will remain until relevant laws mature.Source: §6 · p. 20