Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generation

1 Introduction

• Single-image 4D synthesis requires inferring plausible future motion from a static observation while preserving structural coherence across time and viewpoints.Generate-then-reconstruct methods can suffer geometric inconsistency and structural collapse, while reconstruct-then-generate methods struggle with large-scale, self-initiated movement.Source: §1 · p. 2
• MoGe4D predicts dense future point trajectories from an initial geometric prior inferred from the input image, integrating geometry and motion generation.Its 4D-STraG models the trajectories in a unified denoising process rather than treating geometry and dynamics as separate steps.Source: §1 · pp. 2–3
• MoGe4D consistently outperforms existing baselines in quantitative and qualitative comparisons, with more pronounced dynamics and stronger 3D motion consistency.The reported ablations also support the effectiveness of its key modules for coherent and physically plausible motion.Source: §1 · p. 3
• TrajScene-60K contributes 60,000 samples with dense 4D point-cloud trajectories, videos, and text to address scarce high-quality training data.The dataset supports learning scene-level dynamics.Source: §1 · p. 3
• Depth-guided motion normalization and the Motion Perception Module support geometric awareness and plausible motion, while 4D-ViSM renders trajectories from arbitrary novel viewpoints.The pipeline is presented as producing structurally stable motion and high-quality dynamic videos.Source: §1 · p. 3

Video Generation

• Video-generation methods have advanced realism, but most operate in 2D pixel space without explicit 3D scene modeling.Prior work includes VAE, GAN, and large-scale diffusion approaches, along with structure-, image-, and temporal-guidance strategies.Source: §2 · pp. 3–4

Novel-View Synthesis (NVS)

• Novel-view synthesis methods trade geometric consistency against generation flexibility, with generation-based methods often showing inconsistency or object drift under difficult views.Reconstruction-based approaches can ensure strong geometric consistency but may require costly optimization and suffer artifacts.Source: §2 · p. 4

4D Generation

• Image-to-4D is more ill-posed than video-to-4D because it reconstructs 3D structure and temporal dynamics without multi-frame priors.Generate-then-reconstruct approaches can mismatch video and geometry, while reconstruct-then-generate methods are often object-centric and limited in scene complexity.Source: §2 · p. 4

Dataset curation

• The dataset targets complex scene videos with intricate motion and non-rigid dynamics, addressing a shortage of large-scale annotated data.CogVLM2 captions candidate videos, and DeepSeek-V3 evaluates countable entities and self-initiated, non-rigid, or articulated motion.Source: §3 · p. 4
• The curation pipeline filters WebVid-10M videos for structured motion and countable entities, then extracts and refines dense point tracks to produce 60K scenes.Track refinement uses depth filtering and Gaussian Splatting.Source: §3 · p. 4, Figure 2 · p. 5
• Videos dominated by unstructured dynamics or camera motion are discarded to avoid conflating egomotion with object dynamics in pseudo-ground-truth trajectories.This leaves motion targets whose motion originates from the scene itself.Source: §3 · p. 5

Track extraction and quality control

• DELTA tracks are filtered for invalid depths, excessive scene-depth variation, and scale inconsistency, yielding 60,000 samples at 596 × 336 resolution.The resulting dataset contains over 3 million frames and approximately 12 billion 3D point annotations, with occlusion-aware tracking, per-frame depth, and language descriptions.Source: §3 · p. 5
• The curated 4D scenes are rendered from the original camera viewpoint using Gaussian Splatting, with inpainting masks for void regions.The dataset spans diverse real-world indoor and outdoor environments.Source: §3 · p. 5

4.1 Problem Definition

• Dense point trajectories provide a class-agnostic representation that separates geometry from appearance and supports diverse topologies and large deformations.The scene is represented as a sequence of point clouds, with each point tracking motion over time.Source: §4.1 · p. 6
• Depth-guided normalization scales relative point motion by the initial viewing frustum, yielding a scale-invariant representation across different distances.The normalization uses focal-length- and image-size-based scaling factors, and inference reverses it before fusing motion with the initial point cloud.Source: §4.2 · p. 7
• The generator encodes relative displacements as RGB motion maps and conditions the DiT on image, noise, and depth latents, training it with flow matching.Depth latents provide structural priors for scene layout and object relationships.Source: §4.2 · pp. 7–8
• MPM extracts motion-aware patch features and uses token-wise modulation to guide temporal coherence in a geometrically consistent manner.The paper reports that MPM improves motion-trajectory fidelity and spatiotemporal consistency.Source: §4.2 · p. 8

4.3 4D View Synthesis Module (4D-ViSM)

• 4D-ViSM renders time-varying point clouds from arbitrary camera trajectories and uses generative models to complete regions left uncovered in novel views.Per-frame 3D Gaussians are temporally linked into a lightweight dynamic renderer that reduces projection-induced holes.Source: §4.3 · p. 9

4.4 Inference

• At inference, estimated depth and motion features condition trajectory generation; the denormalized motion is fused with the initial point cloud before rendering from arbitrary camera poses.Source: §4.4 · p. 9

5.1 Experimental Setup

• The model is trained on TrajScene-60K and generates 49-frame, 512 × 368 videos; experiments ran on four NVIDIA H20 GPUs.4D-STraG training uses Wan2.1-14B, while 4D-ViSM is fine-tuned with LoRA.Source: §5.1 · p. 9

5.2 Qualitative Results

• MoGe4D point clouds retain structural consistency and reasonable motion, while 4D-ViSM renders them under two user-defined camera trajectories and qualitative comparisons report more diverse, realistic motion than baselines.Figure 5 places baseline results in the first row and MoGe4D results in the second, with the input image and text prompt in the first column.Source: §5.2 · p. 10, Figure 4 · p. 10, Figure 5 · p. 11

5.3 Quantitative Results

• MoGe4D demonstrated superior Dynamics, Aesthetic, and Imaging Quality versus 4Real, outperformed 3D reconstruction baselines in consistency and visual quality, and led 4D generation models in Aesthetic and Imaging Quality under complex trajectories.Source: §5.3 · p. 12
• MoGe4D consistently outperforms baselines on VLM-assessed 4D consistency, with significant margins in Motion-Geometry Coupling and Temporal Texture Stability, and reduces trajectory error while keeping reprojection errors competitive or superior.The evaluation uses eight sampled frames and five VLM criteria; geometric metrics measure trajectory and reprojection errors.Source: Table 2 · p. 12, §5.3 · p. 12, Table 3 · p. 13
• On one A100, MoGe4D generates 49-frame 512 × 368 sequences in six minutes and is 5× faster than Free4D at equivalent resolutions.GenXD is faster at two minutes but produces shorter, lower-resolution outputs.Source: §5.3 · pp. 12–13, Table 3 · p. 13

5.4 Ablation Studies

• Ablations show that the full model performs best across metrics, while depth guidance, MPM design, and large-scale high-quality training data support reported VBench quality.Removing depth-guided normalization drops Consistency to 0.8604 and Aesthetic quality to 0.4672; removing MPM reduces Dynamic from 0.9 to 0.85. Visual ablations show unstable point clouds with baseline min-max normalization and structural or motion-consistency failures without MPM or depth guidance.Source: §5.4 · p. 13, Table 4 · p. 13, Figure 6 · p. 14

5.5 Discussions

• Sequential video-generation pipelines followed by tracking or reconstruction show fragmentation and drift, whereas direct conditioning on initial point-cloud geometry yields more coherent, geometry-consistent trajectories.The comparison uses Wan2.1-I2V followed by DELTA tracking or VGGT reconstruction, with results rendered from a fixed camera viewpoint.Source: §5.5 · p. 14, Figure 8 · p. 15
• Extreme photometric conditions such as harsh backlighting and highly unstructured dynamics such as chaotic crowds can degrade monocular depth estimates, adding noise to the initial geometric prior and making motion less reliable.The authors identify more robust visual foundation models for these conditions as future work.Source: Figure 7 · p. 14, §5.5 · p. 15

• MoGe4D frames single-image 4D synthesis as dense time-varying point trajectories conditioned on an initial geometric prior, addressing weaknesses of decoupled paradigms.Source: §6 · p. 15
• The work introduces TrajScene-60K and 4D-STraG, which uses depth-guided motion normalization and motion-perception priors.The conclusion identifies these as support for training and trajectory generation.Source: §6 · p. 15
• Experiments report superior geometric consistency, dynamic realism, and visual fidelity compared with existing approaches.The paper points to future exploration of more universal dynamic priors and lightweight 4D representations for practical deployment.Source: §6 · p. 15

• VBench measures subject identity and appearance coherence over time, alongside temporal stability and coherence of background elements.Source: §A.1 · p. 16
• VBench assesses the naturalness of object and camera motion and quantifies the intensity and extent of scene dynamics.Source: §A.1 · p. 16
• VBench judges visual appeal, composition, and stylistic merit, and rates technical image attributes such as sharpness and noise.The paper describes the six dimensions as a holistic comparison framework for single-image 4D generation.Source: §A.1 · p. 16

A.2 Details on VLM-based 4D Evaluation

• Eight sampled frames are rated by Qwen2.5-VL-72B-Instruct on a 1–5 scale across five dimensions of 4D video quality.The dimensions cover 3D geometric consistency, temporal texture stability, subject identity, motion-geometry coupling, and background stability.Source: §A.2 · pp. 16–17

A.3 Details on 3D Geometry-Aware Evaluations

• Average Trajectory Error measures mean L2 distance between generated and ground-truth 3D point locations over valid tracked frames.Dense trajectories are extracted from both videos using DELTA under its default settings.Source: §A.3 · p. 17
• The geometry-aware metrics target structural collapse and texture swimming that standard video metrics may fail to penalize.The trajectory metric penalizes unnatural movement and spatial drift, while reprojection assesses cross-frame 3D coherence.Source: §A.3 · p. 17
• 3D Reprojection Error evaluates rigid structural stability by comparing observed coordinates with projections of reconstructed 3D points across co-visible pairs.The method uses DROID-SLAM to refine camera poses and per-pixel depth; low reprojection error indicates coherent geometry across frames.Source: §A.3 · p. 17

B More Discussions

• Sequential pipelines show spatiotemporal fragmentation, while MoGe4D produces more coherent results under a fixed viewpoint.The comparison uses Wan2.1-I2V followed by DELTA tracking or VGGT reconstruction.Source: B More Discussions · p. 18
• Video-generation inconsistencies can be misread downstream as 3D motion, producing background drift, fragmented geometry, and discontinuous trajectories.The discussion attributes these failures to appearance variation, motion blur, and perspective inconsistencies in generated video.Source: B More Discussions · p. 18
• MoGe4D conditions dense trajectory prediction on a geometric prior within a unified diffusion backbone.The discussion says this mitigates the sequential pipelines’ error accumulation.Source: B More Discussions · p. 18
• Qualitative examples show generated 4D point clouds and videos rendered by 4D-ViSM along three distinct user-defined camera trajectories.The examples include an outdoor festival crowd doing synchronized jumping and crowd-surfing.Source: B More Discussions · pp. 18–19

C.1 Qualitative Results of MoGe4D

• MoGe4D renderings maintain temporally coherent motion and high-fidelity visual details across distinct camera paths.The examples include top-down, forward-backward, right-hand circling, and left-hand circling views.Source: §C.1 · pp. 19–20

C.2 Qualitative Results of Motion-Sensitive VAE

• Fine-tuned VAE reconstructions preserve structural and motion consistency with the input when rendered from the original camera viewpoint.The examples show close visual alignment between pseudo-ground truth and VAE renderings at the same pose.Source: §C.2 · p. 20, Figure C · p. 20

C.3 OOD Generalization

• OOD generalization includes cartoons and paintings when their underlying geometric structure remains sufficiently discernible.Robustness to photometric extremes and unstructured scenes remains an open challenge.Source: §C.3 · pp. 20–21

D.1 Video Filtering and Annotation

• TrajScene-60K contains 3 × 106 frames and 1.2 × 1010 3D points, with dense tracking, depth, and captions for real indoor and outdoor scenes.The dataset quality-control process removes samples with invalid or anomalous depth, excessive scene-depth variation, or scale-inconsistent geometry.Source: §D.2 · p. 22

D.3 Comparison with Previous Datasets

• TrajScene-60K combines real-world video, dense occlusion-aware 4D trajectories, and text captions across diverse indoor and outdoor environments.The paper describes more than 3 million frames and approximately 12 billion 3D point annotations, alongside per-frame depth.Source: §D.3 · p. 22

D.4 Dataset Bias and Release

• TrajScene-60K inherits demographic and content biases from WebVid-10M source videos and the LLM/VLM filtering models.The authors plan to release the dataset and filtering prompts to support transparency and future audits.Source: §D.4 · pp. 22–23

E Baseline Introduction

• 4Real, GenXD, DimensionX, and Free4D use generate-then-reconstruct pipelines, while Gen3C uses reconstruct-then-generate.The methods differ in their approaches to video generation, reconstruction, camera and object motion, or novel-view synthesis.Source: E Baseline Introduction · p. 23

F.1 Motion-Sensitive VAE Architecture

• A shallow ResNet-based motion-sensitive VAE preserves fine-grained trajectory detail; training updates the Trajectory Encoder, Trajectory Decoder, and VAE Decoder while freezing the VAE Encoder.Its compact latent space avoids aggressive spatial downsampling for trajectory reconstruction.Source: §F.1 · pp. 23–24

F.2 Details of 4D-ViSM Architecture

• 4D-ViSM completes holes in rendered novel-view videos by conditioning a pretrained video DiT on the incomplete video, noisy latent, rendered latent, and downsampled occlusion mask.The known visual context and missing-region mask support coherent video completion.Source: §F.2 · p. 24

F.3 Inference Pipeline

• At inference, 4D-STraG estimates depth and generates relative motion latents, which are de-normalized and fused with initial point-cloud coordinates before rendering and 4D-ViSM video synthesis.The resulting representation can be rendered from arbitrary camera poses and aligned with the desired camera trajectory.Source: §F.3 · p. 24

G Applications and Future Work

• The authors identify digital content creation and VR/AR as potential applications, and plan to explore unified motion-appearance generation and standardized 4D evaluation metrics.The proposed metrics aim to quantify geometric and textural stability over time more reliably and automatically.Source: G Applications and Future Work · pp. 24–25

H Prompt of VLM-based 4D Evaluation

• The rubric assigns each metric a score from 1 to 5 and requests JSON output with separate scores for all five evaluation dimensions.The scale runs from poor or failure to excellent or flawless.Source: H Prompt of VLM-based 4D Evaluation · pp. 25–26
• The VLM rubric evaluates geometric consistency, temporal texture stability, subject identity preservation, motion-geometry coupling, and background stability.It assesses generated videos of moving scenes viewed by a moving camera.Source: H Prompt of VLM-based 4D Evaluation · pp. 25–26