DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling

1 Introduction

• Existing real-world 4D data focus mainly on indoor or autonomous-driving scenes, while synthetic data can lack real-world fidelity and complexity; metric-scale geometry, non-rigid actor representations, and dynamic-content captions are often absent.The paper identifies a need for large-scale, diverse, physically aware, semantically rich 4D data.Source: §1 · p. 2
• Over 100K distinct 4D scenes, 800K masklets, and 10M video frames make up DynamicVerse, with metric-scale point maps, camera parameters, categorized object masks, and descriptive captions.The dataset is sourced through a combination of massive 2D video datasets and existing 4D datasets.Source: §1 · pp. 2–3
• DynamicGen combines metric-scale geometric and moving-object recovery from raw videos with hierarchical object-, camera-, and scene-level caption generation.The pipeline uses foundation models and dynamic bundle adjustment as part of its annotation process.Source: §1 · pp. 2–3
• DynamicGen is evaluated on video depth, camera pose, and camera intrinsics benchmarks, with qualitative web-video analyses and human and GPT-assisted caption evaluation.The authors also report demonstrating the pipeline's ability to process web-scale videos and extract multimodal information qualitatively.Source: §1 · pp. 2–3

2 Related Work

• Table 1 compares DynamicVerse with large-scale 2D video datasets and existing 4D scene datasets, and its caption characterizes DynamicVerse as expanding data scale and annotation richness.The supplied table excerpt identifies the datasets but does not expose specific column meanings or comparative values.Source: Table 1 · p. 3
• DynamicGen integrates pretrained models for geometry, correspondence, and dynamic-object segmentation; unlike Uni4D [57], it is described as producing globally refined dense geometry, moving-object recovery, and fine-grained captions.The cited components include UniDepthv2 [54], CoTracker3 [53], UniMatch [51], Qwen2.5-VL [55], and SA2VA [56].Source: §2 · p. 3
• Earlier 4D datasets are described as limited in diversity, while large posed-video collections lack detailed geometry and semantics beyond pose.The paper positions DynamicVerse's metric-scale depth, camera parameters, labeled instance segmentation, and captions as addressing this information gap.Source: §2 · p. 4

3 DynamicVerse

• DynamicVerse is a physical-scale, multi-modal 4D modeling framework for real-world video, paired with a dataset comprising over 100K distinct 4D scenes, 800K masklets, and 10M video frames.Source: §3 · p. 4

3.1 4D scene curation

• DynamicGen curates real-world and synthetic video datasets as scalable sources for 4D scene understanding.Sources include DAVIS2017, Youtube-VIS, UVO-dense, and SA-V, alongside PointOdyssey, Spring, and other synthetic 4D datasets.Source: §3.1 · p. 4

3.2 Data filtering strategy

• Video filtering scores proximal depth, focal-length stability, blur, camera-motion smoothness, and non-perspective distortion to select sequences for bundle adjustment.A Random Forest predicts a 0–5 quality score from normalized criteria, using about 1,000 manually scored videos; VLM judgment further excludes unsuitable videos.Source: §3.2 · pp. 4–5

3.3 Moving object recovery

• Qwen2.5-VL identifies moving objects and categories, SA2VA generates their masks, and geometry supports physical-aware 3D bounding-box annotation.Source: §3.3 · p. 5

3.4 Dynamic bundle adjustment

• A five-stage bundle-adjustment framework recovers metric-scale camera parameters and point maps, combining dynamic masking, camera initialization, static and non-rigid optimization, and sliding-window global refinement.The method assumes shared intrinsics across frames and combines reprojection, flow, non-rigid tracklet-consistency, and motion or camera regularization terms; it freezes camera parameters during non-rigid reconstruction and global refinement.Source: §3.4 · pp. 5–7

3.5 Dynamic Content Caption Generation

• Dynamic content captions describe objects, scenes, and camera motion, with human review used to identify and revise or remove problematic annotations.Object captions describe motion, scene captions capture objects and interactions, and camera captions describe trajectory and movement patterns.Source: §3.5 · pp. 7–8

4.1 Video Depth Estimation

• DynamicGen achieves the best overall video-depth performance across Sintel and KITTI on Abs Rel and δ < 1.25, with the cleanest dynamic segmentations and strongest dynamic/static reconstructions.The evaluation uses global shift and scale alignment; MonST3R distorts object geometry and Uni4D has imprecise masks.Source: §4.1 · pp. 8–9, Table 2 · p. 9, Figure 5 · p. 9

4.2 Camera Pose Estimation

• DynamicGen consistently achieves state-of-the-art camera-pose results across Sintel and TUM-dynamics on ATE, RPE trans, and RPE rot.The evaluation aligns camera trajectories with Umeyama alignment.Source: §4.2 · p. 9, Table 3 · p. 10

4.3 Camera Intrinsics Estimation

• On Sintel, DynamicGen achieves the best focal-length estimation performance on both Absolute Focal Error and Relative Focal Error.The study compares focal-length estimation accuracy, with results summarized in Table 4.Source: §4.3 · p. 10, Table 4 · p. 10

4.4 Caption Quality Evaluation

• On 100 sampled SA-V videos, semantic-aware key-frame extraction, hierarchical prompting, caption rephrasing, and Chain-of-Thought prompting improve caption quality across four assessed dimensions.G-VEval was used for ten independent evaluations; reported dimensions were accuracy, completeness, conciseness, and relevance.Source: §4.4 · p. 10, Table 5 · p. 10

• DynamicGen automates video filtering, metric-scale geometry and motion recovery, and hierarchical semantic captioning from raw videos.The paper validates the pipeline through video-depth and camera pose/intrinsics benchmarks, diverse web-video analyses, and human/LLM-based caption evaluations.Source: §5 · p. 10
• DynamicVerse contains over 100K dynamic scenes with rich physically grounded multimodal annotations.The conclusion positions it as a comprehensive resource for advancing 4D scene understanding.Source: §5 · p. 10
• The work offers a scalable 4D data-generation methodology alongside the dataset.Source: §5 · p. 10

• The appendix includes a section on the framework’s broader impact.Source: A Appendix · p. 17
• The appendix provides dynamic bundle-adjustment details, component ablations, additional hierarchical-caption experiments, and further qualitative results.Source: A Appendix · p. 17
• It also reports DynamicGen inference speed and computational cost and includes a limitations section.Source: A Appendix · p. 17

• DynamicVerse combines long-video 4D reconstruction with window-based bundle adjustment and global optimization for dynamic-scene generation.The authors describe the resulting real-world data as a training ground for realistic, physically plausible, semantically coherent scene generation in content creation and simulation.Source: §A.1 · pp. 17–18
• Its metric-scale 4D geometry, real-world motion, and textual descriptions are presented as data for training 4D vision-language models.The discussion identifies potential applications including temporal question answering in 3D and embodied agents communicating their understanding of dynamic scenes.Source: §A.1 · p. 18
• The dataset links 800K+ instance masks and descriptive captions to evolving 3D structures and motion at physical scale for language-grounded Gaussian splatting.The authors describe potential natural-language querying, editing, and interaction with dynamic 4D representations.Source: §A.1 · p. 18
• The discussion frames DynamicVerse as a resource intended to bridge 2D understanding and 4D world modeling.Source: §A.1 · p. 18

A.2 Details of Dynamic Bundle Adjustment

• The camera-pose parameterization uses rigid transformations, with rotations represented by so(3) rotation vectors for direct optimization.Source: §A.2 · p. 18
• Static-area bundle adjustment minimizes the distance between projected 3D points and observed pixel tracks, using visibility weights and perspective projection.Camera poses also receive a temporal smoothness prior whose penalty is reduced for larger relative motion and increased for smaller relative motion.Source: §A.2 · pp. 18–19
• Dynamic bundle adjustment measures the discrepancy between dynamic 3D point tracks and pixel tracklets, optimizing a 3D point sequence for each observed tracklet.Each dynamic point denotes a point’s 3D location at a particular time.Source: §A.2 · p. 19
• The dynamic motion regularizer combines an as-rigid-as-possible prior with temporal smoothness, and both terms reduce ambiguities in ill-posed 4D structure estimation.The ARAP prior penalizes extreme deformations by preserving relative distances among neighboring tracks; the smoothness term promotes temporal smoothness.Source: §A.2 · p. 19
• The formulation avoids assuming rigid, articulated, or linear-basis motion priors for dynamic structure.Source: §A.2 · p. 19
• An optical-flow projection loss encourages global point maps to match estimated flow in confidently static regions, with global static maps and camera parameters used to update the confident dynamic mask.Source: §A.2 · p. 19

A.3 Ablation Study on Different Components for Dynamic Bundle Adjustment

• The Sintel ablation examines three components intended to improve static–dynamic decomposition and global consistency.Source: §A.3 · p. 20, Table 6 · p. 20
• Epipolar-mask filtering separates static background from dynamic foreground pixels before bundle adjustment, supporting more stable camera-pose estimation and background reconstruction.Source: §A.3 · p. 20
• VLM-based semantic motion analysis supports motion-aware keyframe extraction and provides dynamic-object masks beyond purely geometric or flow-based segmentation.Source: §A.3 · p. 20
• Optical-flow-based sliding-window global refinement addresses error accumulation and temporal drift by enforcing long-range temporal consistency.Source: §A.3 · p. 20

A.4 Additional experiments on generated hierarchical captions

• The generated object masks and labels yield substantial gains in Accuracy and volumetric Intersection over Union on time-sensitive querying against the 4D-LangSplat* baseline.The test uses the “americano” scene from HyperNeRF; Table 7 presents the comparison.Source: §A.4 · p. 20, Table 7 · p. 20
• Scene-level captions score highly on Accuracy, Completeness, Conciseness, and Relevance in the ACCR evaluation of 100 randomly sampled SA-V videos.Table 8 reports the evaluation.Source: §A.4 · pp. 20–21, Table 8 · p. 21
• Over 60.22% of camera-motion captions were rated both clear and fluent, while also receiving high conciseness scores in a human study of 88 filtered-DAVIS videos.Table 9 reports the percentage of captions meeting each quality criterion.Source: §A.4 · p. 21, Table 9 · p. 21

A.5 More qualitative results of dynamic bundle adjustment

• Figure 8 presents qualitative in-the-wild reconstruction results that demonstrate the pipeline’s generalizability and performance.The figure caption identifies the examples as reconstruction results on in-the-wild data.Source: §A.5 · p. 21, Figure 8 · p. 23

A.6 Inference Speed and Computational Cost for DynamicGen

• Table 10 breaks down average processing time and peak VRAM use by pipeline component for 23 Sintel training videos processed on NVIDIA H20 GPUs.The supplied table caption identifies it as a computational cost analysis.Source: §A.6 · p. 21, Table 10 · p. 21

A.7 Limitations

• Noise and quality variation in internet videos can compromise metric-scale geometry and motion recovery, particularly in complex, cluttered, or occluded scenes outside foundation models’ typical training distribution.The passage identifies these as limitations arising from reliance on in-the-wild internet videos.Source: §A.7 · pp. 21–22
• Processing long videos with large-scale models incurs substantial computational overhead, creating practical barriers to real-time performance and scalable deployment.Source: §A.7 · p. 22
• The dataset cannot exhaustively capture the long tail of real-world phenomena, and generalization to novel environments remains tied to the biases and capabilities of the underlying foundation models.Source: §A.7 · p. 22
• Metric-scale reconstructions from web videos can expose sensitive interiors or critical infrastructure, while distribution shift can yield plausible but erroneous geometry and dynamics that misguide downstream planners.The passage names privacy and security risks and identifies robotic or AR planners as possible downstream users.Source: §A.7 · p. 22