Source-linked AI summary
WorldScore: A Unified Evaluation Benchmark for World Generation
Haoyi Duan, Hong-Xing Yu, Sirui Chen, Li Fei-Fei, Jiajun Wu
TL;DR
World-generation research lacks a unified benchmark for evaluating diverse models on comprehensive multi-scene worlds with spatial layout control. WorldScore addresses this gap by standardizing next-scene tasks, diverse test worlds, common video outputs, and metrics for controllability, quality, and dynamics. Its evaluation finds that 3D models generally excel at static world generation, while current approaches still face challenges with dynamics, controllability, long sequences, and outdoor scenes.
Problem
Existing benchmarks mainly evaluate individual scenes and lack unified spatial-layout and camera specifications for diverse world-generation models.
Method
WorldScore decomposes world generation into next-scene tasks with image, text, camera-matrix, and layout inputs, standardizes video outputs, and aggregates metrics across three evaluation aspects.
Results
3D models generally perform better in static world generation, with WonderWorld [91] scoring 72.69 and LucidDreamer [11] 70.40 versus CogVideoX-I2V [88] at 62.15 on WorldScore-Static.
Takeaways & Limitations
WorldScore exposes distinct strengths and challenges across model categories, supporting comparison and research on controllability, extended sequences, outdoor scenes, and dynamic world generation.
Takeaways & Limitations
3D models excel in static world generation, but extending them to 4D representations and incorporating dynamics remains challenging.
Abstract
from arXiv · showhide
We introduce the WorldScore benchmark, the first unified benchmark for world generation. We decompose world generation into a sequence of next-scene generation tasks with explicit camera trajectory-based layout specifications, enabling unified evaluation of diverse approaches from 3D and 4D scene generation to video generation models. The WorldScore benchmark encompasses a curated dataset of 3,000 test examples that span diverse worlds: static and dynamic, indoor and outdoor, photorealistic and stylized. The WorldScore metrics evaluate generated worlds through three key aspects: controllability, quality, and dynamics. Through extensive evaluation of 19 representative models, including both open-source and closed-source ones, we reveal key insights and challenges for each category of models. Our dataset, evaluation code, and leaderboard can be found at https://haoyi-duan.github.io/WorldScore/
1. Introduction
WorldScore addresses the lack of a unified benchmark for generating multi-scene worlds with spatial layout control across diverse model types. It standardizes next-scene tasks, common video outputs, diverse test worlds, and metrics spanning controllability, quality, and dynamics.
- Motivation and gap: Existing benchmarks mainly evaluate single-scene video quality, without explicit spatial layout control or compatibility with camera-conditioned 3D/4D methods.VBench uses curated text-to-video prompts, while many 3D/4D methods require camera specifications or reference images.
- Benchmark design: WorldScore decomposes world generation into next-scene tasks defined by current scene, next scene, and layout, using image, text, camera matrices, and textual layout descriptions.The design supports 3D, 4D, text-to-video, and image-to-video models.
- Benchmark design: All evaluated approaches produce rendered or generated videos, enabling direct comparison across 3D, 4D, image-to-video, and text-to-video methods.The benchmark uses a common output format despite differing model inputs and generation paradigms.
- Evaluation metrics: WorldScore aggregates 10 metrics across controllability, quality, and dynamics to measure control adherence, fidelity and consistency, and accurate stable motion.These three aspects define the benchmark’s overall evaluation score.
- Dataset: The dataset contains 3,000 test examples spanning static and dynamic indoor/outdoor worlds with photorealistic and stylized visual domains.It includes five indoor categories, five outdoor categories, five dynamics types, varying sequence lengths, and stylized counterparts.
- Experiments: The benchmark evaluates diverse model families and reports insights from 18 open-source and 2 closed-source models about current world-generation challenges.The evaluated set includes image-to-video, text-to-video, 3D scene-generation, and 4D-generation models.
2. Related Work
Prior benchmarks largely assess single-scene video quality and do not provide layout control or camera specifications compatible with 3D/4D generation. WorldScore instead unifies video, 3D, and 4D approaches through shared world specifications and video-based evaluation.
- Video generation benchmarks: Existing video benchmarks such as VBench and WorldModelBench focus on single-scene quality without layout control or multi-scene generation.Their designs therefore target video models rather than unified world-generation evaluation.
- Unified evaluation: The benchmark standardizes outputs as videos and evaluates them for controllability, quality, and dynamics across different methodological paradigms.Its world specification combines spatial layout and semantic content to instruct the evaluated models.
- Video generation models: WorldScore accommodates both image and text conditioning strategies within one unified benchmark design.This extends evaluation beyond predominantly text-to-video settings discussed in prior video-generation work.
- 3D and 4D scene generation: WorldScore includes 3D and 4D generation methods, although many existing 4D methods focus on object-level rather than scene-level generation.The benchmark includes 4D-fy because it is openly accessible.
3. The WorldScore Benchmark
WorldScore unifies world-generation evaluation by specifying next-scene tasks with camera-based layouts, diverse curated data, and standardized video outputs. It evaluates controllability, quality, and dynamics across static and dynamic worlds and multiple visual domains.
- World Specification: World generation is decomposed into next-scene tasks specified by the current scene, next-scene prompt, and layout.The current scene includes an image and text prompt; the layout includes a camera trajectory and camera-movement description.
- World Specification: Static tasks assess controllability and quality through scene sequences, whereas dynamic tasks assess in-scene motion under a fixed camera position.Static layouts specify large camera movements; dynamic prompts retain scene content while changing motion.
- Dataset Curation: The benchmark contains 3000 world specifications, including 2000 static and 1000 dynamic examples, with small and large worlds defined by one or three new scenes.Large worlds comprise three next-scene prompts generated autoregressively from prior scene descriptions.
- Dataset Curation: The dataset covers static indoor and outdoor scenes, dynamic motion types, photorealistic worlds, and stylized counterparts.Static scenes use 10 categories, dynamic scenes use 5 motion categories, and each photorealistic example receives a randomly selected stylized counterpart.
- The WorldScore Metrics: WorldScore aggregates metrics for controllability, quality, and dynamics, including camera error, object controllability, text alignment, motion magnitude, and motion smoothness.Camera error combines scale-invariant rotation and translation errors using their geometric mean; motion smoothness is evaluated against interpolated frames.
4. Results
WorldScore evaluates diverse world-generation models across controllability, quality, dynamics, and subdomains, revealing complementary strengths and persistent challenges in camera control, motion, sequence length, and scene type.
- WorldScore metrics align with human preference and remain robust across video resolutions and aspect ratios.
- WonderWorld (72.69) and LucidDreamer (70.40) lead WorldScore-Static, outperforming CogVideoX-I2V (62.15).3D models benefit from camera controllability, content alignment, and 3D and photometric consistency, but do not generate dynamic worlds.
- CogVideoX-T2V achieves the best video-model camera controllability score at 40.22, below every evaluated 3D/4D model.The paper identifies camera conditioning as a possible direction for improving video-based static world generation.
- CogVideoX-I2V scores 62.15 on WorldScore-Static and 59.12 on WorldScore-Dynamic, exceeding both compared closed-source models on these scores.Its strengths and weaknesses differ by aspect: stronger camera controllability but weaker object controllability and content alignment.
- Larger motion often trades off against motion smoothness, while motion magnitude correlates weakly with motion-placement accuracy.Large motion can include unintended camera or irrelevant motion, motivating more robust motion modeling.
- Video models struggle with long sequences and outdoor scenes, while T2V models generally score higher in controllability and motion magnitude and I2V models in quality.The outdoor performance gap between video and 3D models is smaller for indoor scenes.
5. Conclusion
WorldScore identifies distinct limitations across world-generation approaches: 3D models struggle with dynamics, while video models face controllability, long-sequence, and outdoor-scene challenges.
- 3D models excel in static world generation but remain challenged when extended to 4D representations and dynamic scenes.
- Video models face challenges in controllability, long-sequence generation, and outdoor-scene generation.
- The paper points to bridging 3D and 4D representations, improving controllability, and designing architectures for extended scene sequences.
A. Additional Details on World Specification
World specification preprocessing standardizes heterogeneous model inputs while preserving the camera and visual controls needed for unified evaluation.
- WorldScore evaluates 3D, 4D, and video-generation models despite their distinct input requirements.3D/4D models accept precise camera poses, while T2V and I2V models differ in whether they use text, image, or both.
- The preprocessing pipeline standardizes the inputs used by different models.
- The current-scene image is center-cropped and resized for each model, serving as a style reference and an I2V input.T2V models are treated as I2V models that ignore image-based control signals.
- The layout specification includes a predefined camera trajectory used as ground truth for camera controllability.
B. Additional Details on Dataset Curation
The dataset curation pipeline filters starting images for visual quality and suitable perspectives, adds stylized counterparts, and generates static or dynamic next-scene descriptions.
- Dataset construction: Starting images are sourced from existing scene datasets and Unsplash, then filtered because many contain redundancy, unusual viewpoints, or narrow-angle perspectives.
- Dataset construction: CLIP-IQA and CLIP Aesthetic predictors remove poor-quality images, while Perspective Fields filter extreme roll, pitch, and narrow fields of view.
- Visual styles: Each photorealistic image receives a stylized counterpart generated with a text-to-image model using a predefined visual-style set.The predefined set includes seven styles, including anime, cyberpunk, Chinese ink painting, ukiyo-e, impressionism, and postimpressionism.
- Scene descriptions: For static worlds, GPT-4o autoregressively generates descriptions of likely next scenes, prominent entities, and brief scene prompts.
- Benchmark metadata: Table S1 records model details, including average generation time, GPU setting, output resolution or aspect ratio, and precise-camera-pose support.For 4D-fy, generation takes about 20 hours per instance, so iteration steps are reduced to save time.
- Dataset construction: The filtering process includes manual inspection to remove infeasible starting scenes, such as a mid-air city image.
- Scene descriptions: For dynamic worlds, GPT-4o identifies prominent objects capable of motion and describes how they move.
C.1. Camera Controllability
Camera controllability is measured by comparing generated-video camera poses with a ground-truth trajectory. The metric combines rotation and translation errors and averages the resulting error across all generated frames and videos.
- Camera error formulation: Camera error combines scale-invariant rotation and translation errors using their geometric mean.The errors are computed with respect to the ground-truth camera trajectory.
- Pose error components: Rotation error measures angular deviation between ground-truth and estimated camera rotations in degrees.The rotation matrices represent ground-truth and estimated camera orientations.
- Pose error components: Translation error measures scale-invariant Euclidean distance between ground-truth and estimated camera positions.The scale factor is chosen by least-square alignment of the camera positions.
- Aggregation: The final camera controllability error averages camera error across every frame of all generated videos.This aggregates frame-wise deviations into one model-level camera controllability error.
- Empirical bounds: A fixed-camera sequence supplies the empirical high-error baseline for normalization and penalizes outputs that fail to exhibit camera movement.The empirical minimum is zero, which is also the theoretical lower bound.
D. Validation with Human Preference
WorldScore’s metrics are validated against human preferences and remain robust across changes in video resolution and aspect ratio. Human agreement also guides the construction of the subjective-quality metric.
- Human preference study: 400 participants compared video pairs in a 2-alternative forced-choice human preference study.Pairs were sampled from videos generated by CogVideoX-I2V, VideoCrafter1-I2V, DynamiCrafter, WonderJourney, and InvisibleStitch.
- Human preference study: The study used one question asking which video had higher quality, because raters struggled to distinguish aesthetic and perceptual quality dimensions.The preliminary study found a very high correlation between aesthetic and perceptual quality.
- Subjective quality: The arithmetic mean of CLIP-IQA+ and CLIP Aesthetic achieved the highest agreement and was selected for subjective quality.The selected combination is used to compute WorldScore’s subjective-quality metric.
- Metric validation: Higher WorldScore metric scores consistently corresponded to higher human preference when comparing the 90-over-60 and 60-over-30 score buckets.The comparison used score buckets centered at 90, 60, and 30.
- Robustness analysis: Metric differences across high- and low-resolution, cropped-and-resized videos were all ≤0.83, indicating robustness to resolution and aspect-ratio changes.The comparison used EasyAnimate videos at 1344×768 and center-cropped, resized versions at 256×256.
E. Further Visualization
Visualizations show how WorldScore separates generated results and exposes distinct strengths and weaknesses across video and 3D scene generation models. Video models struggle with camera control, while 3D models maintain static consistency but may sacrifice subjective quality.
- Metric visualizations: WorldScore decomposes world-generation capability into 10 independent dimensions, with examples showing how metrics distinguish good and bad generated results.Typical examples are presented for controllability, quality, and dynamics metrics.
- Dynamic-world results: Video-generation models exhibit very low camera controllability, revealing difficulty following predefined camera trajectories.The observation includes Hailuo, VideoCrafter1-I2V, EasyAnimate, and T2V-Turbo.
- Dynamic-world results: Larger motion often comes at the cost of lower motion smoothness, indicating a trade-off between movement magnitude and temporal stability.This trade-off appears in comparisons of motion smoothness and motion magnitude across methods.
- Static-world results: 3D scene models such as WonderWorld achieve high camera controllability, 3D consistency, and photometric consistency in static-world evaluation.The result is reported in the WorldScore-Static visualization.
- Static-world results: These 3D scene models may nevertheless struggle in subjective quality despite geometric and photometric coherence.The visualization suggests a separation between maintaining consistency and producing visually appealing results.