PaperlayerView exact revision on arXiv

Source-linked AI summary

ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation

Shenghai Yuan, Jinfa Huang, Yongqi Xu, Yaoyang Liu, Shaofeng Zhang, Yujun Shi, Ruijie Zhu, Xinhua Cheng, Jiebo Luo, Li Yuan

arXiv:2406.18522v2cs.CVcs.CL

TL;DR

T2V benchmarks largely overlook time-lapse metamorphic amplitude and temporal coherence, limiting evaluation of physically rich video generation. ChronoMagic-Bench addresses this gap with a categorized benchmark, new automatic metrics, model evaluations, and the ChronoMagic-Pro dataset. The paper also reports that its metrics remain imperfectly aligned with human preferences.

  • Problem

    Existing T2V benchmarks primarily assess general videos, visual quality, and textual relevance, overlooking metamorphic amplitude and temporal coherence in time-lapse generation.

    Source: §1 · p. 2

  • Method

    The paper introduces ChronoMagic-Bench with 1,649 categorized prompts and reference videos, MTScore and CHScore, evaluations of T2V models, and the ChronoMagic-Pro dataset.

    Source: §1 · p. 2

  • Results

    ChronoMagic-Bench provides insights into the strengths and weaknesses of various T2V models across time-lapse categories.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The benchmark and dataset provide a framework and resources for evaluating and advancing time-lapse T2V generation with greater physical prior content.

    Source: §1 · p. 2, §6 · p. 10

    The data is primarily sourced from YouTube with variable quality and academic-use licensing constraints, and MTScore and CHScore remain imperfectly aligned with human preferences.

    Source: §7 · p. 10

Abstract

from arXiv · show

We propose a novel text-to-video (T2V) generation benchmark, ChronoMagic-Bench, to evaluate the temporal and metamorphic capabilities of the T2V models (e.g. Sora and Lumiere) in time-lapse video generation. In contrast to existing benchmarks that focus on visual quality and textual relevance of generated videos, ChronoMagic-Bench focuses on the model's ability to generate time-lapse videos with significant metamorphic amplitude and temporal coherence. The benchmark probes T2V models for their physics, biology, and chemistry capabilities, in a free-form text query. For these purposes, ChronoMagic-Bench introduces 1,649 prompts and real-world videos as references, categorized into four major types of time-lapse videos: biological, human-created, meteorological, and physical phenomena, which are further divided into 75 subcategories. This categorization comprehensively evaluates the model's capacity to handle diverse and complex transformations. To accurately align human preference with the benchmark, we introduce two new automatic metrics, MTScore and CHScore, to evaluate the videos' metamorphic attributes and temporal coherence. MTScore measures the metamorphic amplitude, reflecting the degree of change over time, while CHScore assesses the temporal coherence, ensuring the generated videos maintain logical progression and continuity. Based on ChronoMagic-Bench, we conduct comprehensive manual evaluations of ten representative T2V models, revealing their strengths and weaknesses across different categories of prompts, and providing a thorough evaluation framework that addresses current gaps in video generation research. Moreover, we create a large-scale ChronoMagic-Pro dataset, containing 460k high-quality pairs of 720p time-lapse videos and detailed captions ensuring high physical pertinence and large metamorphic amplitude. [Homepage](https://pku-yuangroup.github.io/ChronoMagic-Bench/).

1 Introduction

ChronoMagic-Bench addresses gaps in T2V evaluation by targeting time-lapse videos, metamorphic amplitude, and temporal coherence, while introducing aligned metrics and a large-scale time-lapse dataset.

Motivation

  • Existing T2V benchmarks emphasize visual quality and text relevance, overlooking metamorphic amplitude and temporal coherence in time-lapse generation.These omissions limit evaluation of the physical priors represented in generated videos.Source: §1 · p. 2

Benchmark

  • ChronoMagic-Bench evaluates T2V models across four major categories and 75 subcategories using 1,649 prompts with corresponding reference videos.The categories are biological, human creation, meteorological, and physical phenomena.Source: §1 · p. 2

Metrics

  • MTScore and CHScore assess metamorphic amplitude and temporal coherence, respectively, addressing dimensions neglected by existing evaluation metrics.The metrics are designed to align better with human judgment.Source: §1 · p. 2

Evaluation

  • ChronoMagic-Bench evaluations provide insights into the strengths and weaknesses of various T2V models for model selection.The benchmark supports comparisons across diverse time-lapse prompts and categories.Source: §1 · p. 2

Dataset

  • ChronoMagic-Pro contains 460k high-quality 720p time-lapse videos with detailed captions and emphasizes strong physical characteristics, persistence, and variability.The dataset was curated with an automatic collection framework intended to maintain video purity and improve annotation quality.Source: §1 · p. 2

2 Related Work

Prior T2V research relies on large-scale web-scraped video-text datasets and metrics centered on visual quality and text relevance.

Automatic Metrics

  • Existing T2V benchmarks commonly use FID, FVD, CLIPScore, and improved variants to assess visual quality and text relevance.Examples include UMT-FVD and UMTScore, while BLIP can replace CLIPScore's feature extractor for relevance evaluation.Source: §2 · p. 3

Datasets

  • Large-scale, high-quality video-text pairs are treated as essential for training generation models, with WebVid-10M cited as a common web-scraped dataset.Dataset scale and the physical knowledge represented in videos are described as important for learning real-world representations.Source: §2 · p. 3

3 ChronoMagic-Bench

ChronoMagic-Bench constructs a diverse time-lapse benchmark and introduces automatic measures for metamorphic amplitude and temporal coherence, addressing gaps in existing evaluation.

3.1 Benchmark Construction

  • ChronoMagic-Bench uses 1,649 videos across four major categories and 75 subcategories to evaluate time-lapse generation comprehensively.The categories cover biological, human-created, meteorological, and physical phenomena.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 New Automatic Metrics

  • MTScore estimates metamorphic amplitude from retrieval probabilities, balancing inexpensive coarse-grained assessment with more costly fine-grained evaluation.The coarse-grained score integrates probabilities for metamorphic and general retrieval sentences, while GPT-4o provides fine-grained scoring.Source: §3.2 · p. 5
  • CHScore measures temporal coherence using frame-to-frame tracking instability, including missed-point variation, cutting ratio, and consecutive changes.The score combines normalized tracking-based quantities into a coherence measure.Source: §3.2 · pp. 5–6
  • Together with visual-quality and text-relevance metrics, MTScore and CHScore enable four-dimensional evaluation of T2V models.The framework assesses visual quality, textual relevance, metamorphic amplitude, and temporal coherence.Source: §3.2 · p. 5, §3.3 · p. 6

4 ChronoMagic-Pro

ChronoMagic-Pro is a large-scale time-lapse video dataset designed to provide diverse, physically informative training data for T2V models.

4 ChronoMagic-Pro

  • ChronoMagic-Pro contains 460K semantically consistent single-scene time-lapse videos collected from 66,226 original videos.The dataset is constructed through multi-aspect curation and video splitting.Source: §4 · p. 6
  • Most ChronoMagic-Pro videos are high quality: 97% use 720P resolution, 73% receive aesthetic scores from 4 to 6, and 95% of captions exceed 100 words.The dataset spans 75 categories with varied durations, including 27.1% of videos longer than 60 seconds.Source: §4 · p. 6, Figure 4 · p. 7
  • The dataset statistics summarize variation across categories, clip durations, caption lengths, and resolutions, with most videos in 720P.These dimensions characterize the dataset’s coverage and content quality.Source: §4 · p. 6, Figure 4 · p. 7

5 Experiments

Experiments evaluate open- and closed-source T2V models on time-lapse generation using four dimensions: visual quality, text relevance, metamorphic amplitude, and temporal coherence. Results show that most models struggle with substantial temporal transformations, while MagicTime, CogVideoX, and some closed-source systems perform better on metamorphic generation.

Evaluation Setups

  • The benchmark evaluates visual quality, text relevance, metamorphic amplitude, and temporal coherence using four automated metrics across 1,649 prompts and 4,947 videos per model.The evaluated set includes fourteen open-source models and four closed-source models.Source: §5.2 · p. 7, §5.1 · p. 7

Qualitative Evaluation

  • Most models fail to generate time-lapse videos with significant motion and temporal spans, while MagicTime and CogVideoX follow metamorphic prompts more successfully.The evaluated prompts include house construction, flower blooming, bread baking, and ice melting.Source: §5.3 · p. 8, Figure 5 · p. 8
  • MagicTime and CogVideoX achieve the highest metamorphic scores, whereas ZeroScope has the best coherence and T2V-Zero the worst coherence.MTScore and GPT4o-MTScore broadly agree, while U-Net-based models show higher average metamorphic amplitude than DiT-based models.Source: §5.3 · p. 8

Human Preference

  • MTScore, CHScore, and GPT4o-MTScore align with human perception, while UMTScore has the lowest Kendall and Spearman correlations for time-lapse text relevance.The human study sampled videos from 16 prompts and collected votes from 212 participants.Source: §5.3 · p. 9, Figure 6 · p. 9

Extended Analysis of Closed-Source Models

  • Closed-source models generally surpass open-source models in visual quality, while Dream Machine performs better in metamorphic amplitude and Pika-2.0 has the worst text relevance.Most models still obtain low metamorphic scores on complex state changes, and UMT-FVD and UMTScore do not accurately reflect some open-versus-closed-source differences.Source: §5.3 · p. 9, Table 4 · p. 9

Guideline for Model Selection

  • ChronoMagic-Bench recommends considering metamorphic amplitude, temporal coherence, and visual quality jointly because strong single-frame quality can coexist with flickering or incomplete physical processes.These issues are especially evident for T2V-Zero and OpenSora 1.2.Source: §5.3 · p. 10

6 Conclusion

The paper introduces ChronoMagic-Bench as a benchmark for time-lapse video generation and adds metrics and data intended to support broader evaluation and research. It uses the benchmark to analyze T2V model strengths and weaknesses.

Conclusion

  • ChronoMagic-Bench evaluates time-lapse generation beyond standard video quality by measuring metamorphic amplitude and temporal coherence.The paper also introduces MTScore and CHScore, a broad model evaluation, and the ChronoMagic-Pro dataset.Source: §6 · p. 10

7 Limitations and Future Work

The benchmark is limited by variable source-video quality and licensing constraints, and its automatic metrics still do not fully match human preferences. Future work targets closer alignment between metrics and human judgment.

Limitations

  • Most benchmark videos come from YouTube, requiring extensive quality filtering, while licensing conditions limit many videos to academic research use.The passage specifies that most videos are licensed under CC BY 4.0.Source: §7 · p. 10

Future Work

  • MTScore and CHScore still show a clear gap from human preferences, motivating future work on more human-aligned automatic evaluation.Source: §7 · p. 10

B More details about Automatic Metrics

The benchmark introduces MTScore for metamorphic amplitude and CHScore for temporal coherence, combining retrieval-based assessment with tracking-based continuity measures aligned with human perception.

B.1 Construction of retrieval sentences for Metamorphic Score

  • MTScore uses multiple retrieval sentences to distinguish metamorphic time-lapse videos from general videos and improve robustness across linguistic descriptions.The retrieval probabilities are integrated into a coarse-grained metamorphic score.Source: §B.1 · p. 18

B.2 Further Description of Temporal Coherence Score

  • CHScore estimates temporal coherence from tracking-point visibility and frame-to-frame changes in missing points.The method processes videos with a pretrained tracking model using grid size G and threshold T.Source: §B.2 · pp. 18–19
  • CHScore combines missed-point rate, variation, cut ratio, instability counts, and maximum continuity breaks into a single coherence score.The components capture global tracking loss, frame-to-frame stability, editing requirements, frequent large disruptions, and the most severe continuity break.Source: §B.2 · p. 19
  • The tracking procedure excludes points disappearing in the movement direction so temporally coherent camera or object motion does not unfairly reduce CHScore.Movement direction is estimated from tracking points across the video before filtering disappearances.Source: §B.3 · p. 19, §B.2 · p. 19

B.3 Alignment with Human Perception

  • Qualitative score examples show that higher MTScore and CHScore values correspond to greater metamorphic amplitude and temporal coherence, consistently with human perception.Figure 8 provides visual references for different score magnitudes.Source: Figure 8 · p. 20, §B.4 · p. 20

C.1 Data Preprocessing

ChronoMagic-Pro is constructed by filtering and segmenting online videos into semantically consistent clips, then generating captions from uniformly sampled keyframes and aggregating them with temporal positions.

Video preprocessing

  • The preprocessing pipeline filters low-quality online videos and removes transition-heavy content before constructing time-lapse clips.The filtering retains 66,226 original videos after using metadata such as views, comments, and likes.Source: §C.1 · p. 20, Figure 9 · p. 21
  • OpenCV splits videos into clips using frame differences, while ImageBind recombines erroneously split clips into semantically consistent single-scene sequences.Adjacent clips are recombined when their feature-space difference falls below threshold η.Source: Figure 9 · p. 21, §C.1 · p. 21

Caption generation

  • Keyframe-based captioning is preferred because direct video captioning by 8B, 13B, and 34B models produced severe hallucinations on time-lapse content.The captioning ablation reports that inputting keyframes is more effective than directly inputting the video.Source: §C.2 · p. 21, Figure 10 · p. 22
  • ChronoMagic-Pro captions emphasize progressive transformations across biological, human-created, meteorological, and physical categories.Frequent terms include “change,” “transition,” and “progressing,” alongside examples such as flowers, buildings, eclipses, and explosions.Source: §C.3 · pp. 21–22, Figure 11 · p. 22

Dataset composition

  • The dataset combines compressed and uncompressed time-lapse videos, retains 13% of clips with low aesthetic scores for diversity, and deduplicates YouTube data by video ID.Both time-lapse durations are included, with 60s+ videos comprising 27% of ChronoMagic-Pro.Source: §C.5 · pp. 22–23

D.3 Further Verification Experiment on ChronoMagic-Pro

Fine-tuning OpenSoraPlan with ChronoMagic-Pro increases metamorphic state changes while preserving visual quality, text relevance, and coherence, although the Magic Training Strategy performs better than simple fine-tuning.

Fine-tuning results

  • Fine-tuning OpenSoraPlan with ChronoMagic-Pro enables substantial state changes, including blooming flowers and city traffic, beyond lighting or camera motion.The qualitative comparison reports changes in object state after fine-tuning.Source: §D.3 · pp. 25–26
  • ChronoMagic-Pro fine-tuning preserves visual quality, text relevance, and temporal coherence while extending the range of generated transformations.The comparison indicates no loss in these three properties after fine-tuning.Source: §D.3 · p. 26
  • Simple fine-tuning is less effective than the Magic Training Strategy for producing these time-lapse capabilities.The authors note that the strategy was designed for U-Net models and may be less effective for DiT-based models.Source: §D.3 · p. 26, Figure 13 · p. 26

D.5 More Analysis of Closed-Source Models

Closed-source models generally generate simple time-lapse motion rather than complex state transformations, with differences across models in metamorphic amplitude, visual quality, coherence, and prompt adherence.

Metamorphic amplitude

  • Most closed-source models produce simple changes such as traffic flow, while none generate complex transformations such as plant growth or building construction.Only Dream Machine generates a moderately challenging night-to-day transformation in the reported comparison.Source: §D.5 · pp. 26–27, Figure 14 · p. 27, Figure 15 · p. 28

Temporal coherence

  • Closed-source models show comparable temporal coherence, with only minor visible differences.The comparison reports no large separation among the evaluated models on this dimension.Source: §D.5 · pp. 27–28, Figure 16 · p. 28

Visual quality

  • Dream Machine and KeLing achieve better visual quality than U-Net-based methods, including more realistic plants, more saturated skies, and clearer traffic flow.The reported advantage concerns visual realism rather than metamorphic amplitude.Source: §D.5 · p. 28

Text relevance

  • All models generally follow the requested theme, except Pika-2.0, which reverses a day-to-night prompt into night-to-day.This is reported as a text-relevance error rather than a general failure to follow the prompt.Source: §D.5 · p. 28

D.6 Additional Details of Human Evaluation

Human evaluation rates visual quality, text relevance, metamorphic amplitude, and coherence on five-point scales, with procedures designed to improve consistency and reduce bias. Additional analysis reports that MTScore and CHScore align with human perception on disaggregated videos.

Human Evaluation Questionnaire

  • Human evaluators rated Visual Quality, Text Relevance, Metamorphic Amplitude, and Coherence using five-point scales and scoring guidelines.The questionnaire was established following prior evaluation methodologies and aimed to ensure consistent user selections.Source: §D.6 · p. 28, Figure 17 · p. 29

Response Processing

  • Responses were filtered through duplicate prevention, completion-time checks, randomized presentation, manual-verification requirements, and removal of questionnaires dominated by extreme ratings.Responses completed in under 10 minutes were excluded, and questionnaires with more than 50% combined one-point and five-point ratings were discarded.Source: §D.6 · p. 29

Metric Validation

  • MTScore and CHScore were consistent with human perception in an analysis of 32 randomly selected videos represented as individual data points.The comparison was presented in Figure 16 as additional evaluation beyond the model-aggregated analysis in Figure 6.Source: §D.6 · p. 29

Benchmark Content

  • The benchmark’s search terms span biological, human-created, meteorological, and physical time-lapse content, including plant growth, animal behavior, construction, traffic, and day–night transitions.The supplied examples cover categories such as leaves, seeds, blooming, mushrooms, buildings, fireworks, cities, roads, and changing illumination.Source: E More details about 75 subcategories… · p. 30

F Ethics Statement

The paper addresses licensing, privacy, environmental costs, misuse risks, and dataset bias associated with ChronoMagic-Bench and ChronoMagic-Pro. It describes mitigation measures while identifying energy consumption, deceptive content, and English-only captions as important boundaries.

Data Licensing and Privacy

  • ChronoMagic-Bench uses videos from Pexels, MixKit, PixaBay, and YouTube, while ChronoMagic-Pro uses YouTube videos with licensing information and attribution metadata.The paper states that CC0 content permits unrestricted use and that YouTube CC BY 4.0 videos retain video IDs and author information.Source: F Ethics Statement · p. 33

Environmental Impact

  • Training and evaluation require energy-intensive computation, including H100 experiments, raising concerns about carbon emissions and possible climate impacts from wider adoption.The paper recommends optimizing algorithms to reduce energy consumption.Source: F Ethics Statement · pp. 33–34

False Content

  • Fine-tuned models can generate realistic metamorphic and general videos that may be misused for deceptive content and misinformation, motivating authenticity mechanisms and legal and ethical frameworks.The concern is especially emphasized for fast-paced, influential platforms such as social media.Source: F Ethics Statement · p. 34

Dataset Bias

  • English-only captions may limit interpretation and generation across cultural and non-English settings, creating a dataset-bias boundary that future construction should address with multilingual support.The paper links this limitation to potential language inequalities affecting non-English-speaking users.Source: F Ethics Statement · p. 34

Mitigation Measures

  • The datasets and benchmark are released under CC-BY-4.0, code under an Apache license, and data are labeled for academic research to reduce misuse.The authors also provide a contact route for YouTube authors to request removal of invalid videos.Source: F Ethics Statement · p. 34

Related papers

Loading 2406.18522v2…