PaperlayerView exact revision on arXiv

Source-linked AI summary

TALC: Time-Aligned Captions for Multi-Scene Text-to-Video Generation

Hritik Bansal, Yonatan Bitton, Michal Yarom, Idan Szpektor, Aditya Grover, Kai-Wei Chang

arXiv:2405.04682v4cs.CVcs.AIcs.LG

TL;DR

Many pretrained T2V models produce single-scene clips, limiting their fit to descriptions of ordered multi-scene events. TALC aligns video segments with scene-specific captions and can be paired with multi-scene finetuning; human evaluation reports a 29% relative overall-score gain over FreeNoise for the finetuned model.

  • Problem

    Many pretrained T2V models are trained on single-scene video-text data, while real-world scenarios can call for temporally ordered scenes with consistent entities and backgrounds.

    Source: §1 · p. 1, §3.1 · p. 4

  • Method

    TALC aligns each video segment's representations with its scene-description embeddings, and the pretrained model can be finetuned on curated multi-scene video-text data using this framework.

    Source: §1 · pp. 2–3

  • Results

    29% relative gain over FreeNoise in overall score was reported for the TALC-finetuned model in human evaluation.

    Source: §1 · p. 3

  • Takeaways & Limitations

    TALC is designed to be easily adapted into any diffusion-based T2V model.

    Source: §6 · p. 9

    The setup expects object appearance to remain consistent throughout the video unless a change is specified in the description.

    Source: §2.3 · p. 4

Abstract

from arXiv · show

Most of these text-to-video (T2V) generative models often produce single-scene video clips that depict an entity performing a particular action (e.g., 'a red panda climbing a tree'). However, it is pertinent to generate multi-scene videos since they are ubiquitous in the real-world (e.g., 'a red panda climbing a tree' followed by 'the red panda sleeps on the top of the tree'). To generate multi-scene videos from the pretrained T2V model, we introduce a simple and effective Time-Aligned Captions (TALC) framework. Specifically, we enhance the text-conditioning mechanism in the T2V architecture to recognize the temporal alignment between the video scenes and scene descriptions. For instance, we condition the visual features of the earlier and later scenes of the generated video with the representations of the first scene description (e.g., 'a red panda climbing a tree') and second scene description (e.g., 'the red panda sleeps on the top of the tree'), respectively. As a result, we show that the T2V model can generate multi-scene videos that adhere to the multi-scene text descriptions and be visually consistent (e.g., entity and background). Further, we finetune the pretrained T2V model with multi-scene video-text data using the TALC framework. We show that the TALC-finetuned model outperforms the baseline by achieving a relative gain of 29% in the overall score, which averages visual consistency and text adherence using human evaluation.

1 Introduction

Most open T2V models are trained on single-scene data, while real-world prompts can require temporally ordered scenes with consistent entities and backgrounds. TALC aligns scene descriptions to corresponding video segments; human evaluation reports gains over FreeNoise, larger with multi-scene finetuning.

1 Introduction

  • Open T2V models are often trained on single-scene data, whereas real-world scenarios can require ordered scenes with consistent entities and backgrounds.Source: §1 · p. 1
  • TALC aligns earlier and later video representations with their corresponding scene descriptions while temporal modules share information across frames to maintain visual consistency.The scene descriptions are provided to the diffusion model all at once.Source: §1 · p. 2
  • TALC offers a straightforward alternative to FreeNoise [38], whose motion-injection strategy involves multiple hyperparameters, and can be used with real-world multi-scene finetuning.Source: §1 · pp. 2–3
  • The authors propose curating multi-scene video-text data and finetuning a pretrained T2V model with TALC.Source: §1 · p. 3
  • 29% relative gain over FreeNoise in overall score for TALC-FT, versus 21% for training-free TALC; the score averages visual consistency and text adherence.The score includes visual consistency and text adherence.Source: §1 · p. 3, Figure 2 · p. 3

2 Preliminaries

The preliminaries describe diffusion-based text-to-video generation and its text-conditioning mechanism, then define multi-scene generation as temporally ordered, scene-aligned video with consistent object appearance.

2.1 Diffusion Models for Text-to-Video Generation

  • Diffusion-based T2V modeling learns video distributions conditioned on text, using iterative denoising of raw-video or encoded latent representations.The denoiser predicts a target vector from a noise-corrupted video representation and caption embeddings.Source: §2.1 · p. 3

2.2 Text Conditioning Mechanism

  • Text embeddings are repeated across video frames and used as cross-attention keys and values so visual features capture language semantics.Temporal blocks aggregate features across frames and are useful for maintaining visual consistency.Source: §2.2 · p. 4

2.3 Multi-Scene Text-to-Video Generation

  • Multi-scene generation orders events by scene-description index, aligns each video part with its corresponding description, and preserves object appearance unless a change is specified.Source: §2.3 · p. 4

3 TALC: Time-Aligned Captions for Multi-Scene T2V Generation

Existing pretrained T2V models often generate single-event scenes, while TALC aligns video segments with their corresponding scene descriptions. The section also describes a pipeline for creating multi-scene training data from real-world videos.

3.1 Approach

  • Pretrained T2V models tend to synthesize single scenes because training videos may lack multiple events or captions may omit events beyond the major one.Source: §3.1 · p. 4
  • TALC assigns each video segment the language features of its corresponding scene description, aligning visual representations with the described events.The framework allocates L/n frames per scene segment and concatenates the segments temporally.Source: §3.1 · pp. 4–5, Figure 3 · p. 5
  • TALC applies during training and inference, while denoiser temporal modules support visual consistency; the framework can be used with a base T2V model without further training.Source: §3.1 · p. 5

3.2 Multi-Scene Video-Text Data Generation

  • The data pipeline detects scene cuts, samples each scene's middle frame, and uses Gemini-Pro-Vision with the full-video caption to generate coherent per-scene captions.The authors note that higher-quality multi-scene datasets would enhance TALC performance.Source: §3.2 · p. 5

4 Evaluation

The evaluation measures visual consistency and text adherence, with human visual-quality checks, across four-scene scripted prompts and real-video captions. Scores average performance across scene counts, with entity consistency assessed only where characters remain consistent across scenes.

4.1 Metrics

  • Visual consistency covers entity and background stability, while text adherence checks accurate events and actions in temporal order; overall score averages consistency and adherence, alongside human visual-quality checks for flimsy frames, shakiness, and artifacts.Human and automatic evaluators assess consistency and adherence; visual quality is assessed by humans.Source: §4.1 · pp. 5–6

4.2 Task Prompts

  • Prompts cover four-scene scripts with one animal across contexts (S1), changing animals in one context (S2), and 100 open-ended real-video captions (S3).S1 and S3 test entity and background consistency; S2 tests background consistency while characters change.Source: §4.2 · p. 6

4.3 Evaluation Setup

  • Scores average results across two, three, and four scenes; entity consistency uses S1 and S3, while background consistency and text adherence use all prompts.Visual consistency averages entity and background consistency scores.Source: §4.3 · p. 6

5 Experiments

Across the experiments, TALC improves multi-scene generation, with fine-tuned ModelScope achieving the highest reported automatic overall score. Human and automatic evaluations compare TALC with caption- and video-merging baselines, while qualitative examples examine motion, consistency, and transitions.

5.1 Text-to-Video Generative Models

  • The experiments use ModelScope for human evaluation and ModelScope and Lumiere for automatic evaluation; TALC can also be applied to base models at inference without multi-scene fine-tuning.ModelScope is fine-tuned on multi-scene video-text data with TALC and compared with fine-tuning that merges scene captions; the fine-tuning setup uses up to four scenes.Source: §5.1 · p. 7

5.2 Baselines

  • The baselines merge all scene captions into one prompt, merge separately generated scene videos, or use FreeNoise’s noise rescheduling, temporal attention, and motion injection.Caption merging loses temporal boundaries; independently generated scene videos lack access to the other scene descriptions.Source: §5.2 · p. 7

5.3 Human Evaluation

  • Human evaluation finds base and fine-tuned TALC outperform FreeNoise by 21% and 27% on overall score; fine-tuned TALC exceeds base merge-caption and merge-video methods by 12 and 15.5 points.Merge captions with the base model score highest on visual consistency at 96.5 points, while base merge videos score lowest; TALC has better text adherence than merge captions.Source: §5.3 · p. 8

5.4 Automatic Evaluation

  • Base ModelScope with TALC scores 68.6 overall, versus 61.7 for merge captions and 67.5 for merge videos; on Lumiere, TALC exceeds those baselines by 4 and 2 points, respectively.For Lumiere, TALC also gains 7.1% over merge captions on overall score.Source: §5.4 · pp. 8–9, Table 3 · p. 8
  • 75.6 points: TALC-finetuned ModelScope achieves the highest automatic overall score; its visual consistency is 89 points versus 89.9 for TALC-base, and its text adherence is 15.1 points higher.Fine-grained analysis reports TALC-finetuned as highest across scenarios and scene counts, while performance declines on complex real-video prompts.Source: Table 2 · p. 8, §5.4 · p. 9

5.5 Qualitative Analysis

  • Qualitative examples show TALC videos with high text adherence and visual consistency and smooth transitions; FreeNoise has limited motion, while Gen-L-Video does not maintain consistency.The examples include a man whose appearance changes across surfing and skiing scenes in Gen-L-Video.Source: §5.5 · p. 9

6 Conclusion

TALC aligns video segments with scene-specific descriptions and is presented as a method for improving multi-scene generation. The paper describes its evaluation approach and contrasts it with related approaches, while identifying larger-scale multi-scene data and pretraining as future directions.

6 Conclusion

  • TALC aligns video segments with scene-specific descriptions; the conclusion says the framework can be adapted to any diffusion-based T2V model and identifies scaling multi-scene data and using TALC in pretraining as future directions.The authors report that TALC-finetuned models achieve high visual consistency and text adherence, while baselines suffer on one or both metrics.Source: §6 · p. 9

A Related Work

  • The related-work discussion contrasts TALC’s fine-tuning on curated multi-scene data with zero-shot multi-scene methods and notes that Gen-L-Video typically supports at most two scenes, versus TALC’s four.The paper describes an evaluation protocol combining human evaluation and automated assessments using GPT-4V.Source: A Related Work · p. 14

B Evaluator

  • Human evaluation uses qualified AMT annotators to assess entity consistency, background consistency, text adherence, and visual quality; automatic evaluation uses GPT-4-Vision on sampled frames and agrees with human evaluation 77% of the time.The automatic evaluator samples four frames uniformly from each scene and judges consistency and adherence against the multi-scene description.Source: B Evaluator · p. 14

C Multi-scene video data sources

The dataset is built from MSR-VTT and VaTeX videos with human-written whole-video captions, which are segmented and recaptioned as multi-scene descriptions. The experiments retain at most four segments per video because caption accuracy and fine-tuning cost depend on scene count.

C Multi-scene video data sources

  • MSR-VTT and VaTeX videos are segmented with PyScene, and the first four segments are used to construct multi-scene data.One randomly selected whole-video caption is used per video; MSR-VTT videos are mostly 10–30 seconds with 20 captions each, while VaTeX videos are 10 seconds with 10 captions each.Source: C Multi-scene video data sources · p. 14
  • Median-scene keyframes are processed together by Gemini to generate captions that contextualize each scene within the video's sequence.PyScene cuts identify the scene boundaries before keyframe selection.Source: Figure 4 · p. 15
  • The experiments retain the first four video segments and discard any additional segments because scene count affects captioning accuracy and fine-tuning demands.The authors identify scaling beyond four scenes as future work, contingent on greater computing power and stronger multi-scene captioning models.Source: C Multi-scene video data sources · p. 15

D Visual quality of the generated videos.

Human evaluation finds that TALC fine-tuning largely preserves visual quality, unlike fine-tuning with merged captions. The appendices describe the caption-generation prompts, scenario-specific prompt construction, and automatic evaluation procedure.

D Visual quality of the generated videos.

  • 63.4 points is the visual-quality score for merged-caption fine-tuning, compared with 83.3 for TALC-finetuned and 84.5 for TALC-base models.The TALC-finetuned model's score is described as similar to TALC-base, while naive merging of scene descriptions produces undesirable artifacts.Source: D Visual quality of the generated videos · p. 15, Table 4 · p. 15

E Prompt for Multi-Scene Caption Generation

  • Gemini-Pro-Vision receives a frame from each segmented video and the original whole-video caption to generate multi-scene descriptions.The model is used because it can reason over multiple image sequences.Source: E Prompt for Multi-Scene Caption… · p. 15

F.1 Single character in multiple visual contexts

  • GPT-4 prompts are provided for generating multi-scene descriptions featuring one character across multiple visual contexts.The prompt is shown in Figure 6.Source: §F.1 · p. 16, Figure 6 · p. 16

F.2 Different characters in a specific visual context

  • A separate GPT-4 prompt generates multi-scene descriptions featuring different characters in a shared visual context.The prompt is shown in Figure 7.Source: §F.2 · p. 16, Figure 7 · p. 16

G Automatic Evaluation Prompt

H Human Evaluation Screenshot

The human-evaluation layout asks annotators to assess video quality and consistency alongside text adherence. Fine-grained automatic evaluations compare task-prompt types and scene counts, while the appendix provides sampling details.

H Human Evaluation Screenshot

I.1 Task Prompts

  • Across task prompts, TALC-finetuned achieves the highest overall score among baselines, with higher text adherence than merged-caption baselines and visual consistency close to them.Table 5 reports results for single characters across contexts, different characters in one context, and captions from real videos.Source: §I.1 · pp. 17–18, Table 5 · p. 18

I.2 Number of Scenes

  • As scene count increases, text adherence falls for the baselines and TALC, while TALC visual consistency changes little; TALC-finetuned leads all baselines in this setup.Merged-video text adherence remains unchanged because the scenes are generated independently.Source: §I.2 · p. 18, Table 6 · p. 18

J Inference Details

K Multi-Scene Data Statistics

The appendix points to tables detailing multi-scene dataset statistics and the TALC-finetuned ModelScope training setup.

K Multi-Scene Data Statistics

L Finetuning Details

M Single-Scene Video Generation

The single-scene evaluation uses VBench to assess whether multi-scene fine-tuning affects generation performance, alongside qualitative comparisons of generated videos. Multi-scene fine-tuning scored higher than single-scene fine-tuning, and the authors report improved single-scene generation.

M Single-Scene Video Generation

  • VBench evaluates video-generation aspects including text-prompt adherence, stylistic integrity, semantic coherence, and aesthetic quality.The evaluation checks whether the new multi-scene generation function detracts from single-scene performance.Source: M Single-Scene Video Generation · p. 19
  • 0.48 was the average score for single-scene-finetuned ModelScope, down from 0.63 for base ModelScope.The authors suggest single-scene fine-tuning does not fully use the base model’s high-quality-image optimizations.Source: M Single-Scene Video Generation · p. 19
  • 0.59 was the average score for ModelScope fine-tuned on multi-scene data, surpassing the single-scene-finetuned version.The authors say multi-scene data enhances both multi-scene and single-scene video generation.Source: M Single-Scene Video Generation · p. 19
  • TALC with Lumiere showed smooth motion, seamless prompt transitions, and high overall quality, unlike the lower-motion FreeNoise and less-consistent Gen-L-Video examples.In a FreeNoise example, the background stayed static apart from darkening and the jeep did not appear to move; Gen-L-Video sometimes changed the man’s appearance.Source: Figure 10 · p. 20
  • The authors conclude that their approach enables complex multi-scene narratives and improves single-scene video generation.They connect this comparison to data curation and fine-tuning strategies.Source: M Single-Scene Video Generation · p. 20

N FreeNoise failure cases

The authors report that FreeNoise performs poorly on multi-scene prompts, with examples showing missing actions and content as well as visual inconsistencies. TALC with ModelScope, by contrast, supports visually consistent multi-action videos and diverse scenes.

N FreeNoise failure cases

  • FreeNoise performs poorly on the multi-scene prompts, with limited motion and difficulty introducing new content.The authors illustrate these shortcomings with three qualitative examples.Source: N FreeNoise failure cases · p. 20
  • TALC with ModelScope supports videos of one object performing multiple actions with visual consistency, as well as diverse scenes.The caption identifies the final example as demonstrating scene diversity.Source: Figure 11 · p. 21
  • In the wolf example, FreeNoise omits the prowling, scent-following, gleaming eyes, and kill described across the prompt’s scenes.It shows a wolf standing in a forest instead.Source: Figure 12 · p. 21
  • In the chimpanzee example, FreeNoise shows sitting rather than swinging and omits the fruit and companions.The prompt describes swinging through the canopy and gathering fruit with companions.Source: Figure 13 · p. 22
  • In the tiger example, FreeNoise shows no deer or ambush, and an unexpected second tiger appears from the first.The tiger is depicted standing calmly, and the second tiger abruptly changes directions.Source: Figure 14 · p. 22
Loading 2405.04682v4…