PaperlayerView exact revision on arXiv

Source-linked AI summary

ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation

Weiming Ren, Huan Yang, Ge Zhang, Cong Wei, Xinrun Du, Wenhao Huang, Wenhu Chen

arXiv:2402.04324v2cs.CV

TL;DR

I2V generation must preserve the initial frame’s subject, background, style, and coherent motion, but existing methods often lose visual consistency. ConsistI2V combines spatiotemporal first-frame conditioning with low-frequency noise initialization and reports strong results across evaluations, while extending to long-video generation and camera control.

  • Problem

    I2V generation struggles to preserve the initial frame’s appearance and produce fluid, logically progressing video sequences.

  • Method

    ConsistI2V uses spatiotemporal first-frame conditioning and FrameInit, which initializes inference noise from the first frame’s low-frequency component.

  • Results

    ConsistI2V achieves exceptional I2V generation results in automatic and human evaluations across benchmarks, including the proposed I2V-Bench.

  • Takeaways & Limitations

    The approach supports highly consistent videos and applications including autoregressive long-video generation and camera motion control.

  • Takeaways & Limitations

    The method is constrained by low-resolution, watermark-heavy training data, can produce limited or slower motion, and inherits Stable Diffusion’s difficulty rendering human faces and legible text.

Abstract

from arXiv · show

Image-to-video (I2V) generation aims to use the initial frame (alongside a text prompt) to create a video sequence. A grand challenge in I2V generation is to maintain visual consistency throughout the video: existing methods often struggle to preserve the integrity of the subject, background, and style from the first frame, as well as ensure a fluid and logical progression within the video narrative. To mitigate these issues, we propose ConsistI2V, a diffusion-based method to enhance visual consistency for I2V generation. Specifically, we introduce (1) spatiotemporal attention over the first frame to maintain spatial and motion consistency, (2) noise initialization from the low-frequency band of the first frame to enhance layout consistency. These two approaches enable ConsistI2V to generate highly consistent videos. We also extend the proposed approaches to show their potential to improve consistency in auto-regressive long video generation and camera motion control. To verify the effectiveness of our method, we propose I2V-Bench, a comprehensive evaluation benchmark for I2V generation. Our automatic and human evaluation results demonstrate the superiority of ConsistI2V over existing methods.

1 Introduction

Text-to-video offers limited control, motivating image-to-video generation conditioned on a first frame. ConsistI2V addresses persistent appearance and motion inconsistency through spatiotemporal first-frame conditioning and low-frequency noise initialization, with I2V-Bench supporting evaluation.

  • Existing text-to-video generation provides imprecise control, producing variation in subjects, viewpoints, and backgrounds despite the same prompt.Users may need more descriptive prompts or repeated generation to obtain a desired result.
  • Image-to-video generation adds a first-frame image so generated foreground and background contents are conditioned on the depicted objects.The approach is presented as a solution for more controllable video generation.
  • Prior I2V methods often preserve first-frame appearance incompletely and still produce incorrect or jittery motion.Existing approaches use semantic embeddings, concatenated latent features, or both, but motion remains a practical limitation.
  • ConsistI2V combines spatiotemporal first-frame attention with FrameInit, which uses the first frame’s low-frequency component for layout-guided noise initialization.The spatial and temporal mechanisms target fine-grained appearance and temporal coherence, while FrameInit addresses inference-time noise discrepancy.
  • The method is extended to autoregressive long-video generation and camera motion control, alongside the proposed I2V-Bench evaluation benchmark.The benchmark is designed for quantitative evaluation of I2V generation models.

2 Related Work

Related work develops video generation from diffusion-based text-to-video systems, noise-initialization strategies, video editing methods, and first-frame-conditioned image-to-video generation. These lines of work provide the modeling and conditioning context for ConsistI2V.

  • Text-to-Video Generation: Diffusion-based video generation extends text-to-image frameworks with space-time architectures, temporal attention, adaptor layers, or temporal shifts.Later methods commonly insert temporal convolution and attention layers into latent-diffusion U-Nets.
  • Noise Initialization: Noise-initialization research correlates frame noise, reschedules noise for long videos, or uses low-frequency components to reduce training–inference gaps.These strategies motivate noise-based approaches to improving temporal consistency.
  • Video Editing: Video-editing methods commonly use pretrained text-to-image models and cross-frame attention to maintain temporal coherence across edited frames.The cited approaches address editing without relying on paired before-and-after video data.
  • Image-to-Video Generation: Image-to-video methods condition video generation on both a first frame and text, often by modifying the text-to-video U-Net with latent-feature concatenation.This stream targets more controllable video generation than text descriptions alone provide.

3 Methodology

ConsistI2V conditions video diffusion on the first frame through spatial and temporal feature injection, while FrameInit uses low-frequency layout information to stabilize inference. The method targets videos whose appearance remains aligned with the input frame while following the text prompt.

  • 3 Methodology: ConsistI2V generates a video conditioned on an input image and text prompt, keeping later-frame appearance aligned with the first frame while following the prompt.The model produces an N-frame clip whose first frame is the input image.
  • 3.3 Fine-Grained Spatial Feature Conditioning: Spatial conditioning expands attention keys and values with features from all spatial positions in the first frame, enabling fine-grained feature access across frames.The first-frame latent is encoded with a VAE and used as the conditioning signal.
  • 3.4 Window-based Temporal Feature Conditioning: Temporal conditioning augments keys and values with a local first-frame feature window so moving objects can remain associated with their original entities.The authors set K = 3; boundary replication provides complete windows at every spatial position.
  • 3.5 Inference-time Layout-Guided Noise Initialization: FrameInit duplicates the first frame into a static video and uses its low-frequency component as coarse layout guidance during inference.Low-frequency components represent slowly moving parts and overall layout, while high-frequency components capture fast motion and fine details.
  • 3.5 Inference-time Layout-Guided Noise Initialization: FrameInit empirically improves generated-video stabilization, quality, and consistency, and supports autoregressive long-video generation and camera-motion control.The paper presents additional results for both applications in Section 5.5.

4 I2V-Bench

I2V-Bench is an evaluation dataset designed to address limitations of existing video benchmarks in resolution, diversity, and aesthetic appeal. Its framework evaluates both visual quality and visual consistency using multiple video-output dimensions.

  • 4 I2V-Bench: I2V-Bench contains 2,950 high-quality YouTube videos organized into 16 categories, including Scenery, Sports, Animals, and Portraits.The videos were curated using strict resolution and aesthetic standards.
  • 4 I2V-Bench: The benchmark evaluates visual quality through subject and background consistency, temporal flickering, motion smoothness, and dynamic degree.Visual Quality assesses perceptual video-output quality regardless of the input prompts.
  • 4 I2V-Bench: Visual consistency evaluates how well generated videos adhere to the user’s text prompt.The supplied passage introduces this dimension as distinct from visual quality.

5 Experiments

ConsistI2V is evaluated against open-source I2V baselines using standard benchmarks, I2V-Bench, qualitative comparisons, human judgments, and ablations of its conditioning and initialization components.

  • Quantitative Evaluation: ConsistI2V significantly outperforms the baseline models on nearly all reported metrics, except for slightly lower CLIPSIM on MSR-VTT than SEINE.AnimateAnything has better IS and FID but produces mostly near-static videos, reflected by its highest FVD of 642.64.
  • Quantitative Evaluation: On I2V-Bench, ConsistI2V balances motion magnitude and video quality while outperforming the baselines excluding AnimateAnything on motion quality and visual consistency.It also achieves competitive overall video-text consistency.
  • Human Evaluation: Human evaluation ranks ConsistI2V first for both appearance and motion consistency, with motion consistency comparable to SEINE and appearance consistency higher than every other baseline.The evaluation collected 1061 responses from 13 participants across 548 samples.
  • Qualitative Evaluation: Qualitative comparisons show that ConsistI2V better preserves input-frame subjects and produces more natural, logical motion without abrupt changes.Baselines exhibit subject or background changes, unnatural object movements, or minimal motion in the illustrated cases.
  • Ablation Studies: FrameInit improves all ablation metrics and stabilizes videos by reducing sudden object movements and blurry frames.Without FrameInit, videos can remain reasonable but show more abrupt motion and visual artifacts.
  • Ablation Studies: Removing both temporal and spatial first-frame conditioning eliminates the model’s ability to maintain the input appearance, while spatial conditioning alone can cause jittering and object distortion.The ablation study evaluates these components on UCF-101 under consistent experimental settings.

6 Limitations

The authors identify limitations involving training-data quality, motion magnitude, training requirements, and inherited Stable Diffusion weaknesses.

  • WebVid-10M is predominantly low resolution and contains fixed-position watermarks, limiting supported output resolution and potentially corrupting generated videos.
  • FrameInit improves video stability but can produce limited motion magnitude, restricting subject movement.
  • Spatial first-frame conditioning requires tuning spatial U-Net layers, increasing training costs and limiting direct adaptation to personalized T2I models.
  • The model inherits Stable Diffusion limitations, including difficulty rendering human faces and legible text.

7 Conclusion

The conclusion presents ConsistI2V as a consistency-focused I2V framework using spatiotemporal first-frame conditioning and FrameInit, with applications beyond standard video generation and evaluation on I2V-Bench.

  • ConsistI2V combines spatiotemporal first-frame conditioning with FrameInit layout guidance to improve visual consistency in generated videos.
  • The framework supports autoregressive long video generation and camera motion control, with FrameInit enabling training-free camera motion control.
  • The authors report extensive automatic and human evaluations on multiple benchmarks, including the proposed I2V-Bench, with exceptional I2V generation results.
  • Future work will refine the training paradigm and incorporate higher-quality training data to scale ConsistI2V.

Statement of Broader Impact

The broader-impact statement frames conditional video synthesis as a fundamental content-creation and manipulation problem while acknowledging persistent consistency challenges and slower motions in some cases.

  • Conditional video synthesis enables content creation and manipulation by generating videos faithful to given conditions.
  • The work targets missing coherence and consistency in generated subjects, backgrounds, and styles, while reporting promising improvement.
  • The model can produce slower motions in some cases, which the authors identify as a continuing issue.

A.1 Model Architecture

The architecture combines temporal processing, correlated noise initialization, positional embeddings, and FPS conditioning with explicit first-frame conditioning mechanisms.

  • Temporal layers use convolution, self-attention, and text cross-attention, with a learnable γ weighting factor combining spatial and temporal outputs.
  • Correlated noise initialization combines shared and frame-specific noise, with α empirically set to 1.5 to stabilize training and accelerate convergence.
  • Rotary positional embeddings encode frame positions in temporal attention while aligning tokens with corresponding first-frame spatial features.
  • FPS conditioning samples 16 frames with a randomly selected frame interval from 1 to 5 and injects its embedding into U-Net blocks.
  • Explicit spatial and temporal attention conditioning uses first-frame visual cues to render subsequent frames, while the model directly employs the pretrained T2I model.

B.1 Datasets

The evaluation uses UCF-101 and MSR-VTT alongside I2V-Bench, with metrics covering video quality, semantic alignment, consistency, flickering, smoothness, and dynamics.

  • Datasets: UCF-101 evaluation samples 2,048 videos according to the dataset’s action-category distribution and uses PYoCo text prompts.UCF-101 contains 13K videos across 101 action categories.
  • Datasets: MSR-VTT evaluation uses its official test split and randomly selects one text prompt for each video.MSR-VTT contains 10K videos with 20 captions per video.
  • I2V-Bench: I2V-Bench includes annotated object and scene consistency tests based on selected validation videos from object- and scenery-focused categories.Object consistency uses GRiT, while scene consistency uses Tag2Text to detect annotated keywords.
  • Evaluation Metrics: I2V-Bench evaluates overall video-text alignment, background and subject consistency, temporal flickering, motion smoothness, dynamic degree, object consistency, and scene consistency.The benchmark combines embedding similarity, frame differences, motion interpolation error, optical flow, and object or scene recognition measures.
  • Human Evaluation: Human evaluation compares ConsistI2V with four baseline models on 548 generated videos standardized to 256 × 256 resolution and 16 frames.The study uses 274 prompts and collects two-second videos at 8 FPS.

F Additional Quantitative Results

Additional quantitative experiments evaluate ConsistI2V as a generic text-to-video generator on MSR-VTT and report detailed I2V-Bench results across multiple consistency and motion dimensions.

  • T2V Generation: 428 FVD and 0.2968 CLIPSIM position ConsistI2V second-best in FVD and comparable in CLIPSIM for MSR-VTT text-to-video generation.The experiment generates the first frame from the text prompt with Stable Diffusion 2.1-base.
  • I2V-Bench Results: Detailed I2V-Bench results are reported for background consistency, subject consistency, temporal flickering, and motion smoothness.The corresponding results appear in Tables 6–9, with splits including Scenery-Nature, Scenery-City, Animation-Hard, and Animation-Static.
  • I2V-Bench Results: Additional I2V-Bench tables report object consistency, scene consistency, dynamic degree, and text-video overall consistency.These results are presented in Tables 10–13 using category-specific evaluation subsets.

H Additional I2V Generation Results

Additional figures showcase ConsistI2V outputs for varied image-to-video prompts, including animals, musical performance, a liquid-filled sphere, and the Eiffel Tower.

  • Additional Results: The showcased prompts span underwater swimming, stage performance, cinematic object imagery, and dramatic landmark scenes.The examples pair input-frame descriptions with text prompts specifying actions, settings, or visual conditions.
Loading 2402.04324v2…