PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Colorization with Pre-trained Text-to-Image Diffusion Models

Hanyuan Liu, Minshan Xie, Jinbo Xing, Chengze Li, Tien-Tsin Wong

arXiv:2306.01732v1cs.CVcs.AIcs.GR

TL;DR

Video colorization must infer plausible colors while maintaining temporal coherence, and large-scale task-specific training can be prohibitively costly. ColorDiffuser adapts a pretrained text-to-image diffusion model with attention and sampling strategies for grayscale video, optionally guided by text. The authors report state-of-the-art video-colorization performance, with limitations in runtime and compatibility with efficient sampling approaches.

  • Problem

    Video colorization must balance plausible, temporally coherent colors across grayscale frames, while large-scale task-specific training can require prohibitively large computational resources.

    Source: §1 · pp. 1–2

  • Method

    ColorDiffuser adapts a pretrained text-to-image latent diffusion model to grayscale video, optionally using text and color propagation attention with alternated adjacent-frame sampling.

    Source: §1 · p. 2, §3.2.3 · p. 5

  • Results

    ColorDiffuser achieves state-of-the-art video-colorization performance, with reported gains in color accuracy, temporal coherence, controllability, fidelity, and visual quality.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The reported results support adapting pretrained text-to-image diffusion models for realistic and diverse video colorization.

    Source: §5 · p. 10

    Sampling takes around 100 seconds per frame on CPU or 6 seconds on an RTX 3090, and alternated sampling makes efficient samplers and distillation incompatible with the framework.

    Source: §4.5 · p. 10

Abstract

from arXiv · show

Video colorization is a challenging task that involves inferring plausible and temporally consistent colors for grayscale frames. In this paper, we present ColorDiffuser, an adaptation of a pre-trained text-to-image latent diffusion model for video colorization. With the proposed adapter-based approach, we repropose the pre-trained text-to-image model to accept input grayscale video frames, with the optional text description, for video colorization. To enhance the temporal coherence and maintain the vividness of colorization across frames, we propose two novel techniques: the Color Propagation Attention and Alternated Sampling Strategy. Color Propagation Attention enables the model to refine its colorization decision based on a reference latent frame, while Alternated Sampling Strategy captures spatiotemporal dependencies by using the next and previous adjacent latent frames alternatively as reference during the generative diffusion sampling steps. This encourages bidirectional color information propagation between adjacent video frames, leading to improved color consistency across frames. We conduct extensive experiments on benchmark datasets, and the results demonstrate the effectiveness of our proposed framework. The evaluations show that ColorDiffuser achieves state-of-the-art performance in video colorization, surpassing existing methods in terms of color fidelity, temporal consistency, and visual quality.

1. Introduction

ColorDiffuser adapts a pretrained text-to-image diffusion model for grayscale video colorization, addressing the challenge of producing coherent, semantically consistent, vivid colors across frames. The authors report superiority over baselines in color accuracy, temporal coherence, and controllability.

1. Introduction

  • Video colorization must balance temporal coherence, spatial and semantic consistency, and color richness and faithfulness, while large-scale task-specific training is prohibitively costly.The challenge is especially noted for colorizing artificial content.Source: §1 · pp. 1–2
  • Color propagation attention applies during color inference rather than as post-processing, avoiding reliance on unstable optical flow for temporal consistency.The method addresses the pretrained image model’s lack of guaranteed coherence across video frames.Source: §1 · p. 2
  • Experiments on benchmark datasets show superiority over baseline methods in color accuracy, temporal coherence, and controllability.The evaluation compares the adapted text-to-image diffusion model against several baseline methods.Source: §1 · p. 2
  • ColorDiffuser adapts a pretrained text-to-image model to colorize monochrome videos with diverse, consistent colors and optional text descriptions.The adapter redirects the model to match grayscale-frame luminance and structure while drawing on its learned color variety.Source: §1 · p. 2

2. Related Work

Related image-colorization work uses hints, reference images, and automatic learned mappings, while video methods add temporal propagation or post-processing. Latent diffusion models provide a pretrained generative foundation that can be conditioned for image and video generation.

2.1. Image Colorization

  • Image-colorization approaches use hints, reference images, or automatic learned mappings, but face limits involving annotations, unavailable references, color ambiguity, or visual artifacts; ColorDiffuser instead uses pretrained T2I priors and text control.Reference-based methods may lack suitable content-related images, while automatic approaches can produce unnatural or incoherent colors under uncertainty.Source: §2.1 · p. 2

2.2. Video Colorization

  • Video-colorization methods use temporal filtering, exemplar-guided optical-flow propagation, or sequential frame propagation, each with limitations such as washed-out colors, accumulated flow errors, or scene disparity.ColorDiffuser requires no user-provided reference and uses frame-alternating cross-attention for temporal coherence.Source: §2.2 · p. 3

2.3. Diffusion Models

  • Latent diffusion models denoise noisy representations in a compressed autoencoder space, while prior text-to-video extensions rely on large-scale video training and extensive computational resources.The diffusion backbone uses cross-attention to accept conditions, and adapter-based modifications provide an additional conditioning mechanism.Source: §2.3 · p. 3

3. Method

ColorDiffuser adapts a pretrained text-to-image latent diffusion model for grayscale video colorization, optionally guided by text and a reference color frame. It coordinates cross-frame color propagation during diffusion and combines generated color latents with grayscale features for pixel-aligned output.

3.1. Overview

  • The framework generates colorized latents with a coordinated diffusion process, then uses a video colorization VQVAE model to produce pixel-aligned colorization.The inputs include grayscale frames, optional text, and a reference color-frame latent.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2. Adapting text-to-image latent diffusion model towards video colorization

  • An adapter-based approach extends a pretrained text-to-image model to generate color latents for grayscale frames, conditioned on text and reference-frame information.The model coordinates per-frame sampling rather than extending the pretrained model into a video diffusion model that operates on entire frame sequences.Source: §3.2.1 · p. 4

3.2.1 Conditional generation

  • The Diffusion Coordinator guides the pretrained diffusion model toward outputs consistent with the input grayscale frame and the reference frame’s color distribution.It injects coordinator feature maps into corresponding pretrained denoiser layers.Source: §3.2.1 · pp. 4–5

3.2.2 Color Propagation Attention

  • Color Propagation Attention uses current-frame grayscale features as queries and reference grayscale and color features as keys and values to propagate latent color across frames.The reference grayscale and color features are paired by their corresponding spatial locations.Source: §3.2.2 · p. 5

3.2.3 Alternating Sampling Strategy

  • Alternating reference frames between previous and next neighbors across diffusion steps supports bidirectional color propagation and addresses forward-only degradation during long-term generation.The method alternates previous- and next-frame references during sampling.Source: §3.2.3 · p. 5

3.2.4 Loss Function

  • The conditional diffusion model is trained with its pretrained diffusion weights frozen to reduce learning difficulty and use the diffusion prior.The section introduces the loss function for the conditional model but does not provide its full expression in the supplied passages.Source: §3.2.4 · p. 5

3.3. Video Colorization VQVAE Decoder

  • The VQVAE decoder combines generated color latents with grayscale features and temporal convolutions to align output structure and texture while improving temporal consistency.The original decoder weights are frozen while the grayscale encoder and added temporal convolutional layers are optimized.Source: §3.3 · p. 6

4. Experiments

The experiments evaluate ColorDiffuser on benchmark datasets, through user preferences and ablations, and discuss runtime and bias limitations.

4.1. Implementation and Training

  • Training uses WebVid-2M for both models, while evaluation follows established protocols on DAVIS validation and Videvo test splits using perceptual, colorfulness, consistency, and image-quality metrics.The diffusion coordinator is trained for 200,000 steps and the video colorization VQVAE for 100,000 steps.Source: §4.1 · p. 6, §4.2 · p. 7

4.2. Evaluations

  • ColorDiffuser achieves state-of-the-art performance in qualitative and quantitative evaluations, with more vivid colorization across frames; text descriptions improve perceptual quality and colorfulness.The quantitative comparison covers evaluation datasets and methods in Table 1; the framework also produces diverse results through stochastic reverse diffusion.Source: Table 1 · p. 7, §4.2 · p. 7

4.3. User Study

  • Users preferred ColorDiffuser over other methods for both visual quality and temporal consistency.The study presented results from different methods in randomized grids and asked 31 participants to rank them.Source: §4.3 · pp. 7, 9, Table 2 · p. 9

4.4. Ablation Study

  • Color Propagation Attention generates reference-consistent colors, while the tested alternatives fail; Alternated Sampling Strategy outperforms autoregressive and unidirectional sampling.The comparisons are shown in Figures 6 and 7.Source: §4.4 · p. 9, Figure 6 · p. 9, Figure 7 · p. 10
  • The video colorization VQVAE model is reported, based on visual results of decoder outputs, to produce the sharpest and most visually appealing results.Table 3 compares reconstruction quality across decoders, and Figure 8 illustrates the model’s effects and decoder variants.Source: §4.4 · p. 10, Table 3 · p. 10, Figure 8 · p. 10

4.5. Limitations and Discussions

  • Sampling is time-consuming on CPUs, alternated sampling needs enough steps for coherence, and biases inherited from the LAION-trained prior may affect color outputs.The authors report about 100 seconds per frame on an Intel Xeon 6326 CPU and about 6 seconds on an NVIDIA RTX 3090, each at 50 sampling steps and 256 × 256 resolution.Source: §4.5 · p. 10

5. Conclusion

The paper presents a pre-trained text-to-image stable diffusion pipeline for realistic and diverse video colorization, reporting state-of-the-art performance.

5. Conclusion

  • The paper proposes a video colorization pipeline based on pre-trained text-to-image stable diffusion models.The conclusion characterizes the pipeline as effective.Source: §5 · p. 10
  • The pipeline is designed to produce realistic and diverse video colorization.Source: §5 · p. 10
  • Extensive evaluation shows state-of-the-art performance for video colorization.Source: §5 · p. 10

Related papers

Loading 2306.01732v1…