Source-linked AI summary
Deep Exemplar-based Video Colorization
Bo Zhang, Mingming He, Jing Liao, Pedro V. Sander, Lu Yuan, Amine Bermak, Dong Chen
TL;DR
Video colorization must remain temporally consistent while preserving a reference image's style, despite flicker, artifacts, and accumulated propagation errors. The paper introduces a recurrent end-to-end network that unifies semantic correspondence with colorization and guides every frame using a reference and colorization history. It reports quantitative and qualitative improvements over existing methods, with realistic, temporally consistent outputs and customizable references.
Problem
Existing video colorization methods suffer from flickering, artifacts, color fading, or progressively accumulated errors during temporal propagation.
Method
A recurrent end-to-end network jointly learns reference-to-frame correspondence and colorization using the previous colorized frame, aligned reference, and novel losses.
Results
The network outperforms existing methods quantitatively and qualitatively and supports color references from different scenes for customizable multimodal results.
Takeaways & Limitations
A provided reference can guide every video frame while the recurrent history preserves temporal consistency and reduces accumulated propagation error.
Takeaways & Limitations
The smoothness loss assumes neighboring pixels should have similar colors when their ground-truth chrominance is similar.
Abstract
from arXiv · showhide
This paper presents the first end-to-end network for exemplar-based video colorization. The main challenge is to achieve temporal consistency while remaining faithful to the reference style. To address this issue, we introduce a recurrent framework that unifies the semantic correspondence and color propagation steps. Both steps allow a provided reference image to guide the colorization of every frame, thus reducing accumulated propagation errors. Video frames are colorized in sequence based on the colorization history, and its coherency is further enforced by the temporal consistency loss. All of these components, learned end-to-end, help produce realistic videos with good temporal stability. Experiments show our result is superior to the state-of-the-art methods both quantitatively and qualitatively.
1. Introduction
Exemplar-based video colorization must preserve temporal consistency without accumulating propagation errors or losing reference fidelity. The proposed recurrent end-to-end network combines semantic correspondence, colorization, temporal history, and learned natural-color prediction.
- Framewise colorization can flicker, while temporal filtering may cause color fading and blurring.
- First-frame propagation works for short clips but accumulates errors over long videos and ignores the content of all frames.
- The method uses the previous result for temporal consistency and an exemplar reference to guide every frame, reducing accumulation error.
- Its recurrent state contains a correspondence subnet aligning the reference and a colorization subnet using the aligned reference and previous output.
- The jointly trained correspondence subnet is two orders of magnitude faster than the state-of-the-art exemplar-based method's iterative alignment.
- The network outperforms existing methods quantitatively and qualitatively, while supporting same-scene propagation and customizable results from references in different scenes.
2. Related work
Prior colorization research spans interactive hints, exemplar transfer, and fully automatic prediction, while video methods add temporal processing or propagation. Existing video approaches remain vulnerable to washed-out colors, unsuitable scribbles, propagation failures, and bootstrap requirements.
- Interactive Colorization: Interactive colorization propagates local user color points or strokes using coherent-neighborhood assumptions and hand-crafted low-level features.
- Exemplar-based Colorization: Exemplar-based methods transfer chromatic information from a reference image by matching luminance, texture, or hand-crafted correspondence features.
- Fully Automatic Colorization: Fully automatic methods learn grayscale-to-color mappings from large datasets, but conventional approaches lack color ambiguity modeling and cannot generate multimodal results.
- Video Colorization: Video colorization methods use temporal filters, optical-flow scribble propagation, or sequential propagation from one colored frame.
- Video Colorization: These video approaches can wash out colors, use scribbles unsuitable for other frames, fail on particular frames, or require a good bootstrap frame.
3. Method
The method recurrently colorizes video frames using both prior colorization history and a reference image. It jointly learns semantic correspondence, color propagation, and losses enforcing perceptual plausibility, reference faithfulness, spatial smoothness, and temporal coherence.
- Overall framework: The recurrent network conditions each frame’s colorization on the previous colorized frame and a reference image.This history-based formulation is intended to generate temporally consistent videos while retaining reference guidance.
- Colorization subnet: The correspondence subnet produces a warped color map and confidence map, which the colorization subnet uses with the grayscale frame and previous colorized frame.The colorization subnet selects reliable colors and propagates them to predict the current color map.
- Correspondence subnet: A correspondence subnet aligns reference colors to the current grayscale frame using dense semantic feature correspondences.It extracts deep features, computes pairwise similarities, and warps the reference color into an aligned color map.
- Losses: Perceptual, smoothness, adversarial, temporal-consistency, and L1 losses jointly constrain semantic plausibility, edges, realism, temporal stability, and ground-truth color differences.The temporal loss penalizes color changes along flow trajectories, while WLS weights support edge-preserving colorization and reduce bleeding artifacts.
- Losses: The network uses contextual loss to keep output colors close to semantically related regions in the reference image.Forward feature matching is used because some objects in the input frame may not exist in the reference; feature maps span layers 2 to 5.
- Objective: Combining the losses unifies correspondence and color propagation within one network that learns plausible results from an exemplar image.The L1 term additionally handles the common case where the reference comes from the same scene as the video frames.
4. Implementation
The implementation combines a correspondence network with an auto-encoder colorization subnet in a recurrent architecture, trained on diverse video data and paired reference candidates.
- Network Structure: The correspondence network uses four residual blocks, while the colorization subnet uses an auto-encoder with skip-connections and three encoder and decoder blocks.The decoder predicts the current frame’s chrominance, and tanh bounds the chrominance output within the color space.
- Training Data: Training uses 1,052 Videvo videos, filtered to 768 clips, supplemented with Hollywood2 portrait videos.Reference candidates are obtained from the five most similar images in the corresponding class.
- Training Strategy: The correspondence matching is deliberately weakened during training so the colorization subnet learns to propagate colors despite inaccurate correspondence.
- Optimization: The network is trained for 10 epochs with batch size 40 using AMSGrad and a learning rate of 2 × 10^-4 for both generator and discriminator.The training uses β1 = 0.5 and β2 = 0.999 without a decay schedule.
5. Experiments
Experiments evaluate component importance and compare the method with image colorization, automatic video colorization, and video color propagation baselines. The results report improved correspondence, vivid outputs with fewer artifacts, stable propagation, and stronger user preferences.
- Ablation Studies: Learning parameters in the correspondence subnet match mostly correct reference colors and ease propagation for the colorization subnet compared with nearest-neighbor matching.
- Ablation Studies: Removing Lperc increases artifacts, removing Lcontext weakens reference-style resemblance, removing Lsmooth limits coherent-region propagation, and removing Ladv produces washed-out colors.The full model produces vivid colorization with fewer artifacts.
- Image Colorization: On ImageNet 10k, the method achieves the best Top-5 and Top-1 classification accuracy and the lowest FID among the compared image colorization methods.Its colorfulness is slightly below the most vivid baseline but similar to the ground truth, while that baseline exhibits artifacts.
- Automatic Video Colorization: On 116 Videvo clips, the method is comparably temporally stable to post-processed video colorization while avoiding the blurriness introduced by temporal filtering.Figure 10 shows vibrant colors, significantly fewer artifacts, and good temporal consistency across successively colorized frames.
- Color Propagation: The method maintains a more stable PSNR curve over propagated frames than optical-flow, STN, and VPN baselines on DAVIS.Optical-flow methods start with the highest PSNR but deteriorate significantly thereafter; STN and VPN also degrade.
- User Studies: In user studies, the method was 50.66% more likely to receive first rank than per-frame automatic colorization methods and achieved a 79.67% first-rank percentage against propagation methods.The studies used 20 users across 19 Videvo videos and 15 DAVIS videos, respectively.
6. Conclusion
The paper proposes an end-to-end exemplar-based video colorization algorithm that unifies semantic correspondence and colorization in one network, producing temporally consistent and realistic results.
- The method unifies semantic correspondence and colorization into a single network trained end-to-end.
- The resulting videos exhibit temporal consistency and realistic visual effects.
- The approach is presented as the first exemplar-based video colorization algorithm.
Appendix A. Details of network architecture
The network combines a correspondence subnet with a colorization subnet, using image features and temporal colorization history to guide frame-wise color prediction.
- The correspondence subnet extracts VGG features from the input and reference images to compute warped reference colors and a similarity map.
- The architecture includes distinct correspondence and colorization subnet modules.
- The colorization subnet receives the current frame luminance, warped reference color, similarity map, and previous colorized frame.
- An auto-encoder with skip connections predicts current-frame chrominance from hierarchically encoded features.
- A self-attention discriminator is trained adversarially to incorporate global information and improve vividness and realism.
Appendix B. Multimodal colorization
The method supports reference-guided, multimodal colorization and can apply reference colors to both ordinary and legacy videos.
- Users can customize colorization by providing references whose corresponding regions supply colors to the output.
- Figures 11–14 present examples of multimodal colorization according to user references.
- The method colorizes legacy videos with vivid colors that remain faithful to the given reference.
Appendix D. Quantitative ablation study
The ablation study evaluates perceptual quality and shows that different losses trade off saturation, recognition accuracy, input semantics, and reference style.
- The full model achieves the best perceptual quality with the lowest FID score.
- Ladv greatly improves saturation while slightly reducing recognition accuracy.
- Lperc emphasizes input semantics whereas Lcontext emphasizes reference style, producing contradictory effects on saturation.
Appendix E. User study
The user studies compare the proposed video colorization network with automatic framewise colorization and video propagation methods. Users preferred the proposed methods in both comparisons.
- Comparison with automatic video colorization: The first user study compared the proposed method with three automatic per-frame colorization methods using 19 randomly selected test videos.Users ranked four methods by temporal consistency and visual photorealism.
- Users preferred the proposed video colorization methods over other state-of-the-art methods in both user studies.
- Comparison with video propagation: The second user study evaluated the proposed method against video color propagation methods.
Appendix F. Failure case
The method constrains temporal consistency between adjacent frames but can still develop long-term inconsistency. Matching errors may cause an object's color to drift over time.
- Training constrains temporal consistency between adjacent frames, but the method may suffer from long-term temporal inconsistency.
- The reported color drift is attributed to correspondence mismatch.The paper identifies incorporating temporal consistency into dense correspondence as future work.
- Figure 16 shows the train's color gradually changing from red to blue and back to red as matching errors accumulate.