Source-linked AI summary
Harmonizer: Learning to Perform White-Box Image and Video Harmonization
Zhanghan Ke, Chunyi Sun, Lei Zhu, Ke Xu, Rynson W. H. Lau
TL;DR
Image harmonization aims to make composites realistic by reducing foreground–background appearance mismatches, while autoencoder approaches have high-resolution performance and speed challenges. Harmonizer predicts image-level filter arguments and applies white-box filters; it outperforms prior methods and produces consistent video results at 56 fps at 1080P.
Problem
Autoencoder-based pixel-wise translation methods have unsatisfactory performance and slow inference at high resolution, while harmonization must address appearance mismatches affected by lighting and imaging conditions.
Source: §1 · p. 1Method
Harmonizer uses a neural network to predict arguments for white-box image filters, with a cascade regressor and dynamic loss strategy for more stable and precise learning.
Source: §1 · pp. 2–3Results
Harmonizer outperforms prior methods on all four iHarmony4 subsets, reducing average MSE by 35%, and yields consistent video results at 56 fps at 1080P.
Source: §1 · p. 3, §4.1 · p. 10Takeaways & Limitations
Harmonizer is lighter and faster than prior methods while achieving state-of-the-art performance, according to the paper’s conclusion.
Source: §5 · p. 14Harmonizer may fail on color-specific appearance inconsistencies or different lighting conditions between foreground and background.
Source: §5 · p. 14
Abstract
from arXiv · showhide
Recent works on image harmonization solve the problem as a pixel-wise image translation task via large autoencoders. They have unsatisfactory performances and slow inference speeds when dealing with high-resolution images. In this work, we observe that adjusting the input arguments of basic image filters, e.g., brightness and contrast, is sufficient for humans to produce realistic images from the composite ones. Hence, we frame image harmonization as an image-level regression problem to learn the arguments of the filters that humans use for the task. We present a Harmonizer framework for image harmonization. Unlike prior methods that are based on black-box autoencoders, Harmonizer contains a neural network for filter argument prediction and several white-box filters (based on the predicted arguments) for image harmonization. We also introduce a cascade regressor and a dynamic loss strategy for Harmonizer to learn filter arguments more stably and precisely. Since our network only outputs image-level arguments and the filters we used are efficient, Harmonizer is much lighter and faster than existing methods. Comprehensive experiments demonstrate that Harmonizer surpasses existing methods notably, especially with high-resolution inputs. Finally, we apply Harmonizer to video harmonization, which achieves consistent results across frames and 56 fps at 1080P resolution. Code and models are available at: https://github.com/ZHKKKe/Harmonizer.
1 Introduction
Image harmonization seeks realistic composites by reducing foreground–background appearance inconsistencies, a challenging task because lighting and imaging conditions affect appearance and humans notice fine mismatches. Harmonizer replaces pixel-wise translation with image-level filter-argument regression and white-box filters.
1 Introduction
- The task aims to make composites realistic by removing foreground–background appearance inconsistencies, despite appearance variation from lighting and imaging conditions.Humans are sensitive even to fine appearance mismatches.Source: §1 · p. 1
- High-resolution harmonization remains difficult for autoencoder methods, which are often trained at low resolution, computationally expensive, and may alter image textures or details.Traditional methods also overlook semantic information important for closing large appearance gaps.Source: §1 · pp. 1–2
- Harmonizer predicts image-level filter arguments with a neural network, then applies white-box filters to the composite image.Figure 1 contrasts this approach with hand-crafted-statistics matching and black-box pixel-wise translation.Source: §1 · p. 2, Figure 1 · p. 2
- The cascade regressor predicts each filter argument using preceding arguments’ features, addressing interactions that make independent multi-head prediction unsatisfactory.The dynamic loss strategy balances accumulated filter-output errors and emphasizes filters that are harder to learn.Source: §1 · pp. 2–3
- Harmonizer surpasses prior state-of-the-art methods, is lighter and faster, and produces consistent video results at 56 fps at 1080P.The introduction reports these outcomes across image and video harmonization.Source: §1 · p. 3
2 Related Works
Prior harmonization methods include statistics-based matching and autoencoder-based pixel translation. Related white-box editing work combines neural networks with interpretable operations; Harmonizer applies that direction to image harmonization through filter-argument regression.
2.1 Image Harmonization
- The harmonization function processes the masked foreground and combines it with the unmodified background to form a natural image.The task is framed as matching the foreground’s visual appearance to the background.Source: §2.1 · p. 3
- Traditional harmonization methods match low-level statistics such as color distributions, color templates, and gradients, sometimes adding multi-scale statistics or visual realism.These methods focus on appearance statistics.Source: §2.1 · p. 3
- CNN-based methods treat harmonization as pixel-wise translation with autoencoders, using approaches that include semantic features, attention, background guidance, style transfer, intrinsic-image cues, and Transformer context.The cited examples include Tsai et al., Cun et al., Cong et al., Ling et al., Guo et al., and Guo et al..Source: §2.1 · pp. 3–4
- Prior methods have poor performance and slow inference at high resolution, whereas Harmonizer uses image-level regression for consistent high-resolution inference speed with negligible performance degradation.The passage attributes prior methods’ issues to low-resolution training images and autoencoder computation.Source: §2.1 · p. 4
2.2 White-box Image Editing
- White-box image-editing methods combine neural networks with human-understandable algorithms, while existing deep harmonization methods mainly use black-box autoencoders.The passage notes white-box operations can permit further editing or undoing unwanted operations.Source: §2.2 · p. 4
- Unlike Hu et al., which predicts filter types and arguments sequentially and may repeat filters, Harmonizer regresses a set of filter arguments together and applies each filter once.The passage presents this design as avoiding redundant operations and improving efficiency.Source: §2.2 · p. 4
3 Harmonizer
Harmonizer derives a white-box filtering approach from how people harmonize composites, then predicts filter arguments and applies the filters. Its training strategies address interdependent arguments, accumulated errors, and frame-to-frame consistency.
3.1 Design Motivation
- The user study found that experts harmonize composites by adjusting image-filter arguments, while participant ratings and argument distributions guide filter choice and ranges.The study involved 5 image-editing experts and 26 participants; Fig. 2 summarizes filter importance and argument distributions.Source: §3.1 · p. 5, Figure 2 · p. 5
3.2 Architecture
- Harmonizer uses an encoder and regressor to predict arguments for six selected filters, then applies the filters in sequence to the composite image.The selected filters are brightness, contrast, saturation, color temperature, highlight, and shadow; k = 6 is chosen to meet real-time requirements.Source: §3.2 · pp. 5–7
3.3 Training Strategy
- Training composites are generated by applying filters in reverse order, with argument values sampled from user-study-based Gaussian distributions to preserve reversibility in most cases.The user study supplies rough argument ranges; these reduce the chance that sensitive filters make the generated composite impossible to recover.Source: §3.3 · pp. 7–8
- Because filter operations can be asymmetric and some argument targets are unknown, training uses foreground-masked L2 losses on intermediate filter outputs and dynamically reweights them.The dynamic strategy subtracts preceding-filter errors and normalizes by the final loss; when a loss is lower than its predecessor, it is reweighted to zero.Source: §3.3 · pp. 8–9
3.4 Video Harmonizer
- For video harmonization, exponential moving average smoothing of predicted filter arguments is used to make their changes across frames smooth.The EMA coefficient is α = 0.9.Source: §3.4 · p. 9
4 Experiments
Experiments compare Harmonizer with prior methods on iHarmony4 at standard and original resolutions, assess video performance and efficiency, and test components and user preferences.
4 Experiments
- Experiments use iHarmony4's four subsets; each sample contains a natural image, foreground mask, and composite, while fMSE measures error only in the foreground.Training composites are also created through data augmentation without extra data.Source: §4 · p. 10
4.1 Comparison with State-of-the-arts
- At 256×256, Harmonizer outperforms existing methods on all four iHarmony4 subsets and reduces average MSE by 35% versus the state-of-the-art method.Table 1 reports the quantitative comparison at 256×256 resolution.Source: §4.1 · p. 10, Table 1 · p. 10
- At original image resolutions, Harmonizer's MSE is 50% lower than the state-of-the-art method, with only a small performance drop.Existing methods use low-resolution inputs and upsample their outputs, while Harmonizer processes high-resolution inputs directly.Source: §4.1 · pp. 11–12, Table 2 · p. 11
- At 1080P on an RTX3090, Harmonizer runs at 56 fps—about 4× faster than DOVE—and is faster, lighter, and more memory efficient than other methods.Table 3 compares inference speed, model size, and GPU memory.Source: Table 3 · p. 11, §4.1 · p. 12
4.2 Video Harmonization Results
- Harmonizer produces stable harmonization across video frames, whereas prior methods suffer from severe flickering.The visual comparison identifies inconsistent foreground appearances in prior-method results.Source: §4.2 · p. 12, Figure 7 · p. 13
4.3 Ablation Study
- Both the cascade regressor and dynamic loss improve performance; performance rises with filter count, and at least four filters are needed to avoid large degradation.The ablation compares 2, 4, 6, and 8 filters.Source: §4.3 · pp. 12–13, Table 4 · p. 13
4.4 User Studies
- In user studies, Harmonizer achieves the highest Bradley-Terry scores among the compared results.Twelve participants ranked results from methods and humans; prior methods on video scored below the original composite inputs because of flickering.Source: §4.4 · p. 14, Table 5 · p. 14
5 Conclusion
Harmonizer draws on how humans adjust image filters and combines a novel architecture with cascade regression and dynamic loss. It is lighter and faster than prior methods while achieving state-of-the-art performance, but can fail on specific color or lighting inconsistencies.
5 Conclusion
- Harmonizer combines a novel architecture with a cascade regressor and dynamic loss strategy, and achieves state-of-the-art performance while being lighter and faster than prior methods.The design was inspired by studying human image harmonization.Source: §5 · p. 14
- The method may fail on color-specific appearance inconsistencies or differing foreground and background lighting; color-separated filters are proposed as future work.Figure 9 shows a case with a strong hue difference in the blue range.Source: §5 · p. 14
- Without cascade regression, filters can interfere—for example, simultaneous brightness, highlight, and shadow adjustments may make results too bright or dark; without dynamic loss, color-temperature predictions may be inaccurate.These effects are illustrated in the ablation results.Source: Figure 11 · p. 15