Source-linked AI summary
DoveNet: Deep Image Harmonization via Domain Verification
Wenyan Cong, Jianfu Zhang, Li Niu, Liu Liu, Zhixin Ling, Weiyuan Li, Liqing Zhang
TL;DR
Image harmonization needs large, high-quality training pairs, but manual creation is costly and foreground–background domain mismatch degrades composites. The paper constructs iHarmony4 and proposes DoveNet with domain verification; experiments demonstrate effectiveness, while dataset filtering remains bounded by problematic cross-condition or color-transfer changes.
Problem
Large-scale harmonization training pairs are difficult to create manually because producing compatible foregrounds requires extensive skilled effort.
Method
The paper constructs iHarmony4 from four sub-datasets and proposes DoveNet with a domain verification discriminator that aligns foreground and background domains.
Results
Extensive experiments on iHarmony4 demonstrate the effectiveness of DoveNet.
Takeaways & Limitations
iHarmony4 provides four large-scale harmonization sub-datasets, including Hday2night, while DoveNet extends harmonization with domain verification.
Takeaways & Limitations
Color transfer can create implausible hue changes, and condition-based overlays can include essential object changes, requiring manual removal.
Abstract
from arXiv · showhide
Image composition is an important operation in image processing, but the inconsistency between foreground and background significantly degrades the quality of composite image. Image harmonization, aiming to make the foreground compatible with the background, is a promising yet challenging task. However, the lack of high-quality publicly available dataset for image harmonization greatly hinders the development of image harmonization techniques. In this work, we contribute an image harmonization dataset iHarmony4 by generating synthesized composite images based on COCO (resp., Adobe5k, Flickr, day2night) dataset, leading to our HCOCO (resp., HAdobe5k, HFlickr, Hday2night) sub-dataset. Moreover, we propose a new deep image harmonization method DoveNet using a novel domain verification discriminator, with the insight that the foreground needs to be translated to the same domain as background. Extensive experiments on our constructed dataset demonstrate the effectiveness of our proposed method. Our dataset and code are available at https://github.com/bcmi/Image_Harmonization_Datasets.
1 MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University 2 East China Normal University
The paper addresses limited high-quality harmonization data by constructing iHarmony4 and introduces DoveNet, which translates foreground appearance toward the background domain.
- Dataset motivation: Manual harmonization labels are expensive, motivating synthesized composite–real image pairs for large-scale training.The strategy modifies real images to create inconsistent composites while retaining the original real image as the harmonized target.
- Dataset contribution: Hday2night adds aligned images captured under different conditions, producing real composite images closer to real-world applications.The paper identifies this as an additional sub-dataset absent from the prior dataset.
- Method contribution: DoveNet models harmonization as translating the foreground domain to the background domain without requiring domain labels.Real images form same-domain foreground–background pairs, whereas composites form different-domain pairs.
- Method contribution: The method introduces domain verification alongside a global discriminator to guide foreground–background compatibility.The contribution is presented as a novel image harmonization method equipped with a domain verification discriminator.
- Dataset contribution: iHarmony4 contains four sub-datasets: HCOCO, HAdobe5k, HFlickr, and Hday2night.The dataset is built from COCO, Adobe5k, Flickr, and day2night sources.
2. Related Work
The paper situates DoveNet within image harmonization and image-to-image translation, then describes dataset construction through segmentation, appearance adjustment, filtering, and architectural components.
- Related Work: Prior harmonization methods match appearance statistics, use color templates, apply gradient-domain compositing, or assess composite realism.Later CNN methods perform end-to-end harmonization and incorporate semantic or attention features.
- Related Work: Unlike general paired GANs, DoveNet explicitly verifies whether foreground and background belong to the same domain.This targets the distinctive domain mismatch in image harmonization.
- Dataset Construction: Dataset construction begins with foreground segmentation and adjusts foreground appearance using either cross-image color transfer or aligned counterpart images.COCO and Flickr use same-category references, while Adobe5k and day2night use corresponding regions from alternate versions or conditions.
- Dataset Construction: Adobe5k uses photographer-edited counterparts, whereas day2night uses aligned images from different capture conditions and excludes pairs with object changes.The day2night procedure preserves the same object while varying capture conditions.
- Composite Image Filtering: Filtering combines an aesthetics model, a binary CNN classifier, and manual annotation to remove unrealistic or unsuitable composites.Manual filtering addresses occlusion, implausible foregrounds, and excessive hue changes.
- DoveNet Architecture: DoveNet combines an attention-enhanced U-Net generator with global and domain verification discriminators.The generator maps the composite image and foreground mask to a harmonized image.
- Dataset Differences: iHarmony4 extends the earlier dataset with Hday2night and additional automatic and manual filtering for diversity and quality.The authors describe Hday2night as closer to real-world application and emphasize quality control.
4. Our Method
DoveNet reconstructs a harmonized image with an attention-enhanced U-Net and two discriminators: one for global realism and one for foreground–background domain compatibility.
- Overall architecture: DoveNet takes a composite image and foreground mask as inputs and outputs a harmonized image intended to reconstruct the real image.The harmonized output is expected to be as close as possible to the corresponding real image.
- Generator: An attention-enhanced U-Net generator uses skip links and three attention blocks combining spatial and channel attention.Encoder and decoder features are attended separately before being concatenated.
- Global discriminator: The global discriminator distinguishes real images from generated images, encouraging plausible outputs through adversarial training.Its hinge-loss training encourages large scores for real images and small scores for generated images, while the generator seeks large scores.
- Domain verification discriminator: The domain verification discriminator evaluates whether foreground and background regions belong to the same domain by comparing their extracted representations.Real images provide positive foreground–background pairs, whereas composite images provide negative pairs.
- Training objective: Matching foreground and background domains is intended to produce compatible foreground and background, and its adversarial objective is combined with reconstruction loss.The generator loss uses the reconstruction term together with global and domain-verification adversarial losses.
5. Experiments
Experiments evaluate DoveNet and baselines across four iHarmony4 sub-datasets, using quantitative benchmarks, ablations, foreground-ratio analysis, qualitative examples, and a user study. DoveNet achieves the best reported performance across the sub-datasets and foreground-ratio ranges, while its verification discriminator improves harmonization qualitatively and quantitatively.
- Comparison with Existing Methods: The experiments evaluate traditional and deep-learning baselines, adapting DIH by removing its unavailable semantic-segmentation branch.All baselines use released code when available or an authors' implementation otherwise.
- Experimental Setup: The merged training set is evaluated separately on each sub-dataset and on the whole test set.Training and test splits prevent the same target image from appearing in both sets.
- Comparison with Existing Methods: DoveNet outperforms all baselines by a large margin and achieves the best results on HCOCO, HAdobe5k, HFlickr, and Hday2night.Deep-learning methods trained on paired data generally outperform traditional methods, while S2AM improves over DIH.
- Ablation Studies: Adding the attention block, global discriminator, and verification discriminator progressively improves the U-Net backbone, with the full model achieving the best results on all four sub-datasets.The ablation compares U-Net, attention, adversarial, and verification components.
- Ablation Studies: DoveNet outperforms all baselines on both MSE and foreground MSE in every foreground-ratio range, especially when the foreground is large.Foreground MSE is introduced because background reconstruction error can overwhelm foreground harmonization error when the foreground is small.
- Qualitative Analyses: Qualitative comparisons show DoveNet outputs are more harmonious and closer to real images, while the verification discriminator pushes the foreground domain toward the background domain.The comparison includes DoveNet without verification and the full DoveNet model.
- Comparison with Existing Methods: On 99 real composite images without ground truth, a Bradley-Terry user study ranks DoveNet highest among the compared methods.MSE and PSNR cannot be computed for these images, so subjective pairwise comparisons are used.
6. Conclusions
The work contributes iHarmony4, an image harmonization dataset with four sub-datasets, and proposes DoveNet, a deep harmonization method using a domain verification discriminator. Experiments on the dataset demonstrate the method’s effectiveness.
- iHarmony4 comprises four sub-datasets: HCOCO, HAdobe5k, HFlickr, and Hday2night.
- DoveNet is a deep image harmonization method with a domain verification discriminator.
- Extensive experiments on iHarmony4 demonstrate the effectiveness of DoveNet.
Supplementary Material for DoveNet: Deep Image Harmonization via Domain Verification
The supplementary material documents the DoveNet attention block, analyzes iHarmony4’s construction and foreground-ratio distributions, presents filtered and final dataset samples, and reports results on 99 real composite images.
- The supplement introduces DoveNet’s attention block in Section S1.
- It analyzes iHarmony4 by foreground ratio, color transfer method, and semantic category.
- It shows manually filtered and final dataset samples in Sections S5 and S6.
- It reports results for different methods on 99 real composite images in Section S7.
S1. Details of Attention Block
The attention block enhances U-Net by learning separate full attention maps for encoder and decoder features after concatenating those features. Unlike prior work, it does not require explicit mask control because the mask is included in the generator input.
- The block concatenates encoder and decoder features before learning attention maps.
- It learns separate full attention maps for encoder and decoder features by integrating spatial and channel attention.
- Unlike [S11], the block uses no explicit mask control because the generator input already includes the mask.
- The proposed attention block is described as simpler yet effective than the block in [S11].
S2. Analyses of Foreground Ratio
iHarmony4 covers a wide, long-tailed range of foreground ratios, with distributions varying across sub-datasets. Its overall foreground-ratio distribution is close to that of 99 real composite images.
- Foreground ratios in iHarmony4 are mainly distributed between 0% and 70% and exhibit a long-tail distribution.
- The four sub-datasets have substantially different foreground-ratio distributions because their acquisition processes differ.
- HCOCO has relatively small foreground regions because larger synthesized regions are more likely to be filtered as low quality.
- HAdobe5k and HFlickr have relatively large foreground regions after selecting images with prominent or obvious subjects.
- Hday2night has relatively small foreground regions because selected foregrounds are static objects consistent across capture conditions.
- The whole-dataset foreground-ratio distribution is close to that of 99 real composite images.
S3. Analyses of Color Transfer Methods
The analysis evaluates four color-transfer methods across complementary method categories and finds that diverse transfer strategies improve dataset coverage, while DoveNet remains robust across them.
- Method selection: Four representative methods span parametric and non-parametric transfers in correlated and decorrelated color spaces.The methods are global transfer in Lαβ, global transfer in RGB, cumulative histogram matching, and iterative color distribution transfer.
- Method selection: The four methods are randomly selected when generating HCOCO and HFlickr composites to enrich synthesized-image diversity.
- Dataset distributions: HCOCO and Hday2night contain more small-foreground images, whereas HAdobe5k and HFlickr contain more large-foreground images; iHarmony4 resembles real composites overall.
- Filtering analysis: 17,009 HCOCO composites remained after filtering with iterative color distribution transfer, exceeding the 9,581, 8,119, and 8,119 composites from the other methods.The reported counts correspond respectively to iterative transfer, global Lαβ transfer, global RGB transfer, and cumulative histogram matching as listed in the passage.
- Robustness analysis: HCOCO subsets produced by different transfer methods have considerably different input MSE and PSNR statistics, supporting the use of multiple methods.
- Robustness analysis: DoveNet achieves the best results on all four HCOCO color-transfer subsets, indicating robustness to transfer-method variation.
S4. Analyses of Semantic Category
The semantic-category analysis shows that harmonization gains depend on category variability: low-variance categories improve more than high-variance categories.
- Category analysis: HCOCO semantic segmentation masks provide category labels for comparing foreground harmonization across object categories.
- Category analysis: Categories with small intra-category variance, such as mouse and keyboard, show significant fMSE improvement.
- Category analysis: Categories with large intra-category variance, such as person, show relatively small fMSE improvement.
S5. Examples of Manual Filtering
Manual filtering removes composites whose foregrounds are occluded, implausibly recolored, or changed for reasons outside capture-condition harmonization.
- Filtering criteria: Manual filtering removes highly occluded foregrounds that are not meaningful targets for harmonization.COCO may label a fragment such as a hand or shoulder as a person foreground.
- Filtering criteria: HCOCO and HFlickr filtering removes composites with dramatic hue changes caused by transferring color between different same-category objects.The paper gives a red-shirt-to-yellow-shirt example that should not be harmonized back to the original color.
- Filtering criteria: Hday2night filtering removes essential foreground changes unrelated to capture conditions, such as Eiffel Tower lights switching on or snow appearing.
- Dataset quality: Filtering these cases is intended to preserve the high quality of the iHarmony4 dataset.
- Dataset examples: Examples present each sub-dataset using real images, synthesized composites, and foreground masks.
S6. Examples of Our iHarmony4 Dataset
The supplementary examples visualize iHarmony4 construction and compare DoveNet with existing methods on real composite images across varied scenes and objects.
- Dataset examples: Figure S4 shows HCOCO, HAdobe5k, HFlickr, and Hday2night examples, each paired with a real image, synthesized composite, and foreground mask.
- Real-composite results: Figures S5–S15 display results for 99 real composite images, comparing DoveNet with five existing harmonization methods.The set includes 48 images from Xue et al. and 51 from Tsai et al.
- Real-composite results: Across the shown scenes and objects, DoveNet generally produces satisfactory harmonized images.