DoveNet: Deep Image Harmonization via Domain Verification

1. Introduction

• Deep harmonization requires many composite-target pairs, but manual harmonization is expensive; prior synthesized data [43] was unavailable publicly and limited in diversity, realism, and real composites.Synthesizing composites from real images offers a scalable alternative to expert editing.Source: §1 · p. 1
• iHarmony4 comprises HCOCO, HAdobe5K, HFlickr, and Hday2night, built from COCO, Adobe5k, Flickr, and aligned day2night images.Flickr images are manually filtered for unsuitable backgrounds, clutter, and apparent unreality; day2night enables composites from real images captured under varied conditions.Source: §1 · pp. 1–2
• DoveNet uses a domain verification discriminator to make a harmonized image’s foreground-background pair resemble real-image pairs, translating the foreground toward the background domain without domain labels.The discriminator learns to distinguish real-image positive pairs from composite-image negative pairs, while the generator aims to fool it.Source: §1 · p. 2

2. Related Work

• Traditional methods matched color distributions, used harmonious color templates, gradient-domain compositing, or transferred multi-scale statistics; later work also considered visual realism.Source: §2 · p. 2
• Prior work used CNNs for realism assessment [54] and end-to-end harmonization with segmentation [43], while attention separated foreground/background features [45]; the proposed method instead translates foreground to background domains using a domain verification discriminator.Source: §2 · p. 2
• A paired GAN such as [11], designed for paired training data, can be applied to image harmonization but does not consider its uniqueness; the model extends paired GANs with a domain verification discriminator.Source: §2 · p. 2

3.1. Composite Image Generation

• For aligned Adobe5k and day2night images, the corresponding foreground region is overlaid at the same location; day2night pairs with changed or missing objects are excluded.Adobe5k provides five photographer-edited styles per image, while day2night scenes provide aligned captures under different conditions.Source: Figure 1 · p. 3, §3.1 · p. 3
• Across all four sub-datasets, foregrounds are segmented, kept to reasonable image areas, and selected to cover a wide range of categories before adjustment.COCO uses its provided masks for 80 categories; Adobe5k, Flickr, and day2night regions are manually segmented.Source: §3.1 · p. 3
• For unaligned COCO and Flickr images, same-category references supply color transfers chosen randomly from four methods; Flickr references are retrieved from ADE20K using scene parsing.COCO references come from COCO itself, and the four methods represent parametric or non-parametric transfer in decorrelated or correlated color spaces.Source: §3.1 · p. 3

3.2. Composite Image Filtering

• Filtering combines an aesthetics model, a mask-informed binary CNN, and human review to remove unrealistic composites, unreasonable COCO regions, and implausible hue changes.The CNN is trained with real images as positive samples and composites flagged as unrealistic by the aesthetics model as negative samples; human review removes residual unrealistic examples.Source: §3.2 · pp. 3–4

3.3. Differences between Our Dataset and [43]

• iHarmony4 extends [43] with Hday2night real composites, broader attention to synthesis diversity and quality, and both automatic and deliberate manual filtering.The authors describe Hday2night as closer to real-world application than the other sub-datasets.Source: §3.3 · p. 4

4. Our Method

• DoveNet uses an attention-enhanced U-Net generator and two discriminators to produce plausible images whose foreground and background belong to the same domain.The generator receives the composite image and foreground mask; the global discriminator judges realism, while the domain verification discriminator checks foreground-background domain consistency.Source: §4 · p. 4

4.1. Attention Enhanced Generator

• Three full-attention blocks enhance U-Net [39] features by separately attending to encoder and decoder features before concatenation, while L1 reconstruction encourages closeness to the real image.The attention integrates spatial and channel attention [48]; the design is inspired by [45].Source: §4.1 · p. 5

4.2. Global Discriminator

• The global discriminator uses spectral normalization after each convolution and hinge loss [28], training against real images and generated images to encourage realistic outputs.It scores real images higher than generated images, while the generator seeks to fool it and obtain high scores.Source: §4.2 · p. 5

4.3. Domain Verification Discriminator

• The domain verifier treats real-image foreground-background pairs as positive and composite-image pairs as negative, then uses masked partial convolutions to extract separate domain representations.Partial convolution aggregates features only from masked regions, avoiding leakage from unmasked regions and corruption from zero padding.Source: §4.3 · p. 5
• The verifier computes foreground-background inner-product similarity scores, and its training encourages high scores for positive pairs while the generator seeks high scores for generated pairs.Matching foreground and background domains is intended to make the generated image's regions compatible.Source: §4.3 · p. 5

5.1. Dataset Statistics

• The four sub-datasets contain 42,828 HCOCO, 21,597 HAdobe5k, 8,277 HFlickr, and 444 Hday2night composite–real image pairs.Pairs are split into training and test sets without placing the same target image in both; Hday2night’s 444 pairs are retained without degrading dataset quality despite stringent selection.Source: §5.1 · pp. 5–6

5.3. Comparison with Existing Methods

• DoveNet outperforms all baselines by a large margin and achieves the best results on all four sub-datasets.The model is trained on the merged training sets and evaluated on each sub-dataset’s test set and the combined test set; deep learning methods using the training data generally outperform traditional methods, and S2AM outperforms DIH.Source: Table 1 · p. 6, §5.3 · p. 6

5.2. Implementation Details

• The generator uses eight downsampling and eight upsampling layers; each discriminator has seven layers, and evaluation uses RGB MSE and PSNR on 256 × 256 images.Training uses Adam with learning rate 0.002; the generator’s downsampling blocks use 4 × 4 convolutions with stride 2, LeakyReLU, and instance normalization.Source: §5.2 · p. 6

5.4. Ablation Studies

• The full U-Net+att+adv+ver method achieves the best results on all four sub-datasets, while attention and either discriminator improve on U-Net+att.The U-Net backbone also outperforms DIH’s backbone.Source: Table 3 · p. 7, §5.4 · p. 7

5.5. Qualitative Analyses

• DoveNet produces harmonized images closer to the real-image ground truth, and removing verification yields less harmonious results.The verification discriminator pushes the foreground domain toward the background domain.Source: §5.5 · pp. 7–8, Figure 3 · p. 8

5.6. User Study on Real Composite Images

• DoveNet achieves the highest B-T score among methods on 99 real composite images, where quantitative MSE and PSNR comparison is unavailable.Fifty raters compared pairs of outputs, yielding 51,975 pairwise results; scores were calculated using the Bradley-Terry model.Source: Table 5 · p. 7, §5.6 · p. 8

• iHarmony4 comprises four image harmonization sub-datasets: HCOCO, HAdobe5k, HFlickr, and Hday2night.Source: §6 · p. 8
• DoveNet is a proposed deep image harmonization method with a domain verification discriminator, and experiments on the dataset demonstrate its effectiveness.Source: §6 · p. 8

S1

• Section S1 introduces the attention block used in DoveNet.Source: Supplementary Material for DoveNet:… · p. 11

S2–S4

• Sections S2–S4 analyze iHarmony4 by foreground ratio, color transfer method, and semantic category.Source: Supplementary Material for DoveNet:… · p. 11

S5–S6

• Sections S5–S6 show samples of manually filtered images and final images in the dataset.Source: Supplementary Material for DoveNet:… · p. 11

S7

• Section S7 exhibits results of different methods on all 99 real composite images.Source: Supplementary Material for DoveNet:… · p. 11

S1. Details of Attention Block

• The block concatenates encoder and decoder features, learns separate full spatial-and-channel attention maps, applies them element-wise, then concatenates the attended features.Each map is produced by a 1 × 1 convolution followed by Sigmoid activation.Source: S1. Details of Attention Block · p. 11
• The encoder attention map is intended to emphasize background features because foreground features may not yet be fully harmonized.Source: S1. Details of Attention Block · p. 11
• Unlike [S11], which uses explicit mask control for separate foreground and background attention maps, DoveNet argues its block can use mask information from the generator input automatically.The authors describe their attention block as simple yet effective compared with [S11].Source: S1. Details of Attention Block · p. 11

Analyses of Foreground Ratio

• Foreground ratios are mainly in [0%, 70%], have a long tail, and differ substantially across the four sub-datasets.Figure S2a reports the distributions for HCOCO, HAdobe5k, HFlickr, Hday2night, and the whole dataset.Source: S2. Analyses of Foreground Ratio · p. 11
• HCOCO foregrounds exceed 1% and are below 80% of image area, but filtering low-quality composites makes larger foreground regions more likely to be removed.COCO segmentation masks provide the annotations used to select foregrounds.Source: S2. Analyses of Foreground Ratio · p. 11
• HAdobe5k and HFlickr have relatively large foreground regions, reflecting prominent subjects in Adobe5k and Flickr filtering for obvious foregrounds and unblurred backgrounds.Because these datasets lack segmentation masks, foreground regions are manually selected and segmented.Source: S2. Analyses of Foreground Ratio · p. 11
• Hday2night has relatively small foreground regions because selection favors static objects that remain consistent across capture conditions, excluding moving, deformable, or substantially changing objects.These constraints are used to select reasonable foregrounds for day2night composites.Source: S2. Analyses of Foreground Ratio · p. 11
• The whole-dataset foreground-ratio distribution is close to that of 99 real composites (48 from Xue et al., 51 from Tsai et al. [S9]), supporting its real-world relevance.The comparison is shown in Figure S2b.Source: S2. Analyses of Foreground Ratio · p. 11

Method selection

• The methods span parametric and non-parametric mappings in correlated and decorrelated spaces, covering four quadrants to diversify composites despite the tradeoffs of each choice.Correlated RGB requires modifying all channels coherently, whereas decorrelated spaces enable channel-wise processing but may miss local color detail and inter-channel relationships.Source: S3. Analyses of Color Transfer Methods · p. 11

Color-transfer methods

• Parametric transfer uses global statistics: Reinhard et al. match channel means and standard deviations in Lαβ, while Xiao et al. transfer means and covariance in RGB.The RGB method uses image-specific principal-component axes in place of rotation to Lαβ.Source: S3. Analyses of Color Transfer Methods · p. 12
• Fecker et al. match target color histograms using cumulative mapping in Y CbCr, while Pitié et al. transfer 3D distributions by iteratively matching 1D projections.The iterative method can increase graininess, especially when the two images’ color dynamics differ greatly; Pitié et al. proposed a second-stage post-processing algorithm to reduce this artifact.Source: S3. Analyses of Color Transfer Methods · p. 12

Method selection

• The four selected methods are global transfer in Lαβ and RGB, cumulative histogram matching, and iterative color-distribution transfer.One method is randomly selected when generating HCOCO and HFlickr composites.Source: S3. Analyses of Color Transfer Methods · p. 12

Results

• The proposed method achieves the best results on all four HCOCO test subsets, while their differing MSE and PSNR statistics support using multiple transfer methods.The subsets correspond to the four color-transfer methods.Source: Table S1 · p. 13, S3. Analyses of Color Transfer Methods · p. 13

Dataset construction

• After filtering, iterative transfer yields 17,009 HCOCO composites, more than the other methods, indicating it more often produces realistic, reasonable composites.The remaining counts for,, and are 9,581, 8,119, and 8,119, respectively.Source: S3. Analyses of Color Transfer Methods · p. 13

S4. Analyses of Semantic Category

• Using HCOCO’s COCO-derived foreground labels, the analysis reports input and harmonized-result fMSE by category, defining easy and hard categories by relative improvement over the input composite.Table S2 separates hard categories (S2a) from easy categories (S2b).Source: S4. Analyses of Semantic Category · p. 13, Table S2 · p. 14
• Categories with small intra-category variance, such as mouse and keyboard, show significantly greater fMSE improvement than high-variance categories such as person.The comparison covers HCOCO categories.Source: S4. Analyses of Semantic Category · p. 13, Table S2 · p. 14

S5. Examples of Manual Filtering

• Highly occluded COCO person annotations, such as a hand or shoulder alone, were removed from HCOCO because they were not meaningful harmonization targets.Foreground regions in HCOCO were based on COCO semantic segmentation annotations.Source: S5. Examples of Manual Filtering · p. 14
• HCOCO and HFlickr composites were removed when color transfer changed an object’s hue dramatically, such as a red shirt becoming yellow.The mismatch was especially likely for categories with large within-category variance, such as person; harmonizing the altered shirt would be nonsensical.Source: S5. Examples of Manual Filtering · p. 14
• Hday2night composites were removed when foregrounds gained changes unrelated to capture conditions, such as Eiffel Tower lights turning on or snow appearing on a roof.The authors considered these changes outside the scope of image harmonization.Source: S5. Examples of Manual Filtering · p. 14
• Filtering these unqualified examples was intended to ensure the high quality of the iHarmony4 dataset.Manual filtering supplemented two automatic filtering steps for HCOCO, HFlickr, and Hday2night.Source: S5. Examples of Manual Filtering · p. 14

S6. Examples of Our iHarmony4 Dataset

• Figure S4 shows examples from all four iHarmony4 sub-datasets, with each row corresponding to one sub-dataset.Source: S6. Examples of Our iHarmony4 Dataset · p. 15
• Each Figure S4 example includes the original real image, its synthesized composite, and the foreground mask.Source: S6. Examples of Our iHarmony4 Dataset · p. 15

S7. Results on Real Composite Images

• The real-image set contains 99 composites: 48 from Xue et al. and 51 from Tsai et al. [S9].Source: S7. Results on Real Composite Images · p. 15
• DoveNet generally produces satisfactory harmonized images across varied real-image scenes and objects.The comparisons cover real composites and harmonization results from DoveNet and existing methods.Source: S7. Results on Real Composite Images · p. 15
• The study compares its proposed harmonization method with Lalonde and Efros, Xue et al., Zhu et al. [S14], DIH [S9], and S2AM [S11], reporting generally satisfactory results across scenes and objects.Source: S7. Results on Real Composite Images · p. 15
• Figure S5 presents an input composite, five state-of-the-art methods, and DoveNet results on real composite images.Source: Figure S5 · p. 15