Source-linked AI summary
Region-aware Adaptive Instance Normalization for Image Harmonization
Jun Ling, Han Xue, Li Song, Rong Xie, Xiao Gu
TL;DR
Image harmonization must reconcile foreground and background visual styles, a need existing mapping-based methods do not explicitly model. The paper reframes harmonization as background-to-foreground style transfer and introduces RAIN, which transfers background statistics to foreground features. Across reported experiments, RAIN improves existing harmonization networks, while the authors identify remaining behavior and scope limitations.
Problem
Image harmonization lacks explicit modeling of visual-style consistency between foreground and background images, despite requiring compatible foreground appearance for realistic composites.
Method
RAIN reframes harmonization as background-to-foreground style transfer and applies background-derived channel-wise statistics to normalized foreground features.
Results
RAIN improves existing harmonization networks and outperforms previous normalization methods by a large margin across the reported experiments.
Takeaways & Limitations
RAIN can serve as a drop-in module for existing image harmonization networks and produce performance gains.
Takeaways & Limitations
The authors report unclear encoder-only benefits and softened sharp foregrounds on dark backgrounds, requiring further investigation.
Abstract
from arXiv · showhide
Image composition plays a common but important role in photo editing. To acquire photo-realistic composite images, one must adjust the appearance and visual style of the foreground to be compatible with the background. Existing deep learning methods for harmonizing composite images directly learn an image mapping network from the composite to the real one, without explicit exploration on visual style consistency between the background and the foreground images. To ensure the visual style consistency between the foreground and the background, in this paper, we treat image harmonization as a style transfer problem. In particular, we propose a simple yet effective Region-aware Adaptive Instance Normalization (RAIN) module, which explicitly formulates the visual style from the background and adaptively applies them to the foreground. With our settings, our RAIN module can be used as a drop-in module for existing image harmonization networks and is able to bring significant improvements. Extensive experiments on the existing image harmonization benchmark datasets show the superior capability of the proposed method. Code is available at {https://github.com/junleen/RainNet}.
1. Introduction
Image harmonization seeks realistic composites by making foreground appearance compatible with each background’s visual style. The paper reframes this as background-to-foreground style transfer and proposes RAIN to adapt foregrounds accordingly.
- Motivation: Image harmonization reduces manual editing by adjusting foreground objects to integrate seamlessly with new backgrounds.The task targets realistic composites for editing and augmentation, especially for non-expert users.
- Motivation: Visual style discrepancy arises when foreground and background properties such as illumination, color temperature, saturation, hue, and texture are inconsistent.The paper identifies style consistency as necessary for realistic composites.
- Approach: The paper reframes harmonization as background-to-foreground style transfer, adapting foreground appearance to each new background.Foregrounds may require different appearances when pasted into different environments.
- Approach: RAIN learns style from background features and applies it to foreground objects using channel-wise mean and variance alignment.The module uses convolutional features and the foreground mask as inputs.
- Contribution: RAIN captures style only from background features and is designed as a plug-and-play module for existing harmonization networks.The authors describe it as simple, effective, and applicable to existing architectures.
- Results: Extensive experiments report that the proposed method surpasses state-of-the-art methods by a large margin.This is the paper’s principal reported outcome.
2. Related work
Prior harmonization methods use handcrafted appearance cues, learned realism or semantic constraints, while related style-transfer and normalization methods provide context for RAIN’s design.
- Image harmonization: Traditional harmonization methods match foreground and background using handcrafted color, gradient, multiscale statistical, or semantic features.These methods focus on appearance matching with limited attention to visual realism.
- Neural style transfer: Neural style transfer separates content preservation from visual-style rendering and commonly transfers feature statistics from convolutional representations.The paper situates RAIN within this style-transfer perspective.
- Normalization: Normalization methods include unconditional BN, IN, LN, and GN, alongside conditional CBN, CIN, SPADE, RN, and AdaIN.Unconditional methods align feature statistics without external guidance, whereas conditional methods use external data for affine parameters.
- RainNet: RainNet’s generator is presented through an architecture overview using convolutional and transposed-convolutional components.The figure legend defines Conv. as convolution and Trans. as transposed.
- Normalization: SPADE uses spatially varying semantic-mask transformations, while RN addresses mean-variance shifts without modeling background–foreground semantic connections.The paper explains why these methods do not directly match the harmonization setting.
3. Our approach
The approach composes foregrounds and backgrounds under a mask, then uses RAIN to transfer background statistics into normalized foreground features. Experiments support background-only style transfer and decoder placement.
- Problem formulation: The composite image is formed as Ic = M ◦ If + (1 − M) ◦ Ib, where M selects the foreground region.The background mask is defined as 1 − M, and ◦ denotes the Hadamard product.
- Problem formulation: The harmonization generator G maps the composite image and mask to an output optimized toward the ground truth with an L1 reconstruction loss.The model is defined as Ĩ = G(Ic, M).
- RAIN module: RAIN takes convolutional features and a resized foreground mask, separates foreground and background regions, and normalizes the foreground features.The module is defined at layer i using activations Fi and mask Mi.
- RAIN module: RAIN obtains scale and bias parameters from background features and applies them channel-wise to normalized foreground activations.This transfers background style while excluding inconsistent foreground statistics.
- RAIN module: Unlike AdaIN, RAIN transfers statistics within the same image and is trained specifically for harmonization rather than external-image stylization.The paper states that RAIN’s style parameters are fitted to foreground adjustment operations.
- Experiments: RAIN improves over IN and BN by a large margin, while those methods outperform RN in the reported comparison.The comparison is presented as evidence supporting background-guided statistic transfer.
4. Implementation
The experiments use iHarmony4, a four-subdataset benchmark with 73,147 synthesized composite/ground-truth pairs, following prior train-test splits. Training uses Adam for 100 epochs with 256×256 inputs and batch size 12.
- iHarmony4 contains HCOCO, HAdobe5K, HFlicker, and Hday2night, with 73,147 synthesized composite-image and ground-truth pairs.
- Models are trained with Adam at learning rate 0.0002 for 100 epochs on 256×256 inputs with batch size 12.
5. Experimental Results
RainNet improves harmonization quality across benchmark comparisons, qualitative examples, normalization ablations, and implementation studies. The strongest configuration uses RAIN in the decoder with IN in the encoder, while limitations remain for sharp foregrounds against dark backgrounds.
- Comparison with existing methods: RainNet improves the baseline across iHarmony4 subdatasets and achieves the best results on HCOCO, HAdobe5K, and HFlickr.
- Qualitative comparisons: RainNet produces more visually consistent foreground-background integration than competing methods, including underexposed backgrounds with brighter foreground objects.
- Ablation study: RAIN-Decoder outperforms other normalization strategies on average, whereas RAIN-Encoder performs comparably to IN.
- Ablation study: RAIN-1 slightly outperforms RAIN-2, RAIN-3, and RAIN-4, while RAIN configurations significantly improve over IN, BN, and RN.
- Ablation study: Adding RAIN to DIH raises PSNR from 33.36dB to 33.84dB, a +0.48dB improvement.
- Discussions and limitations: RainNet improves over DoveNet by 1.37dB PSNR and 12.07 estimation-error units, although generator settings partly contribute to these gains.
- Discussions and limitations: RAIN can soften sharp foreground objects in samples with dark backgrounds, reducing visual style discrepancy but leaving a noted limitation.
6. Conclusion
The paper presents RAIN as a solution to visual style inconsistency in image harmonization and reports improvements over prior normalization methods and existing networks.
- RAIN addresses visual style inconsistency by providing a simple normalization module that outperforms previous normalization methods by a large margin.
- Applying RAIN to existing networks such as DIH produces performance gains over those models.
Supplementary Material
The supplementary material provides implementation details, additional RAIN ablations, real-composite comparisons, and failure-case discussion.
- Supplementary sections cover the attention-block architecture and model training objective.
- Additional experiments examine RAIN application strategies and real composite-image comparisons, followed by failure-case discussion.
B. Implementation Details
The implementation combines attention blocks, reconstruction and adversarial objectives, and domain verification to train the harmonization model. Training uses spectral normalization and partial convolutions to stabilize discrimination and prevent unmasked-region information leakage.
- Attention block: Three attention blocks are added to the baseline U-Net decoder to improve feature fusion.Each block concatenates encoder and decoder features, computes a sigmoid coefficient map with a 1×1 convolution, and modulates features element-wise.
- Training objectives: The model optimizes an L1 reconstruction loss that makes the harmonized output approach the ground-truth image.The composite image and foreground mask are inputs to G, which produces the harmonized image.
- Training objectives: Adversarial training uses a discriminator that distinguishes natural images from harmonized samples while encouraging G to produce realistic outputs.The adversarial objective is included in the full loss together with domain verification.
- Training objectives: Domain verification compares foreground-background feature representations from real and harmonized images to encourage similar domain features for positive pairs.A domain encoder extracts region representations, and a domain discriminator measures their similarity.
- Implementation: Spectral normalization stabilizes both discriminators, while partial convolutions extract domain codes from irregular masked regions without unmasked-region leakage.The discriminators share the structure used in prior work.
C. More ablation studies
The ablations show that RAIN placement strongly affects optimization and convergence. Decoder and outermost-layer placements are more effective than sparse or middle-layer placements, with diminishing returns after sufficient decoder layers.
- Decoder placement: More RAIN layers in the decoder produce more stable training and better convergence than using only one layer.Adding a fifth decoder layer provides no further benefit after four layers, where RAIN can become equivalent to instance normalization at small feature sizes.
- Outermost placement: Symmetric normalization improves optimization, with Baseline+RAIN-5 and Baseline+RAIN-6 outperforming other type-II designs.Baseline+RAIN-1 performs slightly better than Baseline+RAIN-2, Baseline+RAIN-3, and Baseline+RAIN-4.
- Middle-layer placement: RAIN-5 inserted in the middle of the network performs much worse than outermost-layer RAIN-5 but better than middle-layer RAIN-3.The paper relates this pattern to visual style properties being low-level features learned near the encoder input and decoder output.
D. Results on real composite images
On real composite images, the proposed method produces better visual consistency between foreground and background images and outperforms competing methods in most cases.
- Real composite images: The proposed method achieves better visual consistency and outperforms other methods in most real-composite-image examples.The comparisons use real composite images from datasets associated with prior work.
E. Failure case
The proposed RainNet fails when a sharp foreground object is combined with a blurred background. In that setting, S2AM performs better, while all compared methods still show boundary artifacts.
- Failure case: RainNet fails on composites with a sharp foreground object and a dim, blurred, or blurred background.The failure concerns harmonizing incompatible foreground sharpness and background blur.
- Failure case: S2AM performs better than RainNet and other methods on the blurred-background, sharp-foreground case.The compared methods nevertheless fail to produce consistent boundaries and introduce visible artifacts.
- Comparison setup: Real-composite examples compare the proposed model with three state-of-the-art methods using foreground masks and test-dataset samples.The captions report better harmonized visual results for the proposed method in these examples.