Weak Segmentation-Guided GAN for~Realistic Color Edition

• The goal is to change an object’s perceived color while automatically handling lighting, geometry, and texture variation.The stated aim is to keep user interaction simple by treating the object as having a single color.Source: §1 · p. 2
• The proposed evaluation considers colorization accuracy, realism, stability, and restraint toward unedited areas.Source: §1 · p. 2
• The central challenge is producing realistic results while retaining precise control over the chosen color.Source: §1 · p. 2
• The contributions include a generic model and training strategy, plus a versatile object-aware color-editing tool based on segmentation.Source: §1 · p. 2
• Experiments across varied datasets and use cases report the approach’s best trade-off against state-of-the-art models.The comparison covers color edition, color transfer, and colorization use cases.Source: §1 · p. 2

• Reference-based color transfer uses a source image’s color statistics, while more complex histogram transformations offer broader palette changes at greater computational cost.A neural method matches semantically meaningful entities before applying color transfer, but finding a suitable reference can be difficult and limits palette choice.Source: §2 · p. 3
• Automatic grayscale colorization methods, including conditional GAN approaches, improve realism but do not let users choose the resulting color; stroke-based control lacks precision measures and can affect backgrounds.Source: §2 · p. 3
• Segmentation with histogram matching or interpolation gives users control over selected pixels, but can look unrealistic near mask edges and requires complex color formats.Source: §2 · p. 3
• Conditional GANs can target color changes, but CAGAN has precision and background-modification challenges, while MagGAN requires masks and does not support continuous color space.The paper aims to bridge GAN-based realism and segmentation-based object selection.Source: §2 · pp. 3–4

3 Proposed model

• G takes an input image and desired color, while C, D, R, and OF train it for color accuracy, realism, stability, and restraint.G predicts a difference applied to the input image; the segmentation model S can be any differentiable function and is trained separately and frozen in the described implementation.Source: §3 · p. 4, Figure 2 · p. 4

3.1 Color Estimation module C

• C estimates object color as a segmentation-weighted mean and compares the generated object's color with the desired color using an L2 loss in CIEL*a*b* space.The segmentation model extracts the object color for control; the weighted mean is described as characterizing uniformly colored objects.Source: §3.1 · p. 5, Figure 3 · p. 5

3.2 Discriminator module D

• D adversarially trains G with a standard Wasserstein GAN loss with gradient penalty to promote realistic edited images.The discriminator distinguishes real images from images edited by G.Source: §3.2 · p. 5

3.3 Reconstruction module R

• R uses cycle consistency: G is trained to reconstruct the original image from its own edited output, preserving object geometry and information.The reconstruction module is illustrated in Figure 4.Source: §3.3 · p. 6, Figure 4 · p. 6

3.5 General objective

• OF penalizes changes outside the segmentation mask so that background pixels remain unchanged during object-color edits.The loss compares generated and original images while focusing its penalty on pixels outside the object mask.Source: §3.4 · p. 6, Figure 5 · p. 7
• The final objective combines weighted losses, with λ values controlling their relative importance; extending the framework to multiple objects requires summing losses across objects.S is trained separately beforehand, while G and D are trained through the described workflow.Source: §3.5 · pp. 6–7

4.1 Datasets

• The experiments cover hair and lips, pets, and garments, using segmentation models trained for those datasets or a previously trained pet model plugged in without further adaptation.The hair and lips segmentation network uses CelebA-HQ, supplemented with FFHQ for diversity; the pet use case uses an existing segmentation network.Source: §4.1 · p. 7

4.2 Implementation

• Models use 256×256 images resized by bicubic interpolation and a U-Net segmentation network.The segmentation model is described as trained for the experiments; the generator and discriminator architectures are given in the appendix.Source: §4.2 · p. 8

4.3 Evaluation Protocol

• Evaluation selects random 10,000-image test subset X1 and an equally sized X2, compares generated X1→2 images with X2 using FID, and assesses realism through 150 annotations by five annotators across 30 random triplets.Source: §4.3 · p. 8

4.4 Comparison with other approaches

• Our method balances color precision and realism, beats alpha blending on FID and other approaches on all metrics, and receives most user votes, particularly over CAGAN.In hair-coloring examples, alpha blending produces mask-edge artifacts, CAGAN edits outside the hair, and StyleGAN can alter faces and fail to reproduce the input perfectly.Source: §4.4 · pp. 9–10, Table 1 · p. 10

4.5 Ablation Study

• Object Focus limits edits outside the colored object; reconstruction improves all metrics alongside it, and a differentiable segmentation function considerably improves results.The study also compares predicted masks with ground-truth masks for Object Focus and combined Color Estimation and Object Focus variants.Source: §4.5 · p. 10

5 Conclusion

• The framework introduces an object-aware colorization model with specific losses intended to improve realism and color precision.The ablation table caption reports that Object Focus improves realism and background preservation, while reconstruction positively affects all metrics.Source: §5 · p. 10, Table 2 · p. 11
• The method works with any type of object image and needs only a pretrained, differentiable segmentation model; the authors report state-of-the-art results on several datasets and applications.The stated applications include color edition, color transfer, and colorization, with a reported balance of realism, accurate object color, and maintained geometry and background.Source: §5 · p. 11
• Future work will address spatial color distribution so users can manipulate more than one general object color while retaining simplicity of use.Source: §5 · p. 11

6.1 Implementation Details

• The encoder-decoder generator concatenates spatially duplicated color conditions with decoder features and adds the input image to its output; the discriminator has four convolutional and one fully connected layers.Source: §6.1 · p. 14, Figure 7 · p. 14

6.2 λfoc analysis

• λfoc = 600 is the best trade-off across metrics for this dataset, balancing reduced off-object changes against realism and color precision.Higher λfoc reduces changes outside the colored object but can negatively affect realism and color precision.Source: Table 5 · p. 15, §6.2 · p. 16

6.3 Color control using StyleGAN

• StyleGAN hair control maps desired RGB color changes to StyleSpace edits learned from 1,000 generated images; this is a simple comparison workaround, not an optimized control method.The workflow encodes the input image, computes a StyleSpace edit, and generates the result from the edited vector.Source: §6.3 · p. 17

6.4 Additional results

• Additional examples cover hair and lips, pets, and clothes; reference-image transfer uses extracted colors, while grayscale-trained colorization keeps object-color choices controllable.For colorization, backgrounds tend to remain gray while hair and lips colors are particularly well reconstructed.Source: §6.4 · p. 17