Source-linked AI summary
ManiGAN: Text-Guided Image Manipulation
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, Philip H. S. Torr
TL;DR
Text-guided image manipulation seeks to edit described attributes while preserving unrelated image content, a setting where prior methods produce low-quality results and struggle with complex scenes. ManiGAN combines cross-modal affine region selection and detail correction to generate aligned attributes and reconstruct missing content, and experiments on CUB and COCO demonstrate superiority over existing methods.
Problem
Text-guided manipulation must modify described image attributes while preserving text-irrelevant contents, but existing methods produce low-quality images and fail on complex scenes.
Method
ManiGAN uses an affine combination module for text-relevant region selection and original-content reconstruction, plus a detail correction module for mismatched attributes and missing contents.
Results
Experiments on CUB and COCO show that ManiGAN outperforms existing state-of-the-art methods qualitatively and quantitatively.
Takeaways & Limitations
ManiGAN supports fine-grained semantic image manipulation while reconstructing text-irrelevant contents and provides a metric covering both objectives.
Takeaways & Limitations
COCO and CUB do not provide paired training data for image manipulation, requiring training without paired original, edited-text, and target-image examples.
Abstract
from arXiv · showhide
The goal of our paper is to semantically edit parts of an image matching a given text that describes desired attributes (e.g., texture, colour, and background), while preserving other contents that are irrelevant to the text. To achieve this, we propose a novel generative adversarial network (ManiGAN), which contains two key components: text-image affine combination module (ACM) and detail correction module (DCM). The ACM selects image regions relevant to the given text and then correlates the regions with corresponding semantic words for effective manipulation. Meanwhile, it encodes original image features to help reconstruct text-irrelevant contents. The DCM rectifies mismatched attributes and completes missing contents of the synthetic image. Finally, we suggest a new metric for evaluating image manipulation results, in terms of both the generation of new attributes and the reconstruction of text-irrelevant contents. Extensive experiments on the CUB and COCO datasets demonstrate the superior performance of the proposed method. Code is available at https://github.com/mrlibw/ManiGAN.
1. Introduction
Text-guided image manipulation edits described attributes while preserving unrelated image content, but prior methods struggle with quality and complex scenes. ManiGAN addresses this with cross-modal region selection, content reconstruction, detail correction, and a joint evaluation metric.
- Task and challenges: Natural-language image manipulation edits selected image aspects while preserving contents not described by the user.The task spans attributes such as colour and texture as well as higher-level semantics.
- Task and challenges: Existing text-guided methods produce low-quality images and fail to manipulate complex COCO scenes effectively.These limitations are illustrated in Figure 1.
- ManiGAN: ManiGAN uses a text-image affine combination module to select relevant regions, align them with semantic words, and encode original features for reconstruction.The module targets accurate attribute generation while preserving text-irrelevant contents.
- ManiGAN: A detail correction module rectifies mismatched attributes and completes missing contents in the synthetic image.Together, the proposed modules produce high-quality manipulation with fine-grained details.
- Evaluation: ManiGAN introduces a metric covering both new-attribute generation and reconstruction of text-irrelevant contents.Experiments on CUB and COCO report qualitative and quantitative superiority over existing state-of-the-art methods.
2. Related Work
Prior work includes text-to-image generation and conditional image synthesis, but these approaches generally generate or translate images rather than manipulate specific attributes from natural-language descriptions. Text-guided manipulation methods remain relatively few and use related GAN-based architectures.
- Text-to-image generation: Text-to-image methods generate new photorealistic images from descriptions rather than manipulating specific visual attributes in an existing image.Attention mechanisms have been used to explore fine-grained word-level information.
- Conditional image synthesis: Conditional image synthesis includes paired and unpaired image-to-image translation, but mainly targets same-domain translation instead of cross-domain text-guided manipulation.The distinction is between image translation and manipulation using natural-language descriptions.
- Text-guided image manipulation: Existing text-guided manipulation studies use GAN-based encoder-decoder architectures to disentangle image and text semantics or provide word-level discriminator feedback.These methods establish a related baseline for natural-language image manipulation.
3. Generative Adversarial Networks for Image Manipulation
ManiGAN combines text and image features through ACM to select and modify text-relevant regions while preserving other image content, then uses DCM to refine details and complete missing contents. Its training uses adversarial objectives and image-reconstruction regularization to support stable, non-identity manipulation.
- Architecture: ManiGAN takes an input image and user text, generating an edited image aligned with the text while preserving text-irrelevant contents.
- Text-Image Affine Combination Module: ACM fuses hidden text features with regional image features before each upsampling block, selecting relevant regions and encoding original representations for reconstruction.Regional image features are processed into learned scaling and bias terms that modulate hidden text features.
- Text-Image Affine Combination Module: The affine combination replaces simple concatenation by re-weighting image feature maps, helping correlate semantic words with attributes and identify regions for manipulation.The multiplication operation serves regional selection, while the bias term preserves image information for stable reconstruction.
- Detail Correction Module: DCM uses word-level attention and fine-grained visual features to refine manipulated attributes, enhance details, and complete missing contents.Its inputs include final ACM features, word features, and visual features extracted from a pretrained VGG-16 network.
- Training: Training alternates optimization of the generator and discriminators, adding a regularization term that penalizes identity mappings and promotes diversity.The training data simulate manipulation using paired image-description data because CUB and COCO lack paired modified-image targets.
4. Experiments
ManiGAN is evaluated on CUB and COCO using quantitative, qualitative, metric-design, training, and ablation analyses. The experiments show strong manipulation and reconstruction performance, while isolating the roles of ACM and DCM.
- Experimental setup: Experiments compare ManiGAN with SISGAN and TAGAN on CUB and COCO using IS, sim, diff, and MP.Higher IS, sim, and MP are better, while lower pixel difference is better.
- Training: Training the DCM for 100 CUB epochs and 12 COCO epochs provides an empirically selected balance between attribute generation and content reconstruction.The stopping choice is based on manipulative precision values.
- Evaluation metric: Manipulative precision combines text-image similarity with L1 pixel difference to evaluate both generated attributes and reconstructed image contents.The metric addresses the separate aspects measured by image-similarity and text-image-similarity metrics.
- Quantitative and qualitative comparison: ManiGAN achieves the highest IS and MP values on both datasets, indicating high-quality manipulation, attribute matching, and reconstruction of irrelevant contents.The comparison is reported quantitatively against state-of-the-art approaches.
- Quantitative and qualitative comparison: Qualitatively, ManiGAN manipulates complex COCO scenes more accurately than SISGAN and TAGAN while preserving text-irrelevant contents.One example changes green grass to dry grass and edits a cow into a sheep.
- Ablation studies: Ablations show that ACM supports regional attribute selection and text correlation, while DCM corrects mismatched attributes and reconstructs missing or irrelevant contents.Removing ACM harms realism or manipulation, removing DCM causes missing or altered contents, and removing the main module produces identity mapping.
5. Conclusion
ManiGAN enables semantic image manipulation from natural-language descriptions while preserving text-irrelevant content. Its ACM and DCM support attribute editing, content reconstruction, and high-quality results.
- ManiGAN semantically manipulates input images using natural-language descriptions.
- The ACM selects text-relevant image regions, correlates them with semantic words, and encodes original features for reconstructing text-irrelevant content.
- The DCM rectifies mismatched visual attributes and completes missing contents in synthetic images.
- Experiments demonstrate superior image-manipulation effectiveness and high-quality generation.
A. Architecture
The architecture builds on ControlGAN and adds image and text encoders with ACM fusion before upsampling and image-generation blocks.
- ManiGAN adopts ControlGAN as its basic framework and replaces generator batch normalisation with instance normalisation except in the first stage.
- A pretrained Inception-v3 image encoder extracts regional image representations for text-image fusion.
- A pretrained RNN encodes text representations, which ACM fuses with regional image representations before each upsampling block at each stage.
A.1. Residual Block
The residual block used in the detail correction module consists of convolution, instance-normalisation, and gated nonlinear components.
- Each residual block contains two convolutional layers, two instance normalisation layers, and one GLU nonlinear function.
- Figure 8 depicts the architecture of the residual block used in the detail correction module.
B. Objective Functions
ManiGAN trains its main and detail-correction modules separately with alternating generator and discriminator optimisation. Its objectives combine adversarial, text-alignment, reconstruction, and regularisation terms.
- The main and detail correction modules are trained separately, while each module alternates generator and discriminator optimisation.
- Generator objective: The generator loss follows ControlGAN and adds a regularisation term.
- Generator objective: The regularisation term prevents identity mapping by penalising large perturbations when the generated image becomes the input image.
- Generator objective: The ControlGAN loss combines DAMSM, text-image correlation, and reconstruction terms weighted by λ2, λ3, and λ4.
- Unconditional adversarial loss promotes realism, conditional adversarial loss aligns generated images with text, and DAMSM measures word-level text-image similarity.
- Discriminator objective: The discriminator objective follows ControlGAN, with the detail correction discriminator using the main module’s final-stage discriminator objective.
C. Trend of Manipulation Results
Manipulation quality changes across training epochs, balancing text-aligned attribute generation against reconstruction of text-irrelevant image contents. Excessive training increasingly erases requested attributes and makes outputs resemble the originals.
- C. Trend of Manipulation Results: Epoch increases reveal a trade-off between generating requested visual attributes and reconstructing text-irrelevant contents.Examples include bird colours and zebra backgrounds versus bird shape and zebra appearance.
- C. Trend of Manipulation Results: At an intermediate stage, original images are smoothly modified to balance new attributes with preserved content.The CUB examples add blue head, blue wings, and yellow belly while preserving bird shape and background; the COCO example adds a dirt background while preserving zebra appearance.
- C. Trend of Manipulation Results: When epochs become larger, text-aligned attributes are gradually erased and synthetic images become increasingly similar to the originals.This trend is reported for both CUB and COCO datasets.
D. Additional Comparison Results
Additional comparisons present ManiGAN alongside SISGAN and TAGAN on CUB and COCO datasets. The figures include text-conditioned bird and scene manipulation examples, but the supplied passages do not state a comparative outcome.
- D. Additional Comparison Results: Figures 12–15 compare ManiGAN, SISGAN, and TAGAN on the CUB and COCO datasets.The paper directs readers to an accompanying video for detailed comparison.
- D. Additional Comparison Results: The comparison layouts include given text, original images, SISGAN results, TAGAN results, and ManiGAN results.This ordering is explicitly shown in the supplied comparison labels.
- D. Additional Comparison Results: Example text conditions describe bird colours, markings, wings, bills, bellies, and scene attributes such as a blue boat and green grass.The supplied examples include “Blue boat, green grass” and multiple bird descriptions.