Source-linked AI summary
Image Inpainting via Conditional Texture and Structure Dual Generation
Xiefan Guo, Hongyu Yang, Di Huang
TL;DR
Image inpainting methods can struggle with global structure and unstable structural priors, especially when structure and texture are not properly coupled. This paper introduces a two-stream network with bidirectional feature fusion and contextual aggregation, and reports superior qualitative and quantitative performance across CelebA, Paris StreetView, and Places2.
Problem
Existing inpainting methods have limited interaction between structure and texture, making global structure recovery difficult and structural priors unstable.
Method
A coupled two-stream network jointly performs structure-constrained texture synthesis and texture-guided structure reconstruction, with Bi-GFF and CFA modules for feature integration and refinement.
Results
The model significantly outperforms state-of-the-art counterparts qualitatively and quantitatively on CelebA, Paris StreetView, and Places2.
Takeaways & Limitations
Coupling structure and texture generation yields semantically reasonable structures and detail-rich textures for image inpainting.
Abstract
from arXiv · showhide
Deep generative approaches have recently made considerable progress in image inpainting by introducing structure priors. Due to the lack of proper interaction with image texture during structure reconstruction, however, current solutions are incompetent in handling the cases with large corruptions, and they generally suffer from distorted results. In this paper, we propose a novel two-stream network for image inpainting, which models the structure-constrained texture synthesis and texture-guided structure reconstruction in a coupled manner so that they better leverage each other for more plausible generation. Furthermore, to enhance the global consistency, a Bi-directional Gated Feature Fusion (Bi-GFF) module is designed to exchange and combine the structure and texture information and a Contextual Feature Aggregation (CFA) module is developed to refine the generated contents by region affinity learning and multi-scale feature aggregation. Qualitative and quantitative experiments on the CelebA, Paris StreetView and Places2 datasets demonstrate the superiority of the proposed method. Our code is available at https://github.com/Xiefan-Guo/CTSDG.
1. Introduction
Image inpainting methods increasingly use deep generative and structural priors, but incomplete structure–texture interaction leaves global reconstruction difficult and structural errors unstable. The proposed coupled two-stream network addresses this with bidirectional feature fusion and contextual aggregation, achieving strong benchmark results.
- Deep generative inpainting captures high-level semantics but commonly struggles to recover global image structure.
- Serial structure-guided methods improve textures and structures, yet unstable structures inferred from corrupted images can produce large errors.
- Single-generator approaches do not fully model the complementary relationship between image structures and textures.
- The proposed two-stream network couples structure-constrained texture synthesis with texture-guided structure reconstruction so the tasks facilitate each other.
- Bi-GFF exchanges and combines structure–texture features, while CFA models long-term spatial dependency to enhance consistency and details.
- The method achieves state-of-the-art performance qualitatively and quantitatively across public benchmarks including CelebA, Paris StreetView, and Places2.
2. Related Work
Related work spans diffusion, patch-based, deep generative, and structure-guided inpainting methods. The paper positions its coupled two-stream design as a more complementary alternative for jointly modeling structures and textures.
- Diffusion-based methods use neighboring appearance, whereas patch-based methods search and paste similar undamaged patches, with higher computational cost.
- Deep generative methods recover semantically meaningful contents with high visual fidelity through powerful feature learning.
- Structure-guided methods improve synthesis by incorporating edges, contours, or edge-preserved smooth images as structural priors.
- The proposed method jointly considers structure-constrained texture synthesis and texture-guided structure reconstruction in separate coupled streams.
- The two subtasks facilitate each other, producing more convincing textures and structures in dual generation.
3. Approach
The approach uses a two-stream generator that jointly reconstructs textures and structures, with cross-branch fusion and contextual aggregation refining consistency and detail. A two-stream discriminator and joint losses further supervise realistic, semantically reasonable outputs.
- Generator: The U-Net-based generator separately encodes texture and structure, then lets each decoder use the other branch's features for complementary reconstruction.The texture decoder synthesizes structure-constrained textures, while the structure decoder recovers texture-guided structures.
- Backbone: Partial convolutions condition processing on uncorrupted pixels, while skip connections combine low- and high-level features across multiple scales.These design choices target irregular boundaries and more sophisticated predictions.
- Bi-directional Gated Feature Fusion (Bi-GFF): Bi-GFF exchanges decoded texture and structure information through soft gating, producing features that are simultaneously texture- and structure-aware.The module adaptively merges each branch into the other, with trainable residual scaling parameters initialized to zero.
- Discriminator and Loss Functions: A two-stream discriminator evaluates texture and structure jointly, and training combines reconstruction, perceptual, style, adversarial, and intermediate losses.The generator maps damaged image, edge, grayscale, and mask inputs to reconstructed image and edge outputs; the reconstruction loss uses ℓ1 distance.
4. Experiments
Experiments evaluate the method qualitatively, quantitatively, and through ablations on CelebA, Paris StreetView, and Places2. The comparisons report stronger visual quality and benchmark performance, while ablations support the proposed architecture and modules.
- Experimental Setup: Experiments use three public datasets with subjective and objective evaluations, alongside ablation studies of the architecture and modules.Images are resized to 256 × 256 pixels, and irregular masks are grouped by hole size.
- Qualitative Comparison: The qualitative comparison includes PatchMatch, PConv, DeepFillv2, RFR, MED, the proposed method, and ground-truth images across three benchmarks.Figure 5 presents corrupted inputs, competing outputs, the proposed outputs, and ground truth.
- Qualitative Comparison: The proposed method produces more detailed and less distorted results than representative methods, while competing approaches exhibit artifacts, over-smoothing, or inadequate sharpness.The reported examples include faces, skies, and facades.
- Qualitative Comparison: Compared with EdgeConnect and PRVS, the proposed model reconstructs more reasonable and sharper structures, leading to better filled results.Figure 6 separates reconstructed structures from their corresponding completed images.
- Quantitative Comparison: On Places2, the method outperforms competing approaches on LPIPS, PSNR, and SSIM and receives more favorable user-study votes by a large margin.The objective comparison covers irregular mask ratios of 0-20%, 20-40%, and 40-60%; ten volunteers answered fifteen randomly sampled questions each.
- Ablation Study: Ablations show that dual generation improves structures and textures, Bi-GFF improves consistency, and CFA improves texture consistency and quantitative scores.The comparisons are conducted on Paris StreetView, with Figure 7 visualizing architecture and module effects.
5. Conclusion
The paper presents a two-stream inpainting method that jointly models structure-constrained texture synthesis and texture-guided structure reconstruction. Bi-GFF and CFA refine consistency and detail, and experiments report superior performance over state-of-the-art counterparts.
- Conclusion: The method simultaneously models structure-constrained texture synthesis and texture-guided structure reconstruction so the subtasks exchange useful information.The architecture is designed to produce semantically reasonable structures and detail-rich textures.
- Conclusion: Bi-GFF and CFA are introduced to refine the generated results, supporting consistency, semantic structure, and detailed textures.The conclusion presents both modules as refinements to the two-stream method.
- Conclusion: Experiments show that the model outperforms state-of-the-art counterparts.The reported conclusion covers the proposed image inpainting model generally rather than a single benchmark.