Source-linked AI summary
EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Z. Qureshi, Mehran Ebrahimi
TL;DR
Image inpainting methods can generate coherent structures but often produce blurry or over-smoothed missing regions, leaving fine-detail reconstruction as a key challenge. EdgeConnect uses an adversarial edge generator followed by an image completion network conditioned on hallucinated edges, and the authors report state-of-the-art results across CelebA, Places2, and Paris StreetView. The model also supports inpainting with multiple irregular missing regions and interactive editing applications.
Problem
Many deep learning inpainting methods generate meaningful structures but remain blurry or artifact-prone when reconstructing high-frequency information.
Method
EdgeConnect combines an adversarial edge generator that hallucinates missing-region edges with an adversarial image completion network that fills regions using those edges as a priori.
Results
The authors report state-of-the-art results on CelebA, Places2, and Paris StreetView and show that edge information improves every evaluated metric in their ablation.
Takeaways & Limitations
Edge information plays an important role in image inpainting, and the proposed model can handle images with multiple irregularly shaped missing regions.
Takeaways & Limitations
The edge generator sometimes fails to depict edges accurately in highly textured areas or when a large portion of the image is missing.
Abstract
from arXiv · showhide
Over the last few years, deep learning techniques have yielded significant improvements in image inpainting. However, many of these techniques fail to reconstruct reasonable structures as they are commonly over-smoothed and/or blurry. This paper develops a new approach for image inpainting that does a better job of reproducing filled regions exhibiting fine details. We propose a two-stage adversarial model EdgeConnect that comprises of an edge generator followed by an image completion network. The edge generator hallucinates edges of the missing region (both regular and irregular) of the image, and the image completion network fills in the missing regions using hallucinated edges as a priori. We evaluate our model end-to-end over the publicly available datasets CelebA, Places2, and Paris StreetView, and show that it outperforms current state-of-the-art techniques quantitatively and qualitatively. Code and models available at: https://github.com/knazeri/edge-connect
1. Introduction
Image inpainting must produce perceptually plausible filled regions with fine details, yet many existing methods generate blurry or over-smoothed results. EdgeConnect addresses this with adversarial edge generation followed by image completion conditioned on hallucinated edges.
- Motivation: Existing inpainting methods often produce over-smoothed or blurry regions that fail to reproduce fine details.Such artifacts are especially noticeable when surrounding image regions contain sharp details.
- Approach: EdgeConnect divides inpainting into edge generation and image completion.The edge generator hallucinates missing-region edges, while the completion network estimates RGB intensities using those edges.
- Approach: Both stages use adversarial frameworks and deep-feature losses to promote visual consistency and perceptual realism.
- Motivation: The model conditions image completion on hallucinated edges because image structure is well represented in edge masks.The approach separates recovery of high-frequency and low-frequency information in the missing region.
- Evaluation: The authors evaluate EdgeConnect on CelebA, Places2, and Paris StreetView against current state-of-the-art schemes.
- Contributions: The proposed end-to-end network combines edge generation with completion to fill missing regions exhibiting fine details and supports applications such as object removal and scene generation.
2. Related Work
Traditional diffusion- and patch-based methods have important structural or computational limitations, while deep learning approaches improve coherence but often retain blur and artifacts. Related work also includes methods that provide additional priors or address edge prediction and image generation separately.
- Traditional Inpainting: Diffusion-based methods propagate neighboring information but remain restricted to locally available information and struggle with meaningful or large missing regions.
- Traditional Inpainting: Patch-based methods copy similar source regions into targets, but require costly target-source similarity computations and assume suitable textures exist elsewhere.
- Deep Learning Inpainting: Context encoders use an encoder-decoder architecture, but recovered regions can contain artifacts and blur caused by an information bottleneck.
- Deep Learning Inpainting: Other deep learning methods introduce additional priors, but some are slow during inference or specialized for frontal human face eye regions.Yeh et al.'s method requires back-propagation for every image, while the eye-region method does not generalize well.
- Image-to-Edges vs. Edges-to-Image: The paper frames inpainting as two linked problems: Image-to-Edges and Edges-to-Image.Its edge maps are computed using the Canny edge detector.
- Image-to-Edges vs. Edges-to-Image: Traditional Edges-to-Image methods struggle to construct fine-grained details near object boundaries, motivating learning-based image generation from line sketches.
3. EdgeConnect
EdgeConnect separates inpainting into adversarial edge generation and image completion, using predicted missing-region edges to condition RGB reconstruction. Its losses and training design target perceptual realism, stable optimization, and reduced artifacts.
- Architecture: EdgeConnect uses two adversarial generator-discriminator stages: G1 predicts missing-region edges, while G2 completes the image from those edges.The stages are named edge generator and image completion network, with pairs (G1, D1) and (G2, D2).
- Edge Generator: G1 takes masked grayscale intensities, masked ground-truth edges, and the mask to predict the missing-region edge map.The mask marks missing pixels with 1 and background pixels with 0.
- Training Objectives: The edge stage uses adversarial and discriminator feature-matching losses, while the completion stage uses ℓ1, adversarial, perceptual, and style losses.Feature matching compares intermediate discriminator activation maps; perceptual and style losses use pretrained-network activations, with style loss targeting checkerboard artifacts.
- Image Completion Network: G2 combines the incomplete color image with a composite edge map containing background ground-truth edges and generated missing-region edges.It returns a completed color image at the input resolution.
- Qualitative Evaluation: The qualitative comparison includes ground truth, masked input, two existing models, EdgeConnect end-to-end, and G2 conditioned on Canny edges with σ = 2.This setup compares the full pipeline with an image-completion-only variant using fixed Canny edges.
4. Experiments
The experiments train EdgeConnect with Canny-derived edge labels and evaluate both regular and irregular masks. Training proceeds from separate generator optimization to end-to-end fine-tuning.
- Edge Labels: Canny edge labels are used to train G1, with Gaussian smoothing σ ≈ 2 selected empirically as producing the best results.The study examines how edge-map quality affects overall image completion.
- Masking: Regular masks cover 25% of image pixels in centered random squares, while irregular masks are augmented through rotations and horizontal reflection.Irregular masks are categorized by size relative to the full image in 10% increments.
- Optimization: Generators G1 and G2 are first trained separately with Canny edges, using learning rates 10^-4 and then 10^-5 until convergence.The networks are trained on 256 × 256 images with batch size eight using Adam with β1 = 0 and β2 = 0.9.
- Optimization: The networks are finally fine-tuned end-to-end after removing D1 with learning rate 10^-6, while discriminators use one tenth of the generators’ learning rate.This final stage continues until convergence.
5. Results
EdgeConnect produces visually detailed inpainting results and is evaluated through quantitative, perceptual, and ablation studies. The experiments examine model comparisons, mask size, edge information, edge quantity, and alternative edge detectors.
- 5.1. Qualitative Comparison: The model generates photo-realistic results with minimal blurriness and no checkerboard artifacts after including style loss.Qualitative comparisons report results closer to ground truth than other state-of-the-art methods.
- 5.2. Quantitative Comparison: Evaluation uses relative ℓ1, SSIM, PSNR, and FID, with FID added because pixel-wise metrics can favor perceptually inaccurate results.FID measures Wasserstein-2 distance between real and inpainted image features from a pre-trained Inception-V3 model.
- 5.2. Quantitative Comparison: The quantitative evaluation compares EdgeConnect components and competing methods on Places2, while also varying mask sizes and reporting statistics from competing pre-trained models where available.Partial Convolution results were taken from its paper because source code was unavailable, and its ℓ1 calculation may differ from the authors’ procedure.
- 5.2. Quantitative Comparison: Human perceptual evaluation uses 2AFC and JND tests over 300 images per model and mask size, with each image shown 10 times.The 2AFC ground-truth score is 94.6 ± 0.5%.
- 5.3. Ablation Study: Adding edge information improves every metric, even when masks remove 40–50% of the image.This comparison uses 10,000 random masks and contrasts the full model with G2 trained without edges.
- 5.3. Ablation Study: The best results occur with Canny edges generated using σ ∈[1.5, 2.5]; too few or too many edges reduce inpainting quality.The authors use σ = 2 for ground-truth edge maps, and combined Canny⊙HED edge maps do not improve results.
6. Discussions and Future Work
EdgeConnect combines adversarial edge generation with image completion and achieves state-of-the-art benchmark results while supporting irregular missing regions and interactive editing. The authors note that edge generation remains unreliable in highly textured areas and when large image portions are missing.
- Discussions and Future Work: EdgeConnect uses an adversarial edge generator and image completion network for image inpainting.The model is presented as a two-stage system in which edge information guides completion.
- Discussions and Future Work: The trained model supports interactive editing by manipulating edge maps and transforming them into generated images, including unwanted-object removal.Figure 10 combines an input image with edge maps from another image, while Figure 11 illustrates object removal.
- Future Work: The edge generator sometimes fails to depict edges accurately in highly textured areas or when a large portion of the image is missing.The paper illustrates such failures in Figure 9 and identifies improved edge generation as future work.
A. Network Architectures
The generator architecture uses convolutional, downsampling, residual, and upsampling layers, with output layers specialized for edge prediction or RGB completion.
- Network Architectures: The layer notation defines 7 × 7 stride-1 convolutions, 4 × 4 stride-2 downsampling or transposed-convolution upsampling, and residual blocks by channel size.
- Network Architectures: The generators follow a convolutional architecture with downsampling, eight residual blocks, and upsampling stages.The stated sequence is c64, d128, d256, eight R256 blocks, u128, u64, and c*.
- Network Architectures: The edge generator outputs one sigmoid-activated channel for edge prediction, while the completion network outputs three tanh-scaled channels for RGB intensities.Spectral normalization is also removed from all layers of the image completion network.
A.2. Discriminators
The two discriminators use a shared 70 × 70 PatchGAN design that classifies overlapping image patches as real or fake.
- Discriminators: Discriminators D1 and D2 use the 70 × 70 PatchGAN architecture.
- Discriminators: Their final convolution predicts real-or-fake scores for overlapping 70 × 70 image patches.
- Discriminators: The discriminator stack is C64-2, C128-2, C256-2, C512-1, and C1-1, with LeakyReLU slope 0.2.
B. Experimental Results
The experiments provide additional results across CelebA, Places2, and Paris StreetView, with dataset-specific preprocessing and an edge-generator evaluation using precision and recall across mask sizes.
- Experimental Results: Additional model results are reported on CelebA, Places2, and Paris StreetView.
- Experimental Results: CelebA contains 202,599 images, Places2 contains more than 10 million images, and Paris StreetView contains 14,900 images.
- Experimental Results: Paris StreetView images are divided into three 537 × 537 crops and scaled to 256 × 256, producing 44,700 images for the model.
- Experimental Results: The edge generator is evaluated across all three datasets using precision and recall for various mask sizes.The stated purpose is to assess how closely generated edges approach oracle edges rather than to maximize precision and recall.
B.2. Comprehensive Results
The model is reported to outperform existing methods on CelebA and Paris StreetView, with additional qualitative results presented across three datasets. Source code and pretrained models are publicly available.
- The model produces noticeably better results than existing methods on CelebA and Paris StreetView.The comparison excludes PConv on those datasets because its source code was unavailable.
- Additional inpainting results are shown for CelebA, Places2, and Paris StreetView.
- Source code, pretrained models, and additional results are available online.
C. Alternative Edge Generating Systems
The paper examines alternative edge representations and their effects on edge generation and inpainting. It compares hybrid HED⊙Canny edges with Canny edges and illustrates generated edges and completed images across datasets.
- Alternative edge representations: Hybrid HED⊙Canny edges produce the best quantitative performance for generated images, but G1 cannot accurately generate them during training.G1 trained on HED⊙Canny has the poorest performance among the compared methods, despite the corresponding ground-truth edges performing best.
- Qualitative results: Figures 12–17 visualize generated edges and completed images for hybrid edge generation, CelebA, Places2, and Paris StreetView.The sample figures distinguish generated edges from ground-truth edges using blue and black, respectively.
- Alternative edge representations: Table 5 compares Hybrid HED⊙Canny and Canny edges on CelebA for generated edges and ground-truth edges.The table marks whether each metric is lower-is-better or higher-is-better.
- Quantitative comparisons: Tables 6 and 7 compare CelebA and Paris StreetView results across CA, GLCIC, end-to-end EdgeConnect, and EdgeConnect using Canny edges.Figures 13 and 14 show the effect of relative mask sizes on ℓ1, SSIM, PSNR, and FID for CelebA and Paris StreetView.