Source-linked AI summary
StructureFlow: Image Inpainting via Structure-aware Appearance Flow
Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H. Li, Shan Liu, Ge Li
TL;DR
Image inpainting must recover both reasonable structures and fine-grained textures, but existing deep networks can produce over-smoothed boundaries or texture artifacts. StructureFlow separates the task into structure reconstruction and texture generation, using edge-preserved smooth images and appearance flow, and reports competitive results on multiple public datasets. The paper also identifies a trade-off in how much texture the structure representation should remove.
Problem
Deep inpainting methods may recover coherent structures yet produce over-smoothed boundaries or texture artifacts because they do not effectively separate structure and texture information.
Method
StructureFlow uses a structure reconstructor trained with edge-preserved smooth images, followed by a texture generator that uses appearance flow to sample features from related existing regions.
Results
StructureFlow achieves competitive results compared with several state-of-the-art methods on multiple public datasets, with ablation studies supporting its modifications.
Takeaways & Limitations
Separating global structure recovery from high-frequency texture synthesis provides an effective structure-aware framework for recovering corrupted images with meaningful structures and vivid textures.
Takeaways & Limitations
The structure representation involves a trade-off: removing too few textures makes structure reconstruction more difficult.
Abstract
from arXiv · showhide
Image inpainting techniques have shown significant improvements by using deep neural networks recently. However, most of them may either fail to reconstruct reasonable structures or restore fine-grained textures. In order to solve this problem, in this paper, we propose a two-stage model which splits the inpainting task into two parts: structure reconstruction and texture generation. In the first stage, edge-preserved smooth images are employed to train a structure reconstructor which completes the missing structures of the inputs. In the second stage, based on the reconstructed structures, a texture generator using appearance flow is designed to yield image details. Experiments on multiple publicly available datasets show the superior performance of the proposed network.
1 . Introduction
StructureFlow addresses image inpainting’s difficulty in recovering both coherent structures and realistic textures by separating the task into structure reconstruction and texture generation. Its two-stage design uses edge-preserved smooth images for global structures and appearance flow to sample details from related existing regions.
- Patch-based methods synthesize vivid textures but struggle with realistic structures in images containing non-repetitive patterns because they cannot capture high-level semantics.
- Deep conditional-generation methods learn meaningful semantics but often produce over-smoothed boundaries or texture artifacts because structure and texture information are not effectively separated.
- StructureFlow uses a structure reconstructor followed by a texture generator to divide inpainting into structure reconstruction and texture generation.
- Appearance flow establishes long-term relationships by sampling features from regions with similar reconstructed structures for texture synthesis.
- The structure reconstructor uses edge-preserved smooth images to recover global structures while removing high-frequency textures and retaining sharp edges and low-frequency structures.
- Gaussian sampling and a sampling correctness loss are introduced to ease appearance-flow optimization and constrain whether sampled regions are suitable.
- Experiments on multiple public datasets report competitive results against several state-of-the-art methods, with ablation studies evaluating the proposed modifications.
2 . Related Work
Prior image-inpainting methods include diffusion-based, patch-based, and deep-learning approaches, each addressing missing content through neighborhood propagation, remote patch reuse, or learned semantic generation. StructureFlow builds on these lines of work by combining global structure recovery with texture synthesis.
- Diffusion-based methods propagate neighborhood appearance but generally handle only small holes in background inpainting and may fail to generate meaningful structures.
- Patch-based methods use remote information to recover lost areas, while early deep-learning methods formulate inpainting as conditional generation to create new content from learned semantics.
- Context Encoder uses an encoder-decoder architecture to extract features and reconstruct outputs but struggles to maintain global consistency.
- StructureFlow combines global structure generation with texture generation and appearance-flow feature sampling from existing regions.
- Contextual attention, partial convolutions, gated convolutions, and EdgeConnect address long-range dependency, invalid pixels, feature selection, or structural guidance in different ways.
2 .2. Optical Flow and Appearance Flow
Optical flow describes motion between consecutive video frames, whereas appearance flow transfers features between source and target regions. StructureFlow adapts appearance flow to inpainting by using reconstructed structures to guide feature warping and texture generation.
- Optical Flow: Optical flow describes the motion of objects, surfaces, and edges between consecutive video frames.
- Optical Flow: Existing optical-flow methods require difficult ground-truth fields or use unsupervised learning, while unsupervised approaches can struggle with large motions.
- Appearance Flow: Gaussian sampling replaces bilinear sampling in StructureFlow’s appearance-flow operation.
- Appearance Flow: Appearance flow predicts 2-D coordinate vectors by calculating correlations between source and target scenes or objects.
- Appearance Flow: StructureFlow uses appearance flow to warp input features so texture generation can exploit long-term relationships between regions with similar reconstructed structures.
3 .1. Structure Reconstructor
The structure reconstructor Gs recovers global structures from masked inputs using edge-preserved smooth guidance, combining reconstruction and adversarial objectives.
- Gs predicts global structures from the masked input image and its observed structures, using edge-preserved smooth images as structural targets.The mask identifies missing regions, while Sin retains the ground-truth structures outside those regions.
- The input mask M is binarized, with 1 marking missing regions and 0 marking background.
- The reconstruction loss measures the ℓ1 distance between predicted structures Ŝ and ground-truth structures Sgt.
- An adversarial framework is applied so Gs can mimic the distribution of target structures Sgt.The discriminator Ds is jointly trained with Gs.
- The adversarial-loss weight λadv is set to 1 in all experiments.
3 .2. Texture Generator
The texture generator uses appearance flow to transfer texture features into corrupted regions, with Gaussian sampling and a sampling correctness loss supporting long-range, reliable feature selection.
- After reconstructing the structure image, the texture generator produces vivid textures from the corrupted input, reconstructed structure, and mask.
- Appearance flow warps input features so vivid texture information can flow into corrupted regions.
- Gaussian sampling expands the receptive field to capture long-term correlations that limited-receptive-field Bilinear sampling may miss.
- The sampling correctness loss constrains appearance flow fields by evaluating sampled features against ground-truth features using relative cosine similarity.
- The texture generator combines reconstruction, contextual, and adversarial objectives, with λt ℓ1 = 5, λt c = 0.25 and λt adv = 1.
4 .1. Implementation Details
StructureFlow is trained on three public datasets using RTV-generated edge-preserved smooth images as structure-reconstruction labels, staged optimization, and fixed 256 × 256 training settings.
- The generators use autoencoder structures, residual blocks, warped features, convolutional features, and two multi-scale PatchGAN discriminators.
- The model is trained on Places2, Celeba, and Paris StreetView with irregular masks spanning hole sizes from 0–20% and larger ratios.
- RTV edge-preserved smoothing provides the training labels for the structure reconstructor.
- Training first optimizes the structure reconstructor and texture generator separately, then fine-tunes the texture generator using reconstructed structures.
- Training uses 256 × 256 images, batch size 12, Adam, and learning rate 10^-4; the best empirical smoothing parameter is σ ≈3.
4 .2. Comparisons
StructureFlow is compared with CA, PConv, and EdgeConnect using distortion, perceptual, subjective, and qualitative evaluations across public inpainting datasets.
- The evaluation compares StructureFlow with CA, PConv, and EdgeConnect through subjective and objective experiments.
- Objective evaluation uses SSIM and PSNR for distortion and FID for perceptual quality.
- The user study samples 600 images per dataset with mask ratios from 0%–60%, and each image is judged five times by different volunteers.
- StructureFlow performs better than competitors on highly structured Celeba and Paris StreetView scenes and achieves competitive results on Places2.
- Qualitatively, CA produces artifacts, EdgeConnect recovers global structures but may create wrong edge details, while StructureFlow produces meaningful structures and vivid textures.
4 .3. Ablation Studies
Ablation studies examine how structure reconstruction, edge-preserved smoothing, and appearance flow affect StructureFlow’s inpainting performance. The experiments identify trade-offs in smoothing and show that the full design benefits from both structural guidance and flow-based texture generation.
- Structure Ablation: Removing the structure reconstructor still retains appearance flow, allowing a direct test of whether reconstructed structures improve the final inpainting model.The comparison is reported in Table 3.
- Structure Ablation: The structure reconstructor can bring stable performance gain to the model.
- Structure Ablation: Using ground truth images containing high-frequency textures as structural labels leads to performance degradation compared with StructureFlow.
- Structure Ablation: Smoothing introduces a trade-off: removing few textures makes structure reconstruction harder but texture generation easier, so the two tasks must be balanced.
- Structure Ablation: On the Paris dataset, σ = 3 gives the best results, while both smaller and larger values degrade model performance.
- Flow Ablation: Without sampling correctness loss, the model cannot sample correct features for large missing regions, while appearance flow improves performance over its removal.
4 .4. User case
StructureFlow supports interactive image editing by generating realistic alternative content in masked regions. Users can remove objects and edit reconstructed structure images to add or copy content.
- User case: Users can remove unwanted objects by interactively drawing masks on input images.
- User case: StructureFlow generates realistic alternative contents for the missing regions.
- User case: Directly editing structure images lets users copy or add new objects and contents to images.
5 . Conclusion
The paper concludes that StructureFlow separates image inpainting into structure reconstruction and texture generation to recover meaningful structures and vivid textures. It uses edge-preserved smooth images for global structure information and appearance flow for feature sampling, achieving competitive results against several state-of-the-art methods.
- Conclusion: StructureFlow divides image inpainting into two subtasks: structure reconstruction and texture generation.
- Conclusion: Edge-preserved smooth images represent global structure information and play an important role in image inpainting.
- Conclusion: Appearance flow samples features from relative regions and brings stable performance gain to the final results.
- Conclusion: StructureFlow obtains competitive results compared with several state-of-the-art methods.