Source-linked AI summary
Diverse Image Inpainting with Bidirectional and Autoregressive Transformers
Yingchen Yu, Fangneng Zhan, Rongliang Wu, Jianxiong Pan, Kaiwen Cui, Shijian Lu, Feiying Ma, Xuansong Xie, Chunyan Miao
TL;DR
Image inpainting must produce realistic, diverse completions despite arbitrary missing regions and the limitations of unidirectional autoregressive context. BAT-Fill combines bidirectional autoregressive modeling with a texture-generation stage, and experiments report superior quality and diversity across multiple datasets.
Problem
Existing methods struggle to generate diverse, realistic, and consistent completions for complex missing regions, while unidirectional autoregressive context is suboptimal for arbitrarily shaped corruption.
Method
BAT-Fill combines bidirectional context modeling and autoregressive output dependency in BAT, followed by CNN-based texture synthesis for high-resolution details.
Results
BAT-Fill achieves superior image-inpainting performance in both quality and diversity across multiple datasets.
Takeaways & Limitations
The framework provides realistic and diverse image inpainting by jointly modeling bidirectional context and autoregressive dependencies.
Abstract
from arXiv · showhide
Image inpainting is an underdetermined inverse problem, which naturally allows diverse contents to fill up the missing or corrupted regions realistically. Prevalent approaches using convolutional neural networks (CNNs) can synthesize visually pleasant contents, but CNNs suffer from limited perception fields for capturing global features. With image-level attention, transformers enable to model long-range dependencies and generate diverse contents with autoregressive modeling of pixel-sequence distributions. However, the unidirectional attention in autoregressive transformers is suboptimal as corrupted image regions may have arbitrary shapes with contexts from any direction. We propose BAT-Fill, an innovative image inpainting framework that introduces a novel bidirectional autoregressive transformer (BAT) for image inpainting. BAT utilizes the transformers to learn autoregressive distributions, which naturally allows the diverse generation of missing contents. In addition, it incorporates the masked language model like BERT, which enables bidirectionally modeling of contextual information of missing regions for better image completion. Extensive experiments over multiple datasets show that BAT-Fill achieves superior diversity and fidelity in image inpainting qualitatively and quantitatively.
1 Nanyang Technological University 2 DAMO Academy, Alibaba Group
The paper’s illustrative materials include Figure 1 samples drawn from three public image datasets.
- Figure 1 presents sample images from CelebA-HQ, Places2, and Paris StreetView, in top-to-bottom order.
1. Introduction
Image inpainting must generate diverse, realistic completions despite ambiguous missing content and complex corrupted regions. BAT-Fill addresses this challenge by combining bidirectional context modeling with autoregressive generation and reports superior performance across multiple datasets.
- Image inpainting permits numerous realistic and semantically reasonable completions, making diversity a central challenge.
- Large corrupted regions with complex textures and structures make it difficult to preserve realism, integrity, and consistency.
- GAN-based methods commonly learn one-to-one mappings, limiting diverse outputs, while VAE-based methods can compromise quality when generating complex structures.
- BAT models deep bidirectional contexts autoregressively by combining autoregressive modeling with masked-language-modeling principles.
- BAT-Fill first recovers diverse coherent structures with BAT, then uses a CNN-based texture generator to synthesize texture details.
- Extensive experiments over multiple datasets report superior inpainting quality and diversity compared with state-of-the-art methods.
2. Related Work
Prior image-inpainting research includes diffusion, patch-based, GAN, VAE, and transformer approaches, each addressing different aspects of completion quality, diversity, or global modeling. BAT specifically adapts transformers to combine bidirectional context with output dependency for image inpainting.
- Image Inpainting: Image inpainting is broadly categorized into deterministic and diverse approaches because realistic completion is an ill-posed problem.
- Traditional Methods: Diffusion methods can blur or fail on large missing regions, while patch-based methods struggle with large complex scenes because they rely on low-level feature matching.
- GAN-based Methods: GAN-based inpainting methods use adversarial learning and local discriminators to improve completion and local consistency.
- Diverse Inpainting: VAE-based methods target pluralistic fillings through dual pipelines or reference images, but variational training limits completion quality.
- Transformers in Vision: Transformers model interactions across sequences and have been applied to image patches, detection, and image classification.
- Transformers in Vision: BAT applies transformers directly to image inpainting so the model learns bidirectional context and output dependency simultaneously.
3. Proposed Method
BAT-Fill combines a diverse-structure transformer with a texture generator to complete images in two stages. Its BAT component jointly models bidirectional context and autoregressive dependencies, while the texture generator upsamples sampled structures with image-conditioned details.
- Overall Framework: BAT-Fill first samples diverse, coherent low-resolution structures and then generates high-resolution texture conditioned on each structure and the masked input.The diverse-structure generator models global structural information; the texture generator synthesizes fine-grained details to produce the final inpainting results.
- Diverse-Structure Generator: The 32 × 32 × 3 low-resolution representation reduces transformer complexity, while a 512-color palette preserves the main image structure for discrete autoregressive generation.Direct 8-bit RGB modeling would require 256^3 classes per pixel; k-means color quantization reduces this dimensionality to 512.
- Bidirectional and Autoregressive Transformer: BAT combines autoregressive output dependency with bidirectional contextual information, addressing the arbitrary shapes and varied neighboring backgrounds of missing regions.Unlike MLM's independent masked-token predictions, BAT preserves dependencies among predicted tokens while accessing context from both directions.
- Bidirectional and Autoregressive Transformer: BAT permutes unmasked tokens before masked tokens, preserves masked-token order and positions, and uses a customized attention mask to prevent future-token information leakage.During prediction, the model can attend to non-predicted and previously predicted tokens, but not future masked tokens.
- Inference: At inference, BAT predicts masked tokens in raster-scan order with top-K sampling, concatenating each prediction as a condition for the next token.Repeated sampling produces discrete structures that are converted back to RGB values before texture refinement.
4.1. Experimental Settings
Experiments use three public datasets with different image characteristics and compare BAT-Fill against deterministic and diverse inpainting methods using five standard metrics.
- Datasets: The evaluation covers CelebA-HQ, Places2, and Paris StreetView, representing aligned faces, diverse natural scenes, and Paris street-view images.CelebA-HQ uses 28,000 training and 2,000 validation images; Places2 evaluation uses 800 validation images; Paris StreetView contains 14,900 training and 100 validation images.
- Compared Methods: The compared methods include deterministic approaches GC, EC, and MEDFE, plus diverse approaches PIC and ICT.GC uses gated convolutions, EC predicts salient edges, MEDFE separates structural and textural features, PIC uses a VAE, and ICT combines transformers with CNNs.
- Evaluation Metrics: Performance is evaluated with FID, mean ℓ1 error, PSNR, SSIM, and LPIPS.FID measures perceptual distribution distance, while LPIPS measures diversity between random pairs of sampled inpainting results.
- Implementation: The network is trained in PyTorch on 256×256 images with random irregular masks.The diverse-structure and texture generators use separate optimization configurations, including AdamW for the structure generator and Adam for the texture generator.
4.2. Quantitative Evaluation
Quantitative evaluations use irregular masks across three datasets and compare BAT-Fill with deterministic and diverse methods, with the reported results favoring BAT-Fill across datasets and metrics.
- Evaluation Protocol: Quantitative evaluations cover three datasets with mask-ratio categories and an additional random category sampling ratios from 20% to 60%.Compared-model performance comes from publicly available pretrained models or implementation code.
- Evaluation Protocol: BAT-Fill is compared with deterministic and diverse inpainting methods using random samples rather than discriminator-based result sorting.The authors note that ℓ1, SSIM, and PSNR favor deterministic methods because predictions are directly compared with ground truth.
- Quantitative Results: On Paris StreetView, BAT-Fill achieves the best FID scores across mask ratios and outperforms PIC in both FID and LPIPS.The comparison includes deterministic methods GC, EC, and MEDFE and diverse method PIC.
- Quantitative Results: On CelebA-HQ, BAT-Fill consistently outperforms all compared methods, especially in FID, while on Places2 it is comparable to deterministic methods and generally has better FID scores.BAT-Fill also shows numerical superiority over PIC and better FID scores than ICT on Places2.
4.3. Qualitative Evaluations
Qualitative comparisons across CelebA-HQ, Places2, and Paris StreetView show BAT-Fill producing realistic, diverse completions with fewer artifacts than several competing methods.
- CelebA-HQ: On CelebA-HQ, BAT-Fill produces a wider range of facial makeups and features with fewer artifacts than EC, GC, and PIC.EC and GC generate deterministic results and may produce distorted facial structures, while PIC has limited diversity.
- Places2 and Paris StreetView: On Places2 and Paris StreetView, competing methods exhibit blurs, corrupted textures, unreasonable semantics, or limited diversity, whereas BAT-Fill produces realistic inpainting with fewer artifacts.The passage specifically contrasts BAT-Fill with EC, GC, MEDFE, PIC, and ICT.
4.4. Ablation Study
The ablation study separates BAT’s bidirectional context and autoregressive modeling, showing that their combination improves reconstruction quality while retaining diversity.
- Ablation settings: Removing bidirectional context yields an autoregressive model with unidirectional attention over previous missing tokens.This variant isolates the effect of bidirectional context in the BAT ablation.
- Ablation settings: Removing autoregressive modeling yields an MLM-equivalent model that reconstructs missing tokens independently.This variant isolates the effect of output dependency.
- Ablation results: BAT clearly outperforms MLM in reconstruction quality, mainly reflected by FID, while achieving comparable diversity measured by LPIPS.The ablation uses the Paris StreetView validation set with irregular masks and 40%-60% mask ratios.
- Ablation results: AR greatly degrades reconstructed structures, and its high LPIPS diversity is largely attributed to poor reconstruction quality.The result indicates that diversity alone does not guarantee faithful structural completion.
- Interpretation: BAT models output dependency to align future predictions with previously predicted tokens, improving reconstructed-structure consistency.Overall, the ablation shows that BAT addresses constraints of both AR and MLM.
5. Conclusion
The paper presents BAT-Fill for realistic and diverse image inpainting by combining autoregressive transformers with bidirectional context and output dependency. Experiments report superior image-inpainting quality and diversity, while future work considers other recovery and generation tasks.
- Conclusion: BAT-Fill leverages autoregressive transformers to achieve realistic and diverse image inpainting.The framework uses their long-dependency modeling capacity.
- Conclusion: BAT models bidirectional context and output dependency simultaneously to improve inpainting quality and diversity.This is the paper’s proposed mechanism within BAT-Fill.
- Conclusion: Extensive experiments show that BAT-Fill achieves superior image inpainting in terms of both quality and diversity.The conclusion states this result without restricting it to a single dataset or metric.
- Future work: Future work will explore adapting the approach to other image recovery or generation tasks using semantic labels, edges, or poses as conditions.The proposed adaptation replaces BAT’s non-predicted part with alternative conditions.