Diverse Image Inpainting with Bidirectional and Autoregressive Transformers

1. Introduction

• Image inpainting admits numerous realistic solutions, but large corrupted regions with complex textures and structures make diverse, coherent completion challenging.Restored content must remain integral and consistent with uncorrupted regions.Source: §1 · pp. 1–2
• GAN methods [32] [50] [30] [26] tend toward one-to-one outputs, while VAE methods [64] can compromise quality and prior autoregressive CNNs [33] omit future context.MLM-based work [41] predicts masked tokens independently, which may oversimplify dependencies [37] and cause inconsistencies; BAT combines bidirectional context with autoregressive output dependency.Source: §1 · p. 2
• BAT sorts valid and missing pixels and begins autoregressive modeling at the first missing pixel, enabling bidirectional context and spatial dependency modeling together.A CNN-based texture generator then upsamples the recovered coarse structures and synthesizes texture details.Source: §1 · p. 2
• BAT-Fill achieves superior inpainting performance to state-of-the-art methods in inpainting quality across multiple datasets.The introduction reports extensive experiments but provides no quantitative results here.Source: §1 · p. 2

2.1. Image Inpainting

• Ill-posed image inpainting methods broadly fall into deterministic and diverse categories based on their outcomes.Source: §2.1 · p. 2

2.1.1 Deterministic Image Inpainting

• Traditional diffusion blurs on large holes and patch matching struggles with complex scenes; later adversarial approaches still target ground-truth reconstruction, yielding deterministic completions.Source: §2.1.1 · p. 2

2.1.2 Diverse Image Inpainting

• VAE-based methods offer diversity but have limited completion quality; later work explores co-modulated GANs, autoregressive VQ-VAE structures, and BERT-objective transformers, while BAT combines bidirectional and autoregressive modeling.Source: §2.1.2 · p. 3

2.2. Transformers in Vision

• Where vision transformers have served high-level tasks or autoregressive pixel generation, BAT targets inpainting by learning both bidirectional context and output dependency.Source: §2.2 · p. 3

3. Proposed Method

• The diverse-structure generator produces coherent low-resolution structures from a masked image, and the texture generator uses sampled structures to create diverse, detailed inpainting results.Sampling different structures enables multiple inpainting results after the full model is trained.Source: §3 · pp. 3–4

3.1.1 Context Representation

• The structure generator represents images at 32 × 32 × 3 resolution and uses a 512-color palette, following Chen et al. and clustering RGB values from ImageNet.The palette reduces the 256^3 possible 8-bit RGB values per pixel to 512 classes while preserving the main image structure.Source: §3.1.1 · p. 4

3.1.2 Bidirectional and Autoregressive Transformer

• BAT combines bidirectional context from unmasked tokens with autoregressive dependencies among masked-token predictions, while preventing attention to future predictions.Unmasked tokens are permuted to the front, and masked tokens retain their positions; each prediction can use unmasked context and earlier predicted tokens. Figure 3 illustrates the attention pattern.Source: §3.1.2 · p. 4, Figure 3 · p. 5

3.1.3 Transformer Architecture

• The BAT network adapts GPT as a decoder-only transformer, using a customized attention mask and a final linear projection to parameterize each pixel’s conditional distribution.At inference, masked tokens are predicted in raster-scan order using top-K sampling; each sampled token conditions the next prediction, and the discrete sequence is converted back to RGB.Source: §3.1.3 · pp. 4–5

3.2.1 Network Architecture

• The texture generator maps low-resolution structures to completed high-resolution images using CNNs and adversarial training, with SPADE [31] upsampling and gated convolutions [50].Separate encoders represent the structures and masked input image; their features are concatenated and processed by residual blocks with different dilation rates.Source: §3.2.1 · p. 5

3.2.2 Loss Functions

• The texture generator combines reconstruction, adversarial, and perceptual losses, weighted λrec=1.0, λadv=1.0, and λperc=0.2.The adversarial loss uses hinge loss [19], and perceptual loss [20] penalizes perceptual and semantic discrepancy using VGG-19 features.Source: §3.2.2 · pp. 5–6

4.1.1 Datasets

• CelebA-HQ [21], a high-quality CelebA [28] version with 30,000 aligned faces, follows split [50]: 28,000 training and 2,000 validation, with 1,000 sampled for evaluation.Source: §4.1.1 · p. 6
• Places2 has over 1.8M images across 365 scenes, with 800 validation images used for evaluation [41]; Paris StreetView [32] has 14,900 training and 100 validation images.Source: §4.1.1 · p. 6

4.1.2 Compared Methods

• GC [50] (DeepFill v2) is a two-stage gated-convolution method; EC [30] predicts salient edges to guide generation, and MEDFE [26] treats deep and shallow features as structures and textures.Source: §4.1.2 · p. 6
• PIC [64] uses a VAE for diverse inpainting, while ICT [41] combines transformers and CNNs for high-fidelity inpainting.Source: §4.1.2 · p. 6

4.1.3 Evaluation Metrics

• Evaluation uses FID [17], mean ℓ1 error, PSNR, SSIM with a 51-pixel window, and LPIPS [60], averaged across random pairs of sampled inpaintings.FID measures distribution distance between synthesized and real images; LPIPS evaluates generated-image diversity.Source: §4.1.3 · p. 6

4.1.4 Implementation Details

• The PyTorch models train on 256×256 images with random irregular masks [25], apply learning-rate decay to both networks, and run on four NVIDIA Tesla V100 GPUs.The diverse-structure generator uses AdamW with β1 = 0.9, β2 = 0.95, and learning rate 3e-4; the texture generator uses Adam with β1 = 0, β2 = 0.9, learning rates 1e-4 for the generator and 4e-4 for discriminators.Source: §4.1.4 · p. 6

4.2. Quantitative Evaluation

• Evaluations use three datasets with irregular masks grouped by mask ratio, plus a random category sampling mask ratios from 20% to 60%.Compared-method performance was obtained using publicly available pretrained models or implementation code.Source: §4.2 · p. 6
• Because ℓ1, SSIM, and PSNR favor deterministic methods, BAT-Fill uses top-50 sampling and all random samples for comparison without PIC’s [64] discriminator-based sorting.This compares BAT-Fill’s stochastic inpainting directly, without additional filtering.Source: §4.2 · p. 6
• BAT-Fill achieves the best FID across Paris StreetView mask ratios, beats PIC [64] in FID and LPIPS, consistently outperforms methods on CelebA-HQ, and is comparable to deterministic methods on Places2 while generally achieving better FID.The numerical results also show better FID than ICT [41].Source: §4.2 · p. 7, Table 1 · p. 8

4.3. Qualitative Evaluations

• On CelebA-HQ, BAT-Fill generated more diverse facial features and makeups with fewer artifacts than PIC [64], while EC [30] and GC [50] were deterministic and distorted facial structures.PIC [64] also generated diverse inpainting, but its makeups and facial features were similar; BAT-Fill’s results were described as more realistic.Source: §4.3 · p. 8
• On Places2, ICT [41] and BAT-Fill achieved realistic inpainting with fewer artifacts and better diversity than other methods; on Paris StreetView, BAT-Fill was more diverse and plausible than PIC [64].On Paris StreetView, BAT-Fill achieved comparable or better inpainting quality than deterministic methods; EC [30], GC [50], and MEDFE [26] produced blur or corrupted texture on Places2, while PIC [64] showed unreasonable semantics, artifacts, and limited diversity.Source: §4.3 · p. 8
• Table 2 compares BAT-Fill with state-of-the-art methods on 100 Paris StreetView [32] validation images with irregular masks [25].It marks the best score overall in bold and the best score among PIC [64] and BAT-Fill in underline.Source: Table 2 · p. 9

4.4. Ablation Study

• The ablation compares BAT with versions removing bidirectional context or autoregressive modeling on Paris StreetView validation images under irregular 40–60% masks at 32 × 32 resolution.Reconstructed structures were evaluated without the texture generator; MLM diversity was measured using iterative Gibbs sampling.Source: §4.4 · p. 9, Table 3 · p. 9
• BAT clearly outperforms MLM in reconstruction quality, mainly reflected by FID, while achieving comparable LPIPS diversity.The study reports that AR's high LPIPS diversity is largely attributable to poor reconstruction quality.Source: §4.4 · p. 9, Table 3 · p. 9
• By aligning future predictions with previously predicted tokens, BAT improves reconstructed-structure consistency and addresses limitations of AR and MLM.MLM exploits bidirectional context, while AR predicts missing tokens from previous tokens using unidirectional attention.Source: §4.4 · p. 9

• BAT-Fill achieves superior image inpainting in both quality and diversity, according to extensive experiments.Source: §5 · p. 9
• The framework leverages autoregressive transformers’ long-dependency modeling to produce realistic and diverse inpainting.Source: §5 · p. 9
• BAT jointly models bidirectional context and output dependency to improve inpainting quality and diversity.Source: §5 · p. 9
• Future work will explore adapting BAT to other image recovery or generation tasks using conditions such as semantic labels, edges, or poses.Source: §5 · p. 9