D-Feat Occlusions: Diffusion Features for Robustness to Partial Visual Occlusions in Object Recognition

1. Introduction

• Occlusion can hide the discriminative cues recognition models rely on, disrupting their predictions.The introduction notes that deep systems often focus on cues such as a dog's ear for breed classification.Source: §1 · p. 1
• Real-world occlusion robustness is underexplored, in part because datasets with real occlusions are scarce.The authors use PartImageNet's mutually exclusive pixel part segmentations for single-object images to explore occlusion.Source: §1 · p. 1
• Input augmentation inpaints occluded image regions, while feature augmentation uses intermediate diffusion features to augment recognition features.Figure 1 shows the input branch using Stable Diffusion inpainting and the feature branch extracting frozen U-Net features.Source: §1 · p. 1, Figure 1 · p. 2
• The study asks whether synthetic image completion or generative features can strengthen robustness to occlusions.Figure 2 illustrates original, blacked-out, and inpainted training images; the inpainted images are used for input augmentation.Source: §1 · p. 1, Figure 2 · p. 3
• The proposed augmentations make Transformers and ConvNets more robust to partial object occlusions on simulated and real-world data.The paper introduces D-Feat, a real-world occlusion dataset, alongside evaluations on simulated ImageNet occlusions.Source: §1 · p. 2

Robustness to Occlusions

• Earlier object-recognition studies lacked approaches for hallucinating object visual features behind occluders.The cited compositional CNN approaches address partial occlusion with occluder kernels, but focus on ConvNets and use a sequential pipeline.Source: §2 · p. 3
• Occlusion robustness work includes detection and localization, CutMix-style evaluation, and compositional CNNs; this paper studies partial object-part occlusions across Transformers and ConvNets.The related work contrasts the proposed compact pipeline with prior sequential methods focused on ConvNets.Source: §2 · p. 3

Classical Image Augmentation

• Prior occlusion methods include regional CutMix and random erasing, while this work drops part-annotated object regions and augments features with diffusion features.The related work contrasts regional dropout and part-based image regions with the proposed feature augmentation.Source: §2 · p. 3

Diffusion Models and Image Augmentation

• Diffusion models have been used for inpainting, denoising, and generation; this paper applies their generative features to hallucinate occluded visual features for recognition.The related-work discussion also describes diffusion augmentation that generates entire synthetic images from prompts or captions.Source: §2 · pp. 3–4
• Unlike whole-image synthetic generation approaches, this work uses diffusion to reconstruct occluded regions and extract features for occlusion robustness.The paper's two uses are diffusion-based inpainting and informative diffusion features for inferring object structure beyond occlusions.Source: §2 · p. 4

• D-Feat proposes input- and feature-space augmentations using part annotations and diffusion models to improve robustness to partial object occlusions.Source: §3 · p. 4
• Input augmentation occludes annotated object parts, inpaints the missing pixels with Stable Diffusion Inpainting, and trains on real and augmented images.The ground-truth class supplies the text prompt.Source: §3 · p. 4
• For each image, the method selects half its annotated parts, choosing combinations with the largest occluded area before inpainting or applying a baseline.Source: §3 · p. 4
• Feature augmentation adds intermediate Stable Diffusion U-Net features, extracted with a null prompt, to the classification features.The authors hypothesize these features capture information useful for reconstructing missing pixels.Source: §3 · p. 4
• The baselines black out parts, use CutMix, or replace parts with pixels from another image; the loss weights classification of augmented images and masks.Source: §3 · pp. 4–5
• Figure 3 contrasts simulated ImageNet Validation occlusions, including a 60% example, with real-world D-feat images from classes overlapping PartImageNet.Source: Figure 3 · p. 5

4.1. Datasets

• PartImageNet trains the classifiers, while evaluation uses ImageNet1K Val on the overlapping classes with randomly blacked-out 16 × 16 patches for simulated occlusion.The study also evaluates real-world occlusions on D-Feat, curated with three images for each of 100 PartImageNet classes.Source: §4.1 · p. 5

4.3. Results

• For simulated occlusions, proposed augmentations contain the highest top-1/top-5 accuracy for each information-loss column, with larger gains at higher information loss.Table 1 evaluates the 158 classes shared by ImageNet1K Val and PartImageNet.Source: Table 1 · p. 6, §4.3 · p. 7

4.2. Experimental Setup

• The study compares pretrained Swin-Base and ConvNeXt-Base, using PartImageNet for classification fine-tuning and a frozen Stable Diffusion model for inpainting and feature extraction.The models have similar parameter counts and FLOPs.Source: §4.2 · p. 6

4.3. Results

• On D-Feat’s 100 real-world occlusion classes, both architectures benefit from proposed augmentations, which significantly outperform other augmentation techniques.D-Feat is evaluation-only because it lacks part annotations.Source: Table 2 · p. 6, §4.3 · p. 7
• ConvNeXt and Swin show similar occlusion robustness and similar benefits from the proposed augmentations; diffusion-based methods outperform other methods for most occlusion levels.Swin’s apparent accuracy advantage is qualified by its higher baseline accuracy.Source: §4.3 · p. 7

5. Conclusions

• Diffusion feature augmentation performs similarly to diffusion-based inpainting while being significantly more efficient.The paper reports improved robustness on a simulated-occlusion benchmark and higher performance than prior input augmentation techniques on real occlusions.Source: §5 · p. 7
• The paper proposes input-space augmentation with inpainted training images and embedding-space augmentation with diffusion features.The authors describe diffusion features as integrating information about missing or noisy pixels.Source: §5 · p. 7
• The authors propose D-Feat as a real-world occlusion evaluation dataset and anticipate diffusion features could support more occlusion-robust recognition in high-stakes applications.Autonomous vehicles recognizing pedestrians are given as an example.Source: §5 · p. 7

6. Limitations and Negative Impacts

• Real-world occlusion percentages are not quantifiable in the evaluation, while simulated occlusions quantify the occluded area but do not control which object regions are hidden.The authors note segmentation with human input could be used to estimate real-world occlusion percentages.Source: §6 · p. 7
• Input augmentation inherits Stable Diffusion inpainting’s limitations and biases, which may affect augmented-data quality.Source: §6 · p. 7
• Augmentation can amplify biases in training data and increase training compute, energy use, and carbon emissions.The authors used frozen weights whenever possible to mitigate the computational burden.Source: §6 · p. 8