Source-linked AI summary
Combining Compositional Models and Deep Networks For Robust Object Classification under Occlusion
Adam Kortylewski, Qing Liu, Huiyu Wang, Zhishuai Zhang, Alan Yuille
TL;DR
Deep networks and compositional models have complementary strengths, but deep networks struggle with partially occluded objects, especially when occlusion patterns differ from training. This paper combines them, outperforming a standard deep network on occluded-object classification while retaining high discrimination for non-occluded objects.
Problem
Deep networks classify partially occluded objects poorly when they have not seen occlusion in training, and generalize poorly when occluder patterns differ from training.
Source: §5 · p. 8Method
The model clusters DCNN features into spatially located part dictionaries and uses a compositional classifier when the DCNN prediction is uncertain.
Source: §3.1 · pp. 3–4, §3.3 · p. 5Results
19.4% on MNIST digits and 13.9% on PASCAL3D+ objects in absolute classification performance over a standard deep network under partial occlusion.
Source: §5 · p. 8Takeaways & Limitations
The combined model recognizes partially occluded objects without occlusion exposure during training while retaining high discriminative performance for non-occluded objects.
Source: §1 · p. 2Part activations are assumed independent, so the model does not encode dependencies between parts.
Source: §3.1 · p. 4
Abstract
from arXiv · showhide
Deep convolutional neural networks (DCNNs) are powerful models that yield impressive results at object classification. However, recent work has shown that they do not generalize well to partially occluded objects and to mask attacks. In contrast to DCNNs, compositional models are robust to partial occlusion, however, they are not as discriminative as deep models. In this work, we combine DCNNs and compositional object models to retain the best of both approaches: a discriminative model that is robust to partial occlusion and mask attacks. Our model is learned in two steps. First, a standard DCNN is trained for image classification. Subsequently, we cluster the DCNN features into dictionaries. We show that the dictionary components resemble object part detectors and learn the spatial distribution of parts for each object class. We propose mixtures of compositional models to account for large changes in the spatial activation patterns (e.g. due to changes in the 3D pose of an object). At runtime, an image is first classified by the DCNN in a feedforward manner. The prediction uncertainty is used to detect partially occluded objects, which in turn are classified by the compositional model. Our experimental results demonstrate that combining compositional models and DCNNs resolves a fundamental problem of current deep learning approaches to computer vision: The combined model recognizes occluded objects, even when it has not been exposed to occluded objects during training, while at the same time maintaining high discriminative performance for non-occluded objects.
1. Introduction
DCNNs struggle with occlusion patterns that may not appear in training, while compositional models are more robust but less discriminative. The paper combines their strengths and reports improved classification of partially occluded objects.
1. Introduction
- Because occlusion patterns are too numerous and variable to cover during training, vision systems must generalize beyond their training data.Source: §1 · p. 1
- DCNNs generalize poorly to partial occlusion and mask attacks, while compositional models are robust to occlusion but less discriminative.Source: §1 · p. 2
- The proposed model learns compositional object models from DCNN features, which are robust to illumination, background clutter, and non-rigid part deformations.Source: §1 · p. 2
- Mixtures of compositional models represent 3D objects using components for particular viewpoints or 3D structures, and outperform single models at classification.Source: §1 · p. 2
- The combined model outperforms a standard DCNN by 19.4% on MNIST digits and 13.9% on PASCAL3D+ in absolute classification performance.Source: §1 · p. 2
2. Related Work
Prior compositional models can handle occlusion, but pixel-based approaches must model nuisances, and several DCNN approaches encode parts only implicitly. This work learns compositional models from DCNN features and uses an explicit model to classify occluded objects.
2. Related Work
- Compositional models have recognized occluded parts and simple shapes, while this work targets robust classification of 3D objects in natural scenes under strong occlusion.Source: §2 · p. 2
- Learning compositional models from DCNN features avoids explicitly modeling nuisances such as illumination and object deformation, which challenge pixel-based approaches.Source: §2 · p. 2
- Earlier approaches encourage part-like activations or sparse, disentangled features, but their compositional models remain implicit and do not provide occlusion robustness.Source: §2 · pp. 2–3
- The proposed inference uses the DCNN first and invokes the compositional branch when its prediction is uncertain, combining discriminative and occlusion-robust predictions.Source: Figure 2 · p. 3
3. A Robust Model Combining Deep Networks and Compositional Models
The model builds spatial compositional representations from DCNN features, adds mixtures for varying part layouts and an occlusion model, then combines them with DCNN predictions. The DCNN handles confident predictions; uncertain ones are classified by the compositional model.
3.1. A Dictionary-Based Compositional Model of DCNN Features
- At a fixed CNN layer, feature vectors at spatial positions correspond to image patches, preserving their spatial layout for modeling.The feature map is the output of a CNN layer, and its position is defined on a 2D lattice.Source: §3.1 · p. 3
- K-means clusters DCNN feature vectors into dictionary components called parts because they activate recurring, similar image patterns.The illustrated activations can share semantic meanings and often correspond to patterns recurring in a particular image class.Source: §3.1 · p. 3
- Cosine-distance thresholding encodes detected dictionary parts as a sparse binary map, with at least one active component at each position.The reported threshold is δ = 0.45; the class-conditional Bernoulli model preserves spatial positions while assuming parts are independently distributed.Source: §3.1 · pp. 3–4
- Mixtures represent distinct spatial part distributions, accommodating viewpoint or 3D-structure changes that a single compositional model cannot represent well.The mixture assignments are initialized by spectral clustering with Hamming distance on part-detection maps; the parameters and assignments are then estimated iteratively.Source: §3.1 · p. 4
3.2. Augmenting the Compositional Model with an Occlusion Model
- The occlusion model treats each position as object or locally independent clutter, avoiding explicit models of the many possible occluding objects.It assumes the background has no spatial structure, estimates occlusion variables using independence, and can localize the occluder.Source: §3.2 · p. 4
3.3. Combining Compositional Models and DCNNs
- The combined classifier uses the DCNN when its top-class probability exceeds a threshold and otherwise uses the compositional model.The paper describes the branches as complementary: the DCNN performs well on non-occluded objects, while the compositional model handles partial occlusion.Source: §3.3 · p. 5
4. Experiments
The experiments evaluate classification under synthetic and natural-object occlusion, including settings where training occlusions are spatially or visually biased. The combined model improves recognition of occluded objects while retaining comparable performance on non-occluded objects.
4. Experiments
- The experiments use MNIST digits and PASCAL3D+ vehicles, with simulated occlusions from noise, textures, white masks, or natural objects.Occlusion levels increase with masked object area; the occluding objects differ from the objects being classified.Source: §4 · p. 5, Figure 5 · p. 6
4.1. DCNNs Do Not Generalize Well Under Partial Occlusion
- VGG performance drops by more than 10% under stronger PASCAL3D+ occlusion, and biased training occlusions impede generalization to unseen positions and appearances.White masks at unseen positions can improve performance, while unseen noise and textures reduce it.Source: §4.1 · p. 6, Table 2 · p. 7
4.2. The Proposed Model Classifies Partially Occluded Objects Robustly
- On MNIST, mean absolute gains over VGG are 12.1% for level-1, 25.7% for level-2, and 27.5% for level-3 occlusions.Mixtures also outperform a single compositional model on these digits; each mixture focuses on a particular writing style.Source: §4.2 · p. 6, Table 1 · p. 7
- On PASCAL3D+, absolute gains over VGG reach 29.9% for level-3 mask attacks and 44.5% for level-3 natural-object occlusion.The combined model also retains comparable performance on non-occluded objects.Source: §4.2 · p. 6, Table 1 · p. 7
- Overall, the combined model generalizes better to previously unseen partial occlusion than a standard DCNN while matching it on similarly distributed, non-occluded data.The two branches provide mutual benefit compared with either branch individually.Source: §4.2 · p. 7
- Compositional models with an occlusion model exceed VGG in absolute performance by 8.6% for CompOcc and 11.5% for CompMixOcc.The mixture is superior to a single compositional model when both include an occlusion model.Source: §4.2 · p. 7
- Compositional-model visualizations show detected parts, training patches that activate those parts, and locations estimated as occluded.The occlusion map uses positive log-likelihood ratios between the background and compositional models.Source: §4.2 · p. 7, Figure 6 · p. 8
5. Conclusion
The conclusion argues that DCNNs remain vulnerable when occlusion patterns differ from training, and combines compositional models with DCNN features to address this gap. The combined model exceeds a standard deep network under partial occlusion on MNIST digits and PASCAL3D+ objects.
5. Conclusion
- DCNNs generalize poorly to partial occlusion when training lacks occlusion, and remain weak when trained on severe occlusion but occluder appearance or spatial distribution is biased.Source: §5 · p. 8
- Learning compositional models from DCNN features uses features robust to nuisances, enabling representation of complex objects in natural scenes.The cited conclusion contrasts this with learning from pixels, which requires modeling processes such as local deformation and illumination.Source: §5 · p. 8
- Mixtures of compositional models represent 3D objects and outperform single compositional models at object classification.Source: §5 · p. 8
- 19.4% on MNIST digits and 13.9% on PASCAL3D+ objects in absolute classification performance: the combined model outperforms a standard deep network under partial occlusion.Source: §5 · p. 8