Source-linked AI summary
Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Parts
Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, Alan Yuille
TL;DR
Object detection remains challenging under deformation, occlusion, and low resolution, especially when holistic objects or body parts are inconsistently detectable. The paper uses a fully connected holistic-and-part model with switchable nodes and supervised body-part annotations. It reports improved animal detection on PASCAL VOC and a richer representation of objects in terms of body parts.
Problem
Object detection remains difficult for highly deformable animals under large deformation, occlusion, and low resolution, particularly when body parts are hard to detect.
Method
A fully connected graphical model represents holistic objects and semantic body parts separately, using switch variables to decouple components that are hard to detect.
Results
4.1% AP improvement over the state-of-the-art is reported for animal categories on PASCAL VOC 2010, with reliable body-part localization and richer object descriptions.
Takeaways & Limitations
The model represents many combinations of holistic objects and body parts to accommodate different detectability patterns caused by deformation, occlusion, and low resolution.
Takeaways & Limitations
Inference uses an approximate solution because the model is loopy and each node has a large space of possible positions and scales.
Abstract
from arXiv · showhide
Detecting objects becomes difficult when we need to deal with large shape deformation, occlusion and low resolution. We propose a novel approach to i) handle large deformations and partial occlusions in animals (as examples of highly deformable objects), ii) describe them in terms of body parts, and iii) detect them when their body parts are hard to detect (e.g., animals depicted at low resolution). We represent the holistic object and body parts separately and use a fully connected model to arrange templates for the holistic object and body parts. Our model automatically decouples the holistic object or body parts from the model when they are hard to detect. This enables us to represent a large number of holistic object and body part combinations to better deal with different "detectability" patterns caused by deformations, occlusion and/or low resolution. We apply our method to the six animal categories in the PASCAL VOC dataset and show that our method significantly improves state-of-the-art (by 4.1% AP) and provides a richer representation for objects. During training we use annotations for body parts (e.g., head, torso, etc), making use of a new dataset of fully annotated object parts for PASCAL VOC 2010, which provides a mask for each part.
Abstract
The paper introduces a part-based detection approach that separately represents holistic objects and body parts, allowing difficult components to be decoupled. On six PASCAL VOC animal categories, it improves state-of-the-art performance by 4.1% AP and provides richer object representations.
- The model separately represents holistic objects and body parts using a fully connected arrangement of templates.It automatically decouples components that are hard to detect, supporting many holistic-object and body-part combinations.
- The approach addresses large deformation, partial occlusion, and low-resolution animals whose body parts may be difficult to detect.
- 4.1% AP improvement over state-of-the-art is achieved on six animal categories in PASCAL VOC.Training uses fully annotated body-part masks from a new PASCAL VOC 2010 dataset.
1. Introduction
Animal detection remains difficult under deformation, occlusion, and low resolution because different components can vary in detectability. The proposed model adaptively switches off unreliable holistic or part nodes, representing and combining whichever cues remain detectable.
- Large deformation can make holistic templates unreliable even when individual body parts remain detectable.
- Occlusion, truncation, and local ambiguity can make some body parts unreliable, requiring automatic selection of a dependable subset.
- At small scales, weak body-part cues motivate ignoring parts and relying on the holistic object.
- The model decouples undetectable body parts or holistic objects and models detectability rather than visibility.A visible but highly deformed holistic body may still be switched off when it is difficult to detect.
- Switch variables in a fully connected graphical model encode spatial and scale relationships while enabling many holistic-object and body-part combinations.
- 7.3% and 4.1% AP improvements over DPM and segDPM, respectively, are reported, alongside richer body-part descriptions and localization.The method is evaluated on highly deformable PASCAL VOC animals using a new annotated body-part dataset.
2. Related Work
Prior work includes holistic-root detectors with fixed parts, models allowing missing or variable parts, and multi-resolution approaches. This paper differs by modeling semantic body-part detectability, allowing the root itself to switch off, and supporting body-part recovery through strong supervision.
- Fixed-root part-based detectors are vulnerable to occlusion, while missing-part and variable-part models address flexibility differently.
- Unlike prior missing-part DPMs, this method models detectability instead of visibility and can switch off the holistic root.
- Human pose methods often omit holistic templates, whereas this model can switch the root on or off when individual parts are difficult to detect.
- Unlike detectors using latent discriminative parts, the method uses semantic body parts intended to recover actual object components.
- Additional annotations have been used for supervised part-based detection, poselet clustering, and distinctive-part detection in animals.
3. Model
The model represents holistic objects and body parts in a fully connected graph, using switch variables to ignore undetectable nodes and accommodate varied detectability patterns. It scores appearance, spatial, and scale relationships, learns from supervised annotations, and uses approximate inference and post-processing to produce detections.
- Graphical model: The model represents a holistic object and body parts as nodes in a fully connected graph connected by spatial and scale relationships.Fully connecting the nodes avoids structural breaks when non-leaf nodes are switched off.
- Graphical model: Each node has position, size, and a binary switch variable indicating whether it should be included in detection.A switched-off node contributes neither its appearance term nor its relationships with other nodes.
- Scoring function: The configuration score combines node appearance, pairwise spatial and scale terms, and a bias modeling the prior of each detectability pattern.Appearance is modeled within windows defined by node position and size; the spatial and scale features are used to encode geometric consistency.
- Detectability patterns: Switch variables represent many holistic-object and body-part combinations for low-resolution, deformed, or partially occluded objects.The model can rely on the holistic object when parts are hard to detect, on parts when the holistic appearance is difficult to model, or on subsets of parts under occlusion.
- Inference: Inference searches for the highest-scoring configuration, whose switches, positions, and scales specify detected components and whose score measures detection confidence.Because the model is loopy and the state space is large, the method uses an approximate procedure, reducing hypotheses with appearance thresholding and non-maximum suppression before exhaustive evaluation of the remaining configurations.
- Learning: Training uses supervised holistic-object and body-part annotations, appearance activations to assign switch labels, and linear max-margin learning for the model parameters.A node is labeled switched off when its appearance activation fails the detection threshold; the learned feature vector concatenates appearance, bias, spatial, and scale terms.
- Post Processing: Post-processing estimates the holistic bounding box from the locations of switched-on body parts and removes duplicate detections through greedy part-based suppression.For n active parts, a 4n-dimensional vector of part-box corners is mapped to the holistic box using independently learned linear functions for each detectability pattern.
4. Experimental Evaluation
Experiments on six PASCAL VOC 2010 animal categories evaluate the model against prior detectors and test its adaptive detectability patterns, small-object behavior, and part localization.
- Experimental setup: Experiments use six highly deformable PASCAL VOC 2010 animal categories, with added body-part masks and bounding boxes for training and evaluation.The experiments use trainval for training and test for evaluation; bounding boxes, rather than masks, are used by the model.
- Comparison with other methods: 32.3% AP is achieved after adding the holistic object node, 7.3% AP above DPM performance.The model without the holistic node is already competitive, while the added node further boosts performance.
- Detectability patterns: Performance degrades drastically when all nodes are forced on, supporting adaptive switching off of unreliable holistic or part nodes.The no-switch-off condition is evaluated both with and without the holistic object node.
- Small scale objects: The holistic-only detectability pattern occurs most often for extra-small instances and less often for larger size classes.This pattern is intended for cases where body-part cues are difficult to detect at low resolution.
- Part localization: Part localization is evaluated with PCP, while POP measures the percentage of detected objects for which each part type is estimated.Head and torso are used more often than legs, with head more frequent for cats and dogs and torso more reliable for birds and sheep.
5. Conclusion
The method uses a flexible part-based model that can switch off body parts or the holistic object, improving detection while providing richer object descriptions and body-part localization.
- 5. Conclusion: The model switches off individual body parts or the holistic object during inference to handle highly deformed, occluded, or low-resolution objects.It performs inference on a loopy graph modeling spatial and scale relationships among all body parts and the holistic object.
- 5. Conclusion: The method provides richer object descriptions by outputting bounding boxes for body parts in addition to holistic object detections.Figure 5 illustrates head, torso, legs, holistic-object, and generated holistic-object bounding boxes.
- 5. Conclusion: Part localization performance is reported on PASCAL VOC 2010 using POP and PCP measures.POP denotes the Percentage of Objects with the Part estimated, while PCP denotes the Percentage of Correctly estimated body Parts.
- 5. Conclusion: 4.1% AP improvement over the state-of-the-art is reported for animal categories on PASCAL VOC 2010.The method is compared with strongly supervised methods and state-of-the-art detection on the dataset.