Source-linked AI summary
Zoom Better to See Clearer: Human and Object Parsing with Hierarchical Auto-Zoom Net
Fangting Xia, Peng Wang, Liang-Chieh Chen, Alan L. Yuille
TL;DR
Object-part parsing in natural images is difficult because object and part scales vary widely, while fixed-scale methods lack local scale adaptation. HAZN hierarchically estimates regions and scales, zooms into objects and then parts, and refines their part parsing. On PASCAL datasets, it improves parsing for humans, horses, and cows, including a reported 5% mIOU gain over state-of-the-art methods.
Problem
Object-part parsing in wild images must handle large variations in scale, location, occlusion, and pose, while existing methods lack adaptation to object size.
Method
HAZN uses two sequential Auto-Zoom Nets and fully convolutional networks to estimate object and part regions, resize them adaptively, and refine part scores.
Results
HAZN performs significantly better by 5% mIOU than state-of-the-art methods for parsing humans, horses, and cows on challenging PASCAL datasets.
Takeaways & Limitations
Searching first for objects and then parts enables fine regional zooming that avoids the memory cost of applying very large scales to the entire image.
Takeaways & Limitations
The models still suffer from heavy occlusion and unusual poses.
Abstract
from arXiv · showhide
Parsing articulated objects, e.g. humans and animals, into semantic parts (e.g. body, head and arms, etc.) from natural images is a challenging and fundamental problem for computer vision. A big difficulty is the large variability of scale and location for objects and their corresponding parts. Even limited mistakes in estimating scale and location will degrade the parsing output and cause errors in boundary details. To tackle these difficulties, we propose a "Hierarchical Auto-Zoom Net" (HAZN) for object part parsing which adapts to the local scales of objects and parts. HAZN is a sequence of two "Auto-Zoom Net" (AZNs), each employing fully convolutional networks that perform two tasks: (1) predict the locations and scales of object instances (the first AZN) or their parts (the second AZN); (2) estimate the part scores for predicted object instance or part regions. Our model can adaptively "zoom" (resize) predicted image regions into their proper scales to refine the parsing. We conduct extensive experiments over the PASCAL part datasets on humans, horses, and cows. For humans, our approach significantly outperforms the state-of-the-arts by 5% mIOU and is especially better at segmenting small instances and small parts. We obtain similar improvements for parsing cows and horses over alternative methods. In summary, our strategy of first zooming into objects and then zooming into parts is very effective. It also enables us to process different regions of the image at different scales adaptively so that, for example, we do not need to waste computational resources scaling the entire image.
1 Introduction
The paper targets object-part parsing in wild images, where scale, location, occlusion, and pose vary substantially. HAZN addresses this by estimating scales and locations hierarchically, then zooming into objects and parts to refine parsing.
- Motivation: Object parsing in the wild must handle large variations in scale, location, occlusion, and pose.The authors use PASCAL images because they contain substantial object-scale variation and are less constrained than purpose-built human-parsing datasets.
- Motivation: Existing fully convolutional methods can make mistakes on small or large objects because they lack a mechanism to adapt to object size.This limitation motivates explicit scale adaptation rather than processing all objects at fixed scales.
- Approach: HAZN jointly estimates object and part scales and parses objects at image-, object-, and part-level granularities.It sequentially combines object-scale and part-scale Auto-Zoom Nets, using FCNs to estimate regions, zoom them, and refine parsing.
- Approach: Adaptive regional scaling searches a larger range of scales than fixed-scale processing while avoiding whole-image enlargement.On PASCAL, the largest zooming ratio is 2.5 at part level, compared with 1.5 when zooming the whole image.
- Evaluation: The method is evaluated on challenging human, horse, and cow parsing datasets and is especially effective for small object parts.The experiments report performance above previous state-of-the-art methods by a large margin.
2 Background
Prior object and human parsing work often relied on constrained scenes or fixed components. The paper positions HAZN as a more unified coarse-to-fine approach for scale variation and wild-scene parsing.
- Prior work: Human parsing studies often assume well-localized instances, simple poses, and relatively constrained appearances.Such assumptions limit direct application to real-world images containing varied poses, scales, and occlusions.
- Prior work: Deep segmentation models improved object-part parsing in the wild but can require complex training and propagate object-mask errors into part segments.The cited drawbacks concern multi-component pipelines and inaccurate or confused object masks.
- Positioning: HAZN follows a coarse-to-fine strategy while using three FCNs with the same structure and avoiding premature decisions.This provides a more unified alternative to pipelines whose object-mask errors can affect part segmentation.
- Positioning: Earlier scale-handling methods commonly used low-level features or constrained scene layouts, making wild-scene variation difficult to handle with DCNNs.The paper motivates a scale strategy integrated with deep convolutional networks.
3 The Model
HAZN addresses scale variation in object parsing by combining image-, object-, and part-level FCNs with sequential Auto-Zoom refinement. Each stage estimates regions and scales, zooms them, and refines part scores before merging results.
- Architecture: HAZN uses three FCN levels—image, object, and part—to jointly estimate scale/location and parse semantic parts.All three FCNs use the modified DeepLab-LargeFOV structure.
- Object-scale AZN: The object-scale AZN estimates object ROIs, zooms them to standard-sized regions, and predicts refined part scores within those regions.ROI proposals are obtained by thresholding confidence scores and applying non-maximum suppression.
- Score merging: The model merges scores from overlapping zoomed regions into whole-image maps, while uncovered pixels retain image-level scores.Overlapping ROI contributions are combined using confidence-weighted score merging.
- Part-scale AZN: Part-scale AZN repeats ROI estimation and zooming within object regions to refine scores for object parts.Part ROIs are similarly thresholded, suppressed, resized, parsed, and merged.
- Extension: HAZN can extend to finer levels, adding one FCN for each additional Auto-Zoom stage to focus on smaller parts.The paper gives human eyes as an example of a finer-scale target.
- Training: Training combines part parsing with SEN-based ROI regression and confidence estimation derived from ground-truth object or part regions.The SEN is trained by minimizing negative log likelihood, with balanced cross entropy for confidence seeds and Euclidean regression loss for bounding boxes.
4 Experiments
Experiments evaluate HAZN on human and animal part parsing, including scale-specific, instance-wise, qualitative, and ablation analyses. HAZN improves parsing accuracy, particularly for small instances and parts, while remaining limited by heavy occlusion and unusual poses.
- Experimental setup: The experiments use PASCAL-Person-Part for human parsing, with six merged part classes and 1,716 training and 1,817 validation images.The evaluation uses mIOU and instance-wise AP r part.
- Human parsing results: 57.5% performance is 5.8% better than DeepLab-LargeFOV and 4.5% better than DeepLab-LargeFOV-CRF.HAZN improves segmentation accuracy across all parts without CRF post-processing.
- Human parsing results: HAZN surpasses Multi-Scale Attention by 1.15% mIOU overall and improves upper-leg parsing by around 3%.The advantage is larger for small instances, reaching 9.5% at size XS and 5.5% at size S.
- Ablation analysis: Removing object-scale AZN decreases mIOU by 2.8%, while removing part-scale AZN decreases it by 1%.Object-scale refinement contributes strongly to overall performance, while part-scale refinement improves detailed predictions such as lower arms by around 3%.
- Scale-specific analysis: HAZN improves DeepLab-LargeFOV by 14.6% for size XS and 10.8% for size S, with roughly 5% gains for sizes M and L.The four sizes are defined by bounding-box scale, from XS [0, 80] to L [220, 520].
- Animal parsing results: For horses and cows, HAZN improves over the state-of-the-art by roughly 5% mIOU and exceeds 10% improvement for some small parts.Head parsing for horses or cows and cow-tail detection show the most noticeable small-part gains.
5 Conclusions
HAZN parses objects in the wild by adaptively estimating object and part scales through two Auto-Zoom stages. It achieves a 5% mIOU improvement over state-of-the-art methods on humans, horses, and cows, while future work targets finer parts such as hands and eyes.
- HAZN performs per-pixel object-part segmentation using a two-stage process that estimates scales for objects and their parts.
- 5% mIOU improvement over state-of-the-art methods is reported for humans, horses, and cows.
- Adaptive object-then-part zooming can enlarge small image regions without scaling the entire image at prohibitively expensive memory costs.
- Future work includes extending HAZN to finer parts such as human hands and eyes.