Deep Human Parsing with Active Template Regression

Xiaodan Liang; Si Liu; Xiaohui Shen; Jianchao Yang; Luoqi Liu; Jian Dong; Liang Lin; Shuicheng Yan

Source-linked AI summary

Deep Human Parsing with Active Template Regression

Xiaodan Liang, Si Liu, Xiaohui Shen, Jianchao Yang, Luoqi Liu, Jian Dong, Liang Lin, Shuicheng Yan

arXiv:1503.02391v1Mar 9, 2015cs.CV

TL;DR

Human parsing must decompose images into semantic fashion and body regions while handling contextual relationships among labels, masks, and spatial layouts. The ATR framework predicts template-based masks and active shape parameters with separate CNNs, and evaluations report substantially stronger performance than prior baselines.

Problem
Human parsing decomposes human images into semantic fashion and body regions, but these regions carry complex contextual relationships involving labels, masks, and spatial layouts.
Method
ATR predicts each label’s mask from learned template combinations and morphs it using position, scale, and visibility parameters through separate CNNs.
Results
19.62% higher F1-score than PaperDoll and 22.58% higher than Yamaguchi et al., while foreground accuracy reaches 71.04%.
Takeaways & Limitations
The ATR framework provides an end-to-end human-parsing approach that directly predicts label masks and morphs them into parsing results without explicitly designing feature representations or contextual interactions.
Takeaways & Limitations
The authors identify the separate-network design as a limitation, suggesting that jointly regressing template coefficients and shape parameters may further improve performance.

Abstract

from arXiv · show

In this work, the human parsing task, namely decomposing a human image into semantic fashion/body regions, is formulated as an Active Template Regression (ATR) problem, where the normalized mask of each fashion/body item is expressed as the linear combination of the learned mask templates, and then morphed to a more precise mask with the active shape parameters, including position, scale and visibility of each semantic region. The mask template coefficients and the active shape parameters together can generate the human parsing results, and are thus called the structure outputs for human parsing. The deep Convolutional Neural Network (CNN) is utilized to build the end-to-end relation between the input human image and the structure outputs for human parsing. More specifically, the structure outputs are predicted by two separate networks. The first CNN network is with max-pooling, and designed to predict the template coefficients for each label mask, while the second CNN network is without max-pooling to preserve sensitivity to label mask position and accurately predict the active shape parameters. For a new image, the structure outputs of the two networks are fused to generate the probability of each label for each pixel, and super-pixel smoothing is finally used to refine the human parsing result. Comprehensive evaluations on a large dataset well demonstrate the significant superiority of the ATR framework over other state-of-the-arts for human parsing. In particular, the F1-score reaches $64.38\%$ by our ATR framework, significantly higher than $44.76\%$ based on the state-of-the-art algorithm.

1 INTRODUCTION

Human parsing is framed as predicting structured semantic masks rather than labeling isolated pixels or hypotheses. ATR combines learned mask templates with active shape parameters in an end-to-end CNN framework to generate parsing results.

Motivation: Existing methods rely on pose estimation or bottom-up hypotheses, whose errors or coverage assumptions can limit human parsing.Pose-estimation errors can degrade parsing, while parselets and co-parsing assume generated hypotheses tightly cover semantic regions.
Motivation: Structural relations among labels, masks, and spatial layouts help explain visibility, shape, and location dependencies between neighboring human regions.For example, skirt visibility affects dress/pants probabilities and constrains leg visibility and locations; neighboring masks guide one another.
ATR formulation: ATR formulates human parsing as direct prediction and localization of each label mask, using normalized masks morphed by position, scale, and visibility parameters.Each label mask is represented through a template dictionary learned from training masks, allowing shape variability through template coefficients.
Network design: Two CNN networks predict complementary structure outputs: max-pooling selects template coefficients, while a no-pooling network preserves position sensitivity for active shape parameters.The template coefficients and active shape parameters are fused to generate the parsing result.
Contributions: ATR provides an end-to-end human-parsing approach that avoids explicitly designing feature representations, model topology, or contextual interactions among labels.The framework directly predicts label masks and morphs them into parsing results with active shape parameters.
Contributions: The framework uses learned template combinations and active shape regression to produce semantically meaningful label masks and achieve strong comparative performance.The authors report comprehensive evaluation on a large dataset and emphasize the framework’s superiority over prior state-of-the-art methods.

2 RELATED WORK

Earlier human-parsing systems commonly decompose images into local units and apply hand-designed or learned classifiers. The reviewed work instead emphasizes an end-to-end mapping from the image to structured parsing outputs.

Background: Human parsing supports fashion-related and human-centric applications but remains incompletely solved.Prior methods are described as following either hand-designed or deep-learning pipelines.
Hand-designed pipelines: Traditional pipelines use carefully tuned processing steps built from low-level over-segmentation and pose estimation.Composite And-Or graph templates are used to model and parse clothing configurations.
Deep-learning pipelines: Deep CNN methods have shown potential for vision tasks, but earlier parsing-related approaches commonly classify pixels, super-pixels, or region hypotheses locally or semi-locally.Some approaches also require complex post-processing for accurate prediction.
Contrast with prior work: The ATR approach contrasts with these pipelines by directly relating the input image to label masks and corresponding shape parameters.This is presented as a simpler and more efficient use of CNNs for human parsing.

3 ACTIVE TEMPLATE REGRESSION

Active Template Regression represents each semantic human region with learned mask templates and active shape parameters, then combines these structure outputs into refined parsing maps. Separate CNNs predict template coefficients and spatial shape parameters because these tasks require different invariances and sensitivities.

3 ACTIVE TEMPLATE REGRESSION: ATR represents each of K semantic label masks as a linear combination of its learned template dictionary and predicts the associated template coefficients.The template dictionaries encode common shape patterns among regions of the same label.
3 ACTIVE TEMPLATE REGRESSION: For each image, the predicted coefficients reconstruct normalized masks, which are morphed by predicted shape parameters and fused into foreground and background confidence maps.The input human bounding box is enlarged by factor 1.2 before reconstruction and morphing.
3 ACTIVE TEMPLATE REGRESSION: Active shape parameters encode each label mask’s position, scale, and visibility, enabling normalized masks to be morphed into precise image coordinates.Visibility determines whether the associated label mask is considered.
3 ACTIVE TEMPLATE REGRESSION: Two separate neural networks predict template coefficients and active shape parameters because template selection resembles classification whereas precise localization resembles detection.The active template network predicts coefficients, while the active shape network predicts spatial parameters.
3.3 Structure Output Combination and Superpixel Smoothing: Foreground and background confidence maps are refined with super-pixel smoothing to produce the final pixel-level parsing result.The combination stage derives background confidence from foreground and background seeds before smoothing.
3.1 Active Template Network: The active template network uses a pooled CNN to regress 850 = 17 × 50 normalized template-coefficient outputs with an ℓ2 objective.Training constructs per-label dictionaries from resized cropped binary masks and learns nonnegative template coefficients using NMF with ℓ2 regularization.

4 EXPERIMENTS

Experiments evaluate ATR on a large combined dataset, compare it with state-of-the-art methods, and analyze its template, shape, and network components. ATR achieves substantially stronger parsing performance, while ablations support separate networks and position-sensitive shape prediction.

Experimental Settings: The combined dataset contains 7,700 images from three public datasets and 1,833 challenging Human Parsing in the Wild images, with 18 merged categories.The public datasets contribute 685, 2,682, and 2,500 images, respectively.
Results and Comparisons: 64.38% average F1-score is achieved by ATR, compared with 44.76% for the state-of-the-art baseline.On 1,000 test images, ATR also reaches 71.04% foreground accuracy, 71.69% precision, and 60.25% recall.
Results and Comparisons: ATR exceeds Yamaguchi et al. and PaperDoll by over 26.43% and 22.5% average F1-score, respectively, on the 229-image Fashionista test set.The comparison uses versions trained with 6,000 images.
Results and Comparisons: ATR improves small-label parsing, reaching 57.07% F1-score for scarf and 53.66% for bag versus substantially lower baseline scores.Qualitative results also show successful detection of small labels such as sunglasses and hats.
Ablation Studies of Our Networks: 98.67% accuracy and 95.45% average precision are obtained by the template-reconstruction upper bound, indicating few reconstruction errors from learned template coefficients.The upper bound uses ground-truth template coefficients and fixed active shape parameters.
Ablation Studies of Our Networks: 91.01% accuracy and 62.78% average F1-score are achieved by the no-pooling shape network, exceeding the lessfcfilters variant and supporting position-sensitive feature preservation.Bounding-box refinement further improves the no-pooling network's average F1-score by 1.6%.

5 CONCLUSIONS

ATR formulates human parsing as regression of mask templates and active shape parameters, using two CNNs to generate structure outputs. Experiments compare ATR with and without super-pixel smoothing, while future work targets richer low-level and fine-grained information.

ATR uses two separate CNNs to predict mask template coefficients and active shape parameters for human parsing.The template network uses max-pooling, while the shape network omits max-pooling to preserve position sensitivity.
The reported visual comparison includes PaperDoll, ATR without super-pixel smoothing, and the full ATR method.The methods are shown sequentially for each image.
The template dictionaries contain 21 NMF-learned templates for each of eight semantic labels.Brighter pixels indicate parts most important for distinguishing label masks.
Future work will explore low-level information, integrate fine-grained semantic-label attributes, and extend the framework to generic image parsing.The authors specifically mention edges, super-pixels, scene parsing, and human pose estimation.