Source-linked AI summary
Matching-CNN Meets KNN: Quasi-Parametric Human Parsing
Si Liu, Xiaodan Liang, Luoqi Liu, Xiaohui Shen, Jianchao Yang, Changsheng Xu, Liang Lin, Xiaochun Cao, Shuicheng Yan
TL;DR
Human parsing needs both supervision from annotated data and flexibility for newly annotated images and labels, while existing pipelines or CNNs have complementary limitations. The paper proposes a quasi-parametric framework using M-CNN to match and transfer KNN semantic regions, and reports effectiveness on a dataset of 7,700 annotated human images.
Problem
Existing human parsing pipelines can be bottlenecked by sequential components, while deep models cannot be easily updated with new semantic labels.
Method
The quasi-parametric framework retrieves KNN images and uses an M-CNN with multi-layer cross-image filters to predict region matching confidence and displacements for label transfer.
Results
The framework demonstrates effectiveness on a large dataset with 7,700 annotated human images.
Takeaways & Limitations
The model combines supervision from annotated data with flexible use of newly annotated images and labels in an end-to-end human parsing framework.
Abstract
from arXiv · showhide
Both parametric and non-parametric approaches have demonstrated encouraging performances in the human parsing task, namely segmenting a human image into several semantic regions (e.g., hat, bag, left arm, face). In this work, we aim to develop a new solution with the advantages of both methodologies, namely supervision from annotated data and the flexibility to use newly annotated (possibly uncommon) images, and present a quasi-parametric human parsing model. Under the classic K Nearest Neighbor (KNN)-based nonparametric framework, the parametric Matching Convolutional Neural Network (M-CNN) is proposed to predict the matching confidence and displacements of the best matched region in the testing image for a particular semantic region in one KNN image. Given a testing image, we first retrieve its KNN images from the annotated/manually-parsed human image corpus. Then each semantic region in each KNN image is matched with confidence to the testing image using M-CNN, and the matched regions from all KNN images are further fused, followed by a superpixel smoothing procedure to obtain the ultimate human parsing result. The M-CNN differs from the classic CNN in that the tailored cross image matching filters are introduced to characterize the matching between the testing image and the semantic region of a KNN image. The cross image matching filters are defined at different convolutional layers, each aiming to capture a particular range of displacements. Comprehensive evaluations over a large dataset with 7,700 annotated human images well demonstrate the significant performance gain from the quasi-parametric model over the state-of-the-arts, for the human parsing task.
1. Introduction
The paper proposes a quasi-parametric human parsing framework that combines annotated-data supervision with flexibility for newly annotated images and labels. Its M-CNN matches KNN semantic regions to testing images using multi-ranged cross-image filters, enabling end-to-end label transfer and fusion.
- Human parsing partitions people into semantic regions, supporting applications including clothing classification and retrieval.
- Transferred label maps from matched KNN regions are combined into label probability maps and refined by superpixel smoothing.
- Existing pipelines combine sequential handcrafted components, whose weaknesses can bottleneck performance, while end-to-end CNNs are difficult to update with new semantic labels.
- The quasi-parametric framework combines annotated-data supervision with flexible extension to newly annotated images and labels.
- M-CNN matches each KNN semantic region to the testing image by predicting matching confidence and displacements, using cross-image filters at multiple convolutional layers.
- The framework integrates traditional pipeline components into one unified end-to-end deep CNN framework and is evaluated on 7,700 annotated human images.
2. Related Work
Prior human parsing research includes parametric, non-parametric, and deep learning methods. These approaches model structure, transfer labels, or learn segmentation with CNNs, but the paper positions its quasi-parametric model as integrating their advantages.
- Related work reviews parametric human parsing, non-parametric methods, and deep learning-based methods.
- Parametric methods: Parametric methods learn region appearance and label structure from annotations but may rely on manually designed models that fit specific data suboptimally.
- Non-parametric methods: Non-parametric methods transfer information using pixels, superpixels, object proposals, or dense image correspondences.
- Non-parametric methods: Superpixel-based label transfer can lack semantic meaning, while correspondence optimization can introduce noise and outliers during transfer.
- Deep learning methods: The proposed quasi-parametric model integrates parametric and non-parametric advantages through M-CNN.
3. Quasi-parametric Human Parsing
The quasi-parametric parsing pipeline retrieves annotated KNN images, uses M-CNN to estimate region matching confidence and displacement, and fuses transferred label maps into the final parsing result. Its cross-image filters operate across convolutional layers to handle semantic regions with different spatial displacements.
- Pipeline: The pipeline retrieves KNN images, forms label-specific regions, predicts matching confidence and displacement with M-CNN, and post-processes their label maps.KNN images are retrieved from the annotated corpus; each semantic region is paired with the input image before label-map generation and final post-processing.
- Matching outputs: M-CNN estimates whether a KNN region matches the input and predicts the matched-region displacement using confidence and coordinate-regression outputs.The confidence indicates match quality, while the four displacement values encode differences between source and matched-region coordinates; training minimizes an ℓ2 loss.
- M-CNN architecture: The architecture combines two single-image convolutional paths with a cross-image path, then fuses their feature maps to estimate matching confidence and displacement.The single-image paths represent the input and KNN region hierarchically, while the cross-image path models their displacement relationship.
- Multi-range matching: Cross-image matching filters are embedded at different convolutional layers so increasing receptive fields capture near-, middle-, and far-range displacements.This multi-range design addresses cases where corresponding regions, such as bags, appear at substantially different locations.
- Parsing inference: For each label, matching confidences across eligible KNN regions are averaged and thresholded to determine visibility before matched label maps are transferred.Only labels predicted as visible receive transferred maps; invisible labels do not generate a transferred label map.
4. Experiments
Experiments on 7,700 annotated images compare M-CNN with state-of-the-art methods and architectural variants. M-CNN achieves stronger overall and small-region parsing, while multiple cross-image filters and nine KNN regions provide the best reported settings.
- 7,700 images are divided into 6,000 training, 700 validation, and 1,000 testing images, with accuracy, average precision, average recall, and average F1 evaluated.
- M-CNN obtains 73.98% foreground accuracy, 64.56% precision, and 65.17% recall, exceeding both reported baselines on each metric.
- For small regions, M-CNN achieves F1 scores of 43.38% for scarf, 57.87% for bag, and 38.45% for belt, above both baselines.
- Adding cross-image filters through conv5, conv4, conv3, and conv2 raises F1 from 56.99% to 62.81%, whereas adding one at conv1 lowers it to 61.53%.The reported interpretation is that deeper receptive fields better cover semantic labels, while conv1 covers only part of a label.
- Nine KNN regions yield the highest F1 score of 63.58%, while one region remains competitive at 56.92%.
5. Conclusion and Future Work
The paper concludes that its quasi-parametric framework combines supervision from annotated data with extensibility to newly annotated images and labels. It attributes this capability to M-CNN and reports significant gains over state-of-the-art methods.
- Conclusion: The quasi-parametric framework combines parametric supervision with non-parametric flexibility for human parsing.It uses annotated data while remaining extensible to newly annotated images and labels.
- Conclusion: M-CNN uses two single-image convolutional paths and a cross-image path with matching filters to characterize multi-ranged matching.The cross-image filters are embedded into convolutional layers.
- Conclusion: Extensive experiments demonstrate significant performance gains over state-of-the-art human parsing methods.
- Future Work: The authors plan to extend the framework to exemplar-based tasks such as face parsing and explore more powerful network structures.GoogLeNet is given as an example of a future network structure.