Source-linked AI summary
Understanding Humans in Crowded Scenes: Deep Nested Adversarial Learning and A New Benchmark for Multi-Human Parsing
Jian Zhao, Jianshu Li, Yu Cheng, Li Zhou, Terence Sim, Shuicheng Yan, Jiashi Feng
TL;DR
Crowded-scene understanding remains limited because existing perceptual tasks do not fully distinguish fine-grained semantics across multiple interacting, occluded people. The paper introduces the MHP v2.0 benchmark and NAN, a nested end-to-end model, and reports superiority over state-of-the-art methods across multiple datasets. The benchmark and baseline are intended to support further multi-human parsing research.
Problem
Existing detection, instance segmentation, and human parsing approaches do not fully address instance-level semantic understanding of multiple people in crowded scenes.
Method
The paper introduces MHP v2.0 and NAN, whose three GAN-like sub-networks perform semantic saliency prediction, instance-agnostic parsing, and instance-aware clustering jointly.
Results
NAN consistently outperforms existing state-of-the-art solutions on MHP v2.0 and several other datasets.
Takeaways & Limitations
MHP v2.0 and NAN provide a benchmark and strong baseline for advancing multi-human parsing toward crowded real-world scenes.
Takeaways & Limitations
Heavy occlusion, extreme poses, intensive interactions, and small semantic categories remain difficult for NAN to identify and segment.
Abstract
from arXiv · showhide
Despite the noticeable progress in perceptual tasks like detection, instance segmentation and human parsing, computers still perform unsatisfactorily on visually understanding humans in crowded scenes, such as group behavior analysis, person re-identification and autonomous driving, etc. To this end, models need to comprehensively perceive the semantic information and the differences between instances in a multi-human image, which is recently defined as the multi-human parsing task. In this paper, we present a new large-scale database "Multi-Human Parsing (MHP)" for algorithm development and evaluation, and advances the state-of-the-art in understanding humans in crowded scenes. MHP contains 25,403 elaborately annotated images with 58 fine-grained semantic category labels, involving 2-26 persons per image and captured in real-world scenes from various viewpoints, poses, occlusion, interactions and background. We further propose a novel deep Nested Adversarial Network (NAN) model for multi-human parsing. NAN consists of three Generative Adversarial Network (GAN)-like sub-nets, respectively performing semantic saliency prediction, instance-agnostic parsing and instance-aware clustering. These sub-nets form a nested structure and are carefully designed to learn jointly in an end-to-end way. NAN consistently outperforms existing state-of-the-art solutions on our MHP and several other datasets, and serves as a strong baseline to drive the future research for multi-human parsing.
1. Introduction
Multi-human parsing addresses the need to distinguish fine-grained body parts and fashion items for each person in crowded scenes. The paper introduces the MHP benchmark and NAN model, with evaluations showing NAN’s superiority over state-of-the-art methods.
- Multi-human parsing identifies body parts and fashion items at the instance level, unlike detection, instance segmentation, or instance-agnostic human parsing.It is motivated by crowded scenes containing multiple people with varied interactions, poses, and occlusion.
- NAN combines semantic saliency prediction, instance-agnostic parsing, and instance-aware clustering in three GAN-like sub-networks.The sub-networks are trained jointly through gradient backpropagation in an end-to-end architecture.
- 25,403 images with pixel-wise annotations for 58 semantic categories form the proposed large-scale benchmark.
- NAN is evaluated on MHP v2.0, MHP v1.0, PASCAL-Person-Part, and Buffy, where comprehensive experiments verify superiority over state-of-the-art solutions.
2. Related Work
Existing human-parsing datasets and methods provide limited support for fine-grained, instance-aware analysis in crowded scenes. The paper positions MHP v2.0 and NAN as broader and more integrated alternatives.
- LIP provides densely pixel-wise annotations but has limited semantic categories and an instance-agnostic setting with few multi-person interactions.These properties restrict fine-grained and realistic crowded-scene understanding.
- MHP v1.0 contains 4,980 images and 18 semantic labels, whereas MHP v2.0 expands to 25,403 images and 58 fine-grained semantic part labels.
- Prior approaches commonly use separate stages for instance localization, human parsing, and result refinement.
- NAN instead produces multi-human parsing results through a single forward pass without tedious pre- or post-processing.
3. Multi-Human Parsing Benchmark
MHP v2.0 is a large-scale benchmark designed for realistic, fine-grained multi-human parsing. It combines diverse crowded-scene imagery, detailed instance annotations, and controlled dataset splits for evaluation.
- MHP v2.0 contains 25,403 images with 58 fine-grained labels covering body parts, clothing, accessories, and background.The images include varied viewpoints, poses, occlusion, interactions, resolution, and complex backgrounds.
- Images are collected from real-world scenarios designed to include diverse human relationships, scenes, viewpoints, poses, and interactions.Underlying relationships and possible scenes guide searches for Creative Commons imagery.
- Professional annotators count foreground persons and assign fine-grained pixel-wise labels separately to each person instance.Multi-scale superpixels accelerate annotation, and instances are annotated in left-to-right order.
- The dataset uses 15,403 training images, 5,000 validation images with public annotations, and 5,000 test images with withheld annotations.
- The benchmark defines 11 body parts and 47 clothing or accessory categories to support detailed instance-level analysis.Left-right subdivisions and confusing or small-scale categories increase task difficulty.
- MHP v2.0 is intended to provide a realistic benchmark and standard evaluation server for fine-grained multi-human parsing.
4. Deep Nested Adversarial Networks
NAN decomposes multi-human parsing into three nested GAN-like sub-nets that jointly predict saliency, semantic parts, and person instances end-to-end. Their adversarially coupled design supports holistic training and single-pass inference without tedious pre- or post-processing.
- Architecture: NAN decomposes multi-human parsing into semantic saliency prediction, instance-agnostic parsing, and instance-aware clustering sub-nets.The three sub-tasks are designed as simpler components of the original task and jointly perform multi-human parsing.
- Semantic Saliency Prediction: The saliency sub-net segments foreground humans from background and provides a basic prior for subsequent processing.It uses an FCN-8s generator for binary pixel-wise labeling, trained with adversarial and semantic saliency losses.
- Instance-Agnostic Parsing: The parsing sub-net combines RGB information with the saliency prior to estimate fine-grained semantic labels for body parts and fashion items.Its prediction is downsampled by 8 to trade accuracy against speed, and it supplies semantic guidance for instance-aware clustering.
- Instance-Aware Clustering: The clustering sub-net combines RGB, saliency, and parsing priors to infer pixel-wise instance locations and the number of persons.It uses sibling branches and multi-scale features, associating semantic masks with individual person instances.
- Training and Inference: Nested adversarial losses provide deep supervision and mutually boost the sub-nets, enabling end-to-end optimization and single-forward-pass inference.Testing obtains parsing, instance-location, and instance-number predictions from NAN before applying an off-the-shelf clustering method.
5. Experiments
The experiments evaluate NAN qualitatively and quantitatively across multi-human parsing, instance-agnostic parsing, and instance-aware clustering settings using several benchmark datasets.
- Datasets and Tasks: NAN is evaluated on MHP v2.0, MHP v1.0, PASCAL-Person-Part, and Buffy under varied settings and granularities.Multi-human parsing is tested on the MHP and PASCAL-Person-Part datasets, while Buffy evaluates instance-agnostic parsing and instance-aware clustering byproducts.
5.1. Experimental Settings
The experimental settings specify network initialization and image dimensions, while evaluation uses human-centric metrics for part-level parsing quality.
- Implementation Settings: RGB inputs, semantic saliency predictions, discriminator inputs, and G2 inputs are fixed at 512×512.Other prediction resolutions are described separately in the implementation settings.
- Implementation Settings: The three discriminators are separately initialized from ImageNet-pretrained VGG-16 networks and adapted with new input and output layers.Each discriminator receives a new convolutional input-adaptation layer and a sigmoid-activated one-way fully connected output.
- Evaluation Metrics: Multi-human parsing is evaluated with Average Precision based on part (APp) and Percentage of Correctly parsed semantic Parts (PCP).APp uses part-level pixel IoU within person instances, while PCP measures the proportion of semantic categories correctly parsed per instance.
- Evaluation Metrics: APp averages IoU across semantic part categories over IoU thresholds from 0.1 to 0.9 in increments of 0.1.PCP complements APp by reflecting how many semantic parts are correctly parsed within each person instance.
5.2. Evaluations on the MHP v2.0 Benchmark
On MHP v2.0, NAN is evaluated through component analysis, benchmark comparisons, and qualitative cases. It outperforms baselines and competing methods, including under intensive human interactions, while remaining time-efficient, though heavy occlusion and small parts remain difficult.
- Benchmark: 25,403 images with 58 fine-grained labels define the MHP v2.0 benchmark, split into training, validation, and withheld testing sets.The benchmark reports APp and PCP on validation and testing data.
- Quantitative comparison: 10.33% and 6.78% APp gains over Mask R-CNN and MH-Parser, respectively, show NAN’s strong validation-set performance.The comparison is on MHP v2.0 validation data.
- Component analysis: 2.16%, 4.66%, and 2.42% drops across all metrics without semantic saliency prediction verify its role as a useful human-region prior.Removing G1 causes the reported performance decreases.
- Quantitative comparison: 7.15%, 5.70%, and 5.27% gains over the second-best method occur on the full MHP v2.0 testing set.NAN also leads on the more challenging Inter20% and Inter10% subsets.
- Efficiency and limitations: NAN processes one 512×512 image in about 1 second, while heavy occlusion, extreme poses, intensive interactions, and small categories remain difficult.The speed result is reported alongside the testing comparison; the failure cases are described separately.
5.3. Evaluations on the MHP v1.0 Benchmark
On MHP v1.0, NAN is compared with three state-of-the-art methods using APp and PCP on the testing set. Nested adversarial learning yields consistent gains over the second-best method, supported by qualitative comparisons.
- Quantitative comparison: 4.41% APp, 6.95% APp vol, and 8.04% PCP0.5 gains over the second-best method are reported on MHP v1.0.The comparison evaluates the testing set using the paper’s reported metrics.
- Method and interpretation: NAN’s nested adversarial learning combines semantic saliency prediction, instance-agnostic parsing, and instance-aware clustering in the MHP v1.0 evaluation.The reported gains are attributed to the nested adversarial learning framework.
- Qualitative comparison: Qualitative results further validate NAN’s advantages over existing solutions on MHP v1.0.The comparison is presented in Figure 8.
5.4. Evaluations on the PASCAL-Person-Part Benchmark
NAN is evaluated on PASCAL-Person-Part, a six-part human parsing benchmark, with quantitative and qualitative comparisons. It substantially exceeds the second-best method on the reported testing-set metric.
- Benchmark: PASCAL-Person-Part contains 3,535 images with pixel-wise labels for six human body parts, split into 1,717 training and 1,818 testing images.The reported parts are head, torso, upper-/lower-arms, and upper-/lower-legs.
- Qualitative comparison: Qualitative NAN outputs show high concordance with corresponding ground truths on PASCAL-Person-Part.The results are visualized in Figure 9.
5.5. Evaluations on the Buffy Benchmark
On the Buffy benchmark, NAN is evaluated for instance segmentation across episodes 4, 5, and 6. It achieves the best reported scores on every metric and improves substantially over the second-best method.
- Benchmark: The Buffy dataset contains 748 images annotated with 12 semantic labels and evaluates instance segmentation using F and B scores.The reported evaluation covers episodes 4, 5, and 6.
- Quantitative comparison: 6.13% for F score and 7.98% for B score improve over the second-best method, with an average boost of 7.05%.NAN consistently achieves the best performance across Buffy episodes 4, 5, and 6.
- Qualitative comparison: Qualitative instance-agnostic parsing and instance-aware clustering results show NAN’s potential for fine-grained understanding of humans in crowded scenes.These results are visualized in Figure 10.
6. Conclusions
The work contributes MHP v2.0, a large-scale benchmark for multi-human parsing, and NAN, a deep model evaluated on this task and related datasets.
- MHP v2.0 contains 25,403 images richly labelled with 59 semantic categories.
- The authors propose NAN to address multi-human parsing in crowded scenes.
- NAN is evaluated against current state-of-the-art methods on MHP v2.0 and several other datasets.
- The authors intend MHP v2.0 and NAN to support research toward real-world scenes containing multiple persons and complex interactions.
- Future work will expand the benchmark with more images and more detailed semantic category annotations.