Source-linked AI summary
DeepFashion2: A Versatile Benchmark for Detection, Pose Estimation, Segmentation and Re-Identification of Clothing Images
Yuying Ge, Ruimao Zhang, Lingyun Wu, Xiaogang Wang, Xiaoou Tang, Ping Luo
TL;DR
Existing fashion benchmarks leave gaps through sparse annotations, single-item images, and missing pixel masks. DeepFashion2 addresses these gaps with a large, richly annotated multi-task benchmark and Match R-CNN, while evaluations expose consistent difficulty under challenging visual conditions.
Problem
Existing fashion benchmarks such as DeepFashion are limited by single-item images, sparse landmarks, and absent per-pixel masks, while real images involve deformation, occlusion, and cross-domain differences.
Method
The paper constructs DeepFashion2 with dense clothing annotations and multiple understanding tasks, and proposes Match R-CNN built on Mask R-CNN for end-to-end retrieval.
Results
Evaluations report that detection, landmark estimation, and segmentation perform better on moderate-scale, lightly occluded, non-zoomed, frontal items than on challenging subsets.
Takeaways & Limitations
DeepFashion2 provides a unified benchmark for evaluating fashion understanding across tasks and variations including scale, occlusion, zoom-in, and viewpoint.
Takeaways & Limitations
The authors identify future scope in harder synthesis tasks, multi-domain learning, and additional evaluation metrics such as size, runtime, and memory consumption.
Abstract
from arXiv · showhide
Understanding fashion images has been advanced by benchmarks with rich annotations such as DeepFashion, whose labels include clothing categories, landmarks, and consumer-commercial image pairs. However, DeepFashion has nonnegligible issues such as single clothing-item per image, sparse landmarks (4~8 only), and no per-pixel masks, making it had significant gap from real-world scenarios. We fill in the gap by presenting DeepFashion2 to address these issues. It is a versatile benchmark of four tasks including clothes detection, pose estimation, segmentation, and retrieval. It has 801K clothing items where each item has rich annotations such as style, scale, viewpoint, occlusion, bounding box, dense landmarks and masks. There are also 873K Commercial-Consumer clothes pairs. A strong baseline is proposed, called Match R-CNN, which builds upon Mask R-CNN to solve the above four tasks in an end-to-end manner. Extensive evaluations are conducted with different criterions in DeepFashion2.
1. Introduction
DeepFashion2 addresses gaps between existing fashion benchmarks and real-world image understanding by combining richer annotations, multiple tasks, and consumer-commercial image pairs in a large-scale benchmark.
- Fashion image understanding remains challenging because clothing undergoes large deformations and occlusions, while consumer and commercial images differ across domains.
- DeepFashion is limited to one clothing item per image, sparse category-shared landmarks, and no per-pixel masks.Its bounding boxes are inferred from labeled landmarks, making them noisy.
- DeepFashion2 contains 491K images spanning detection, recognition, landmark and pose estimation, segmentation, verification, and retrieval.The benchmark provides comprehensive tasks and annotations for fashion image understanding.
- 801K clothing items are labeled with scale, occlusion, zooming, viewpoint, bounding boxes, dense landmarks, and per-pixel masks, and form 873K consumer-commercial pairs.The items are grouped into 43.8K clothing identities; the pairs are reported as 3.5 times larger than DeepFashion.
- The benchmark contributes clothing pose definitions for 13 categories and Match R-CNN, which aggregates category, pose, and mask features for end-to-end retrieval.The authors also extensively evaluate Mask R-CNN on DeepFashion2.
- DeepFashion2 is positioned as a unified benchmark and framework covering detection, landmark prediction, segmentation, and retrieval across scale, occlusion, zoom-in, and viewpoint variations.
2. DeepFashion2 Dataset and Benchmark
DeepFashion2 is a large, versatile benchmark designed for real-world fashion understanding, combining multiple tasks with dense annotations, diverse conditions, and cross-domain clothing identities.
- Versatility: The benchmark supports clothes detection and classification, dense landmark and pose estimation, instance segmentation, and cross-domain instance-level retrieval.Four benchmarks use shared images and labels, with training, validation, and test splits.
- Expressivity: DeepFashion2 allows multiple clothing items per image and defines category-specific poses with 23 landmarks per category on average.Contours and skeletons capture clothing shapes and structures rather than human pose.
- Diversity: Items vary across scale, occlusion, zoom-in, and viewpoint, with each property assigned three difficulty levels.More than 50% of items have medium or heavy occlusion, and more than 30% are zoomed in.
- Cross-Domain Data: The dataset pairs consumer and commercial images of the same clothing identities while allowing style differences such as color and printing.These pairs support retrieval across the two image domains.
- Data Labeling: Per-item masks are generated from contours and refined by human annotators when complex poses make automatic masks inaccurate.The two-stage process combines automatic generation with human correction.
3. Match R-CNN
Match R-CNN is an end-to-end framework built on Mask R-CNN that jointly supports clothing perception and consumer-to-shop retrieval. It uses feature extraction, perception, and matching components with task-specific losses and streams.
- Overview: Match R-CNN jointly learns clothes detection, landmark estimation, instance segmentation, and consumer-to-shop retrieval.The framework stacks a Siamese module over task streams to aggregate learned features.
- Feature extraction: Its feature network uses a ResNet-FPN backbone, region proposal network, and RoIAlign to extract region features.RoIAlign extracts features from different levels of the feature pyramid.
- Perception network: The perception network separates landmark estimation, clothes detection, and mask prediction into three streams.These streams process extracted RoI features for landmarks, classification and box regression, and segmentation masks.
- Matching network: The matching network transforms RoI features into retrieval vectors and computes similarity scores between clothing items from two images.Its similarity learning network receives feature vectors from the two input images.
- Optimization: The model minimizes a weighted combination of classification, box, pose, mask, and pair losses.The pair loss uses binary cross-entropy for matched versus unmatched clothing items.
- Inference: At inference, proposals are classified and regressed, filtered with non-maximum suppression, and passed to landmark, mask, and retrieval stages.For retrieval, the highest-confidence unique detected clothing item in a consumer image is selected as the query.
4. Experiments
Experiments evaluate Mask R-CNN and Match R-CNN across detection, landmark estimation, segmentation, and retrieval under varied clothing conditions. Results show substantial difficulty from scale, occlusion, zoom-in, and viewpoint, with retrieval additionally sensitive to box quality and feature choice.
- Experimental setup: Evaluations cover clothes detection, landmark estimation, instance segmentation, and consumer-to-shop retrieval across difficulty subsets.The validation set varies scale, occlusion, zoom-in, and viewpoint.
- Clothes detection: Detection performance is highest for moderate-scale, slightly occluded, unzoomed, frontal clothing and declines for extreme scale, occlusion, deformation, and non-frontal views.Some items occupy less than 2% or more than 90% of the image, creating difficult detection cases.
- Landmark and pose estimation: Occluded-landmark evaluation generally performs worse than visible-landmark evaluation, while Mask R-CNN achieves an overall AP of 0.563.Landmark estimation follows detection trends, with heavy occlusion and zoom-in substantially reducing performance.
- Clothes segmentation: Segmentation performance declines for small and large items, heavy occlusion, large zoom-in, and side or back viewpoints.These conditions produce corresponding failure cases in the segmentation results.
- Consumer-to-shop retrieval: Match R-CNN achieves top-20 retrieval accuracy below 0.7 even with ground-truth boxes, and detected boxes reduce accuracy further.The comparison uses ground-truth versus predicted detection boxes.
- Consumer-to-shop retrieval: Combining learned features generally improves retrieval accuracy, with pose-and-class features outperforming other combinations.Pose features outperform mask features, suggesting landmark locations are more robust across scenarios.
- Consumer-to-shop retrieval: Retrieval is weakest for small, heavily occluded, highly zoomed-in, and side or back-view clothing items.Large zoom-in can remove distinguishing features, while non-frontal views may hide front patterns.
5. Conclusions
DeepFashion2 is presented as a large-scale fashion benchmark with comprehensive annotations and multiple fashion-understanding tasks, supported by an end-to-end Match R-CNN framework. Its rich data and labels are intended to support future algorithm development, while future work includes broader tasks, multi-domain learning, and additional real-world evaluation metrics.
- DeepFashion2 provides richly labeled fashion images and benchmarks for detection, pose estimation, segmentation, verification, and retrieval.Annotations include style, scale, occlusion, zooming, viewpoint, bounding boxes, dense landmarks and pose, pixel-level masks, and consumer-commercial image pairs.
- Match R-CNN builds upon Mask R-CNN to solve the benchmark’s multiple tasks end to end.
- The rich data and labels are intended to facilitate future development of algorithms for understanding fashion images.
- Future directions include synthesizing clothing images, multi-domain learning, and adding size, runtime, and memory-consumption metrics.