PaperlayerView exact revision on arXiv

Source-linked AI summary

ModaNet: A Large-Scale Street Fashion Dataset with Polygon Annotations

Shuai Zheng, Fan Yang, M. Hadi Kiapour, Robinson Piramuthu

arXiv:1807.01394v4cs.CV

TL;DR

Single-image fashion understanding is difficult because clothing varies widely and prior fine-grained datasets are limited in scale and pose diversity. ModaNet provides 55,176 fully annotated street-fashion images built on 1 million weakly labeled PaperDoll images, enabling detailed benchmarks for detection, semantic segmentation, and polygon prediction. The paper concludes that this combination of scale and annotation richness supports promising results across these tasks and related applications.

  • Problem

    Single-image fashion understanding remains difficult because clothing varies widely, while existing fine-grained datasets have limited scale and pose diversity.

  • Method

    ModaNet combines 55,176 fully annotated street-fashion images with 1 million weakly labeled PaperDoll images, providing masks, polygons, and bounding boxes.

  • Results

    Experiments benchmark object detection, semantic segmentation, and polygon prediction, showing promising results that are difficult to obtain from small-scale and weakly annotated datasets.

  • Takeaways & Limitations

    ModaNet supports detailed evaluation and training for multiple fashion-understanding tasks and related applications such as color attribute prediction.

Abstract

from arXiv · show

Understanding clothes from a single image has strong commercial and cultural impacts on modern societies. However, this task remains a challenging computer vision problem due to wide variations in the appearance, style, brand and layering of clothing items. We present a new database called ModaNet, a large-scale collection of images based on Paperdoll dataset. Our dataset provides 55,176 street images, fully annotated with polygons on top of the 1 million weakly annotated street images in Paperdoll. ModaNet aims to provide a technical benchmark to fairly evaluate the progress of applying the latest computer vision techniques that rely on large data for fashion understanding. The rich annotation of the dataset allows to measure the performance of state-of-the-art algorithms for object detection, semantic segmentation and polygon prediction on street fashion images in detail. The polygon-based annotation dataset has been released https://github.com/eBay/modanet, we also host the leaderboard at EvalAI: https://evalai.cloudcv.org/featured-challenges/136/overview.

1 INTRODUCTION

Fashion understanding remains difficult because clothing varies widely in type, appearance, style, and layering, while useful localization and segmentation require costly fine-grained annotation. ModaNet addresses this need with a large, diverse, polygon-annotated street-fashion dataset supporting detection, segmentation, and polygon prediction.

  • Motivation: Fashion understanding remains open because clothing has enormous visual variety, making category recognition, localization, and segmentation difficult in single images.These capabilities matter for applications such as online shopping, recommendation, and virtual try-on.
  • Limitations of prior data: Earlier pixel-level fashion datasets contained only thousands of images and often limited pose variation, restricting the scenarios covered by trained models.Fine-grained annotations are expensive because they require substantial human effort and domain knowledge.
  • Contributions: 55,176 fully annotated street-fashion images extend 1 million weakly labeled Paperdoll images with masks and polygons for 13 fashion categories.The annotations also include bounding boxes and are intended for detection and segmentation research.
  • Evaluation: Experiments evaluate state-of-the-art detection and semantic-segmentation networks and also test polygon prediction on ModaNet.The authors use these experiments to analyze the dataset's usefulness across multiple fashion-understanding tasks.

2 RELATED WORK

Prior fashion datasets support retrieval or parsing but generally trade annotation richness against scale, while ModaNet combines large-scale data with fine-grained annotations. The related work situates this dataset alongside detection and segmentation methods used for fashion analysis.

  • 2.1 Datasets: Fashion datasets range from category-specific or retrieval-oriented collections with image-level labels to smaller parsing datasets with pixel-level annotations.Pixel-level datasets are typically limited to hundreds or thousands of images because such annotations are difficult to obtain.
  • 2.1 Datasets: The diversity of fashion items spans type, appearance, composition, and style, motivating datasets that support more detailed analysis.
  • 2.1 Datasets: ModaNet advances prior fashion datasets by combining more than 55,000 fully annotated images with pixel-level segments, polygons, and bounding boxes across 13 categories.The comparison is presented against other fashion-parsing datasets in Table 1.
  • 2.2 Detection and 2.3 Segmentation: Object detection methods localize fashion items and assign category labels, while semantic segmentation assigns object labels to individual pixels.The related work reviews deep neural-network approaches for detection and feature-learning architectures for segmentation.
  • 2.3 Segmentation: Fashion segmentation research includes human parsing, clothing parsing, retrieval-based parsing, parselet models, video context, and convolutional neural networks.

3 THE MODANET DATASET

ModaNet is constructed from PaperDoll street-fashion images through filtering, quality selection, and human polygon annotation. The resulting dataset groups related clothing labels into 13 meta categories and focuses its first release on single-person images.

  • 3.1 Constructing ModaNet: The dataset construction begins with 1 million PaperDoll images relevant to fashion applications such as street-to-shop and shop-the-look.
  • 3.1 Constructing ModaNet: The first release retains images containing a single person, leaving multi-person scenes for future work because occlusion and scale variation are more challenging.
  • 3.1 Constructing ModaNet: A ResNet-50 quality classifier, fine-tuned from ImageNet pretraining using manually selected examples, filters the initially selected images.The manually selected set included 2,000 unsuitable and 2,000 high-quality images.
  • 3.1 Constructing ModaNet: Human annotators skip ambiguous images, draw polygons around individual fashion objects, and assign labels from predefined categories.Seventeen annotators were trained for two weeks, reached 99.5% internal quality accuracy, and were monitored and checked by supervisors.
  • 3.2 Statistics: The dataset groups highly related labels into 13 meta categories, including footwear, top, outer, pants, bags, dress, skirt, headwear, and scarf&tie.Most images contain 3 to 5 fashion objects, with footwear, top, outer, pants, and bags among the most common.

4.1 Object Detection for Fashion

ModaNet supports fashion-item detection with category labels and evaluates Faster RCNN, SSD, and YOLO using accuracy, category-level difficulty, and IoU robustness analyses.

  • The dataset enables fashion-item detection by localizing each item and assigning a category label for applications such as visual search and product recommendation.
  • The benchmark derives bounding boxes from polygon annotations, merges polygons for each fashion item, discards boxes smaller than 100 pixels, and uses 52,377 training images with 2,799 validation images.The validation split ensures at least 500 instances per category.
  • Faster RCNN achieves the best overall accuracy and leads on every category at an IoU threshold of 0.5.The comparison uses precision-recall curves and mAP, with qualitative results shown separately.
  • At IoU 0.5, pants, headwear, and sunglasses achieve mAP above 0.9, whereas scarf&tie is difficult for all detectors.Pants offer varied pose examples, while headwear and sunglasses are generally rigid and less occluded; scarves and ties are deformable and often occluded.
  • mAP and category AP decrease as the IoU threshold rises from 0.5 to 0.9, testing detector robustness under stricter overlap criteria.Pants and headwear remain above 0.8 AP at IoU 0.8.
  • Sunglasses AP falls from the second-best category at IoU 0.5 to the worst at IoU 0.9, indicating poor bounding-box overlap under strict evaluation.The paper identifies small and deformable objects such as sunglasses, belts, and scarf&tie as particularly difficult.

4.2 Semantic Segmentation for Fashion

ModaNet enables semantic segmentation research through pixel-wise annotations, and the evaluated methods show distinct performance and failure patterns across fashion categories.

  • The evaluation measures mean pixel IoU, precision, recall, and F-1 across representative FCN, CRFasRNN, and DeepLabv3+ approaches.
  • Detector performance comparisons in Figure 6 examine Faster RCNN, SSD, and YOLO, while Figure 7 illustrates color attributes inferred from segmented regions.
  • DeepLabV3+ performs significantly better than the alternative semantic-segmentation approaches across all reported metrics.
  • CRFasRNN improves performance for large objects but can over-smooth small objects such as sunglasses, while preserving shapes for color-consistent outerwear and pants.

4.3 Polygon Prediction for Fashion

The paper investigates polygon prediction for individual fashion items using ModaNet’s human-annotated polygons and establishes a Polygon-RNN++ baseline.

  • Polygon annotations are treated as a detailed alternative to pixel-level masks and are useful for recognizing clothing design and detecting fashion-item keypoints.
  • The Polygon-RNN++ pipeline encodes image features, predicts vertices recurrently with visual attention, selects candidates, and refines polygons with a graph-based network.
  • Polygon-RNN++ provides the baseline for predicting fashion-item polygons from cropped images formed using Faster-RCNN detections.
  • Predicted polygons are converted to mask-like predictions and evaluated on the ModaNet validation set against semantic-segmentation masks.

4.4 Color Attribute Prediction Prototype

The prototype combines semantic segmentation with mean-RGB mapping to predict fine-grained color names and fashion categories for segmented regions.

  • The prototype performs semantic segmentation, then maps each segment’s mean RGB values to a fine-grained color-name space.
  • The supplied tables report per-category IoU, precision, recall, and F-1 scores for evaluated semantic-segmentation approaches.
  • Figure 7 presents predicted masks alongside text containing color attribute names and fashion object category names.

5 CONCLUSIONS

ModaNet is introduced as a large-scale street-fashion dataset with polygon, pixel-level, and bounding-box annotations. Experiments cover detection, semantic segmentation, polygon prediction, and color attribute prediction.

  • ModaNet provides street-fashion photos with varied poses, lighting, occlusion, and deformation, together with polygons, pixel-level masks, and bounding boxes.
  • The dataset supports experiments in object detection, semantic segmentation, and polygon prediction, and also trains models for color attribute prediction.
  • A failure case shows DeeplabV3+ missing ties and confusing boots with shoes.
Loading 1807.01394v4…