PaperlayerView exact revision on arXiv

Source-linked AI summary

Fashionpedia: Ontology, Segmentation, and an Attribute Localization Dataset

Menglin Jia, Mengyun Shi, Mikhail Sirotenko, Yin Cui, Claire Cardie, Bharath Hariharan, Hartwig Adam, Serge Belongie

arXiv:2004.12276v2cs.CVcs.LGeess.IV

TL;DR

Instance segmentation with attribute localization combines object segmentation and fine-grained attribute recognition, a task the paper develops for fashion. The authors introduce an ontology and annotated dataset, plus Attribute-Mask R-CNN; adding an attribute constraint reduces reported box AP from 48.7 to 35.7 for SpineNet-143.

  • Problem

    The paper proposes a task unifying instance segmentation and fine-grained attribute recognition to represent fashion objects with both boundaries and properties.

    Source: §1 · p. 2

  • Method

    The authors introduce a fashion ontology and dataset and present Attribute-Mask R-CNN for the joint task.

    Source: §6 · p. 13

  • Results

    Box AP for SpineNet-143 drops from 48.7 to 35.7 when τF1 is added as a true-positive constraint.

    Source: §5.3 · p. 11

  • Takeaways & Limitations

    The authors expect Fashionpedia models to support applications such as product recommendation, visual search, and resolving ambiguous fashion-related text queries.

    Source: §6 · p. 13

    The reported results leave room for more advanced models on instance segmentation with attribute localization.

    Source: §5.3 · p. 12

Abstract

from arXiv · show

In this work we explore the task of instance segmentation with attribute localization, which unifies instance segmentation (detect and segment each object instance) and fine-grained visual attribute categorization (recognize one or multiple attributes). The proposed task requires both localizing an object and describing its properties. To illustrate the various aspects of this task, we focus on the domain of fashion and introduce Fashionpedia as a step toward mapping out the visual aspects of the fashion world. Fashionpedia consists of two parts: (1) an ontology built by fashion experts containing 27 main apparel categories, 19 apparel parts, 294 fine-grained attributes and their relationships; (2) a dataset with everyday and celebrity event fashion images annotated with segmentation masks and their associated per-mask fine-grained attributes, built upon the Fashionpedia ontology. In order to solve this challenging task, we propose a novel Attribute-Mask RCNN model to jointly perform instance segmentation and localized attribute recognition, and provide a novel evaluation metric for the task. We also demonstrate instance segmentation models pre-trained on Fashionpedia achieve better transfer learning performance on other fashion datasets than ImageNet pre-training. Fashionpedia is available at: https://fashionpedia.github.io/home/index.html.

1 Introduction

The paper frames instance segmentation with attribute localization as a way to combine object boundaries with fine-grained descriptions. It introduces a fashion ontology and annotated dataset, alongside a model and evaluation metric for the task.

1 Introduction

  • Instance segmentation with attribute localization unifies object localization and fine-grained attribute recognition in a structured image representation.The authors focus on fashion as an example and anticipate applicability to other man-made product domains.Source: §1 · pp. 2–3
  • The Fashionpedia ontology organizes 27 main apparel categories, 19 apparel parts, and 294 fine-grained attributes.It is built by fashion experts and informed by product descriptions from the internet.Source: §1 · p. 3
  • Fashionpedia contains 48,825 fashion images annotated with segmentation masks and localized fine-grained attributes.Crowd workers annotated masks, while fashion experts annotated localized attributes.Source: §1 · p. 3
  • Attribute-Mask R-CNN jointly performs instance segmentation and localized attribute recognition, and the paper provides an evaluation metric for the task.Source: §1 · p. 3

2 Related Work

Prior work covers object segmentation and fine-grained recognition, but fashion datasets vary in their annotations and taxonomy. Fashionpedia emphasizes garment-part masks, expert-localized attributes, and an expert-built fine-grained ontology.

2 Related Work

  • Fashionpedia adds masks for 19 garment parts alongside the 27 main garment and accessory categories annotated in prior fashion segmentation datasets.Smaller objects such as collars, pockets, sleeves, zippers, and embroidery are included.Source: §2 · p. 5
  • Fashion experts manually annotate attributes linked to crowd-worker-labeled segmentation masks, whereas prior datasets often derive attributes from potentially noisy product descriptions.Source: §2 · p. 5
  • Fashionpedia’s expert-verified ontology addresses repeated labels, limited categorization depth, and diverse taxonomy structures in prior fashion attribute work.The ontology draws on fashion websites, luxury brands, trend-forecasting companies, and academic resources.Source: §2 · p. 5

3 Dataset Specification and Collection

Fashionpedia’s ontology structures apparel categories, parts, attributes, and their relationships. Its collection and annotation pipeline combines crowd-worker segmentation masks with expert-assigned attributes.

3.1 Ontology specification

  • Fashionpedia represents apparel objects through a structured vocabulary of basic-level categories and fine-grained attributes.Its objects correspond to common apparel items such as jackets, shirts, and dresses.Source: §3.1 · p. 5
  • Images annotate main garments and their parts with tight segmentation masks, averaging 3 main garments, 3 accessories, and 12 garment parts per image.Each object receives a Fashionpedia synset ID.Source: §3.1 · p. 6
  • Main garments and garment parts can carry canonicalized attributes; images average 16.7 attributes, with a maximum of 57.The ontology provides attributes for 13 main outerwear categories and five garment-part types.Source: §3.1 · p. 6
  • Ontology relationships encode part-of links, category-to-attribute relations, and up to four levels of is-an-instance-of hierarchy.Examples include garment-to-part links and fabric types organized from specific to general.Source: §3.1 · pp. 6–7

3.2 Image Collection and Annotation Pipeline

  • The dataset was built from 50,527 collected images, filtered to 48,825 after expert review and duplicate removal.Source: §3.2 · p. 7
  • Annotation was divided between trained crowd workers, who drew segmentation masks, and fashion experts, who labeled attributes per mask.The 28 crowd workers trained for 10 days; 15 experts annotated attributes and another expert checked annotations.Source: §3.2 · p. 7
  • The dataset is split into 45,623 training, 1,158 validation, and 2,044 test images.Source: §3.2 · p. 7

4 Dataset Analysis

The analysis compares Fashionpedia with fashion and general-domain segmentation datasets, examining image resolution, masks, categories, and attributes. It finds diverse, complex masks and a broad but long-tailed annotation set, with some attribute uncertainty linked to occlusion or viewpoint.

4.1 Image Analysis

  • Fashionpedia training images average 1710 × 2151 pixels; the analysis compares the dataset with DeepFashion2, ModaNet, COCO, and LVIS.The paper says high resolution supports detailed apparel views and more accurate, faster mask and attribute annotation.Source: §4.1 · p. 8, §4 · p. 8

4.2 Mask Analysis

  • Fashionpedia averages 7.3 masks per image (median 7, max 74), with the largest median and widest range among the three fashion datasets.Its mask-count range is comparable to COCO, while COCO and LVIS have wider distributions overall.Source: §4.2 · p. 8, Figure 3 · p. 9
  • Fashionpedia mask sizes resemble COCO and LVIS but include fewer masks larger than 0.95 relative area; DeepFashion2 has more zoomed-in garment views.ModaNet has more masks in the 0.2–0.4 relative-area range, while Fashionpedia includes 19 apparel-part categories whose masks are relatively small.Source: §4.2 · p. 8, Figure 3 · p. 9

4.3 Category and Attributes Analysis

  • The dataset has 46 apparel categories and 294 attributes; each mask averages 3.7 attributes, and the category and attribute distributions are long-tailed.Each image averages 7.3 instances, 5.4 categories, and 16.7 attributes; masks have up to 14 attributes.Source: §4.3 · p. 9

4.2 Mask Analysis

  • Fashionpedia masks have the most complex boundaries among five datasets and the second-highest average vertices per polygon, after LVIS.The authors state that this boundary complexity suggests the masks represent apparel silhouettes more accurately than ModaNet and DeepFashion2.Source: §4.2 · p. 9, Table 2 · p. 10

4.3 Category and Attributes Analysis

  • For each attribute superclass, fewer than 15% of masks were labeled “not on the list,” while uncertainty arose for attributes obscured by occlusion or viewpoint.The cited uncertainty examples are Opening Type, Waistline, and Length.Source: §4.3 · pp. 9–10

5 Evaluation Protocol and Baselines

The evaluation extends COCO-style instance segmentation metrics with an attribute constraint and tests an end-to-end model for joint segmentation and attribute localization. Results show the added constraint makes the task substantially harder, while detector errors differ between Fashionpedia- and COCO-trained models.

5.1 Evaluation metric

  • The main metric averages precision across 10 IoU thresholds, 10 macro F1 thresholds, and all categories, requiring each detected object to satisfy an attribute F1 constraint.The COCO-style APIoU metric averages across 10 IoU thresholds and all 80 categories.Source: §5.1 · p. 10

5.2 Attribute-Mask R-CNN

  • Attribute-Mask R-CNN adds a multi-label attribute prediction head to Mask R-CNN and trains end-to-end with sigmoid cross-entropy for joint segmentation and localized attribute recognition.The architecture is illustrated in Figure 4.Source: §5.2 · p. 11, Figure 4 · p. 11

5.3 Results Discussion

  • Adding the attribute F1 constraint reduces SpineNet-143 box AP from 48.7 to 35.7, indicating that joint segmentation and attribute localization is substantially harder than segmentation alone.The authors also report that per-super-category mask results verify this difficulty.Source: §5.3 · p. 11, Table 3 · p. 12
  • COCO-trained detector errors center on localization (+28.3 AP at τIoU=0.75) and background confusion (+15.7), while Fashionpedia-trained errors span localization (+13.4), classification (+6.9), and background confusion (+6.6).The detector-analysis curves are averaged over categories.Source: §5.3 · p. 12, Figure 5 · p. 13
  • The model produces good results on small objects such as shoes and glasses and correctly predicts fine-grained attributes for some masks.The examples are shown in Figure 6.Source: §5.3 · p. 12
  • A qualitative error assigns the nickname “welt” to a pocket, and the authors identify remaining headroom for more advanced models.The authors refer to supplementary material for further baseline analysis.Source: §5.3 · p. 12

6 Conclusion

The paper introduces a task unifying instance segmentation and attribute recognition, alongside Fashionpedia's ontology and dataset, a model, and an evaluation metric. The authors hope these contributions advance fine-grained image understanding in fashion.

6 Conclusion

  • The paper frames instance segmentation with attribute recognition as a new unified task and introduces the Fashionpedia ontology and dataset to address it.The authors describe Fashionpedia as the first dataset combining part-level segmentation masks with fine-grained attributes.Source: §6 · p. 13
  • The paper presents Attribute-Mask R-CNN and a novel evaluation metric for the task.Source: §6 · p. 13
  • The authors expect models trained on Fashionpedia may support product recommendations, visual search, and resolving ambiguous fashion-related words in text queries.Source: §6 · p. 13

8 Supplementary Material

The supplementary material provides additional detail on Fashionpedia’s task, model, ontology, dataset, and annotation process, along with further evaluation and discussion of potential uses and limitations. It also includes an example Fashionpedia image annotated with 74 masks.

8.1 Attribute-Mask R-CNN

  • Metrics requiring both IoU and attribute-F1 constraints are lower than IoU-only metrics across categories, underscoring the task’s difficulty.Garment-part categories also have lower AP and AR than outerwear and accessories.Source: §8.1 · p. 15, Figure 7 · p. 16
  • Ignoring localization errors raises outerwear AP only from 77.5 to 79.1 (+1.6), while part and accessory errors are dominated by background confusion and localization.The authors suggest smaller sizes and lower counts may contribute to the generally lower AP for garment parts.Source: §8.1 · p. 15
  • Binary-macro is the main F1 method; binary-micro scores exceed 0.97, while multi-label macro scores are 0.01–0.03, exposing room to improve attribute localization.The analysis considers four averaging options: micro, macro, binary-micro, and binary-macro. The baseline also overpredicts attributes and confuses fine-grained labels, including sleeve types and opening types.Source: §8.1 · pp. 15, 18

8.2 Fashionpedia Ontology and Knowledge Graph

  • Apparel graphs represent outfit garments, parts, attributes, and their relationships; their union forms a Fashionpedia knowledge graph with potential uses in recommendations, visual search, and text search.The paper also proposes extending existing knowledge graphs such as WikiData with fashion-specific knowledge.Source: §8.2 · pp. 19, 22

8.3 Dataset Analysis

  • Fashionpedia images span diverse resolutions, and polygon vertex counts have the second-widest range after LVIS; fewer than 6% of masks are labeled “not on the list,” excluding nicknames.The authors plan to release resized and original images to address data-loading burdens from high-resolution images.Source: §8.3 · pp. 22, 24

8.4 Fashionpedia dataset creation details

  • Images were collected from Flickr and stock-photo sites and manually screened to limit photographic bias; 28 crowd workers received 10 days of training on the ontology and mask quality.Two fashion experts monitored annotation, answered questions, checked quality, and gave feedback; uncertain masks received their final judgment.Source: §8.4 · pp. 27–28

8.5 Discussion

  • Instance segmentation distinguishes separate objects that semantic segmentation merges; Fashionpedia localizes attributes to garment instances, unlike CUB-200’s global attributes and bird keypoints.Fashionpedia images and labels were newly collected and annotated; one image contains at most 74 masks, most of them garment-part rivets.Source: §8.5 · pp. 29–30

iMat-Fashion Kaggle challenges

Loading 2004.12276v2…