Source-linked AI summary
Clothing Co-Parsing by Joint Image Segmentation and Labeling
Wei Yang, Ping Luo, Liang Lin
TL;DR
The paper addresses joint clothing-image segmentation and semantic labeling when pixelwise annotations are costly but image-level tags are available. It proposes a two-phase co-parsing framework and reports effective, applicable performance against state-of-the-art methods, alongside a new annotated street-fashion dataset.
Problem
Pixelwise clothing labels are costly and inefficient to obtain, motivating joint segmentation of clothing images and transfer of image-level tags to clothing regions.
Method
The system performs image co-segmentation followed by region co-labeling over clothing-image batches, using spatial locations, mutual item relations, and multi-image graphical modeling.
Results
The framework achieves effective and applicable clothing parsing performance compared with state-of-the-art methods on the Fashionista and CCP datasets.
Takeaways & Limitations
The paper contributes a high-resolution street-fashion photo dataset with annotations for evaluating clothing co-parsing under realistic challenges such as disordered backgrounds and varied poses.
Takeaways & Limitations
The framework may produce incorrect results for ambiguous garment patterns, similar item appearances, extremely disordered backgrounds, or poor illumination.
Abstract
from arXiv · showhide
This paper aims at developing an integrated system of clothing co-parsing, in order to jointly parse a set of clothing images (unsegmented but annotated with tags) into semantic configurations. We propose a data-driven framework consisting of two phases of inference. The first phase, referred as "image co-segmentation", iterates to extract consistent regions on images and jointly refines the regions over all images by employing the exemplar-SVM (E-SVM) technique [23]. In the second phase (i.e. "region co-labeling"), we construct a multi-image graphical model by taking the segmented regions as vertices, and incorporate several contexts of clothing configuration (e.g., item location and mutual interactions). The joint label assignment can be solved using the efficient Graph Cuts algorithm. In addition to evaluate our framework on the Fashionista dataset [30], we construct a dataset called CCP consisting of 2098 high-resolution street fashion photos to demonstrate the performance of our system. We achieve 90.29% / 88.23% segmentation accuracy and 65.52% / 63.89% recognition rate on the Fashionista and the CCP datasets, respectively, which are superior compared with state-of-the-art methods.
1. Introduction
The paper addresses joint clothing-image segmentation and semantic labeling when pixelwise annotation is costly, using a two-phase data-driven co-parsing system.
- Motivation: Pixelwise clothing annotation is costly and inefficient, motivating joint segmentation and transfer of image-level tags to clothing regions.
- Contributions: The paper contributes an applicable batch-parsing system and a ground-truthed database designed for disordered backgrounds and varied human poses.
- Challenges: Clothing parsing is difficult because garment appearances, styles, textures, poses, self-occlusions, and fine-grained categories vary substantially.
- Approach: The system performs image co-segmentation followed by region co-labeling, while exploiting spatial locations and mutual relations among clothing items.
- Approach: Co-segmentation iteratively groups superpixels, selects confident foreground regions for E-SVM training, and propagates segmentations across images.
- Approach: Region co-labeling links adjacent and cross-image similar regions in a graphical model, borrowing statistical strength for joint label assignment.
2. Probabilistic Formulation
The probabilistic formulation jointly optimizes clothing regions, labels, E-SVM weights, and segmentation propagations, with separate factors for co-segmentation and co-labeling.
- Variables: Each image is represented by superpixels grouped into coherent regions, garment labels, selected-region E-SVM weights, and segmentation propagations.
- Posterior optimization: The model seeks the maximum-posterior joint assignment of labels, regions, E-SVM weights, and propagations given the image set.
- Posterior optimization: The posterior factorization separates region inference, label assignment, and propagation-related terms, and optimization proceeds through two phases.
- Co-segmentation: Co-segmentation maximizes P(R|C, I) by grouping superpixels into regions using unary spatial-distance potentials and pairwise smoothness potentials.
- Co-segmentation: Selected coherent regions train an ensemble of E-SVMs, with φ(rj) indicating whether region rj is chosen and E(wk, rk) giving the classifier energy.
- Co-segmentation: Segmentation propagation uses top-k E-SVM detections obtained through a sliding-window scheme.
- Co-labeling: Co-labeling assigns garment tags through a multi-image graphical model with singleton, interior-affinity, and exterior-affinity potentials.
3. Clothing Co-Parsing
Clothing co-parsing alternates region discovery and propagation, then jointly assigns garment labels in a contextualized multi-image graphical model. The two phases use E-SVM-guided segmentation and Graph Cuts-based labeling with appearance, location, and garment-interaction cues.
- Image Co-Segmentation: Binary edge variables allow neighboring superpixels to be grouped without specifying the number of regions in advance.Additional mask variables encourage superpixels covered by a propagated segmentation to form one region.
- Image Co-Segmentation: Each selected region trains an E-SVM using the region as a positive exemplar and patches outside it as negatives.The implementation uses HOG features, and segmentation responses are calibrated with a logistic distribution before propagation.
- Contextualized Co-Labeling: The co-labeling graph connects adjacent regions within images and regions across images matched by propagated segmentation.Interior and exterior edges encode within-image adjacency and cross-image correspondence, respectively.
- Image Co-Segmentation: The system alternates superpixel grouping, E-SVM training, and cross-image segmentation propagation until the regions stabilize.Grouping is guided by propagated segmentations, while E-SVMs are trained for selected regions and then used to propagate segmentations.
- Contextualized Co-Labeling: Graph Cuts optimizes joint garment labels using appearance, item location, appearance compatibility, and mutual-interaction potentials.The location model uses a garment-specific 2-D Gaussian, while interaction costs reflect how frequently garment pairs appear as neighbors in training data.
4. Experiments
Experiments evaluate the framework on Fashionista and the newly created CCP dataset using quantitative, component-level, qualitative, and efficiency analyses. The method outperforms the compared systems, handles challenging examples, and has reported practical runtime characteristics, while failures occur under several difficult visual conditions.
- Datasets: The evaluation uses Fashionista and CCP, a new dataset of 2,098 high-resolution fashion photos with varied styles, accessories, garments, and poses.More than 1,000 CCP images have superpixel-level annotations with 57 tags, while the remainder have image-level tags.
- Quantitative Evaluation: The experiments compare the full system with PECS, BSC, and STF using 10-fold cross validation, average Pixel Accuracy, and mean Average Garment Recall.An all-background assignment serves as the baseline because background is the most frequent label.
- Quantitative Evaluation: The full method outperforms PECS, BSC, and STF on both datasets, with a larger advantage on CCP's more complex backgrounds and clothing styles.The authors attribute this to iterative co-segmentation suppressing clutter and co-labeling incorporating clothing priors and contexts.
- Evaluation of Components: Mutual interactions between garment items are important: the exterior-affinity variant achieves the best result among the evaluated component variants.The ablations compare exterior affinity, interior affinity, and superpixel grouping.
- Qualitative Evaluation: Qualitative results show accurate parsing under challenging illumination and complex backgrounds, including small garments, ambiguous patterns, and multiple people in one image.Examples include belts, purses, hats, sunglasses, dotted T-shirts, colorful dresses, and several persons parsed simultaneously.
- Qualitative Evaluation: The framework can produce wrong results with ambiguous patterns, visually similar garment items, extremely disordered backgrounds, or poor illumination.These cases are presented as failure scenarios in the qualitative evaluation.
- Efficiency: Co-segmentation costs about 10 seconds per image per iteration, while co-labeling takes less than 1 minute for 70 images and Graph Cuts converges in 3–4 iterations.The reported measurements were obtained on an Intel Dual-Core E6500 CPU with 8GB RAM.
5. Conclusions
The paper presents a joint clothing-image parsing framework and a high-resolution street-fashion dataset, while identifying future scalability improvements.
- The framework jointly parses batches of clothing images using image-level clothing tags.
- The paper introduces a high-resolution street-fashion photo dataset with annotations.
- Future work will iterate the two inference phases to bootstrap results and study parallel implementation for large-scale applications.