PaperlayerView exact revision on arXiv

Source-linked AI summary

Fashion Landmark Detection in the Wild

Ziwei Liu, Sijie Yan, Ping Luo, Xiaogang Wang, Xiaoou Tang

arXiv:1608.03049v1cs.CV

TL;DR

Fashion analysis needs a representation that captures functional clothing regions despite substantial pose, scale, deformation, and appearance variation. The paper introduces FLD and a three-stage DFA cascade using pseudo-labels and auto-routing; experiments show improved landmark estimation, pose-estimation generalization, and stronger fashion representations than boxes or joints.

  • Problem

    Existing clothing representations based on bounding boxes or human joints do not directly capture functional regions needed to distinguish clothing design and category under large garment variations.

  • Method

    The paper introduces FLD and DFA, which cascades three CNN stages and uses pseudo-labels plus auto-routing to estimate fashion landmarks.

  • Results

    Fashion landmarks are more discriminative than clothing bounding boxes and human joints for fashion attribute prediction and clothes retrieval, while DFA also generalizes to pose estimation.

  • Takeaways & Limitations

    Functional clothing landmarks provide a useful representation for understanding fashion images and support robust landmark and structural prediction.

Abstract

from arXiv · show

Visual fashion analysis has attracted many attentions in the recent years. Previous work represented clothing regions by either bounding boxes or human joints. This work presents fashion landmark detection or fashion alignment, which is to predict the positions of functional key points defined on the fashion items, such as the corners of neckline, hemline, and cuff. To encourage future studies, we introduce a fashion landmark dataset with over 120K images, where each image is labeled with eight landmarks. With this dataset, we study fashion alignment by cascading multiple convolutional neural networks in three stages. These stages gradually improve the accuracies of landmark predictions. Extensive experiments demonstrate the effectiveness of the proposed method, as well as its generalization ability to pose estimation. Fashion landmark is also compared to clothing bounding boxes and human joints in two applications, fashion attribute prediction and clothes retrieval, showing that fashion landmark is a more discriminative representation to understand fashion images.

1 Introduction

The paper introduces fashion landmark detection as a more discriminative alternative to clothing bounding boxes and human joints, addressing the large spatial, appearance, pose, and scale variations of clothing. It proposes DFA, a three-stage cascade that uses pseudo-labels and auto-routing to improve landmark estimation and supports fashion understanding applications.

  • Motivation: Fashion landmarks localize functional clothing regions, such as necklines and cuffs, to capture design, pattern, and category information beyond clothing bounding boxes or human joints.The paper motivates landmarks as useful for extracting features from regions such as collars and hemlines.
  • Challenges: Clothing landmarks are harder to detect than human joints because garments undergo non-rigid deformation and exhibit larger spatial and appearance variances.The paper identifies deformation, spatial variance, and appearance variance as three distinct challenges.
  • Approach: DFA cascades three deep convolutional networks to estimate landmarks while predicting pseudo-labels that encode sample similarities and reduce fashion-image variations.The pseudo-label scheme is designed progressively for landmark configurations, local offsets, and contextual relationships.
  • Approach: DFA uses full-image inputs instead of separate networks for each body part, reducing computation, and auto-routing assigns easy and challenging samples to different CNN branches.These design choices address computational cost and heterogeneous correction difficulty.
  • Results: Experiments show DFA is effective, generalizes to pose estimation, and yields a more discriminative representation than clothing bounding boxes and human joints for attribute prediction and clothes retrieval.The paper evaluates the representation in two fashion applications and compares it with both alternative representations.

2 Fashion Landmark Dataset (FLD)

FLD is a large-scale benchmark for fashion landmark detection, covering varied clothing types, poses, scales, and landmark visibility conditions. It contains more than 120K images, each annotated with eight fashion landmarks and their visibility.

  • Dataset construction: FLD contains more than 120K images, each labeled with eight fashion landmarks and landmark visibility.The visibility states are visible, invisible because of occlusion, or truncated/cut-off outside the image.
  • Dataset composition: The dataset includes upper-body, lower-body, and full-body clothing items with corresponding landmark annotations.These clothing categories are illustrated through sample images and annotations.
  • Dataset composition: FLD is divided into five subsets according to landmark positions and visibility, including normal, medium, and large pose or scale conditions.The subsets include cases with more than one or three cut-off landmarks, respectively.
  • Dataset challenges: More than 30% of images have large pose and zoom-in variations, making FLD a challenging benchmark for landmark detection.The dataset statistics characterize substantial pose and scale variation.

3 Our Approach

DFA aligns fashion landmarks through a three-stage cascade that progressively refines predictions, uses stage-specific pseudo-labels, and routes samples to specialized final-stage branches. The framework combines landmark, visibility, and pseudo-label objectives while reducing computational cost relative to DeepPose.

  • Framework Overview: DFA uses three VGG-16 stages that first predict rough landmarks and configuration pseudo-labels, then estimate local and contextual offsets for refinement.Stage-3 produces the final prediction by adding its estimated offset to the stage-2 landmark estimate.
  • Pseudo-Label: Pseudo-labels encode sample relationships through clustered landmark configurations or offsets, with K = 20 clusters used at every stage.Stage-1 uses landmark-coordinate space, whereas later stages use offset spaces; temperature T softens the labels.
  • Pseudo-Label: Stage-3 incorporates contextual correlations between landmarks by forming contextual offset representations and linearizing outer products for comparable outputs.This representation captures relationships such as those between a left collar and a left sleeve.
  • Auto-Routing: Auto-routing assigns each sample to one of two stage-3 branches according to stage-2 pseudo-label error patterns and a threshold set to ϵ = 0.3.Cluster-specific average errors define the routing function; r_i = 1 selects branch 1 and r_i = 0 selects the other branch.
  • Training: Each stage jointly optimizes landmark positions, visibility, and pseudo-labels, with piecewise balancing weights designed to preserve position-learning performance.The weights α(t) and β(t) are adjusted during training, with t_1 = 2000 and t_2 = 4000 iterations in the implementation.
  • Computations: DFA requires three VGG-16 models instead of DeepPose’s 17 for eight-landmark prediction, saving at least 5 times computational costs.The comparison concerns the number of VGG-16 models trained for the three-stage cascade.

4 Experiments

Experiments show that DFA's cascaded design, pseudo-labels, and auto-routing improve fashion landmark detection, generalize to pose estimation, and provide useful representations for attribute prediction and clothes retrieval.

  • Ablation Study: DFA progressively reduces localization errors across three cascade stages by predicting corrections over previous landmark estimates.Stage-1 produces rough predictions, while later stages refine them using local and contextual correction patterns.
  • Ablation Study: Pseudo-labels substantially improve performance beyond direct regression, with soft configuration labels and offset-based labels providing stage-specific benefits.Contextual landmark offsets further improve stage-3 results because landmark corrections are generally correlated.
  • Ablation Study: Auto-routing outperforms averaging two branches and particularly improves performance on medium and large zoom-in subsets.The routing function allows one stage-3 branch to focus on difficult samples.
  • Benchmarking: DFA consistently outperforms or matches DeepPose and IDPR across fashion landmarks, while collars are easiest and sleeves hardest to detect.For hem landmarks, DeepPose performs better at small thresholds, whereas IDPR catches up at larger thresholds.
  • Benchmarking: DFA outperforms other methods across clothing types and gains most on difficult medium pose or zoom-in subsets, although detection remains challenging.At a 15-pixel threshold, even the normal-pose detection rate is only slightly above 70%.
  • Benchmarking: DFA takes about 100ms per 300 × 300 image on a GTX Titan X, compared with nearly 650ms for DeepPose.This comparison is reported for detecting the full set of fashion landmarks.
  • Generalization of DFA: DFA achieves 74.4 mean strict PCP on pose estimation and shows comparable or better results than several existing methods.Reported part scores are 87, 91, 70, 56, 81, and 76 for torso, head, upper arms, lower arms, upper legs, and lower legs.
  • Applications: Fashion landmarks achieve the highest average top-5 attribute recall at 73%, compared with 27% for full images, 53% for bounding boxes, and 65% for human joints.They outperform the other localization schemes across the representative attributes shown.

5 Conclusions

The paper introduces FLD and DFA for robust fashion landmark detection, then evaluates landmark representations across fashion applications. Results support fashion landmarks as more discriminative than clothing bounding boxes and human joints for fashion-related tasks.

  • FLD provides a large-scale benchmark for fashion landmark detection, with images labeled using eight fashion landmarks and visibility information.
  • DFA performs robust fashion landmark detection by leveraging pseudo-labels and an auto-routing mechanism to reduce variation in fashion images.
  • Fashion landmarks are more discriminative than clothing bounding boxes and human joints for clothing attribute prediction and clothes retrieval.

A More Comparisons between DFA and DeepPose

DFA and DeepPose both use cascaded pipelines, but DFA preserves full-image context and adds landmark-position and pseudo-label signals in later stages. Its architecture also reduces VGG usage, improving training and testing efficiency.

  • DFA and DeepPose use cascaded pipelines, with DeepPose receiving local patches after stage one while DFA continues using full images.DFA also uses estimated landmark positions as additional input in its last two stages.
  • DFA retains contextual information and combines previous landmark positions with pseudo-labels that capture global and local shape structure.
  • DFA’s architecture reduces the number of VGG networks in the cascade, increasing training and testing efficiency.
  • Figure 9 contrasts stage-wise DFA and DeepPose predictions for the left and right sleeves, including pseudo-label similarities and stage-specific input patches.

B Implementation Details

This section introduces the implementation details of Deep Fashion Alignment and DeepPose.

  • The implementation section covers both Deep Fashion Alignment and DeepPose.
  • The section focuses on how the two methods are implemented.
  • Deep Fashion Alignment and DeepPose are treated as the implementation subjects of this section.

B.1 Pre-processing

The preprocessing pipeline uses normalized landmark coordinates and a clothes bounding box, then applies inverse normalization to recover absolute landmark positions after inference.

  • Each cascade stage uses a VGG16 regression model for landmark positions and visibility, initialized from a clothes bounding box.
  • Landmark coordinates are normalized relative to a clothes bounding box defined by its center, width, and height.
  • After inference, inverse normalization converts estimated landmark coordinates back to absolute image coordinates.

B.2 Input Preparation for DeepPose

DeepPose prepares later landmark regressors by first estimating all landmarks from the clothing bounding box, then cropping landmark-centered part boxes to regress local offsets.

  • Input Preparation: Stage-1 regresses all N fashion landmarks from the clothing bounding box within its receptive field.The estimated positions support input preparation for subsequent stages.
  • Input Preparation: Stage-2 uses N separate models, each receiving a part bounding box cropped around one estimated landmark.The example targets the ‘left sleeve’ landmark.
  • Input Preparation: Each stage-2 model regresses the offset for its underlying fashion landmark using the cropped part box.The passage states that the bounding-box size is s, but does not provide its value.

C More Results

The paper presents additional visual results for fashion landmark detection, attribute prediction, and clothes retrieval. These results show DFA across methods and retrieval or attribute outputs, with the authors stating that it handles complex variations under different scenarios.

  • More Results: DFA is reported to handle complex variations under different scenarios in additional visual results.The results cover fashion landmark detection, attribute prediction, and clothes retrieval.
  • More Results: The attribute-prediction and retrieval visualization is organized around a query image, predicted attributes, and top-5 retrieved images.This layout is identified in the accompanying labels.
  • More Results: Figure 10 presents additional attribute-prediction and clothes-retrieval results produced by DFA, alongside predicted landmarks.Correct retrieval matches are marked in green.
  • More Results: The landmark visualization includes Ground Truth, IDPR, and DeepPose method labels.These labels identify methods represented in the visual comparison.
  • More Results: Figure 11 compares visual fashion-landmark results from Ground Truth, IDPR, DeepPose, and DFA across stage-1, stage-2, and full-model outputs.The figure lists DeepPose stage-1 and full-model results, plus DFA stage-1, stage-2, and full-model results.
Loading 1608.03049v1…