Source-linked AI summary
Automatic Spatially-aware Fashion Concept Discovery
Xintong Han, Zuxuan Wu, Phoenix X. Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, Larry S. Davis
TL;DR
The paper tackles clothing search challenges arising from visual variation, weak image-text resources, and the need for higher-level attribute feedback. It jointly models clothing images and descriptions, combines semantic and spatial attribute representations, and learns concept-specific subspaces. Experiments on Fashion200K report effective concept discovery and retrieval, including the highest retrieval accuracy across all five categories after combining semantic and spatial information.
Problem
Clothing search faces severe garment deformation and style variation, while existing datasets lack image-description pairs needed for semantic attribute representations.
Method
The method jointly embeds clothing images and descriptions, combines word vectors with attribute spatial representations, clusters attributes into concepts, and learns concept-specific subspaces.
Results
Combining semantic and spatial information achieves the highest attribute-feedback retrieval accuracy for all five clothing categories.
Takeaways & Limitations
The discovered concepts support structured product browsing and attribute-feedback retrieval through multimodal linguistic regularities and automatic negative-attribute detection.
Takeaways & Limitations
When a concept subspace predicts that a query has none of its attributes, retrieval degenerates to the baseline method.
Abstract
from arXiv · showhide
This paper proposes an automatic spatially-aware concept discovery approach using weakly labeled image-text data from shopping websites. We first fine-tune GoogleNet by jointly modeling clothing images and their corresponding descriptions in a visual-semantic embedding space. Then, for each attribute (word), we generate its spatially-aware representation by combining its semantic word vector representation with its spatial representation derived from the convolutional maps of the fine-tuned network. The resulting spatially-aware representations are further used to cluster attributes into multiple groups to form spatially-aware concepts (e.g., the neckline concept might consist of attributes like v-neck, round-neck, etc). Finally, we decompose the visual-semantic embedding space into multiple concept-specific subspaces, which facilitates structured browsing and attribute-feedback product retrieval by exploiting multimodal linguistic regularities. We conducted extensive experiments on our newly collected Fashion200K dataset, and results on clustering quality evaluation and attribute-feedback product retrieval task demonstrate the effectiveness of our automatically discovered spatially-aware concepts.
1. Introduction
The paper addresses difficult clothing search by automatically discovering spatially-aware concepts from weakly labeled image-description pairs. These concepts support structured browsing and attribute-feedback retrieval through a joint visual-semantic space and concept-specific subspaces.
- Clothing search is challenging because garments undergo severe deformation and style or texture variation, while customer intents remain separated from low-level visual features.
- Interactive relevance feedback can converge slowly, motivating higher-level attributes such as neckline, sleeve length, and dress length.
- The method combines attribute semantic word vectors with spatial representations learned from neural activations to cluster attributes describing the same characteristic.
- Concept-specific subspace embeddings enable structured dataset exploration and attribute-feedback retrieval, including implicit removal of contradictory attributes.
- A joint visual-semantic embedding is trained from clothing images and product descriptions, with fine-tuned GoogleNet providing attribute localization information.
- Fashion200K supplies over 200,000 image-description pairs across five clothing categories for evaluating concept discovery, browsing, and retrieval.
- The paper reports that augmenting semantic attribute vectors with spatial representations effectively clusters attributes into meaningful spatially-aware concepts and supports attribute-feedback retrieval.
2. Related Work
Prior work studies interactive search, clothing attributes, visual concept discovery, and visual-semantic embeddings. This paper differs by automatically deriving clothing concepts through joint image-text modeling.
- Interactive image search: Interactive image search incorporates user feedback on candidate relevance or low-level properties, but relevance feedback may require many iterations to converge.
- Attributes for clothing modeling: Clothing-modeling research uses semantic attributes as mid-level representations, including hierarchical, subspace, relative-attribute, and retrieval-oriented approaches.
- Attributes for clothing modeling: The proposed framework distinguishes itself by jointly modeling images and text to derive collections of automatically identified attributes as clothing concepts.
- Visual concept discovery: Visual concept discovery methods exploit weakly labeled data through visual-semantic clustering, attribute visualness, topic models, pair-concepts, and spatial extent discovery.
- Visual-semantic joint embedding: Visual-semantic embedding research maps images and descriptions jointly using approaches including deep embeddings, encoder-decoder contrastive learning, and cross-view ranking.
3. Fashion200K Dataset
Fashion200K was collected to provide weakly labeled clothing image-description pairs unavailable in existing datasets. It contains over 200,000 images across five categories and uses automatic preprocessing without manual annotations.
- Existing clothing datasets lack image-description pairs, preventing the use of textual semantic representations for attribute concept discovery.
- The collection began with over 300,000 crawled product images and descriptions, then removed descriptions containing fewer than four words.
- The dataset is split into 172,049 training, 12,164 validation, and 25,331 testing images.
- No manual annotations were added because the dataset supports automatic concept discovery and concept-level subspace learning from noisy image-description pairs.
- Figure 3 presents examples of Fashion200K image-description pairs.
4. Our Approach
The approach learns a joint visual-semantic embedding, extracts spatial attribute representations from activation maps, clusters attributes into concepts, and learns concept-specific subspaces for retrieval and browsing.
- Visual-semantic embedding learning: Clothing images and descriptions are jointly embedded by fine-tuning GoogleNet with a bidirectional contrastive objective.Image features are projected into the embedding space and trained to match their associated product descriptions while separating mismatched pairs.
- Spatially-aware concept discovery: Attributes are represented by concatenating normalized spatial and semantic vectors, then clustered with K-means into concepts describing shared characteristics.This combines spatial consistency with semantic similarity rather than relying directly on visual similarity between attributes.
- Spatially-aware attribute representation: Embedded attribute activation maps localize where an attribute appears, and averaging them over positive training images produces its attribute activation map.The map indicates likely attribute presence at each spatial location and aggregates evidence across images whose descriptions contain that attribute.
- Spatially-aware concept discovery: Activation maps capture characteristic clothing regions for attributes such as belts, long sleeves, and off-shoulder designs, while uncertain attributes span much of the image.The dress-category examples show localized patterns for spatially specific attributes and diffuse maps for floral, stripe, and color attributes.
- Spatially-aware concept discovery: The method groups semantically or spatially related attributes into concepts, including colors, sleeve types, and noisy attributes that do not affect attribute-feedback retrieval.Colors are grouped mainly through semantic proximity, whereas sleeve attributes share both semantic and activation-map similarity.
- Concept subspace learning and retrieval: Concept-specific subspaces enable similarity comparisons focused on one characteristic and support feedback retrieval by removing an automatically detected contradictory attribute.For a requested attribute, the method identifies the most likely existing attribute from the same concept and subtracts it from the query embedding; if none is predicted, it falls back to the baseline.
5. Experimental Results and Discussions
Experiments evaluate concept discovery and attribute-feedback retrieval using clothing-category models, showing benefits from combining semantic and spatial attribute information.
- Evaluation of Discovered Concepts: The concept-quality evaluation compares manual category assignments with ACD, Word2vec, and attribute activation maps.Homogeneity, completeness, and V-measure are used as evaluation metrics.
- Evaluation of Discovered Concepts: 0.03 V-measure improvement: jointly clustering semantic and spatial representations outperforms the compared concept-discovery methods.Semantic-only representations perform reasonably, while spatial-only representations perform worst because many attributes lack discriminative spatial information.
- Attribute-feedback Product Retrieval: The proposed method achieves the highest retrieval accuracy across all five categories, especially where attributes provide strong spatial cues.Examples include collar shape, sleeve length, and sleeve shape; pants show only marginal improvement over Word2vec.
- Attribute-feedback Product Retrieval: Retrieval examples show changes such as sleeveless to cap-sleeve and one-shoulder to strapless, with negative attributes automatically detected.These examples use customer feedback attributes to modify retrieved clothing results.
- Subspace Embeddings: Concept-specific subspaces organize images by characteristics such as dress length and top color for structured browsing.The dress-length embedding orders maxi, midi, and mini dresses continuously, while the color embedding separates differently colored tops.
6. Conclusion
The paper automatically discovers spatially-aware concepts from clothing images and descriptions, then learns concept-specific subspaces for retrieval and exploration. Experiments across five clothing categories support the effectiveness of the discovered concepts and subspace features.
- Conclusion: The method combines semantic and spatial attribute representations to cluster attributes describing the same characteristic.It learns spatial representations from a joint visual-semantic embedding and combines them with semantic representations before clustering.
- Conclusion: A separate subspace embedding captures concept-specific information for each discovered concept.These subspaces support structured browsing and attribute-feedback product retrieval.
- Conclusion: Experiments on five clothing categories show the effectiveness of the discovered concepts and learned subspace features.The evaluation covers clustering quality and attribute-feedback product retrieval.