PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-domain Image Retrieval with a Dual Attribute-aware Ranking Network

Junshi Huang, Rogerio S. Feris, Qiang Chen, Shuicheng Yan

arXiv:1505.07922v1cs.CV

TL;DR

The paper addresses retrieving identical or attribute-similar clothing from online shopping images using uncontrolled user photos despite substantial cross-domain discrepancy. It proposes DARN, which combines dual attribute-aware networks, triplet ranking, and a large paired dataset; top-20 retrieval accuracy is 0.570 versus 0.268 for pre-trained CNN features only.

  • Problem

    Cross-domain clothing retrieval must match uncontrolled user photos with professional online shopping images despite domain discrepancy and limited annotated paired data.

  • Method

    DARN uses two domain-specific sub-networks with tree-structured semantic attribute learning and a triplet visual similarity constraint for retrieval feature learning.

  • Results

    0.570 vs. 0.268 top-20 retrieval accuracy is reported for DARN versus pre-trained CNN features only.

  • Takeaways & Limitations

    The framework integrates semantic attributes and visual similarity constraints for cross-domain clothing retrieval, and the dataset is presented as useful for other applications.

  • Takeaways & Limitations

    Existing clothing retrieval methods often do not model the discrepancy between user photos and online clothing images, and large annotated paired training sets are lacking.

Abstract

from arXiv · show

We address the problem of cross-domain image retrieval, considering the following practical application: given a user photo depicting a clothing image, our goal is to retrieve the same or attribute-similar clothing items from online shopping stores. This is a challenging problem due to the large discrepancy between online shopping images, usually taken in ideal lighting/pose/background conditions, and user photos captured in uncontrolled conditions. To address this problem, we propose a Dual Attribute-aware Ranking Network (DARN) for retrieval feature learning. More specifically, DARN consists of two sub-networks, one for each domain, whose retrieval feature representations are driven by semantic attribute learning. We show that this attribute-guided learning is a key factor for retrieval accuracy improvement. In addition, to further align with the nature of the retrieval problem, we impose a triplet visual similarity constraint for learning to rank across the two sub-networks. Another contribution of our work is a large-scale dataset which makes the network learning feasible. We exploit customer review websites to crawl a large set of online shopping images and corresponding offline user photos with fine-grained clothing attributes, i.e., around 450,000 online shopping images and about 90,000 exact offline counterpart images of those online ones. All these images are collected from real-world consumer websites reflecting the diversity of the data modality, which makes this dataset unique and rare in the academic community. We extensively evaluate the retrieval performance of networks in different configurations. The top-20 retrieval accuracy is doubled when using the proposed DARN other than the current popular solution using pre-trained CNN features only (0.570 vs. 0.268).

1. Introduction

The paper targets cross-domain clothing retrieval between uncontrolled user photos and professional online shopping images. It introduces DARN, combining domain-specific attribute learning, visual similarity ranking, and a large paired dataset.

  • Cross-domain retrieval seeks similar images in a target domain given a query from another domain, including clothing search from street photos to online galleries.
  • User-photo and online-shopping discrepancies, plus limited paired annotated datasets, make clothing retrieval difficult.
  • DARN uses two domain-specific sub-networks with tree-structured attribute branches to learn semantic clothing representations and reduce domain discrepancy.
  • Triplet ranking loss constrains matched online-offline pairs to be closer than mismatched pairs, aligning feature learning with retrieval.
  • The dataset contains about 450,000 online images and 90,000 offline counterparts with fine-grained clothing attributes.
  • 0.570 vs. 0.268 top-20 retrieval accuracy is reported for DARN versus pre-trained CNN features only.

2. Related Work

Related work spans clothing datasets, visual analysis, attribute-based representations, and deep retrieval methods. The paper distinguishes its paired user-photo and shopping-image data and its cross-domain architecture from prior approaches.

  • Existing fashion datasets mainly support clothing parsing and attribute estimation, whereas this dataset pairs user photos with corresponding online garments and fine-grained attributes.
  • Clothing visual analysis includes recognition, parsing, person re-identification, and applications such as identification, occupation recognition, and style analysis.
  • Most clothing retrieval methods do not model user-photo versus online-image discrepancy, while one exception uses a different methodology and does not exploit customer-review imagery.
  • Attribute-based representations support zero-shot learning, ranking, retrieval, fine-grained categorization, scene understanding, and image-to-sentence generation.
  • Prior clothing attribute systems enable fine-grained searches such as blue striped polo-style shirts, while this work uses attributes for cross-domain retrieval feature learning.
  • Unlike approaches that implicitly encode attributes or use alignment costs, this work explicitly regularizes deep networks with attribute prediction for cross-domain retrieval.

3. Data Collection

The paper constructs a large cross-scenario clothing dataset by pairing online shopping images with user-uploaded offline photos and fine-grained attributes. These data support semantic representation learning and address the scarcity of annotated online-offline pairs.

  • Online image collection: 453,983 online upper-clothing images were collected from shopping websites, with nine attribute categories and 179 possible attribute values.Attributes include clothing color, collar shape, sleeve shape, and clothing style; color alone has 56 values.
  • Collection strategy: Customer review websites provide links between user clothing photos and the corresponding online product images, enabling collection of otherwise rare cross-scenario pairs.The resulting pairs reveal discrepancies between shopping and street imagery.
  • Online-offline pairing: 381,975 online-offline image pairs were initially crawled and reduced to 91,390 curated pairs after unsuitable images were removed.Each pair depicts the same clothing in different scenarios, with variations in pose, lighting, and background clutter.
  • Dataset suitability: The dataset combines large scale, fine-grained attributes, and paired shopping-street images to support retrieval-model training and semantic clothing representations.The paired images expose real cross-scenario discrepancies and provide information for real-world applications.

4. Technical Approach

DARN uses two domain-specific networks that learn semantic clothing attributes and comparable retrieval features from online and street images. Clothing detection removes background clutter, while triplet ranking combines global and local features across domains.

  • Network overview: DARN integrates semantic attributes, visual similarity constraints, and domain-specific modeling into retrieval feature learning.The architecture uses separate but similarly structured sub-networks for the online shopping and street domains.
  • Network structure: Two NIN-based sub-networks process online and street images separately and produce high-level comparable features.Each sub-network contains stacked convolutional layers followed by MLPConv layers and two fully connected layers.
  • Semantic information embedding: Tree-structured fully connected layers model clothing attributes by sharing low-level representations while learning high-level semantic features separately.Each branch corresponds to an attribute, and its output size matches the number of values for that attribute; missing-attribute branches contribute zero gradients.
  • Learning to rank: The triplet ranking loss requires a matching online-offline pair to be closer than an offline image and a dissimilar online image by at least margin m.The margin is empirically set to 0.3 and the loss is applied across the two domain sub-networks.
  • Global and local features: The ranking features concatenate global FC1 responses with pooled local Conv4 and Conv5 features to retain subtle clothing information.FC1 provides the global representation, while convolutional responses supply complementary local features.
  • Clothing detection: A clothing detector crops foreground clothing before DARN, using attribute-aware features, SVR, proposal refinement, and hard-example mining.The preprocessing is intended to reduce the impact of cluttered backgrounds before retrieval feature learning.
  • Training procedure: Training forms triplets from curated online-offline pairs plus a randomly sampled online image with a different clothing ID.Cropped images are sequentially fed into the sub-network corresponding to their scenario.

5. Experiments

The experiments evaluate dataset construction, detection, retrieval metrics, feature configurations, component contributions, attribute matching, robustness, and runtime. DARN improves retrieval through semantic attributes, triplet ranking, dual structure, and robust feature learning.

  • Experimental Setting: The retrieval experiments train on about 230,000 online and 65,000 offline images, using paired images and randomly sampled dissimilar online images to construct triplets.Testing uses 1,717 online-offline image pairs.
  • Evaluation Metrics: The evaluation measures exact-match top-k retrieval accuracy and attribute-level NDCG@k, whose relevance score is the fraction of matched query attributes.The gallery contains 200,000 images in the reported top-k and NDCG evaluations.
  • Clothing Detection: 74.4% and 68.3% AP are achieved by AN with SVR on online and offline images, respectively, outperforming the other detection baselines.Detection uses a strict IoU threshold of 0.7, and online detection generally performs better than offline detection.
  • Clothing Detection: More precise clothing detection leads to more accurate retrieval when DARN with Conv4-5 is evaluated on a 10,000-image online gallery.This experiment directly links detector quality with downstream retrieval accuracy.
  • Cross-domain Clothing Retrieval Evaluation: 11.6% is the top-20 accuracy increase for AN after attribute fine-tuning, while triplet ranking adds 5.0% for ARN.Retrieval performance improves progressively with NIN, semantic information, learning to rank, and dual structure.
  • Cross-domain Clothing Retrieval Evaluation: 9.9% is the top-20 retrieval accuracy improvement of DARN over ARN, while local features provide an additional smaller benefit.The comparison includes DARN with Conv5 and DARN with Conv4-5.
  • Attribute-aware Clothing Retrieval Evaluation: DARN produces retrieval results with more similar attributes than traditional features under the NDCG@K attribute-matching evaluation.The relevance score counts matched attributes relative to the query’s total attributes.
  • Showing the Robustness: Performance vs. Retrieval Gallery Size: The accuracy increment ratios from the largest to smallest gallery are 115.4% for traditional features, 63.8% for pre-trained NIN, 64.5% for ARN, and 28.2% for DARN.The smaller ratio indicates better robustness; the reported comparison identifies DARN as the most robust among these features.

6. Conclusions

The conclusion presents DARN as a cross-domain retrieval method that combines semantic attributes, visual similarity constraints, and domain-discrepancy modeling. It reports substantial improvement in clothing retrieval and introduces a large-scale clothing dataset for broader use.

  • 6. Conclusions: DARN embeds semantic attribute information and visual similarity constraints into feature learning while modeling the discrepancy between the two domains.The method is demonstrated in a real-world clothing retrieval application.
  • 6. Conclusions: The clothing retrieval application shows substantial improvement over other baselines and is accompanied by a unique large-scale clothing dataset.The conclusion states that the dataset may support other applications.

A. Clothing Detection

The clothing detector is evaluated on online and offline images under varying real-world conditions. R-CNN with attribute-aware fine-tuning and SVR achieves the strongest reported detection performance and handles substantial visual variation.

  • Clothing Detection: 74.4% and 68.3% AP are achieved by R-CNN with AN + SVR on online and offline images, respectively, outperforming the other baselines.The precision-recall comparison evaluates detectors across both domains.
  • Clothing Detection: R-CNN with AN + SVR is reported as robust to illumination changes, cluttered backgrounds, human poses, occlusion, and viewpoint variation.These conditions are illustrated through detection examples.

B. Clothing Retrieval

The retrieval system first crops foreground clothing and then applies DARN with Conv4-5 to retrieve online images. The examples show retrieval of the same clothing and items with similar attributes, while the figures summarize retrieval and detection results.

  • Clothing Retrieval: Foreground clothing is cropped with R-CNN using AN + SVR before DARN retrieval feature extraction.The retrieval examples use DARN with Conv4-5.
  • Clothing Retrieval: DARN with Conv4-5 can retrieve the same clothing from the shop domain and rank online images with similar attributes among the top results.Figure 10 presents top-8 retrieval results, with the first column containing query images.
  • Clothing Retrieval: Figure 8 presents precision-recall curves for upper-clothes detection on online images and offline images.Figure 9 presents example detection results from R-CNN with AN + SVR.
Loading 1505.07922v1…