PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Search at Alibaba

Yanhao Zhang, Pan Pan, Yun Zheng, Kang Zhao, Yingya Zhang, Xiaofeng Ren, Rong Jin

arXiv:2102.04674v1cs.CV

TL;DR

Alibaba’s commercial visual search must bridge real-shot queries and inventory images while operating over massive, noisy, fine-grained data with limited annotation and engagement requirements. The paper builds an end-to-end system combining model/search fusion, weakly supervised joint detection and feature learning, binary indexing, and re-ranking; its fusion method reaches 91.01% category-prediction Accuracy@1 and the system is deployed in Pailitao.

  • Problem

    Alibaba visual search must handle heterogeneous real-shot and inventory images, massive noisy fine-grained collections, costly annotations, and user-engagement requirements.

  • Method

    The system combines model/search fusion for category prediction, click-supervised joint detection and feature learning, binary indexing, and re-ranking.

  • Results

    91.01% category-prediction Accuracy@1 is reported for the fusion approach, a 2.15% increase over the model-based method.

  • Takeaways & Limitations

    The visual-search solution was successfully deployed to Pailitao and integrated into other Alibaba internal applications.

Abstract

from arXiv · show

This paper introduces the large scale visual search algorithm and system infrastructure at Alibaba. The following challenges are discussed under the E-commercial circumstance at Alibaba (a) how to handle heterogeneous image data and bridge the gap between real-shot images from user query and the online images. (b) how to deal with large scale indexing for massive updating data. (c) how to train deep models for effective feature representation without huge human annotations. (d) how to improve the user engagement by considering the quality of the content. We take advantage of large image collection of Alibaba and state-of-the-art deep learning techniques to perform visual search at scale. We present solutions and implementation details to overcome those problems and also share our learnings from building such a large scale commercial visual search engine. Specifically, model and search-based fusion approach is introduced to effectively predict categories. Also, we propose a deep CNN model for joint detection and feature learning by mining user click behavior. The binary index engine is designed to scale up indexing without compromising recall and precision. Finally, we apply all the stages into an end-to-end system architecture, which can simultaneously achieve highly efficient and scalable performance adapting to real-shot images. Extensive experiments demonstrate the advancement of each module in our system. We hope visual search at Alibaba becomes more widely incorporated into today's commercial applications.

1 INTRODUCTION

Alibaba’s Pailitao visual-search system addresses real-shot versus inventory-image mismatch, massive noisy and fine-grained data, costly training-data maintenance, and user engagement. The paper presents an end-to-end approach combining deep learning, search, indexing, and re-ranking, evaluated on Alibaba data.

  • Application: Pailitao addresses commercial visual search through a deployed application intended to make image-based shopping more convenient and support product conversion.The application was developed as Alibaba’s image-intelligence product for searching by images.
  • Motivation: Pailitao faces four challenges: heterogeneous real-shot and inventory images, billions of noisy fine-grained data, expensive training-data maintenance, and user engagement.Real-shot queries can contain uneven quality and semantic or visual gaps, while marketplace images include complex backgrounds and noisy labels.
  • Motivation: Alibaba uses inventory images and natural seller or customer labels as opportunities for large-scale visual-search development.The shopping scenario also supplies broad opportunities for visual search.
  • Contributions: The paper presents an end-to-end Alibaba visual-search system covering category prediction, joint detection and feature learning, large-scale indexing, and image re-ranking.The modules are evaluated on a purpose-built test set, including indexing efficiency and re-ranking effectiveness.

2 RELATED WORK

Prior visual-search research covers CNN retrieval, deep metric embedding, and weakly supervised localization, but Alibaba-scale deployment remains difficult. The paper responds with a scalable system using fusion-based category prediction, weakly supervised joint detection and feature learning, binary indexing, and re-ranking.

  • Prior work: Prior work includes CNN descriptors for retrieval, metric-learning methods for image similarity, and weakly supervised localization approaches.The cited localization approaches are described as multi-stage rather than end-to-end.
  • Research gap: Alibaba-scale visual search must handle billions of images while meeting demanding performance and latency requirements.The collection also contains fine-grained categories, complex backgrounds, and noisy labels.
  • System response: The paper introduces a hybrid scalable and resource-efficient visual-search system for grounding these techniques in a commercial product.The system targets Alibaba-scale datasets and operational constraints.
  • Contributions: Model- and search-based fusion predicts categories, reducing search space while improving scalability and performance for confusion categories and domain restrictions.The approach is contrasted with a traditional model-only method.
  • Contributions: A branched deep CNN jointly learns object detection and retrieval features from user click behavior without additional human annotations.This weakly supervised design aims to reduce background disturbance while learning discriminative features.
  • Deployment: The deployed application uses binary indexing and re-ranking to provide millisecond responses, lossless recall, and scalable service for millions of users.The passage describes this as an end-to-end architecture for Pailitao.

3 VISUAL SEARCH ARCHITECTURE

Pailitao separates visual search into offline inventory indexing and online query processing. Daily offline updates build searchable representations, while online prediction, detection, feature extraction, retrieval, and re-ranking produce final results.

  • Scalability: The system establishes a stable and scalable visual-search architecture as Pailitao’s business grows.The architecture supports the application’s retrieval service.
  • Architecture: Pailitao’s architecture divides visual search into offline and online process flows.Figure 2 presents the overall process.
  • Offline process: The offline flow selects items, extracts features, constructs indexes, and updates the online inventory daily.It builds the index for documents processed every day.
  • Online process: The online flow performs category prediction, object detection, feature extraction, indexed retrieval, and re-ranking after a user uploads a query image.The final output is a ranked result list.

3.1 Category Prediction

The system predicts Taobao categories by combining a deep model with search over a large labeled reference set. This fusion improves category accuracy while narrowing the search space for fine-grained products.

  • 3.1.1 Item inventory selection: The inventory is filtered by shopping preferences and image quality before indexing to reduce redundancy among highly similar Taobao items.The gallery contains multiple image types, including main, SKU, unboxing, and LOG images.
  • 3.1.1 Item inventory selection: The system predicts 14 category sets covering all leaf categories to reflect user preferences and narrow the search space.The category hierarchy includes visually and semantically similar leaf categories such as shoes, dresses, and bags.
  • 3.1.2 Model and search-based fusion: The model uses GoogLeNet V1 trained with softmax loss on diverse inventory images, balancing accuracy against latency.Images are resized to 256 × 256 and randomly cropped to 227 × 227 during training.
  • 3.1.2 Model and search-based fusion: The search-based predictor uses 200 million reference images and estimates category likelihoods from feature distances with a weighted fusion function.The weight function uses λ estimated by maximum likelihood.
  • 3.1.2 Model and search-based fusion: Over 2% absolute Top-1 accuracy improvement comes from fusing model-based and search-based category predictions.The search-based method corrects confused categories using discriminative deep features.

3.2 Joint Detection and Feature Learning

The joint model learns object localization and image representations for mismatched buyer and seller images using weakly supervised click-derived triplets. Its ranking framework suppresses background clutter while aligning identical products without bounding-box annotations.

  • 3.2.1 PVLOG triplet mining: User-clicked PVLOG images form valid triplets that jointly learn object location and features without additional bounding-box annotations.Clicked images are treated as likely identical products, providing hard training examples from existing behavior data.
  • 3.2.1 PVLOG triplet mining: Triplet ranking pulls query embeddings toward identical-product images and pushes them away from different-product images using normalized feature distance with margin δ = 0.1.The CNN feature function is trained end-to-end.
  • 3.2 Joint Detection and Feature Learning: Buyer and seller images are mapped into a common CNN embedding space so heterogeneous sources can be matched reliably.The model addresses low-quality, blurred, unevenly illuminated consumer photos against controlled seller imagery.
  • 3.2.2 Unified deep ranking framework: The deep joint model uses separate detection and feature branches to reduce background impact while learning discriminative representations.The detection branch locates the target, while the feature branch learns the image representation.
  • 3.2.2 Unified deep ranking framework: The unified ranking framework feeds query, positive, and negative triplets into deep joint models to learn features and detection masks simultaneously.The framework maximizes positive-negative discrimination and detects informative regions without bounding-box supervision.
  • 3.2.2 Unified deep ranking framework: The differentiable sigmoid approximation enables end-to-end training of the otherwise nondifferentiable step-function detection mask.This design uses weakly supervised user-click data rather than manually annotated bounding boxes, reducing human-resource costs.

3.3 Image Indexing and Retrieval

Alibaba’s retrieval engine combines distributed binary indexing, fine re-ranking, and quality-aware semantic scoring to serve large-scale visual search efficiently.

  • Large-scale search of billion-scale images: Multi-shard indexing distributes vectors across machines, while merging each shard’s K nearest neighbors supports scalable retrieval.The architecture can dynamically add shards, and each machine handles only a fraction of the vectors.
  • Large-scale search of billion-scale images: Multi-replication partitions concurrent queries across index clusters so fluctuating demand stays within each cluster’s theoretical QPS peak.Alibaba’s promotions can increase query volume by as much as ten times.
  • Large-scale search of billion-scale images: A coarse binary filter rapidly removes mismatches by Hamming distance before fine re-ranking uses visual attributes and local features.Fine re-ranking is slower because metadata is non-binary and may not fit in memory, making cache hit rate important.
  • Quality-aware image re-ranking: Quality-aware re-ranking improves engagement by combining sales, conversion, applause, and user-profile features with Gradient Boosted Decision Trees and Logistic Regression.The system re-ranks the Top 60 appearance-similar results and scales the final score to [0, 1].

4 EXPERIMENT

Experiments on the High Recall Set evaluate category prediction, joint detection and feature learning, click-derived training data, localization, and large-scale retrieval efficiency.

  • 4.1 Evaluation of Category Prediction: 91.01% Accuracy@1 makes the fusion approach outperform the model-based method by 2.15% for category prediction.The search-based method averages 85.51%, versus 88.86% for the model-based method; their complementarity corrects some model misclassifications.
  • 4.2 Evaluation of Search Relevance: The joint detection-and-feature model outperforms all single-branch baseline variants across categories while suppressing background interference.The feature evaluation uses Identical Recall@K with K values of 1, 4, and 20.
  • 4.2 Evaluation of Search Relevance: Mining PVLOG click images without additional annotations increases Identical Recall@1 by 17 percentage points and MAP@1 by 5% over category-trained features.The result indicates a more relevant retrieved list under the reported MAP metric.
  • 4.3 Evaluation of Object Localization: The deep joint model achieves IOU@0.5 of 98.1% and IOU@0.7 of 70.2% against ground-truth boxes, slightly below fully supervised SSD.The learned detection branch also captures object content and achieves competitive results at much faster speed.
  • 4.4 Evaluation of Indexing and Reranking: On 3 billion images, indexing achieves lossless Linear Recall@60 versus linear search, with total response latency in the hundreds of milliseconds.Reported components include 30 ms for model plus search, 40 ms for embedding, 10–20 ms for ranking, and 5 ms for Top 60 quality-aware re-ranking; CVR engagement rises 7.85%.

5 CONCLUSIONS

The paper presents Alibaba’s deployed end-to-end visual search system, combining fusion-based category prediction, click-mined joint detection and feature learning, binary indexing, and re-ranking.

  • 5 CONCLUSIONS: The system was successfully deployed in Pailitao and integrated into other Alibaba internal applications.The conclusion reports millisecond response, lossless recall, high availability, and scalability for the mobile application.
  • 5 CONCLUSIONS: Future work will use object co-segmentation and contextual constraints within images to enhance visual search relevance in Pailitao.
Loading 2102.04674v1…