PaperlayerView exact revision on arXiv

Source-linked AI summary

Embedding-based Product Retrieval in Taobao Search

Sen Li, Fuyu Lv, Taiwei Jin, Guli Lin, Keping Yang, Xiaoyi Zeng, Xiao-Ming Wu, Qianli Ma

arXiv:2106.09297v1cs.IR

TL;DR

E-commerce retrieval must balance personalized product matching with query relevance, while existing embedding-based systems face relevance and training–inference consistency problems. MGDSPR addresses these issues with a revised training objective and relevance-improving methods, yielding offline gains and deployment in Taobao Search.

  • Problem

    E-commerce retrieval must find relevant products at scale while preserving personalized user characteristics, but existing EBR systems show low query relevance and training–inference discrepancy.

  • Method

    MGDSPR uses softmax cross-entropy training, temperature-based smoothing of noisy click data, and generated hard negatives to improve relevance without extra training procedures.

  • Results

    MGDSPR improves over α-DNN by 2.5% in Recall@1000, 13.3% in P_good, and 6.0% in P_f_good, with effectiveness also verified in online A/B tests.

  • Takeaways & Limitations

    MGDSPR was deployed in Taobao Product Search to serve hundreds of millions of users in real time, alongside a practical online deployment scheme.

  • Takeaways & Limitations

    Offline evaluation uses recall because AUC has no positive correlation with online GMV, and it augments testing data with relevant records purchased elsewhere in Taobao.

Abstract

from arXiv · show

Nowadays, the product search service of e-commerce platforms has become a vital shopping channel in people's life. The retrieval phase of products determines the search system's quality and gradually attracts researchers' attention. Retrieving the most relevant products from a large-scale corpus while preserving personalized user characteristics remains an open question. Recent approaches in this domain have mainly focused on embedding-based retrieval (EBR) systems. However, after a long period of practice on Taobao, we find that the performance of the EBR system is dramatically degraded due to its: (1) low relevance with a given query and (2) discrepancy between the training and inference phases. Therefore, we propose a novel and practical embedding-based product retrieval model, named Multi-Grained Deep Semantic Product Retrieval (MGDSPR). Specifically, we first identify the inconsistency between the training and inference stages, and then use the softmax cross-entropy loss as the training objective, which achieves better performance and faster convergence. Two efficient methods are further proposed to improve retrieval relevance, including smoothing noisy training data and generating relevance-improving hard negative samples without requiring extra knowledge and training procedures. We evaluate MGDSPR on Taobao Product Search with significant metrics gains observed in offline experiments and online A/B tests. MGDSPR has been successfully deployed to the existing multi-channel retrieval system in Taobao Search. We also introduce the online deployment scheme and share practical lessons of our retrieval system to contribute to the community.

1 INTRODUCTION

Taobao product retrieval must balance query relevance with personalized user behavior at massive scale. MGDSPR addresses EBR’s training–inference discrepancy and relevance weaknesses through a revised objective, noise smoothing, hard negatives, and relevance control.

  • Motivation: E-commerce retrieval must account for short, ungrammatical product text and users’ historical behaviors while retrieving relevant products.Lexical matching is reliable and controllable but cannot flexibly capture user-specific interests for the same query.
  • Motivation: EBR can personalize retrieval but has relatively low relevance controllability after extended Taobao deployment.The system’s practical weaknesses include insufficient exact matching and degraded retrieval quality.
  • Motivation: Existing training and inference stages are inconsistent because pairwise loss does not directly train the global top-K comparison required at inference.Inference selects the closest products from all candidates, whereas prior methods use hinge loss with random negatives.
  • Approach: MGDSPR uses softmax cross-entropy, smooths click-derived relevance noise, and generates relevance-improving hard negatives without extra knowledge or training procedures.It also adapts a relevance-control module and is evaluated on Taobao data and online A/B tests.
  • Contributions: The paper reports MGDSPR’s design, deployment solution, relevance methods, and effectiveness on a large-scale industrial dataset and Taobao Product Search.The contributions include analyzing effects across search-system stages and improving EBR relevance controllability.

2 RELATED WORK

Prior work established embedding-based retrieval as an efficient way to address semantic gaps and incorporate user context. However, existing e-commerce systems either omit or weaken personalization and do not address insufficient product relevance caused by EBR.

  • Representation-based retrieval: Representation-based retrieval commonly uses two neural towers to encode queries and products, then matches their representations with a simple function.Examples include DSSM, CLSM, LSTM-RNN, and ARC-I.
  • Industrial retrieval: ANN algorithms have become mainstream for efficiently deploying neural retrieval models in industry.Facebook applied EBR to text matching and searcher context, while Amazon used a two-tower model for semantic product retrieval.
  • E-commerce systems: Amazon’s model omits user personalization, while JD’s DPSR uses average-pooled behaviors that weaken personalization characteristics.Neither system studies insufficient product relevance caused by EBR.

3 MODEL

The model section introduces MGDSPR as a two-tower retrieval model that jointly represents query semantics and historical behavior data to retrieve products with good relevance.

  • Model overview: MGDSPR simultaneously models query semantics and historical behavior data for personalized product retrieval.The section presents the model structure before describing its two towers, training objective, and relevance-improving methods.

3.1 Problem Formulation

The task formulates personalized product retrieval from a user’s current query and behavior history. MGDSPR scores items with an efficient two-tower architecture and returns the top-K candidates.

  • Problem formulation: The formulation represents users, queries, products, and three behavior-history windows: real-time, short-term, and long-term.The windows are defined relative to the current time and use sequence representations.
  • Retrieval task: Given a user’s behavior histories and current query, the system returns items satisfying the search request from the product collection.Candidates are selected at time t using scores between the user-query representation and products.
  • Scoring architecture: The retrieval model uses a two-tower design with query-and-behavior and item encoders, scoring candidates through an inner product.This architecture is selected for efficiency.

3.2 User Tower

The user tower combines multi-granular query semantics with real-time, short-term, and long-term behavior representations, using attention to preserve query-relevant personalization.

  • Query representation: Multi-granular query semantics represent unigram, 2-gram, segmentation, sequence, historical-query, and mixed information.The six representations are concatenated into Q_mgs.
  • Behavior representation: Item histories encode identifiers and side information, including leaf category, first-level category, brand, and shop features.The same embedding approach is used for real-time and long-term sequences.
  • Behavior attention: Query attention captures user behaviors related to the current query rather than treating all historical behavior as equally relevant.An all-zero vector is added to handle unrelated historical behaviors and potential noise.
  • Real-time behavior: Real-time behavior sequences use LSTM hidden states followed by multi-head self-attention to aggregate potential points of interest.A zero vector is added before producing the query-related real-time representation.
  • Short- and long-term behavior: Short-term behavior is aggregated with multi-head self-attention, while long-term behavior uses item, shop, leaf-category, and brand attribute representations.The real-time and short-term sequences are click sequences, whereas long-term behavior spans one month and includes click, buy, and collect actions.
  • Fusion: Self-attention fuses Q_mgs with real-time, short-term, and long-term representations into the user tower representation H_qu.The input prepends a [CLS] token, whose output is used as H_qu.

3.3 Item Tower

The item tower represents each product from its item ID and title, using mean pooling for title tokens because product titles are keyword-stacked and lack grammatical structure.

  • Item representation: The item representation H_item uses the product ID and title segmentation result.The item ID is represented by an embedding, and title tokens contribute to the product representation.
  • Title encoding: Mean pooling outperforms LSTM or Transformer encoding for titles because e-commerce titles are keyword-stacked and lack grammatical structure.This design prioritizes the observed effectiveness of simple aggregation for the title representation.

3.4 Loss Function

The loss design aligns training with inference by replacing local pairwise comparison with softmax-based global comparison, while two additional methods target retrieval relevance under noisy supervision.

  • Training–inference consistency: Inference requires selecting top-K items through global comparison, whereas hinge loss performs only local comparison and requires margin tuning.The mismatch motivates changing the training objective.
  • Softmax objective: Softmax cross-entropy is used as the training objective to align training and inference, yielding faster convergence and better performance without additional hyper-parameter tuning.The positive item is the item clicked by the user under the query.
  • Retrieval relevance: The proposed relevance methods aim to retrieve more relevant products while increasing the number of products entering subsequent ranking stages and maintaining efficiency.Both methods are designed without relying on additional knowledge.
  • Smoothing noisy data: The model smooths noisy click and purchase supervision by introducing a temperature parameter into the softmax distribution.These behavioral signals can also reflect images, prices, and user preferences rather than query–product relevance alone.
  • Hard negatives: Relevance-improving hard negatives are generated without additional knowledge, annotated data, or training procedures.The method selects high-scoring negatives and interpolates them with positive-item representations.
  • Hard-negative training: The generated hard negatives are included in the softmax denominator so the model distinguishes the positive item from nearby embedding-space samples.The hardness is controlled by the interpolation coefficient, with values closer to 1 producing samples nearer the positive item.

4 SYSTEM ARCHITECTURE

Taobao combines multi-channel retrieval with multi-stage ranking, deploying MGDSPR through offline indexing and online ANN search. Boolean relevance control addresses embedding retrieval’s weakness in exact matching of query key terms.

  • 4 SYSTEM ARCHITECTURE: Taobao’s search engine generates a deduplicated candidate set through multi-channel retrieval before pre-ranking, relevance ranking, ranking, re-ranking, and mix-ranking.The embedding-based module is the third matching channel, supplementing two existing channels.
  • 4.1 Offline Training and Indexing: MGDSPR is deployed offline-to-online: user/query networks and item embeddings are built offline, then served for real-time inference and ANN retrieval.Item embeddings are indexed across multiple columns using hierarchical clustering, K-means, and INT8 quantization.
  • 4.3 Relevance Control: Embedding retrieval improves personalization and fuzzy matching but can return semantically similar products that miss exact query terms such as brand, type, or color.The paper illustrates this problem with Adidas sports-shoe queries retrieving Nike products.
  • 4.3 Relevance Control: Boolean matching improves EBR relevance by enforcing predefined key-term rules for attributes such as brand, color, style, and audience.The rules are defined through query understanding and supplement embedding-based retrieval.

5 EXPERIMENTS

The experiments evaluate retrieval quality, relevance, and downstream participation using Taobao logs and a large candidate corpus. MGDSPR is assessed against a strong baseline and deployed in Taobao’s production environment.

  • 5 EXPERIMENTS: Offline evaluation uses Recall@K, good rate, and the number of retrieved products entering subsequent ranking stages.Good rate is measured on both the retrieval set and the filtered set.
  • 5 EXPERIMENTS: The evaluation supplements search logs with relevant records purchased elsewhere in Taobao’s mobile recommender system.The authors report that recall correlates with online GMV, whereas AUC does not show positive correlation with GMV.
  • 5 EXPERIMENTS: Pgood estimates the proportion of retrieved products judged relevant by an online relevance model, while Ph_good uses human judgments for displayed products.The relevance model has AUC 0.915 on human-labeled data.
  • 5 EXPERIMENTS: The dataset contains 4.7 billion training records from seven days, with evaluation using 1 million search records and 0.5 million recommender-system purchase logs.The candidate item set contains about 100 million products, matching the online environment.
  • 5 EXPERIMENTS: A well-trained MGDSPR model is deployed in Taobao search production, serving hundreds of millions of user-query requests over a candidate set of about 100 million items.The deployment covers the most active products at Taobao.

5.4 Offline Experimental Results

MGDSPR improves offline retrieval relevance and recall over the strong α-DNN baseline. Its components also reveal a recall–relevance trade-off, while softmax cross-entropy converges faster than pairwise loss and tuned noise smoothing and hard negatives improve relevance.

  • Comparison with the Strong Baseline: MGDSPR improves over α-DNN by 2.5% in Recall@1000, 13.3% in P_good, and 6.0% in P_f_good.These gains indicate improved retrieval-set quality and relevance control.
  • Ablation Study: Multi-granular semantics and dynamic fusion improve Recall@1000 and P_good, while temperature and hard negatives substantially raise P_good.The ablation isolates four components: mgs, trm, τ, and I_mix.
  • Ablation Study: The full model exposes a trade-off between recall and relevance, which may indicate excessive personalization.This comparison uses MGDSPR+all against MGDSPR and MGDSPR+mgs+trm+τ.
  • Ablation Study: Softmax cross-entropy converges in about three days versus about six days for pairwise loss, while achieving faster convergence and better performance.The comparison evaluates test Recall@1000 across training steps.
  • Ablation Study: Every non-zero N improves P_good over N=0, but P_good peaks at N=684 and then declines.The τ experiment also finds that τ=0.1 decreases relevance, indicating noisy training data.

6 CONCLUSION

The paper proposes MGDSPR to address low relevance and resource waste in Taobao’s previous EBR system. Offline and online A/B tests validate the model, which was deployed in Taobao Product Search for large-scale users.

  • Conclusion: MGDSPR addresses performance degradation and online computing resource waste caused by low retrieval relevance in Taobao’s previous EBR system.The paper also shares model-design, evaluation, deployment, and relevance-control lessons.
  • Conclusion: Offline experiments and online A/B tests verify MGDSPR’s effectiveness, and the model was deployed on Taobao Product Search.The deployment serves hundreds of millions of users in real-world search.
Loading 2106.09297v1…