PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Product Search

Priyanka Nigam, Yiwei Song, Vijai Mohan, Vihan Lakshman, Weitian, Ding, Ankit Shingavi, Choon Hui Teo, Hao Gu, Bing Yin

arXiv:1907.00937v1cs.IRcs.CL

TL;DR

Semantic product search must retrieve semantically related products despite lexical matching’s weaknesses with meaning, morphology, and spelling. This paper trains a neural model on customer behavior data, introducing thresholded loss handling and token representations that improve product matching and online retail KPIs.

  • Problem

    Semantic product search seeks to retrieve related products despite lexical matching failures involving synonyms, morphology, antonyms, and spelling errors.

  • Method

    The paper trains a neural network on purchase and click signals, using a three-part thresholded hinge loss, average pooling with n-grams, shared embeddings, and token hashing.

  • Results

    The model improves offline matching metrics and produces significant increases in online retail KPIs, with end-to-end embedding training yielding over a 3X improvement in Recall@100 and MAP.

  • Takeaways & Limitations

    Separating impressed-but-not-purchased items from random negatives and handling unseen tokens are central practical insights for semantic product matching.

  • Takeaways & Limitations

    The paper focuses on factorized matching models, excludes latent-factor ranking models, and constrains architecture choices by a 20-millisecond response target at thousands of queries per second.

Abstract

from arXiv · show

We study the problem of semantic matching in product search, that is, given a customer query, retrieve all semantically related products from the catalog. Pure lexical matching via an inverted index falls short in this respect due to several factors: a) lack of understanding of hypernyms, synonyms, and antonyms, b) fragility to morphological variants (e.g. "woman" vs. "women"), and c) sensitivity to spelling errors. To address these issues, we train a deep learning model for semantic matching using customer behavior data. Much of the recent work on large-scale semantic search using deep learning focuses on ranking for web search. In contrast, semantic matching for product search presents several novel challenges, which we elucidate in this paper. We address these challenges by a) developing a new loss function that has an inbuilt threshold to differentiate between random negative examples, impressed but not purchased examples, and positive examples (purchased items), b) using average pooling in conjunction with n-grams to capture short-range linguistic patterns, c) using hashing to handle out of vocabulary tokens, and d) using a model parallel training architecture to scale across 8 GPUs. We present compelling offline results that demonstrate at least 4.7% improvement in Recall@100 and 14.5% improvement in mean average precision (MAP) over baseline state-of-the-art semantic search methods using the same tokenization method. Moreover, we present results and discuss learnings from online A/B tests which demonstrate the efficacy of our method.

1 INTRODUCTION

Product search needs semantic matching because lexical retrieval misses related products, mishandles linguistic variation, and can return products that contradict customer intent. The paper proposes learning semantic matching from customer behavior data to complement lexical retrieval.

  • 1 INTRODUCTION: Lexical matching misses synonyms, hypernyms, and antonyms, such as retrieving neither “sneakers” for “running shoes” nor “red dress” for “burgundy dress.”It can also retrieve contradictory products when all query words appear textually, as with latex-free versus latex examination gloves.
  • 1 INTRODUCTION: 10-15% of queries are estimated to be misspelled, creating confusion and frustration when lexical matching returns no results.The paper notes that handling spelling errors directly could simplify matching compared with relying on separate spell correction.
  • 1 INTRODUCTION: Customer behavior logs provide semantic signals for training a deep learning model that retrieves products matching query intent.The paper frames this as a question of whether rich behavior data can support semantic product retrieval despite its scale and noise.
  • 1 INTRODUCTION: Product search differs from web search because queries are shorter, purchases are sparser than clicks, and sessions may contain multiple simultaneous intents.Click-oriented models may therefore favor accessories over the main product, while shoppers browse for several related purposes.
  • 1 INTRODUCTION: The proposed neural network complements lexical retrieval using purchase and click signals, with a thresholded loss separating random negatives, impressed negatives, and purchased positives.The paper also highlights average pooling with n-grams and consistent token hashing as contributions.

2 RELATED WORK

Prior semantic search work includes latent factor, factorized, and interaction models, largely developed for semantic ranking. The paper focuses on factorized matching because it offers comparable performance with constant memory per product under strict latency and query-volume constraints.

  • 2 RELATED WORK: Semantic search models are commonly categorized as latent factor, factorized, or interaction models based on how they represent queries and documents.The categories range from content-free embeddings to neural processing of query-document interaction matrices.
  • 2 RELATED WORK: DSSM established a factorized neural approach, while later systems combined semantic and lexical matching for web-search ranking.The related work traces these developments from latent semantic analysis and word embeddings through DSSM and Duet.
  • 2 RELATED WORK: ARC-II and Match Pyramid achieve sub-50% Recall@100 and MAP on the paper’s product-matching dataset despite state-of-the-art ranking results elsewhere.This motivates evaluating semantic methods directly on ad hoc product matching rather than ranking alone.
  • 2 RELATED WORK: The paper selects a factorized architecture because it has constant memory per product and can meet a 20-millisecond response target for thousands of queries per second.Latent factor models are excluded because the study focuses on matching rather than their ranking-oriented ability to memorize query-product associations.

3.1 Neural Network Architecture

The model uses a shared Siamese embedding architecture for queries and products, then forms fixed-length representations with average pooling and normalizes them before similarity computation. Product embeddings can be precomputed for nearest-neighbor retrieval.

  • 3.1 Neural Network Architecture: Shared query-product embeddings in a Siamese network capture local word-level matching before training.The embedding layer has |V| × N parameters, where rows represent word parameters and the same embeddings are used for both inputs.
  • 3.1 Neural Network Architecture: Average pooling produces fixed-length query and product embeddings with less than 0.5% difference in MAP and Recall@100 from recurrent approaches.It also reduces computation, training time, and inference latency relative to LSTM and GRU approaches.
  • 3.1 Neural Network Architecture: Batch normalization follows pooling because shorter queries and longer product content produce embeddings with different magnitudes.The normalization layers are applied separately to the query and product arms.
  • 3.1 Neural Network Architecture: Precomputed product embeddings are searched with k-nearest neighbors to retrieve products similar to a query.This retrieval procedure is used during online A/B testing.

3.2 Loss Function

The loss function uses score thresholds to separate purchased, impressed-but-not-purchased, and random products, addressing threshold-based matching and overlapping negative distributions.

  • 3.2 Loss Function: Threshold-based pruning motivates a pointwise loss that constrains positive scores above ϵ+ and negative scores below ϵ−.The 2-part hinge uses ϵ+ = 0.9 and ϵ− = 0.2 after validation tuning.
  • 3.2 Loss Function: More than 2X offline matching improvement over MSE was observed with the 2-part hinge, despite overlapping positive and negative score distributions.The negative distribution was bimodal, and manual inspection identified impressed-but-not-purchased products in the overlap region.
  • 3.2 Loss Function: The 3-part hinge separates purchased, impressed-but-not-purchased, and random products, improving distinction among matching classes.It treats impressed negatives separately because they are often valid results and more similar to positives than random negatives.

3.3 Tokenization Methods

The tokenization method combines word and character features in a hashed bag of n-grams to capture phrases, typos, compound words, and unseen tokens.

  • 3.3 Tokenization Methods: Word unigrams, word n-grams, and character trigrams are combined into a hashed bag of n-grams.The approach uses hashing to handle the large vocabulary created by higher-order n-grams.
  • 3.3 Tokenization Methods: Word n-grams capture phrase-level intent that word unigrams lose, such as distinguishing iPhone accessories from iPhone searches.Bigrams and trigrams preserve short-range token sequences without using recurrent or convolutional architectures.
  • 3.3 Tokenization Methods: Character trigrams provide robustness to typos and compound words while capturing similarities in model parts and sizes.The method explicitly motivates character trigrams with examples such as “iphione” versus “iphonr” and “amazontv” versus “firetvstick.”
  • 3.3 Tokenization Methods: A single bag-of-tokens combining tokenization types performed well for the best model.Table 6 examples illustrate retrieval behavior for synonyms, intents, spelling errors, and overall robustness.

4 DATA

The dataset uses large-scale search logs, weighted query-product aggregates, and tokenized representations to train and evaluate product-matching models.

  • 4 DATA: Training uses 11 months of search logs and evaluation uses 1 month, with 54 billion sampled query-product pairs reduced to 650 million weighted rows.Pairs are grouped by query-product identity over the full period and weighted by aggregated counts.
  • 4 DATA: The data pipeline reduces training time by 3X by preprocessing pairs into tokens and storing them with mmap.This optimization is applied after aggregation into weighted query-product rows.
  • 4 DATA: Each query targets six impressed and seven random products for every purchased query-product pair.The three product categories are purchased, impressed but not purchased, and random; sampling supports matching and ranking training.
  • 4 DATA: Queries are lowercased, whitespace-tokenized, truncated at the 99th length percentile, and right-padded to a fixed length.Frequent vocabulary tokens receive ids, while remaining tokens receive a default or hashing-based identifier.
  • 4 DATA: Representing product attributes as an ordered bag of words avoids the 5% lower recall observed when embedding attributes independently and concatenating them.The evaluated attributes include title, brand, and color, whose structured-data quality and availability varied across products.

5 EXPERIMENTS

The experiments evaluate matching and ranking with controlled comparisons of losses, embedding aggregation, tokenization, normalization, and baselines. Results favor end-to-end embeddings, a three-part L2 hinge loss, average pooling, n-gram features with OOV hashing, and efficient simpler models.

  • Loss functions: The 3-part hinge loss consistently improves matching over the 2-part loss, while both have similar ranking performance.Its separate treatment of impressed negatives improves separation between positives and random negatives; L2 variants outperform L1 variants.
  • Token aggregation: Average pooling performs similarly to or slightly better than LSTM and GRU while requiring significantly less training time.Short product-search queries and titles make averaging sufficient for their short-range dependencies.
  • Tokenization: Adding bigrams and character trigrams improves matching by capturing phrase-level information and robustness to spelling errors.Character trigrams provide additional generalization beyond unigram features.
  • Tokenization: OOV hashing improves matching over equal-parameter unigram models by generalizing to infrequent or unseen terms, despite adding parameters.The equal-parameter comparison isolates the benefit of hashing from parameter count.
  • Baseline comparison: The proposed model improves matching across four baselines, while more complex models have lower performance and substantially longer training and evaluation times.The baselines are DSSM, Match Pyramid, ARC-II, and the proposed model with frozen random embeddings.
  • Embedding training: Over 3X improvement in Recall@100 and MAP comes from end-to-end trained embeddings versus randomly initialized embeddings.Earlier GloVe and word2vec initialization methods performed worse than end-to-end training.

6 TRAINING ACCELERATION

The paper scales model training across GPUs by splitting the embedding computation and reducing communication to partial sums. This approach becomes advantageous for sufficiently large embedding dimensions.

  • The model-parallel design splits the embedding matrix across GPUs and uses average pooling with a Siamese architecture to increase throughput.The input is sent to all GPUs to retrieve partial token embeddings.
  • Training experiments used 8 NVIDIA Tesla V100 GPUs and reported median time scaled to 1 billion examples.
  • Beyond an embedding dimension of 512, splitting the model across GPUs reduces training time despite communication overhead.For dimensions below 256, model parallelism instead increases overall training time.

7 CONCLUSION AND FUTURE WORK

The paper presents a semantic product search model that improved product-discovery KPIs and summarizes practical lessons from its experiments. Future work targets higher precision and reduced reliance on online filtering heuristics.

  • The authors report that a 3-part hinge loss handles impressed-but-not-purchased products better than other loss variants.
  • Hashing unseen tokens improved precision across different tokenization strategies, while more training data improved offline metrics.
  • Future work aims to improve precision and eliminate additional heuristics used to filter irrelevant online results.

A ADDITIONAL EXPERIMENTS

Additional experiments evaluate embedding sharing and out-of-vocabulary hashing across tokenizations and loss functions. Shared embeddings generally improve matching, while larger OOV bins help matching but can hurt ranking.

  • Sharing the embedding layer tends to improve matching across multiple tokenizations and loss functions.The comparison held the number of model parameters constant.
  • Increasing the OOV bin size improves matching performance, whereas ranking performance peaks at smaller bin sizes.
  • OOV hashing supports generalization to unseen tokens by reducing spurious matches from collisions and retaining tokens that would otherwise be excluded.
Loading 1907.00937v1…