PaperlayerView exact revision on arXiv

Source-linked AI summary

Embedding-based Retrieval in Facebook Search

Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padmanabhan, Giuseppe Ottaviano, Linjun Yang

arXiv:2006.11632v2cs.IR

TL;DR

Facebook Search must address semantic matching while modeling query text together with the searcher and context, rather than relying mainly on Boolean matching. The paper presents unified embeddings, hybrid retrieval, and end-to-end optimization for deploying embedding-based retrieval, reporting significant metric gains in online A/B experiments and outlining production lessons and future directions.

  • Problem

    Facebook Search requires semantic matching that accounts for query text, the searcher, and context beyond traditional term matching.

  • Method

    The paper develops unified embeddings for personalized search, integrates embedding KNN with Boolean matching in an inverted-index retrieval stack, and applies full-stack optimization.

  • Results

    Significant metric gains were observed for embedding-based retrieval on Facebook Search verticals in online A/B experiments.

  • Takeaways & Limitations

    Production deployment opens a path for continued retrieval-quality improvement through semantic embedding learning and provides practical experience for implementing embedding-based retrieval.

  • Takeaways & Limitations

    Online hard negative mining may fail to produce sufficiently hard negatives because random samples have a low probability of containing them.

Abstract

from arXiv · show

Search in social networks such as Facebook poses different challenges than in classical web search: besides the query text, it is important to take into account the searcher's context to provide relevant results. Their social graph is an integral part of this context and is a unique aspect of Facebook search. While embedding-based retrieval (EBR) has been applied in eb search engines for years, Facebook search was still mainly based on a Boolean matching model. In this paper, we discuss the techniques for applying EBR to a Facebook Search system. We introduce the unified embedding framework developed to model semantic embeddings for personalized search, and the system to serve embedding-based retrieval in a typical search system based on an inverted index. We discuss various tricks and experiences on end-to-end optimization of the whole system, including ANN parameter tuning and full-stack optimization. Finally, we present our progress on two selected advanced topics about modeling. We evaluated EBR on verticals for Facebook Search with significant metrics gains observed in online A/B experiments. We believe this paper will provide useful insights and experiences to help people on developing embedding-based retrieval systems in search engines.

1 INTRODUCTION

Facebook Search applies embedding-based retrieval to personalized social search, combining semantic representations of the query, searcher, context, and documents with an inverted-index retrieval stack. The paper addresses modeling, hybrid serving, and end-to-end optimization, reporting significant online A/B metric gains.

  • 1 INTRODUCTION: Embedding-based retrieval represents queries and documents as dense semantic vectors and converts retrieval into nearest-neighbor search in embedding space.Embeddings are introduced as dense representations of sparse ID vectors that can capture semantics.
  • 1 INTRODUCTION: At Facebook scale, EBR must handle billions or trillions of indexed documents while combining embedding retrieval with term matching in the retrieval layer.These constraints affect both embedding training and serving, unlike smaller ranking stages that process hundreds of documents per session.
  • 1 INTRODUCTION: Facebook Search must model query text together with the searcher and surrounding context because social-search intent is not determined by text alone.The paper frames this as a text, user, and context understanding problem rather than text-only embedding.
  • 1 INTRODUCTION: The unified embedding model uses a two-sided representation: a search request containing query text, searcher, and context on one side, and the document on the other.Training data is mined from search logs and features are extracted from searchers, queries, contexts, and documents.
  • 1 INTRODUCTION: The hybrid retrieval framework integrates Faiss vector quantization with inverted-index retrieval to jointly score embedding and Boolean matches.It avoids the cost and maintenance burden of separate candidate sets while supporting embedding KNN constrained by term matching.
  • 1 INTRODUCTION: Significant metric gains were observed in online A/B experiments after evaluating EBR on Facebook Search verticals.The paper also describes later-stage optimization that incorporates embeddings into ranking layers and uses feedback from retrieval results.

2 MODEL

The model formulates retrieval as nearest-neighbor ranking in an embedding space, using unified embeddings to combine query text, searcher information, context, and documents. It trains and evaluates these models for personalized retrieval, with label and loss choices materially affecting recall.

  • Retrieval objective: Retrieval is formulated as maximizing recall among top-K results for a query’s target document set.Target results may be based on user clicks or human relevance ratings.
  • Embedding model: Query and documents are encoded into dense vectors, ranked by cosine similarity, and trained with triplet loss.The model uses separate query and document encoders by default, with optional parameter sharing.
  • Unified embedding: Unified embedding extends text embedding by incorporating textual, social, and contextual features for personalized Facebook search.Query-side examples include searcher location and social connections; document-side examples include aggregated group location and social clusters.
  • Evaluation: Offline recall@K on KNN search over the whole index is used to evaluate models before online experiments.The evaluation set was built from 10000 sampled search sessions with query and target-result pairs.
  • Loss function: Triplet-loss margin tuning changes KNN recall by 5-10% across training tasks.The optimal margin varies substantially between tasks.
  • Training labels: Using non-click impressions as negatives caused an absolute 55% recall regression versus random negatives for the people embedding model.The paper attributes this to hard-negative bias, which makes the training set unrepresentative of the mostly easy negatives in the index.
  • Training labels: Click and impression positives produced similar recalls at equal data volume, while adding impression data produced no additional gain.The click-positive, random-negative setup was retained as a reasonable baseline for further hard-mining experiments.

3 FEATURE ENGINEERING

Feature engineering improves unified embeddings beyond text-only representations by adding character and word n-grams, location, and social-graph information. Gains vary by vertical and depend on identifying informative features.

  • Feature engineering: Unified embeddings improved recall by +18% for events search and +16% for groups search versus text embeddings.The paper reports these gains across verticals and notes that effectiveness depends on informative feature design.
  • Text features: Character n-grams provide compact, out-of-vocabulary-robust text representations, while adding word n-grams yields a further +1.5% recall gain.Word n-gram vocabularies can be very large, so hashing is used to reduce embedding-table size.
  • Text features: Text embeddings address fuzzy matching and optional terms that Boolean matching may miss.Examples include matching “kacis creations” with Kasie’s creations and dropping “nw” from “mini cooper nw.”
  • Location features: Adding location features ranks groups sharing the searcher’s location more highly, such as groups associated with Louisville, Kentucky.The comparison is between text-only and text-plus-location embeddings for groups search.
  • Social embedding features: A separate social-graph embedding model embeds users and entities so unified embeddings can incorporate broader social information.This supplements social-graph information that may not otherwise be fully available to the unified model.

4 SERVING

The serving system integrates approximate nearest-neighbor search into Facebook’s inverted-index retrieval stack, using quantization and hybrid Boolean–embedding queries. ANN configurations are tuned offline and online, with radius-based matching selected for production.

  • ANN serving: Inverted-index ANN reduces embedding storage through quantization and integrates more easily with Facebook’s existing retrieval system.Faiss quantizes vectors, while NN search runs in the existing inverted-table scanning system.
  • ANN tuning: Coarse quantization, product quantization, and nprobe are the main ANN components and parameters requiring tuning.The system compares coarse-clustering choices such as IMI and IVF, product-quantization variants, and search breadth.
  • ANN tuning: ANN tuning uses scanned-document count as a performance proxy because imbalanced clusters make equal num_cluster and nprobe settings scan different amounts of data.This issue is especially pronounced for IMI, where roughly half the clusters contain only a few samples.
  • ANN evaluation: 1-recall@10 measures the average recall of retrieving the exact-search top result within the ANN search’s top 10 results.The measure is used to evaluate ANN accuracy alongside scanned-document cost.
  • ANN tuning: Product-quantizer accuracy gains are limited beyond pq_bytes = d/4, while larger codes increase memory and latency.The recommended code size is therefore d/4 bytes.
  • Hybrid retrieval: Embedding KNN is exposed as a Boolean-language operator, enabling hybrid retrieval expressions that combine semantic matching with term constraints.The operator can support fuzzy matching for spelling variations while retaining other retrieval-expression conditions.
  • Hybrid retrieval: Radius-based matching provides the production performance–quality tradeoff, whereas top-K mode may require scanning the whole index.Radius mode also permits constrained NN search under other parts of the matching expression.
  • Model serving: The two-sided embedding model is split into online query inference and offline document inference before embeddings are published with metadata.Query embeddings are generated in real time; document embeddings are batch-computed with Spark.

5 LATER-STAGE OPTIMIZATION

Because later rankers were designed for existing retrieval scenarios, they may rank new embedding-retrieved results sub-optimally. The proposed optimization propagates embedding information into later ranking stages.

  • Problem: Later ranking stages can rank embedding-retrieved results sub-optimally because they were designed for existing retrieval scenarios.Embedding retrieval operates at the bottom of Facebook’s multi-stage ranking stack, whose later models refine its outputs.
  • Embedding as ranking feature: Embedding similarities can be propagated as ranking features to help rankers recognize new results and provide a generic semantic similarity measure.The explored features include cosine similarity, Hadamard products, and raw embeddings.

6 ADVANCED TOPICS

The paper advances embedding-based retrieval through hard mining and ensemble models, addressing retrieval-scale training challenges and differing recall–precision needs. Experiments show gains from carefully selected negatives and positives, and from combining models across hardness levels.

  • 6.1 Hard Mining: Hard mining addresses retrieval training over diverse text, semantic, and social-match distributions where classification-oriented techniques do not directly apply.The paper separates hard mining into hard negative mining and hard positive mining.
  • 6.1.1 Hard negative mining (HNM).: Online hard negative mining improved recall by +8.38% for people search, +7% for groups search, and +5.33% for events search.The best setting used at most two hard negatives per positive; more than two began to regress model quality.
  • 6.1.1 Hard negative mining (HNM).: Mixing easy and hard negatives improved recall, with the easy:hard ratio continuing to help until 100:1, while hard-to-easy transfer learning produced further gains.The results support retaining easy examples because retrieval spans mixed levels of hardness.
  • 6.1.1 Hard negative mining (HNM).: Approximate nearest neighbor search made offline hard-negative generation practical, and searching one random shard was sufficient for effective semi-hard negatives.Offline mining iteratively generates top-K results, selects hard negatives, and retrains the embedding model.
  • 6.1.2 Hard positive mining.: Hard positives mined from searchers’ activity logs matched click-training recall using only 4% of its data and improved recall further when combined with impressions.These positives target results that production retrieval had missed despite subsequent positive activity.
  • 6.2 Embedding Ensemble: Ensemble models combine easy and hard training regimes through weighted concatenation or cascades, balancing broad recall with precision among similar candidates.Weighted ensemble serving combines model-specific cosine similarities using empirical weights, while cascade models rerank first-stage outputs.

7 CONCLUSIONS

The paper presents unified embedding and embedding-based retrieval for social search, along with lessons from initial end-to-end optimization. It identifies deeper modeling, stack-specific improvement, and universal embedding models as future directions.

  • The paper presents unified embedding to model semantics for social search and implements embedding-based retrieval in a classical inverted-index search system.
  • The authors describe model improvement, serving-algorithm tuning, and later-stage optimization as experience from the system’s first implementation.They highlight hard mining and embedding ensemble as selected modeling topics.
  • The authors characterize production deployment as opening a path for continued retrieval-quality improvement through semantic embedding learning.
  • Go deep: Future work includes deeper models and stack-specific optimization guided by full-stack failure analysis.The proposed directions include BERT or task-specific models, serving-algorithm tuning, and ranking-model improvement.
  • Go universal: Future work also includes universal text and query embedding models across tasks and use cases.
Loading 2006.11632v2…