PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Embedding Representation Stability in Recommendation Systems with Semantic ID

Carolina Zheng, Minhui Huang, Dmitrii Pedchenko, Kaushik Rangadurai, Siyu Wang, Gaby Nahum, Jie Lei, Yang Yang, Tao Liu, Zutian Luo, Xiaohan Wei, Dinesh Ramasamy, Jiyan Yang, Yiping Han, Lin Yang, Hangjun Xu, Rong Jin, Shuang Yang

arXiv:2504.02137v1cs.IRcs.AI

TL;DR

Industrial recommendation must handle high item cardinality, skewed impressions, and changing item IDs. This paper applies Semantic ID prefix-ngram parameterization to item representations and reports improved offline stability and production performance. In Meta’s production ads recommendation system, the features yield a 0.15% online performance gain and reduce prediction variance.

  • Problem

    High item cardinality, impression skew, and ID drifting challenge item embedding representations in industrial recommendation systems.

    Source: §1 · p. 1, §5 · p. 4

  • Method

    The paper proposes Semantic ID prefix-ngram, a token parameterization technique built on Semantic ID, and integrates its features into Meta’s production ads recommendation system.

    Source: §1 · pp. 1–2

  • Results

    Semantic ID mitigates embedding instability versus random hashing and individual embeddings, while production features yield a 0.15% online performance gain and reduced downstream ad-delivery variance.

    Source: §1 · p. 2, §8 · p. 8

  • Takeaways & Limitations

    The production findings support using Semantic ID features in Meta’s ads recommendation system, where they improve online prediction stability.

    Source: §1 · p. 2

Abstract

from arXiv · show

The exponential growth of online content has posed significant challenges to ID-based models in industrial recommendation systems, ranging from extremely high cardinality and dynamically growing ID space, to highly skewed engagement distributions, to prediction instability as a result of natural id life cycles (e.g, the birth of new IDs and retirement of old IDs). To address these issues, many systems rely on random hashing to handle the id space and control the corresponding model parameters (i.e embedding table). However, this approach introduces data pollution from multiple ids sharing the same embedding, leading to degraded model performance and embedding representation instability. This paper examines these challenges and introduces Semantic ID prefix ngram, a novel token parameterization technique that significantly improves the performance of the original Semantic ID. Semantic ID prefix ngram creates semantically meaningful collisions by hierarchically clustering items based on their content embeddings, as opposed to random assignments. Through extensive experimentation, we demonstrate that Semantic ID prefix ngram not only addresses embedding instability but also significantly improves tail id modeling, reduces overfitting, and mitigates representation shifts. We further highlight the advantages of Semantic ID prefix ngram in attention-based models that contextualize user histories, showing substantial performance improvements. We also report our experience of integrating Semantic ID into Meta production Ads Ranking system, leading to notable performance gains and enhanced prediction stability in live deployments.

1 Introduction

Industrial recommendation faces high item cardinality, impression skew, and rapidly changing IDs, while random hashing can destabilize embeddings. The paper studies Semantic ID and proposes prefix-ngram parameterization, reporting offline and production gains.

1 Introduction

  • Random hashing can produce contradictory gradient updates, while ID drifting leaves new items with essentially random weights and discards learning from retired items.These challenges arise alongside high item cardinality, impression skew, and short item lifetimes.Source: §1 · p. 1
  • The paper investigates Semantic ID as a fixed, semantically meaningful item space derived from hierarchical clusters of content similarity.The authors characterize item cardinality, impression skew, and ID drifting and connect them to embedding stability.Source: §1 · pp. 1–2
  • Semantic ID prefix-ngram adds a token parameterization technique that brings significant performance gains over the original Semantic ID.The experiments use a simplified version of Meta’s production ads ranking model.Source: §1 · p. 1
  • The authors report productionizing Semantic ID prefix-ngram as both sparse and sequential features, with online performance gains and improved prediction stability.The introduction describes adding these features to Meta’s production ads recommendation system.Source: §1 · p. 2
  • Semantic ID improves generalization and distribution-shift sensitivity versus random hashing, with most gains coming from the item-distribution long tail.The authors also report outsized gains when Semantic ID is used in models that contextualize user interaction histories.Source: §1 · p. 2
  • In production, Semantic ID prefix-ngram features yield a 0.15% online performance gain and significantly reduce prediction variance for the same item.The features serve as the top sparse features by feature importance; the authors connect reduced variance to advertiser trust and stable item rankings.Source: §1 · p. 2

2 Related Work

Prior work addresses item representation through hashing and tokenization, while this paper takes a holistic approach to a stable ID space intended to reduce reliance on hashing.

2 Related Work

  • Random hashing is a simple response to high item cardinality, but collisions can be undesirable; alternatives include collision-free hashing, double hashing, and learned hash functions.Double hashing reduces memory use but retains random collisions; collision-free hashing dynamically frees embeddings for retired items.Source: §2 · p. 2
  • Work on impression skew uses contrastive learning or clustering, approaches the authors describe as complementary to designing a stable ID space.Their stated aim is to minimize the need for hashing and address embedding representation shifts directly.Source: §2 · p. 2
  • Semantic ID builds on tokenization and vector quantization work, including RQ-VAE-based approaches, and is adapted here as a stable ID method for online recommendation.The cited Semantic ID work reports benefits in generalization performance and sequential recommendation, respectively.Source: §2 · p. 2

3 Ranking Model Overview

The ranking model treats recommendation as classification over user- and item-side features. Its DLRM-based architecture processes sparse, dense, and history features, then scores their interactions; normalized entropy is the performance metric.

3 Ranking Model Overview

  • Recommendation is posed as binary classification of user-item features associated with an impression or conversion, labeled by whether the user interacted or converted.The data point includes both user-side and item-side features.Source: §3 · p. 2

3.1 Model

  • The DLRM-based model embeds sparse, dense, and history features separately, combines their vectors for interaction modeling, then produces a probability through an MLP and sigmoid.Sparse feature sets are sum-pooled; user histories cover up to three months and more than one billion items, and are contextualized with Bypass, Transformer, or PMA aggregation.Source: §3.1 · pp. 2–3

3.2 Metrics

  • Normalized entropy is cross-entropy divided by the cross-entropy of predicting the positive-label mean frequency, and lower values are better.The metric is used to measure model performance.Source: §3.2 · p. 3

4 Semantic ID and Parameterizations

Semantic ID clusters items by content so related items can share information, and its token parameterization determines how hierarchical codes contribute to embeddings. Prefix-ngram performs best among the tested parameterizations.

4.1 Overview

  • Semantic IDs are sequences of coarse-to-fine discrete codes learned by training an RQ-VAE on content embeddings from text, images, or video.The RQ-VAE encodes each content embedding, quantizes its latent representation into codes, and decodes the codes to reconstruct the embedding.Source: §4.1 · p. 3

4.2 Token Parameterization

  • The codes form a hierarchy: earlier tokens capture broad categories, while later tokens refine them, such as from food to pizza to pizza-language combinations.This lets the model receive different amounts of Semantic ID information, while using all possible code tuples can be infeasible because of their cardinality.Source: §4.2 · pp. 3–4
  • Token parameterization maps Semantic IDs to embedding-table rows; Prefix-ngram uniquely includes all possible tuples across granularities.The parameterizations trade off token cardinality against how much Semantic ID information the model receives.Source: §4.2 · p. 4
  • Prefix-ngram is the best parameterization; increasing its depth and RQ-VAE cardinality improves NE performance.The authors attribute its advantage to incorporating hierarchical clustering in the embedding-table mapping, allowing knowledge sharing among more items than a flat mapping.Source: §4.2 · p. 4, Table 2 · p. 4

5 Item Impression Distribution Issues

Raw IDs present challenges from item cardinality, impression skew, and rapid turnover, which can undermine embedding quality and stability. Semantic IDs use shared content-based codes to support knowledge transfer and reduce representation drift when broad semantic categories remain stable.

Item cardinality

  • Randomly assigning many raw IDs to a limited embedding table creates collisions that can harm embedding quality and impede effective knowledge sharing.The number of distinct items can exceed the feasible embedding-table size, and raw IDs are randomly generated when items are created.Source: §5 · p. 4

Impression skew

  • The top 0.1% of items account for 25% of impressions, the next 5.5% for 50%, and the remaining 94.4% for 25%.Tail items have few training examples, and random hashing does not let them effectively share knowledge with semantically similar head or torso items.Source: §5 · p. 4

ID drifting

  • Half of the original item corpus exits after six days as an equal number of new items enters, while random-hash embeddings represent different items over time.This turnover, called raw ID drifting, reflects the short lifespan of many ads and is associated with embedding representation drift.Source: §5 · p. 4, Figure 3 · p. 5

Item representation with Semantic ID

  • When broad product categories remain temporally stable, new and retired ads can share Semantic IDs or prefixes, reducing distribution drift in Semantic ID space.The new ad's fine-grained content may differ from the retired ad's even when their broad semantic category remains the same.Source: §5 · pp. 4–5
  • Semantically similar tail and head or torso items can share Semantic IDs or prefixes, allowing knowledge transfer and yielding a less skewed item impression distribution.The resulting embeddings are equal or similar when items share codes or prefixes.Source: §5 · p. 5

6 Offline Experiments

The offline experiments compare individual embeddings, random hashing, and Semantic ID in a simplified ads-ranking model. They examine item segments, representation geometry, and user-history aggregation, finding benefits for tail-item generalization and contextualized history modeling.

6.1 Baselines

  • The experiments compare individual embeddings, random hashing, and Semantic ID using a simplified production ads-ranking model.The target item is the only sparse feature retained; user-history features are randomly hashed in the segment and representation analyses.Source: §6 · p. 5, §6.1 · p. 5

6.2 Segment Analysis

  • Semantic ID improves generalization for tail items, is neutral for head items, and is slightly beneficial for torso items compared with the baselines.The experiments also evaluate new items; Semantic ID's largest gains there are over random hashing and individual embeddings.Source: §6.2 · pp. 5–6
  • Compared with random hashing, Semantic ID scales better with training data over a longer period.The comparison uses models trained on 20 days versus the last four days of the period.Source: §6.2 · p. 6, Table 4 · p. 6

6.3 Item Representation Space

  • Semantic ID partitions have lower intra-cluster variance than random hashing, while pairwise-distance results are mixed.The authors hypothesize that low distances among the top 1,000 clusters reflect RQ-VAE placing multiple centroids in dense regions.Source: §6.3 · p. 6, Table 5 · p. 6

6.4 User History Modeling

  • Semantic ID with PMA or Transformer aggregation brings larger gains than with Bypass, which does not contextualize the history sequence.Semantic ID models also show lower entropy, self-attention, and padding-token attention, and higher attention on the first source token.Source: §6.4 · pp. 6–7, Table 6 · p. 7

7 Productionization

Semantic ID features are productionized in Meta’s ads recommendation system, with codes generated from content embeddings and served as sparse and sequential features. The production features yield online performance gains, correlate prediction similarity with semantic similarity, and reduce A/A prediction differences.

7 Productionization

  • Semantic ID features have been in Meta’s ads recommendation system for more than a year and rank among its top sparse features by feature importance.The online serving pipeline supports both sparse and sequential features.Source: §7 · p. 7

7.1 Offline RQ-VAE Training

  • Production RQ-VAEs use six layers, codebook size 2048, prefix-5gram, and an embedding table of O(50M); a frozen checkpoint serves online.The models are trained offline on ad IDs and content embeddings sampled from the past three months.Source: §7.1 · p. 7

7.2 Online Semantic ID Serving System

  • At ad creation, content embeddings are converted to Semantic ID signals, stored, and later joined with item and user-history IDs to form serving features.The precomputed features are fetched for ranking requests.Source: §7.2 · p. 7, Figure 4 · p. 7

7.3 Production Performance Improvement

  • Across multiple ads-ranking models, Semantic ID features yield a 0.15% gain in the top-line online performance metric.The features include six sparse and one sequential feature from text, image, and video embeddings; the reported NE gain is for the flagship model.Source: §7.3 · p. 7, Table 8 · p. 7

7.4 Semantic and Prediction Similarity

  • Deeper Semantic ID prefixes monotonically reduce click loss rate, supporting a correlation between prediction similarity and semantic similarity.The click-loss-rate decrease is summarized in Figure 5.Source: §7.4 · p. 8, Figure 5 · p. 8

7.5 A/A Variance

  • The production model with six Semantic ID sparse features achieves a 43% reduction in average AAR versus the same model without those features.The authors believe most of the reduction comes from tail items.Source: §7.5 · p. 8

8 Conclusion

The paper presents Semantic ID as a stable item-representation space and proposes prefix-ngram parameterization to improve its ranking performance. Offline and production results report mitigated embedding instability, performance gains, and reduced downstream ad-delivery variance.

8 Conclusion

  • Semantic ID prefix-ngram significantly improves Semantic ID’s performance in ranking models.The conclusion also reports mitigated embedding-instability effects offline and production performance gains with reduced downstream ad-delivery variance.Source: §8 · p. 8
  • Offline experiments find that Semantic ID mitigates harmful embedding-instability effects compared with random hashing and individual-embedding baselines.Source: §8 · p. 8
  • The production system obtains significant performance gains and reduced downstream ad-delivery variance from Semantic ID features.Source: §8 · p. 8

A Aggregation Module Architectures

The section describes Transformer and PMA aggregation over embedding sequences. PMA changes the attention queries while retaining the Transformer module’s equations.

A Aggregation Module Architectures

B Distributions of Clicks for Semantic ID

The marginal click distribution is less skewed in Semantic ID space than in raw ID space.

B Distributions of Clicks for Semantic ID

Loading 2504.02137v1…