Source-linked AI summary
One Hierarchy, Two Systems: Semantic Product IDs for Discovery-Surface Ranking and Search-Page Query Reformulation
Steven Xu, Sanjyot Thete, Saathvik Dirisala, Raghav Saboo, Nimesh Sinha, Leo Shao, Elyse Winer, Sudeep Das, Martin Wang, Kyle MacDonald
TL;DR
Multi-merchant identifiers fragment behavioral evidence, while expert taxonomies can be too coarse for fine-grained discovery. The paper learns one hierarchical Semantic ID vocabulary from product content and lets ranking and query reformulation use it with separate behavioral and serving contexts. Across both applications, the hierarchy improves task-specific outcomes, including finer intent preservation, better suggestions, and reduced online search effort.
Problem
Equivalent products under different merchant-scoped identifiers fragment cross-merchant evidence, while expert-defined taxonomies may be too coarse for fine-grained ranking and reformulation.
Method
The paper learns a hierarchical Semantic ID representation from product-content embeddings and independently applies its prefixes to ranking features and concept-based query reformulation.
Results
Across ranking and query reformulation, Semantic IDs improve task-specific outcomes, including relevance, finer intent preservation, suggestion quality, and reduced online search effort.
Takeaways & Limitations
A shared semantic product hierarchy can support search and recommendation while each application retains its own behavioral signals and serving context.
Takeaways & Limitations
Semantic compression can merge unrelated behaviors, create irrelevant reformulation hubs, misrepresent original query intent, or separate useful substitutes near quantization boundaries.
Abstract
from arXiv · showhide
Multi-merchant e-commerce catalogs contain equivalent and related products under different merchant-scoped identifiers, fragmenting behavioral evidence across merchants. Expert-defined taxonomies, meanwhile, are often too coarse for fine-grained discovery. We investigate whether a single hierarchical Semantic ID (\sid{}) representation can support personalized ranking and query reformulation. Learned once from product-content embeddings, the hierarchy defines product concepts at multiple granularities that each application combines with its own behavioral and serving context. For ranking, we aggregate consumer affinity and product performance over \sid{} prefixes and derive sequence features for candidate products and consumer histories. Controlled ablations show improved offline relevance, while online evaluation of the full ranking treatment shows stronger top-slot add-to-cart engagement and broader exposure for less-popular products. For query reformulation, we ground queries and session transitions in \sid{} concepts, use the hierarchy for navigation and refinement, and filter suggestions against the merchant's assortment. Offline evaluation shows finer intent preservation than taxonomy and higher-quality suggestions than raw query-string transitions; online evaluation shows reduced search effort and earlier access to purchasable products. These results show that a shared semantic product hierarchy can support both recommendation and search while preserving the task-specific context required by each application.
1 Introduction
The paper proposes a shared hierarchical Semantic ID representation to address fragmented cross-merchant evidence and coarse product taxonomies in personalized ranking and query reformulation. Each application uses the learned hierarchy with its own behavioral data and objectives.
- Equivalent products receive different merchant-scoped listing IDs, fragmenting preference evidence across merchants.
- Expert-defined taxonomies share evidence across broad groups but can be too coarse for fine-grained personalization.
- String-level query transitions fragment evidence across synonymous expressions and can conflate distinct intents across verticals.
- Semantic IDs encode products as discrete code sequences whose shared prefixes represent progressively finer semantic groups.
- The proposed hierarchy is learned once from catalog content, then used independently for ranking features and concept-based hierarchical query reformulation.
- The study develops task-specific SID applications and reports more effective representations than expert-defined taxonomy for ranking and assortment-aware reformulation.
2 Related Work
Related work studies Semantic IDs for retrieval and recommendation, while query suggestion commonly models session transitions between raw query strings. This paper retains transition-graph properties while grounding search behavior in product concepts.
- Prior Semantic ID work encodes items with discrete codes derived from embeddings for generative retrieval and recommendation.
- Behavioral query suggestion commonly represents queries as transition graphs, sometimes adding random walks, click evidence, or semantic transfer for sparse and ambiguous queries.
3 A Shared, Learned Product Hierarchy
The shared hierarchy is built by residual-quantizing product-content embeddings into nested SID prefix partitions. Longer prefixes form more compact and semantically coherent product groups while retaining multiple levels of granularity.
- Product profiles concatenate catalog fields such as name, brand, and size before encoding into product embeddings.
- Residual-quantization K-means constructs a three-stage SID hierarchy with 512 centroids per stage from normalized embedding residuals.
- SID prefixes at depths 1, 2, and 3 define the L1, L2, and L3 hierarchy levels.
- Longer shared prefixes tend to identify more semantically similar products, creating nested concepts between listing IDs and expert taxonomies.
- The hierarchy supports ranking aggregation over multiple prefix depths and query reformulation through concept grounding and hierarchical transition modeling.
- Table 1 compares code-usage balance, cluster separation, and held-out semantic coherence across SID prefix lengths.
- Longer prefixes show lower DBI and higher held-out cosine similarity, while code-usage imbalance remains comparable across levels.
4 Personalized Item Ranking with Semantic IDs
The ranking system uses Semantic ID prefixes and sequence tokens to share behavioral evidence across product concepts at multiple granularities. Offline ablations improve relevance, while the online treatment increases add-to-cart engagement and reduces top-position popularity concentration, though online effects do not isolate SID contributions.
- Feature Design: The ranker combines SID-derived dense aggregates over prefix levels with sequence features representing candidate products and consumer histories.Prefix aggregates capture consumer affinity and global or regional performance, while sequence features model interactions between candidate products and historical preferences.
- Feature Design: Prefix n-grams provide aggregation keys, while SentencePiece subwords provide complementary units for item- and consumer-side sequence features.BPE is used because the Unigram LM vocabulary substantially overlaps with information already captured by prefix-keyed dense features.
- Feature Design: The item- and consumer-side tokens share an embedding table, enabling the ranker to relate candidate-product subwords to consumers’ historical preferences.Input representations are mean-pooled, and the pooled representations and embedding parameters are learned jointly with the ranking objective.
- Offline Evaluation: The offline comparison holds non-SID features fixed and shows that the full candidate substantially outperforms the SID-ablated candidate on MRR@5 and NDCG@5.The same pattern holds at K=3 and K=10; evaluation uses sessions containing at least one conversion.
- Online Evaluation: 0.31% relative increase in subtotal accompanies higher add-to-cart engagement across the carousel and its first three positions in the online treatment.The reported treatment includes SID-derived features alongside concurrent non-SID updates.
- Online Evaluation: 18.1% lower average historical popularity for the first-position item coincides with a 2.1 percentage-point reduction in first-position impressions for blockbuster items.The treatment therefore improves engagement while allocating less top-position exposure to historically dominant products.
- Interpretation: The online experiment cannot isolate SID effects because the production bundle also contains concurrent non-SID updates.Offline ablation and postexperiment analysis nevertheless indicate that SID-derived features account for most of the observed improvement.
5 Query Reformulation with Semantic IDs
The query reformulation system grounds search behavior in hierarchical SID concepts, combining lateral navigation, fine-grained refinement, rendering, and assortment filtering. Compared with taxonomy and raw query-string transitions, this approach preserves more intent distinctions and produces better suggestions while reducing search effort online.
- Method: SID-based reformulation grounds queries and session transitions in catalog concepts, supporting both lateral shopping moves and progressive product refinements.The system replaces query-string pairs with business-vertical-specific SID transitions and adds a separate path from L2 concepts to L3 children.
- Method: Query-to-concept grounding assigns dominant SID prefixes from query-associated add-to-cart events while retaining raw query nodes when evidence is fragmented.Grounding requires at least five events and applies a 30% dominance threshold, conditioned on business vertical.
- Method: Suggestions are rendered into consumer-facing queries, ranked by source-target embedding similarity, and filtered to concepts represented in the merchant’s active assortment.Language generation is limited to rendering; observed behavior and catalog concepts determine candidates.
- Offline Evaluation: 10.9% of intent-changing transitions collapse under SIDs versus 18.8% under taxonomy, preserving more behavioral signal for candidate generation.Transitions that map to the same concept become self-loops and cannot generate reformulation candidates.
- Offline Evaluation: Rank-one judged quality rises from 0.522 for the query-string graph to 0.734 for the catalog-grounded SID system.The comparison uses queries served by both systems and an LLM judge evaluating usefulness, target-text quality, and distinctiveness.
- Online Evaluation: Online, the reformulation module improves purchase MRR while reducing ATC position and search scroll depth, indicating earlier access to relevant purchasable items with less effort.The complete module was evaluated in a consumer-randomized experiment against a control without suggested-query reformulations.
6 Qualitative Analysis
SID compression enables evidence pooling across merchant listings and multiple semantic granularities, but broad or mixed prefixes can connect unrelated behaviors. The applications therefore restore task-specific information outside the SID itself.
- Shared Representation: Products with similar semantics can share SID codes across merchant-scoped listings, allowing behavioral evidence to transfer across merchants.The hierarchy also offers coarser prefixes for pooling and deeper prefixes for distinguishing specific product intents.
- Trade-offs: Semantic compression trades statistical support for possible semantic mixing, creating risks such as irrelevant graph hubs, concept-misaligned query grounding, and substitute or attribute mismatches in ranking.These issues arise when broad prefixes connect unrelated behaviors or when nearby products receive different codes.
- Mitigations: Ranking restores context through multi-depth prefixes and consumer- and product-specific signals, while reformulation uses business-vertical conditioning, query-specific L3 evidence, and assortment filtering.The SID hierarchy functions as a transferable semantic prior rather than a complete representation of task intent.
7 Cross-System Findings
Both applications use the same SID hierarchy to pool behavioral evidence semantically, then restore application-specific context before producing outputs. SIDs complement rather than replace exact identifiers and taxonomies.
- Shared Pattern: Both systems replace fragmented behavioral units with semantic SID units over which evidence can be pooled.They differ in hierarchy depths, attached behavioral signals, and the information restored before output.
- Hierarchy Use: Coarser prefixes provide support and transferability, while deeper prefixes recover specificity; ranking combines depths, whereas reformulation assigns L2 to navigation and L3 to refinement.The appropriate operating depth depends on the application.
- Relationship to Existing Representations: Exact product IDs remain necessary for identity and serving, and taxonomies remain useful for business organization; SIDs add a reusable fine-grained hierarchy across discovery tasks.Applications need not share the same model or decision logic to reuse the SID representation.
8 Conclusion
A single hierarchical SID vocabulary supports both recommendation and search by providing shared semantic product concepts. Each application combines that hierarchy with its own behavioral signals and serving context, improving task-specific outcomes without a jointly trained model.
- Conclusion: A single hierarchical SID vocabulary supports discovery systems spanning personalized recommendation and search.Ranking uses SID prefixes as transferable feature keys, while reformulation grounds queries, mines transitions, and generates assortment-constrained suggestions.
- Conclusion: Across both applications, SIDs improve task-specific outcomes while allowing search and recommendation to share a representation without sharing a jointly trained model.SIDs complement exact identifiers and taxonomies with a fine-grained hierarchy that applications combine with their own behavioral and serving context.
A Qualitative Examples and Cross-System Comparison
Table 6 illustrates SID transferability and hierarchical granularity, while Table 7 compares how personalized ranking and query reformulation reuse the shared hierarchy.
- Table 6 illustrates how SID transferability and hierarchical granularity benefit the two applications.
- Table 7 summarizes how personalized ranking and query reformulation reuse the shared SID hierarchy.