Source-linked AI summary
MMQ: Multimodal Mixture-of-Quantization Tokenization for Semantic ID Generation and User Behavioral Adaptation
Yi Xu, Moyu Zhang, Chenxuan Li, Zhihao Liao, Haibo Xing, Hao Deng, Jinxin Hu, Yu Zhang, Xiaoyi Zeng, Jing Zhang
TL;DR
Static ItemIDs face challenges in dynamic corpora and long-tail settings, while multimodal tokenizers must balance cross-modal synergy with modality-specific information and behavioral alignment. MMQ combines shared-specific experts with behavior-aware fine-tuning. It consistently outperforms state-of-the-art baselines in generative retrieval and discriminative ranking.
Problem
ItemIDs face stability and scalability challenges in dynamic corpora and poorly generalized representations for long-tail items, while multimodal tokenizers must balance synergy with modality-specific details.
Source: §1 · pp. 1–2Method
MMQ trains shared and modality-specific experts with orthogonal regularization, then fine-tunes semantic IDs using downstream recommendation objectives while preserving modality information.
Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 3Results
MMQ consistently outperforms state-of-the-art baselines in generative retrieval and discriminative ranking across industrial datasets, public benchmarks, and online A/B tests.
Source: §6 · p. 10Takeaways & Limitations
MMQ offers a scalable, flexible approach to semantic ID generation, with potential applications in personalized search, advertising, and cross-domain content understanding.
Source: §6 · p. 10The two-stage framework adds training complexity, may require substantial multimodal data, and its task-dependent fine-tuning may limit transfer to unrelated tasks.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Recommender systems traditionally represent items using unique identifiers (ItemIDs), but this approach struggles with large, dynamic item corpora and sparse long-tail data, limiting scalability and generalization. Semantic IDs, derived from multimodal content such as text and images, offer a promising alternative by mapping items into a shared semantic space, enabling knowledge transfer and improving recommendations for new or rare items. However, existing methods face two key challenges: (1) balancing cross-modal synergy with modality-specific uniqueness, and (2) bridging the semantic-behavioral gap, where semantic representations may misalign with actual user preferences. To address these challenges, we propose Multimodal Mixture-of-Quantization (MMQ), a two-stage framework that trains a novel multimodal tokenizer. First, a shared-specific tokenizer leverages a multi-expert architecture with modality-specific and modality-shared experts, using orthogonal regularization to capture comprehensive multimodal information. Second, behavior-aware fine-tuning dynamically adapts semantic IDs to downstream recommendation objectives while preserving modality information through a multimodal reconstruction loss. Extensive offline experiments and online A/B tests demonstrate that MMQ effectively unifies multimodal synergy, specificity, and behavioral adaptation, providing a scalable and versatile solution for both generative retrieval and discriminative ranking tasks.
1 INTRODUCTION
Static ItemIDs face challenges in dynamic, long-tail settings, while multimodal semantic IDs must balance cross-modal synergy with modality-specific information and user behavior. MMQ addresses these challenges with shared-specific experts and behavior-aware fine-tuning.
1 INTRODUCTION
- Static ItemIDs face stability and scalability challenges in dynamic corpora, while long-tail sparsity leads to poorly generalized representations.Source: §1 · p. 1
- Content-derived semantic IDs map similar items into a shared space, supporting knowledge transfer, new or long-tail recommendations, and fixed-vocabulary scalability.Source: §1 · p. 1
- Text and vision contain synergistic and modality-unique information; alignment can obscure modality-specific details, whereas separation can miss cross-modal synergy.Source: §1 · pp. 1–2
- Semantic IDs learned in semantic space may misrepresent behavioral preferences, motivating dynamic adaptation to downstream recommendation tasks.Source: §1 · p. 2
- MMQ uses modality-shared and modality-specific experts with orthogonal regularization, then behavior-aware fine-tuning to align semantic IDs with recommendation objectives.A multimodal reconstruction loss preserves modality information during fine-tuning.Source: §1 · p. 2
2 RELATED WORK
Related work develops content-derived semantic tokens through quantization and clustering, explores collaborative signals, and applies different strategies to multimodal tokenization. Existing multimodal approaches face a trade-off between preserving modality uniqueness and capturing synergy.
2 RELATED WORK
- Semantic indexing generates discrete item tokens from content, commonly using quantization or clustering techniques.Source: §2 · p. 2
- TIGER [17] uses RQ-VAE for semantic IDs, while SPM-SID [24] and PMA [25] combine IDs and language-model subwords using n-grams or SentencePiece.Source: Figure 1 · p. 2
- RQ-Kmeans [18] addresses RQ-VAE collapse, unordered tokenization [20] supports faster parallel generation, and VQ-Rec [26] and ActionPiece [27] use alternative tokenization strategies.VQ-Rec applies OPQ to BERT text vectors; ActionPiece uses unordered feature sets and context-aware token merging.Source: Figure 1 · p. 2
- Recent approaches incorporate collaboration or behavior into tokenization: EAGER [28] combines semantic information with collaborative signals, ColaRec uses interaction signals in hierarchical K-means clustering, and IDGenRec [32] generates semantic IDs from behavior-sequence descriptions.Source: §2 · pp. 2–3
- Modality alignment risks losing uniqueness, while modality separation risks ignoring synergy; prior work had not unified their explicit modeling while mitigating quantization issues.Source: §2 · p. 3
3 METHODOLOGY
MMQ converts multimodal item embeddings into semantic IDs through shared and modality-specific experts, then fine-tunes the tokenizer with downstream recommendation objectives. Reconstruction and orthogonality losses preserve multimodal information while encouraging distinct expert representations.
3.1 Problem Formulation
- MMQ maps pretrained text and vision embeddings to a discrete sequence of semantic IDs for each item.The tokenizer is intended to preserve both synergistic and modality-unique information.Source: §3.1 · p. 3
3.2.1 modality-shared and modality-specificscalability Expert.
- The shared-specific tokenizer combines modality-shared experts for cross-modal synergy with modality-specific experts for unique modality properties.Specific experts are dynamically weighted by an input-dependent gating network, while inputs are deterministically assigned to shared experts.Source: §3.2.1 · p. 3
3.2.2 Orthogonal-Constrained Vector-Qantized Code Learning.
- Shared experts capture common cross-modal knowledge and can reduce redundancy among specific experts, supporting more parameter-efficient specialization.The tokenizer also uses orthogonality constraints to encourage distinct expert projections and complementary input representations.Source: §3.2.1 · p. 3, §3.2.2 · p. 5
3.2.1 modality-shared and modality-specificscalability Expert.
- Cosine similarity replaces L2 distance in codebook lookup so assignments depend on embedding direction rather than magnitude.This is intended to reduce scale mismatches across modality-specific encoders and better match multimodal embedding geometry.Source: §3.2.1 · pp. 3–4
3.2.2 Orthogonal-Constrained Vector-Qantized Code Learning.
- MMQ trains semantic IDs with multimodal reconstruction and orthogonality losses to preserve shared semantics and modality-unique cues while reducing expert redundancy.An auxiliary model-specific reconstruction loss also facilitates learning modality-specific latent representations.Source: §3.2.2 · pp. 4–5
3.3 Behavior-aware Fine-tuning Stage
- Behavior-aware fine-tuning uses differentiable soft indices during backpropagation to jointly optimize the tokenizer and recommendation model while retaining discrete IDs for forward prediction.A reconstruction objective is included to preserve the tokenizer’s semantic knowledge during joint optimization.Source: §3.3 · p. 5
4 EXPERIMENTS
The experiments evaluate MMQ on an industrial advertising dataset and Amazon Beauty, covering quantization quality and downstream generative retrieval, discriminative ranking, and online performance. Comparisons are organized around modality-aligned and modality-separated tokenization paradigms and mainstream quantization methods.
4 EXPERIMENTS
- The experiment questions cover baseline performance, component contributions, long-tail recommendations, transfer of behavior-aware fine-tuning, scaling with ID count, and parameter efficiency.They span both generative and discriminative recommendation tasks.Source: §4 · pp. 5–6
4.1.1 Dataset.
- Experiments assess MMQ on an industrial dataset and the Amazon Product Reviews Beauty subset.The industrial data cover 30 million users and 40 million advertisements; Beauty uses chronological sequences and a 5-core filter for generative retrieval.Source: §4.1.1 · p. 6
4.1.2 Evaluation Metrics.
- Evaluation covers reconstruction fidelity, codebook utilization, token-distribution entropy, retrieval and ranking quality, and online advertising outcomes.Generative retrieval uses Recall@N and NDCG@N at N=5,10; discriminative ranking uses AUC and GAUC; online metrics include revenue, CVR, and orders.Source: §4.1.2 · p. 6
4.1.3 Baselines.
- Tokenizer baselines represent both modality-aligned and modality-separated paradigms and include mainstream semantic-ID quantization methods.The described methods include RQ-VAE, RQ-Kmeans, and OPQ.Source: §4.1.3 · p. 6
4.1.4 Experiment Setup.
- HeterRec is the generative retrieval backbone, while PPNet is used for discriminative ranking.The tokenizer uses codebooks of size 3,000 on industrial data and 100 on public data, with six semantic IDs in all settings.Source: §4.1.4 · p. 6
4.2 Overall Performance (RQ1)
MMQ outperforms modality-aligned and modality-separated baselines across the evaluated tasks and datasets. Learned semantic IDs also generally outperform traditional ItemID-based ranking, while modality-separated methods fare better on the large industrial dataset than on the smaller Amazon Beauty dataset.
4.2 Overall Performance (RQ1)
- Modality-separated methods consistently outperform modality-aligned methods on the large industrial dataset across ranking and retrieval, while the gap is less pronounced on Amazon Beauty.The authors suggest that forcing modalities into one shared space increases quantization difficulty as item vocabulary and data complexity grow.Source: §4.2 · pp. 6–7
- MMQ surpasses both modality-aligned and modality-separated baselines across tasks and datasets, with the lowest reconstruction loss and high codebook utilization.Its semantic IDs yield superior accuracy in generative retrieval and discriminative ranking.Source: Table 1 · p. 7, §4.2 · p. 7
- Learned semantic IDs generally outperform the traditional ItemID-based PPNet backbone in discriminative ranking on both industrial and public datasets.The passage reports a consistent trend but gives no numeric values.Source: §4.2 · pp. 7–8
4.3 Ablation Study (RQ2)
Ablations indicate that orthogonal regularization, model-specific reconstruction loss, cosine quantization, and behavior-aware fine-tuning each contribute to MMQ's representation quality or downstream performance.
4.3 Ablation Study (RQ2)
- Removing orthogonal regularization sharply reduces codebook utilization and token-distribution entropy, indicating expert collapse and redundant semantic IDs.The authors connect this loss of specialization with weaker downstream recommendation performance.Source: §4.3 · p. 8
- Removing the modality-specific auxiliary reconstruction loss significantly degrades codebook utilization and reconstruction loss.The loss guides specific encoders toward discriminative representations and the shared encoder toward common characteristics.Source: §4.3 · p. 8
- Cosine distance improves codebook utilization and token-distribution entropy over L2 distance, promoting more uniform and diverse codebook use.Source: §4.3 · p. 8
- Removing behavior-aware fine-tuning consistently lowers Recall and NDCG across settings, consistent with a gap between semantic space and user behavior.Fine-tuning aligns item-semantic ID mapping with downstream task objectives.Source: §4.3 · p. 8
4.4 Item Popularity Stratified Performance Analysis (RQ3)
On industrial data, semantic-ID methods improve AUC over an ItemID-based model for long-tail items, and MMQ delivers the largest gains in that group.
4.4 Item Popularity Stratified Performance Analysis (RQ3)
- Semantic-ID methods improve AUC over a traditional ItemID-based model for long-tail items, with MMQ delivering the largest gains.Items were stratified by accumulated impressions into Popular Items (Top 25%) and Long-tail Items (Bottom 25%); the improvement is reported for both modality-aligned and modality-separated strategies.Source: §4.4 · p. 8
- The authors interpret the long-tail gains as evidence that semantic IDs mitigate data sparsity and that MMQ learns a more meaningful item-to-semantic mapping.Source: §4.4 · p. 8
- The popularity-stratified evaluation uses industrial-dataset items grouped by accumulated impressions into the top and bottom 25%.Source: §4.4 · p. 8
4.5 The Compatibility of the Behavior-aware Fine-tuning Framework (RQ4)
Behavior-aware fine-tuning improves RQ-VAE downstream performance under both modality-aligned and modality-separated paradigms, supporting its compatibility across quantization methods.
4.5 The Compatibility of the Behavior-aware Fine-tuning Framework (RQ4)
- Integrating behavior-aware fine-tuning with RQ-VAE yields consistent, significant downstream performance gains under both modality-aligned and modality-separated paradigms.The authors describe the fine-tuning as model-agnostic and portable across vector-quantization methods.Source: §4.5 · p. 8, Figure 4 · p. 9
- Behavior-aware fine-tuning dynamically adjusts cluster centroids using downstream recommendation objectives to align item semantics with user behavioral patterns.Source: §4.5 · p. 9
4.6 The Scalability of Semantic ID Length (RQ5)
Longer semantic ID sequences improve recommendation accuracy while reconstruction loss and token-distribution entropy stay stable. The shared-expert comparison favors MMQ over specific-only variants, including one with more experts.
4.6 The Scalability of Semantic ID Length (RQ5)
- Recommendation accuracy measured by NDCG@5 and NDCG@10 consistently improves as semantic ID length increases, while reconstruction loss and token-distribution entropy remain stable.The tested lengths are 6, 9, 12, 15, and 18.Source: §4.6 · p. 9
4.7 Experiment Analysis on the Contribution of Modality-Shared Experts (RQ6)
- Specific-Only (10 experts) still performs below MMQ (Shared=2, Specific=4), supporting the shared expert’s role in parameter-efficient specialization.Specific-Only (4 experts) also performs substantially worse than MMQ (Shared=2, Specific=4).Source: §4.7 · p. 9, Figure 6 · p. 9
5 ONLINE EXPERIMENTS
A 30-day online A/B test compared recommendations based on MMQ-generated semantic IDs with the existing Item ID-based system. The semantic-ID group improved advertising revenue, conversion rate, and orders.
5 ONLINE EXPERIMENTS
- The semantic-ID group increased Advertising Revenue by 0.90%, Conversion Rate (CVR) by +4.33%, and Orders by 3.52% versus the Item ID-based control.The test ran July 6–August 6, 2025, with 10% of random user traffic allocated to the experimental group.Source: §5 · p. 9
6 CONCLUSION
MMQ combines modality-shared and modality-specific experts in a two-stage training framework that aligns content-derived semantic IDs with recommendation behavior. Evaluations across industrial datasets, public benchmarks, and online A/B tests report gains across generative retrieval and discriminative ranking, alongside limitations in training complexity, data needs, and transferability.
6 CONCLUSION
- MMQ combines modality-shared and modality-specific experts with shared-specific pretraining and behavior-aware fine-tuning to align semantic and behavioral spaces.The framework generates compact semantic IDs from heterogeneous modalities.Source: §6 · p. 9
- MMQ consistently outperforms state-of-the-art baselines in generative retrieval and discriminative ranking, with reported generalization across domains.Evaluations cover industrial datasets, public benchmarks, and online A/B tests.Source: §6 · p. 10
- Two-stage training adds complexity, substantial multimodal data may be needed to exploit modality-specific experts, and task-dependent fine-tuning may limit transfer to unrelated tasks.The authors suggest exploring lightweight or continual-learning variants, additional modalities, and self-supervised or few-shot adaptations.Source: §6 · p. 10