Source-linked AI summary
Empowering Large Language Model for Sequential Recommendation via Multimodal Embeddings and Semantic IDs
Yuhao Wang, Junwei Pan, Xinhang Li, Maolin Wang, Yuan Wang, Yue Liu, Dapeng Liu, Jie Jiang, Xiangyu Zhao
TL;DR
LLM-based sequential recommendation faces embedding collapse from mapping low-dimensional collaborative embeddings into LLM space and forgetting of quantized-embedding information. MME-SID combines multimodal embeddings and semantic IDs, with trained code embeddings initializing the latter. It reports improved performance across three datasets and lower embedding collapse in its analyses.
Problem
LLM-based sequential recommendation faces embedding collapse when mapping pre-trained collaborative embeddings into LLM space and loss of learned code-embedding distance information when semantic-ID embeddings are trained from scratch.
Source: §1 · p. 2Method
MME-SID combines original multimodal embeddings with semantic-ID embeddings initialized from trained MM-RQ-VAE code embeddings, using MMD reconstruction and contrastive learning.
Source: §1 · p. 2, §4.2.2 · p. 5Results
MME-SID significantly improves performance and surpasses all baselines on three datasets, beating the best baseline by 10.47%, 4.42%, and 8.12% on nDCG@5.
Source: §5.2 · p. 7Takeaways & Limitations
The experiments report that multimodal embeddings reduce embedding collapse while trained multimodal code embeddings help retain quantized-embedding information.
Source: §5.3 · p. 8, §7 · p. 9
Abstract
from arXiv · showhide
Sequential recommendation (SR) aims to capture users' dynamic interests and sequential patterns based on their historical interactions. Recently, the powerful capabilities of large language models (LLMs) have driven their adoption in SR. However, we identify two critical challenges in existing LLM-based SR methods: 1) embedding collapse when incorporating pre-trained collaborative embeddings and 2) catastrophic forgetting of quantized embeddings when utilizing semantic IDs. These issues dampen the model scalability and lead to suboptimal recommendation performance. Therefore, based on LLMs like Llama3-8B-instruct, we introduce a novel SR framework named MME-SID, which integrates multimodal embeddings and quantized embeddings to mitigate embedding collapse. Additionally, we propose a Multimodal Residual Quantized Variational Autoencoder (MM-RQ-VAE) with maximum mean discrepancy as the reconstruction loss and contrastive learning for alignment, which effectively preserve intra-modal distance information and capture inter-modal correlations, respectively. To further alleviate catastrophic forgetting, we initialize the model with the trained multimodal code embeddings. Finally, we fine-tune the LLM efficiently using LoRA in a multimodal frequency-aware fusion manner. Extensive experiments on three public datasets validate the superior performance of MME-SID thanks to its capability to mitigate embedding collapse and catastrophic forgetting. The implementation code and datasets are publicly available for reproduction: https://github.com/Applied-Machine-Learning-Lab/MME-SID.
1 Introduction
The introduction frames sequential recommendation as modeling users’ changing interests from interaction histories, then identifies embedding collapse and catastrophic forgetting as obstacles in LLM-based recommendation. MME-SID combines multimodal embeddings with semantic IDs and trained code embeddings to address these challenges.
1 Introduction
- Traditional item-ID-only methods are vulnerable to cold start, while LLM-based SR uses text generation or semantic IDs as item tokens.Semantic IDs encode item embeddings into codes that can serve as new LLM tokens.Source: §1 · pp. 1–2
- Embedding collapse maps low-dimensional collaborative embeddings into a low-dimensional subspace of the LLM representation, while semantic-ID training can forget learned code-embedding distance information.The paper reports over 98% of embedding dimensions collapsed and over 94% of partial-order distance information lost in its analyses.Source: §1 · p. 2
- Addressing both failures is difficult: collaborative embeddings risk collapse, whereas randomly initialized embeddings are costly at scale and discard previously acquired knowledge.The introduction also notes that traditional multi-embedding solutions fail in LLM-based recommendation.Source: §1 · p. 2
- MME-SID uses MM-RQ-VAE to generate multimodal semantic IDs, with MMD reconstruction loss for distance preservation and contrastive learning for inter-modal correlations.The framework is intended to address embedding collapse and forgetting together.Source: §1 · p. 2
- MME-SID combines original collaborative, textual, and visual embeddings with semantic-ID embeddings, initializes semantic IDs from trained code embeddings, and fine-tunes the LLM with frequency-aware LoRA.Source: §1 · p. 2
- Experiments on three public Amazon datasets report superior recommendation performance and analyses of embedding collapse and catastrophic forgetting.The paper also presents its work as systematically addressing both issues and examining multimodal information and semantic-ID use.Source: §1 · p. 2
2 Background
The background introduces sequential recommendation and residual quantization as the context for representing item embeddings with hierarchical semantic IDs.
2.1 Problem Formulation
- A conventional sequential recommender uses each user’s behavioral item sequence to predict a target item and optimizes binary cross-entropy against its true label.The prediction uses the recommender output and target-item embedding.Source: §2.1 · p. 2
2.2 RQ-VAE
- RQ-VAE encodes an original embedding into a latent semantic embedding, then quantizes it through a hierarchy of codebooks to produce semantic IDs.The quantized code embeddings are summed and decoded to reconstruct the original embedding.Source: §2.2 · pp. 2–3
- At each residual-quantization level, the assigned code corresponds to the current residual, which is updated by subtracting that code embedding.The initial residual is the latent embedding, and the codebook assignment uses L2 distance.Source: §2.2 · p. 2
3 Preliminary Analysis
The preliminary analysis examines why LLM-based recommendation can suffer embedding collapse and catastrophic forgetting, using rank arguments and distance-order comparisons.
3.1 Embedding Collapse
- Linear projection cannot expand the rank of low-dimensional collaborative embeddings, so mapping them into LLM token space leaves them in a low-dimensional subspace.The paper notes collaborative embeddings commonly have dimensions such as 64 or 128; it treats nonlinear mappings empirically because their rank effects lack a unified theoretical conclusion.Source: §3.1 · p. 3
3.2 Catastrophic Forgetting
- On Amazon Beauty, quantized embeddings preserve 37.14% of the original distance-order information, but fine-tuning randomly initialized code embeddings leaves only 5.50% preserved.The reported Kendall’s tau values are 0.3714 before downstream fine-tuning and 0.0550 afterward, corresponding to 94.50% of previously learned information forgotten.Source: §3.2 · pp. 3–4
4 Method
MME-SID combines multimodal embeddings and semantic IDs in encoding and fine-tuning stages to address embedding collapse and catastrophic forgetting. Its method uses MM-RQ-VAE for multimodal quantization and a frequency-aware, LoRA-based LLM fine-tuning process.
4.1 Overview
- MME-SID has encoding and fine-tuning stages that obtain multimodal embeddings and semantic IDs, then tune the LLM for sequential recommendation.The framework combines multimodal embeddings with trained code embeddings to address embedding collapse and catastrophic forgetting.Source: §4.1 · p. 4
4.2 Encoding Stage
- Items are encoded from collaborative, textual, and visual embeddings, which are quantized into multimodal semantic IDs.The paper denotes these embeddings as E_c, E_t, and E_v.Source: §4.2 · p. 4
4.2.1 Multimodal Embedding Encoding.
- LLM2CLIP supplies textual and visual embeddings, addressing encoder-space alignment and long-text limitations identified for prior multimodal methods.The paper also extracts collaborative embeddings from an item-ID-based sequential recommender.Source: §4.2.1 · p. 4
4.2.2 Multimodal Embedding Quantization.
- MM-RQ-VAE quantizes embeddings from collaborative, textual, and visual modalities, using MMD reconstruction and contrastive alignment between collaborative and textual or visual codes.The characteristic-kernel MMD is used to preserve distribution information, while visual and textual embeddings are not separately aligned because LLMCLIP maps them into the same space.Source: §4.2.2 · pp. 4–5
4.3 Fine-tuning Stage
- MME-SID initializes semantic-ID embeddings with trained code embeddings to mitigate catastrophic forgetting and combines projected original embeddings with semantic-ID embeddings as LLM input.The concatenated representation is passed through an MLP; the fine-tuning stage also uses multimodal frequency-aware fusion and LoRA.Source: §4.3 · p. 5
- Frequency-aware fusion adapts the weights for LLM output and item embeddings across modalities based on item frequency.The method records training-set item frequency as an input feature for fusion.Source: §4.3 · p. 5
4.4 Discussions
- MME-SID can rank the whole item set and output top-k items, while the paper contrasts this with TIGER’s autoregressive retrieval and semantic-ID collision handling.The paper also states that MME-SID takes a D_LLM × N input rather than TIGER’s D_LLM × N × L input, and reports higher inference efficiency.Source: §4.4 · p. 6
5 Experiments
Experiments compare MME-SID with collaborative and multimodal baselines on three Amazon 5-core datasets, using HR@k and nDCG@k. MME-SID outperforms all baselines across the datasets.
5.1.1 Datasets. We experiment on three categories of Amazon2
- The evaluation uses Amazon 5-core Beauty, Toys & Games, and Sports & Outdoors data, with HR@k and nDCG@k at k = 5, 10, and 20.Each user and item has at least five interactions; the task predicts ratings above 3, and the last interaction is held out for testing.Source: §5.1.1 · p. 6, §5.1.2 · p. 6
5.1.3 Baselines.
- The baseline set spans collaborative-only and multimodal methods, including SASRec, E4SRec, ME, Concat, CTRL-MM, TIGER-MM, MOTOR, and LETTER.LLM-based methods use Llama3-8B-instruct, while RQ-VAE generates semantic IDs.Source: §5.1.3 · pp. 6–7
5.1.4 Implementation Details.
- The experiments use product titles and images for multimodal encoding, and report results averaged over three runs on A100 GPUs.The collaborative embedding dimension is 64 and textual and visual embedding dimensions are each 1280; the LLM token embedding dimension is 4096.Source: §5.1.4 · p. 7
5.2 Overall Performance (RQ1)
- LLM-based methods generally outperform traditional SASRec and MOTOR, while E4SRec surpasses SASRec and ME adds only limited gains on Beauty and Toys & Games.The authors attribute ME’s limited gains speculatively to its use of collaborative data alone.Source: §5.2 · p. 7
- Concat, Concat&MLP, and CTRL-MM perform worse than E4SRec, while TIGER-MM, MOTOR, and LETTER have the weakest accuracy among the multimodal methods.The authors note that this challenges approaches that use semantic IDs alone for generative retrieval.Source: §5.2 · p. 7
- MME-SID beats the best baseline by 10.47%, 4.42%, and 8.12% on nDCG@5 across the three datasets.Table 3 presents overall performance on Beauty, Toys & Games, and Sports & Outdoors; the supplied text does not assign each improvement value to a particular dataset.Source: §5.2 · p. 7, Table 3 · p. 7
datasets.
On Beauty, MME-SID combines multimodal embeddings and semantic IDs with the best recommendation performance and the lowest measured collapse across most embedding dimensions. Tests of ReLU also show degraded accuracy and persistent forgetting.
5.3 Alleviating Embedding Collapse (RQ2)
- MME-SID achieves the best recommendation performance and the lowest collapse from dimensions 65 through 4096, which comprise over 98% of the embedding dimensions.The target-item embedding result also indicates that a separate target-item embedding table is necessary to alleviate collapse.Source: §5.3 · p. 8
- Compared with the variant without ReLU, the ReLU variant does not significantly improve embedding rank, degrades recommendation accuracy, and still exhibits catastrophic forgetting.The authors suggest forgetting may persist because nonlinearity disrupts distance information in the original embedding.Source: §5.3 · p. 8
- The results conclude that relying only on pre-trained low-dimensional collaborative embeddings leads to collapse, whereas MME-SID alleviates it using multimodal embeddings and semantic IDs.The section compares SASRec, E4SRec, ME, SE-SID-MMD, and MME-SID; embedding collapse is measured using embedding-table singular values, with higher values indicating less collapse.Source: §5.3 · p. 8
5.4 MMD-based Reconstruction Loss (RQ3)
The section finds that MMD-based reconstruction is associated with less forgetting than MSE, and that initializing semantic ID embeddings with trained code embeddings mitigates forgetting. Its Beauty ablations also highlight the roles of multimodal correlations and frequency-aware fusion.
5.4 MMD-based Reconstruction Loss (RQ3)
- τ = 0.4436 for SE-SID-MMD, versus τ = 0.3714 for SE-SID-MSE, indicating less forgetting with MMD-based reconstruction.The variants have comparable collapse, so the authors attribute SE-SID-MMD’s advantage more appropriately to mitigating forgetting.Source: §5.4 · p. 8
5.5 Embedding Initialization (RQ4)
- τ = 0.2727 for MME-SID after fine-tuning, versus τ = 0.0508 for MME-SID-random, indicating that trained code embeddings relieve forgetting.Figure 5(a) compares code-embedding initialization using nDCG@k; the authors conclude that random initialization leads to catastrophic forgetting.Source: §5.5 · p. 8, Figure 5 · p. 8
5.6 Ablation Study
- On Beauty, parameter-matched MME-random performs worse than MME-SID, while SE-random performs worse than E4SRec and removing fusion decreases performance.The authors associate MME-random’s gap with its inability to leverage intra- and inter-modal correlations, and the fusion ablation with the multimodal frequency-aware fusion module.Source: §5.6 · p. 8, Figure 5 · p. 8
6 Related Work
Related work covers semantic IDs in generative and traditional recommendation, then LLM-based and multimodal recommendation. The section contrasts MME-SID with prior approaches on code initialization, model structure, and inference.
6.1 Semantic IDs for Recommendation
- Generative semantic-ID methods such as TIGER [35] use item content to produce semantic tokens, but may discard trained code embeddings and randomly initialize them downstream.The section associates that practice with catastrophic forgetting.Source: §6.1 · p. 9
- QARM [30] uses vector and residual quantization as auxiliary features, but its gains are described as constrained by traditional model structure.The section contrasts this with MME-SID’s use of multimodal embeddings and trained quantized embeddings in an LLM.Source: §6.1 · p. 9
6.2 LLM4Rec & Multimodal Recommendation
- LLM4Rec progressed from text-formatted tuning such as TALLRec [2] to multimodal approaches such as LC-Rec [66], which the section says suffer from autoregressive inference latency.MME-SID instead scores output embeddings against target-item embeddings directly, which the authors say leads to high efficiency.Source: §6.2 · p. 9
7 Conclusion
The conclusion identifies embedding collapse and catastrophic forgetting in existing LLM-based sequential recommendation, then presents MME-SID and its multimodal quantization approach. It reports efficacy on three public Amazon datasets.
7 Conclusion
- The paper identifies embedding collapse and catastrophic forgetting as issues in existing LLM-based sequential recommendation.These are the challenges the proposed framework aims to address.Source: §7 · p. 9
- MME-SID combines multimodal embeddings with semantic IDs initialized using trained code embeddings.The paper proposes MM-RQ-VAE with maximum mean discrepancy reconstruction loss and contrastive learning to preserve distance information and learn inter-modal connections.Source: §7 · p. 9
- Experiments on three public Amazon datasets validate the efficacy of the proposed method.Source: §7 · p. 9
A Pseudo-code
The pseudo-code divides MME-SID into encoding and fine-tuning stages. It encodes collaborative and multimodal item representations, quantizes them, then fine-tunes an LLM as a sequential recommender.
A Pseudo-code
- The procedure has an encoding stage and a fine-tuning stage, and outputs a trained LLM as a sequential recommender.Its inputs include users, items, interaction histories, target items, and labels.Source: A Pseudo-code · p. 9
- The encoding stage obtains collaborative embeddings from a pretrained sequential recommender and textual and visual embeddings using LLM2CLIP, then updates MM-RQ-VAE using reconstruction, alignment, and RQ-VAE losses.Source: A Pseudo-code · p. 9
- In fine-tuning, the procedure retrieves multimodal embeddings and semantic IDs, computes an LLM output and fusion-based prediction score, calculates BCE loss, and updates the LLM using LoRA.Source: A Pseudo-code · p. 9
B Experimental Settings
The experiments exclude items without titles or images and use AdamW with specified hyperparameters and LoRA modules. Only about 0.19% of parameters are updated.
B Experimental Settings
- Only about 0.19% of all parameters are updated in the experiments.LoRA targets gate_proj, down_proj, and up_proj.Source: B Experimental Settings · p. 9
- Items lacking a title or image in the original dataset are removed during data processing.Source: B Experimental Settings · p. 9
- The experiments use AdamW [29] with α=1, β=1e-3, and γ=1.The hyperparameters are shown in Tab. 4.Source: B Experimental Settings · p. 9