Source-linked AI summary
RecGOAT: Graph Optimal Adaptive Transport for LLM-Enhanced Multimodal Recommendation with Dual Semantic Alignment
Yuecheng Li, Hengwei Ju, Zeyu Song, Wei Yang, Chi Lu, Peng Jiang, Kun Gai
TL;DR
LM representations can conflict with ID-based collaborative signals in multimodal recommendation, motivating RecGOAT’s dual-granularity alignment framework. RecGOAT combines graph-based semantic enrichment with CMCL and OAT, and achieves state-of-the-art benchmark performance with large-scale deployment scalability.
Problem
LM-enhanced multimodal recommendation still lacks effective alignment between large-model modality representations and interaction IDs, while prior methods often omit global distribution-level alignment.
Method
RecGOAT combines multimodal attentive graphs and LLM-inferred user preferences with instance-level CMCL and distribution-level OAT alignment.
Results
RecGOAT demonstrates state-of-the-art performance on three Amazon datasets, while A/B testing on a large-scale advertising platform confirms scalability.
Takeaways & Limitations
The aligned unified representations provide theoretically supported alignment consistency and fusion comprehensiveness, with ablations confirming the necessity of OT-based distribution alignment.
Takeaways & Limitations
Existing multimodal recommendation methods remain limited by insufficient model scale for deep semantic abstraction and reasoning.
Abstract
from arXiv · showhide
Integrating large language model (LLM) representations into multimodal recommendation has shown promise, yet a fundamental challenge remains largely overlooked: the semantic heterogeneity between generative LM representations and the ID-based collaborative signals that recommendation systems rely on. Naively injecting LM features without alignment degrades recommendation performance rather than improving it. To resolve this, we propose RecGOAT, a dual-granularity semantic alignment framework built on graph neural networks and optimal transport theory. RecGOAT first enriches collaborative semantics through multimodal attentive graphs that capture item-item, user-item, and user-user relationships, initializing user representations via LLM-inferred behavioral preferences. It then aligns LM-derived modality representations with recommendation IDs at two complementary granularities: (1) instance-level alignment via cross-modal contrastive learning (CMCL), which produces discriminative per-sample representations; and (2) distribution-level alignment via optimal adaptive transport (OAT), which minimizes the 1-Wasserstein distance between ID distributions and LLM semantics to produce a unified, consistently aligned feature space. Theoretically, we prove that the unified representation achieves strictly lower target error than any single-modality representation, with the gap bounded by the Wasserstein distance and the InfoNCE loss, providing rigorous guarantees for both alignment consistency and fusion comprehensiveness. Extensive experiments on three public benchmarks demonstrate state-of-the-art performance. Deployment on a large-scale online advertising platform further validates RecGOAT's industrial scalability. Our code is available at https://github.com/6lyc/RecGOAT-LLM4Rec.
1 Introduction
Multimodal recommendation uses rich content and large-model semantics to address sparse interactions, but semantic heterogeneity with ID signals remains a central challenge. RecGOAT addresses this through graph-enhanced collaborative modeling and dual-granularity alignment.
- Sparse user-item interactions constrain recommendation performance, while multimodal content can complement interaction signals and improve personalization.
- GNN-based multimodal methods capture structural relations but have limited semantic understanding and often initialize user IDs randomly or from interacted-item features.
- LM-enhanced recommendation models add semantic representations but still exhibit deficiencies aligning modal signals with interaction IDs.
- RecGOAT proposes dual semantic alignment for LLM-enhanced multimodal recommendation.
- Its attentive graphs strengthen item-item, user-item, and user-user collaborative signals while LLMs encode item modalities and infer users’ multidimensional preferences.
- RecGOAT combines instance-level CMCL with distribution-level OAT, and experiments on public datasets and an online advertising platform demonstrate effectiveness and scalability.
2 Related Work
Related work advances multimodal representation, structural modeling, and LM-enhanced recommendation, but existing approaches remain limited in semantic abstraction or global distribution alignment. RecGOAT targets this unresolved alignment gap.
- Multimodal recommendation methods use CNNs, GNNs, Transformers, and diffusion models to integrate content and interaction information.
- Prior multimodal methods’ core limitation is insufficient model scale for deep semantic abstraction and reasoning.
- LM-enhanced methods employ fine-tuning, in-context learning, cross-attention, contrastive learning, and late fusion for semantic integration.
- Existing LM-enhanced alignment mechanisms largely focus on instance-level or pair-wise local alignment rather than global distribution-level alignment.
- The paper introduces distribution alignment to account for overall cross-modal feature distributions and uses Wasserstein distance from optimal transport theory.
3 Methodology
RecGOAT combines LLM-enhanced modality features, attentive graph learning, and user-preference reasoning to model collaborative semantics. It then performs dual-granularity alignment before fusion and recommendation.
- RecGOAT augments modalities with large models, learns structural relations through attentive graphs, aligns representations at instance and distribution levels, and performs final fusion.
- Item-item multimodal graph learning: Pretrained LLMs and LVLMs encode item text and images, while KNN graphs and graph attention aggregate high-order within-modality collaborative signals.
- ID embedding learning: The user-item interaction graph represents implicit feedback, and propagated ID embeddings encode high-order collaborative relations for later cross-modal alignment.
- User-user graph learning: Personalized behavioral prompts use users’ interaction histories and item descriptions to elicit multidimensional preferences from QwQ-32B.
- User-user graph learning: The generated textual answers are encoded as users’ textual modal features and refined through semantically connected user graphs.
3.2 Cross-modal: Dual-Granularity Alignment of LLM-enhanced Modalities and ID Signals
RecGOAT aligns LLM-enhanced modalities with ID-based collaborative signals at instance and distribution levels, then fuses the aligned representations for comprehensive item embeddings.
- Instance-level Alignment: Instance-level CMCL treats same-item ID, text, and visual pairs as positives while using different items as negatives.InfoNCE encourages higher similarity for positive pairs, with cosine similarity, temperature τ, and batch negatives defining the objective.
- Instance-level Alignment: CMCL-refined representations become semantically discriminative across modalities and provide the similarity structure used to construct the optimal-transport cost matrix.The resulting transport cost captures deep semantic discrepancy rather than relying only on raw feature distances.
- Distribution-level Alignment: Distribution-level OT transports each LLM-enhanced modality distribution toward the collaborative ID distribution by minimizing total semantic transport cost.The source is the empirical distribution P_m of an LLM-enhanced modality, while the target is the empirical ID distribution Q_id.
- Distribution-level Alignment: The feature-wise OT cost uses normalized L1 distance between LLM-enhanced and ID features, with a scaling factor s for numerical stability.The cost matrix is computed for batch feature representations and scaled to control numerical behavior.
- Distribution-level Alignment: Entropy-regularized OT is solved with Sinkhorn-Knopp, producing an approximate transport plan that adapts through a learnable residual matrix.The residual provides task-specific semantic corrections to the geometry-driven coupling before transporting LLM-enhanced features toward the ID space.
- Unified Representation: The three aligned item representations are fused with modality weights to form a unified embedding combining semantic consistency and multimodal-collaborative comprehensiveness.The unified representation is used as the final item embedding for downstream preference prediction.
3.3 Theoretical Guarantees for Alignment Consistency and Fusion Comprehensiveness
RecGOAT’s theory analyzes item-side errors under bounded-user and Lipschitz assumptions, showing that dual-granularity alignment supports consistency and comprehensive fusion.
- Problem Setup and Assumptions: The analysis fixes user embeddings and defines modality-specific and unified representation errors through expected prediction discrepancies.The model score is an inner product, and the unified item representations follow distribution Q.
- Problem Setup and Assumptions: Bounded user embeddings make the inner-product scoring function K-Lipschitz with respect to item representations.The bound follows from the Cauchy-Schwarz inequality.
- Problem Setup and Assumptions: The true preference function is assumed to be L*-Lipschitz with respect to item representations, expressing smoothness of user preferences.This is stated as an explicit assumption for the theoretical analysis.
- Supporting Lemmas: The supporting lemmas bound modality-to-unified error using both distribution-level Wasserstein distance and instance-level representation distance.This connects the two alignment granularities in the theoretical analysis.
- Main Theorem: Theorem 3.5 states consistency and fusion-comprehensiveness guarantees for every modality and fixed user embedding.The theorem explicitly separates consistency and comprehensiveness guarantees.
- Main Theorem: Reducing W1 and CMCL improves alignment bounds, while unified error remains no greater than any single-modality error plus dual-granularity alignment error.The result supplies the paper’s theoretical foundation for bridging LLM-enhanced modalities and ID-based interaction signals.
3.4 Preference Optimization for Recommender
RecGOAT optimizes recommendation preferences with BPR using fused user representations and the unified item representation.
- Preference Optimization: The downstream objective is Bayesian Personalized Ranking over observed user, positive-item, and negative-item triplets.The scoring function is the user-item inner product, and higher predicted scores rank items as stronger recommendation candidates.
- Preference Optimization: The fused user representation combines user ID and enhanced textual user representations before optimization with the unified item embedding.The item representation is the unified embedding defined earlier in the framework.
3.5 Complexity Analysis
RecGOAT’s OAT module aligns feature distributions rather than matching individual nodes, yielding complexity that scales linearly with sample size.
- Complexity Analysis: OAT relies on feature dimension d instead of sample size N for distribution alignment, improving scalability over standard OT node matching.The module uses Sinkhorn-Knopp with at most L iterations.
- Complexity Analysis: Training time is O((N + L) · d^2) and space complexity is O(N · d^2), giving linear scalability in sample size.These bounds are reported for the Sinkhorn-Knopp implementation.
4 Experiments
Experiments evaluate RecGOAT on three Amazon datasets and an industrial advertising platform, covering benchmark performance, alignment ablations, theoretical properties, and online impact.
- Experiments use Baby, Sports, and Electronics datasets with user-item interactions plus visual and textual item descriptions.
- RecGOAT is compared against ID-based, multimodal, and LM-enhanced recommendation methods using Recall@10 and NDCG@10 averaged over 10 runs.
- Naive concatenation or summation of LLM-enhanced modalities can underperform or inconsistently match ID-only LightGCN, while OAT consistently outperforms CMCL across datasets.
- RecGOAT achieves statistically significant state-of-the-art performance on every reported metric across all three Amazon datasets.
- The fused aligned representation outperforms each single aligned modality, while performance remains robust to modality weights, supporting alignment consistency and fusion comprehensiveness.
- Online A/B testing over 5% of production traffic and approximately 20 million users yields a 1.5% ADVV lift overall and a 2.3% lift on long-tail data.
5 Hyperparameter Sensitivity Analysis (RQ5)
RecGOAT remains stable across the tested nearest-neighbor settings, indicating limited sensitivity to the item-item graph parameter K.
- Recall@10 and NDCG@10 remain highly stable across all three datasets as K varies from 10 to 50.
6 Runtime Analysis (RQ6)
Runtime analysis compares RecGOAT with representative ID-based, multimodal, and LM-enhanced baselines on the Baby dataset under identical evaluation conditions.
- Training and inference times are reported for RecGOAT, LightGCN, FindRec, and IRLLRec on the Baby dataset.
7 Conclusions
RecGOAT aligns large-model representations with recommendation ID signals at instance and distribution levels, with theoretical guarantees for the resulting representations. Experiments report SOTA performance on three Amazon datasets, while A/B testing confirms scalability on a large-scale advertising platform.
- RecGOAT combines cross-modal contrastive learning and optimal adaptive transport to align large-model representations with recommendation ID signals.The framework performs alignment at instance and distribution granularities.
- Theoretical analysis establishes consistency and comprehensiveness for RecGOAT’s aligned representations.
- SOTA performance is demonstrated against relevant baselines on three Amazon datasets, and A/B testing confirms scalability on a large-scale advertising platform.The paper identifies future work on multiple optimal transport alignments and omni-modal large recommendation models.