PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Graph Retrieval-Augmented Sequential Recommendation via Collaborative Filtering Paths

Jason Marcell Setiadi, Xin Cao, Lina Yao

arXiv:2610.11228v1cs.LG

TL;DR

Existing MLLM-based sequential recommenders may omit collaborative signals from neighboring users or require repeated inference over long histories. MGRASRec retrieves candidate-conditioned collaborative paths, extends them with multimodal similarity, and selects relevant history items for an augmented MLLM prompt. It outperforms strong baselines on all metrics across three public datasets.

  • Problem

    Existing MLLM recommenders may omit neighboring-user collaborative signals, while long histories are difficult to fit into prompts without potentially discarding relevant items or repeatedly invoking the MLLM.

    Source: §1 · p. 2

  • Method

    MGRASRec retrieves candidate-conditioned paths from the user-item graph, extends coverage with multimodal similarity, and uses the paths to select relevant history items for an augmented MLLM prompt.

    Source: §1 · p. 2

  • Results

    MGRASRec outperforms strong baselines on all metrics across three public datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Structured graph-based collaborative paths are the primary driver of MGRASRec’s gains and indicate a promising direction for multimodal sequential recommendation.

    Source: §5 · p. 11

    MMSRARec is excluded from quantitative comparison because it has no released code and its preprocessing and evaluation protocol differ, making reported numbers incomparable.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have demonstrated strong potential for sequential recommendation through their ability to reason over complex multimodal data. However, existing approaches either rely solely on the target user's own interaction history, neglecting collaborative signals from neighboring users, or incur substantial computational overhead through repeated MLLM inference over long interaction histories. To address these challenges, we propose MGRASRec, a multimodal graph retrieval-augmented framework for sequential recommendation. MGRASRec injects collaborative filtering signals conditioned on the candidate item directly into the MLLM prompt by retrieving structured paths from a user-item interaction graph, extended via multimodal similarity to increase coverage beyond exact co-interaction overlap. This retrieval also surfaces the history items most relevant to the candidate at no additional cost, removing the need for recurrent summarization and keeping inference to a single forward pass per candidate. All components are unified into an augmented prompt for parameter-efficient fine-tuning of an MLLM. Extensive evaluations across three publicly available datasets validate the effectiveness of MGRASRec, achieving the best performance on all metrics with particularly strong gains in ranking quality.

1 Introduction

Existing MLLM-based sequential recommenders may miss collaborative signals from neighboring users and struggle to fit long histories into prompts without costly repeated inference. MGRASRec retrieves candidate-conditioned graph paths, extends coverage with multimodal similarity, and uses the paths to select relevant history items for a single-pass MLLM prompt.

1 Introduction

  • Existing MLLM recommenders can miss neighboring-user behavioral patterns, while exact co-interaction sparsity leaves many user-item pairs without collaborative signals.Recency- or frequency-based history truncation may discard candidate-relevant items; recurrent summarization preserves full histories but requires repeated MLLM inference.Source: §1 · p. 2
  • MGRASRec injects candidate-conditioned collaborative signals into the MLLM prompt through structured graph paths, extended with multimodal similarity beyond exact co-interaction overlap.The paths also surface candidate-relevant history items at no additional inference cost, and the augmented prompt is used for parameter-efficient fine-tuning.Source: §1 · p. 2
  • Across three public datasets, MGRASRec outperforms strong baselines on all metrics.The contribution summary identifies structured graph path retrieval as the primary driver of gains across the datasets.Source: §1 · p. 2

2 Related Work

Prior recommendation work uses LLMs and MLLMs as feature encoders or directly as text-generation recommenders. MGRASRec is positioned as candidate-conditioned graph retrieval for collaborative evidence in prompt-based MLLM recommendation.

2 Related Work

  • Feature-encoder approaches extract reasoning, knowledge, or multimodal representations, but decouple semantic understanding from the final recommendation decision.The cited examples include LLMRG, KAR, NoteLLM-2, MLLMRec [2], and Molar [13].Source: §2 · pp. 2–3
  • Prompt-generation approaches include P5 and TALLRec, while MLLM-MSR recurrently summarizes histories at a cost that grows linearly with history length.MMSRARec [16] retrieves embedding-similar users, but its collaborative signal is not conditioned on the candidate item.Source: §2 · p. 3
  • MGRASRec instead uses structured graph-based collaborative retrieval to condition behavioral evidence on the candidate item.The related-work discussion presents this as a comparatively unexplored direction in prompt-based MLLM frameworks.Source: §2 · p. 3

3 Method

MGRASRec is a three-stage framework that retrieves candidate-conditioned collaborative paths, selects relevant history items, and fine-tunes an MLLM using an augmented prompt. The method formalizes next-item prediction from a user’s history and a candidate item.

3.1 Problem Formulation

  • Given a user’s history and a candidate item, the task is to predict the probability that the candidate is the user’s next interaction.Items have textual titles and images, and the history is a sequence of previously interacted items.Source: §3.1 · p. 4

3.2 Graph-based Collaborative Filtering Retrieval

  • A bipartite user-item interaction graph supports offline path retrieval, with held-out positive pairs excluded to avoid leakage.Retrieved paths connect the target user to the target item through bridge users.Source: §3.2 · p. 4

Co-Interaction Path Retrieval

  • Three-hop co-interaction paths connect the target user and candidate through a bridge user and their shared item, providing direct collaborative evidence.Exact overlap leaves many user-item pairs, especially negative candidates, without a collaborative signal.Source: Co-Interaction Path Retrieval · p. 5

Similarity Path Retrieval

  • Four-hop similarity paths widen coverage by linking a history item to a similar bridge item while requiring the bridge user to have interacted directly with the candidate.The method uses CLIP image and title embeddings and FAISS similarity search; adding similarity paths covers almost all pairs across the three datasets for positive and negative candidates.Source: Table 1 · p. 4, Similarity Path Retrieval · p. 5

Path Priority Ranking

  • The method ranks paths by the number of unique bridge users, fixes K at 5, and reserves at least 60% of slots for co-interaction paths.A count summary of available paths is included in the prompt; K was selected after a sweep that found no significant performance difference among tested values.Source: Path Priority Ranking · p. 6

3.3 Relevant History Selection

  • History selection extracts first-hop items from top-ranked paths, pads sparse or empty cases with frequent items, and restores chronological order.This surfaces history items structurally relevant to the candidate as a byproduct of retrieval, at no additional inference cost.Source: §3.3 · p. 6

3.4 MLLM LoRA Fine-Tuning

  • Retrieved collaborative signals and selected history items are assembled into an augmented prompt for supervised fine-tuning of LLaVA with LoRA.When no paths exist, the prompt omits the collaborative-signals section; the same prompt format is used in training and inference.Source: §3.4 · p. 6
  • At recommendation time, MGRASRec uses a single forward pass per candidate.LoRA is applied under the parameter-efficient fine-tuning framework to reduce training cost while preserving in-context reasoning capability.Source: §3.4 · pp. 6–7

4 Experiments

The evaluation uses three real-world datasets from micro-video, digital goods, and baby-product domains, with histories and data splits constrained for the experiments.

Dataset Description

  • The evaluation uses Microlens, Amazon-Games, and Amazon-Baby, covering micro-video recommendation, digital goods, and baby-product purchasing.Each dataset includes user-item interactions, product descriptions, and images.Source: Dataset Description · p. 7
  • K-core filtering thresholds are (7, 5) for Microlens and (6, 5) for Amazon; histories are capped at 5 items for Microlens and 10 for Amazon.The history caps reflect MLLM context-window limitations, and histories are sorted chronologically.Source: Dataset Description · p. 7
  • Users are split randomly into training, validation, and test sets at 8:1:1, with negative sampling ratios of 1:1 for training and validation and 1:20 for testing.Negatives are sampled uniformly from items the target user has not interacted with.Source: Dataset Description · pp. 7–8

Baseline Methods

The experiments compare MGRASRec with sequential, multimodal, LLM-based, and MLLM-based recommendation baselines.

Baseline Methods

  • GRU4Rec and SASRec are basic sequential-recommendation baselines that rely on item IDs and collaborative signals.GRU4Rec models sequential dependencies with gated recurrent units; SASRec uses self-attention for long-term preferences.Source: Baseline Methods · p. 8
  • MMGCN and FREEDOM represent multimodal recommendation baselines that incorporate modality features through graph-based methods.MMGCN uses message passing; FREEDOM uses modality-specific graph structures and edge pruning.Source: Baseline Methods · p. 8
  • TALLRec is the LLM-based baseline, fine-tuned for recommendation using textual item information.Source: Baseline Methods · p. 8
  • MLLM-based baselines include zero-shot and LoRA-fine-tuned LLaVA without collaborative signals, and MLLM-MSR with repeated reasoning over image and text descriptions.MLLM-MSR summarizes user histories through multiple rounds of MLLM inference.Source: Baseline Methods · p. 8

Evaluation Metrics

Evaluation follows a ranking-stage protocol in which the model scores a shortlist of candidate items using three reported metrics.

Evaluation Metrics

  • The model re-scores a shortlist containing the ground-truth positive and sampled negative candidates rather than searching the full catalog.Source: Evaluation Metrics · p. 8
  • The ranking-stage evaluation protocol is consistent with the convention adopted by the compared baselines.Source: Evaluation Metrics · p. 8
  • Performance is assessed with AUC, HR@5, and MRR@5.Source: Evaluation Metrics · p. 8

Implementation Details

Experiments use LLaVA-v1.6-Mistral-7B with LoRA fine-tuning on a single NVIDIA H200 GPU.

Implementation Details

4.2 Overall Performance

MGRASRec achieves the strongest reported performance across all three datasets, while the comparison excludes a concurrent preprint whose results are not comparable under differing protocols.

4.2 Overall Performance

  • MMSRARec [16] is excluded from quantitative comparison because its code is unavailable and its preprocessing and evaluation protocol differ.Source: §4.2 · p. 8
  • MGRASRec achieves the best performance on all metrics across all three datasets.Source: §4.2 · pp. 8–9, Table 2 · p. 9
  • Multimodal and ID-only sequential models generally underperform fine-tuned MLLMs, and supervised fine-tuning substantially outperforms zero-shot LLaVA.Source: §4.2 · p. 9
  • MLLM-MSR underperforms both fine-tuned LLaVA and MGRASRec despite sharing the backbone and SFT setup; the authors attribute this to item-level information loss from preference summaries.Source: §4.2 · p. 9
  • MGRASRec's gains are particularly pronounced on HR@5 and MRR@5, which the authors say reflects complementary collaborative preference information.Source: §4.2 · p. 9

4.3 Ablation Study

Ablations identify collaborative signals as MGRASRec's most impactful component, with the relative value of path types and history selection varying by dataset.

4.3 Ablation Study

  • The ablation variants remove similarity paths, co-interaction paths, path-guided history selection, collaborative signals, or all components.Source: §4.3 · p. 9
  • Co-interaction paths are more reliable on Amazon datasets, whereas similarity paths matter more on Microlens, where visual semantics correlate more strongly with user interest.Source: §4.3 · pp. 9–10
  • Across three datasets, collaborative filtering signals are consistently the most impactful component, while the relative contributions of co-interaction and similarity paths vary by domain.Source: §4.3 · p. 9, Figure 3 · p. 10
  • MGRASRec requires exactly one GPU forward pass regardless of history length, unlike MLLM-MSR's repeated preference-summarization passes.Source: Figure 4 · p. 10
  • Path-guided history selection improves all three metrics on Microlens but is inconsistent or slightly negative elsewhere; its primary value is surfacing relevant history at no extra cost.Source: §4.3 · p. 10

4.4 Efficiency Analysis

MGRASRec keeps recommendation-time MLLM inference constant with respect to history length by shifting path retrieval offline.

4.4 Efficiency Analysis

  • Inference cost grows linearly with user-history length for MLLM-MSR but remains constant for MGRASRec.Source: §4.4 · p. 10
  • MGRASRec shifts retrieval offline to CPU-bound BFS traversal and lookups in a precomputed item-similarity table.Source: §4.4 · p. 10
  • The constant cost applies only with respect to history length: MGRASRec still uses one binary yes/no query per candidate item scored.Source: §4.4 · pp. 10–11

5 Conclusion

MGRASRec augments multimodal sequential recommendation with graph-retrieved collaborative paths, and the authors identify these paths as the primary driver of its gains.

5 Conclusion

  • The authors identify structured graph-based collaborative paths as the primary driver of MGRASRec’s gains.They describe this as a promising direction for multimodal sequential recommendation.Source: §5 · p. 11
  • MGRASRec injects neighboring users’ collaborative signals into the MLLM prompt, extending path coverage with multimodal bridge items and selecting candidate-relevant history at constant inference cost.The framework retrieves paths connecting the target user to the target item through bridge users.Source: §5 · p. 11
  • The current formulation predicts over a constrained candidate set with one forward pass per candidate; future work could explore open-set next-item generation.The proposed direction uses constrained decoding for item grounding.Source: §5 · p. 11
Loading 2610.11228v1…