PaperlayerView exact revision on arXiv

Source-linked AI summary

Agents as Knowledge Integrator and Utilizer in Multimodal Recommendation

Jinfeng Xu, Zheyu Chen, Shuo Yang, Jinze Li, Puzhen Wu, Zewei Liu, Zheng Lin, Jianheng Tang, Jing Yang, Wei Wang, Xiping Hu, Edith Ngai

arXiv:2608.29410v1cs.IR

TL;DR

Multimodal recommenders can remain misaligned with recommendation objectives when they rely on content-side features or modality similarity alone. AgentMMRec uses Integrator and Utilizer Agents to convert behavior-aware multimodal knowledge into reusable memory, graphs, representations, and reranking, and experiments show consistent gains over recent baselines across challenging settings.

  • Problem

    Multimodal content and user behavior are not fully aligned with recommendation objectives in existing feature-based and similarity-graph methods.

  • Method

    AgentMMRec stores behavior- and multimodal-aware preferences and item properties in memory, then uses that memory to refine graphs, enhance representations, and rerank candidates.

  • Results

    AgentMMRec consistently improves over recent multimodal baselines across metrics, with gains under sparsity and item cold-start settings and transfer to existing backbones.

  • Takeaways & Limitations

    The results support treating LLM-derived multimodal knowledge as reusable recommendation structure rather than only augmented content or a final reranking signal.

  • Takeaways & Limitations

    Validation and test evaluation use a frozen knowledge memory, with preference extraction, graph construction, reranking feedback, and memory updates restricted to training data.

Abstract

from arXiv · show

Online platforms increasingly rely on multimodal recommender systems to rank products, media, and other Web content. Existing methods usually inject visual and textual features into item representations or build homogeneous graphs from modality-level similarity, but the resulting signals can remain misaligned with the recommendation objective. We study this semantic gap from a knowledge-integration perspective: multimodal content should be interpreted together with user behavior before it is used to construct recommendation graphs or adjust rankings. We propose AgentMMRec, an agent-based multimodal recommendation framework with two coordinated roles. The Integrator Agent infers behavior- and multimodal-aware user preferences and item properties from training interactions and item content, then stores them in a reusable knowledge memory. The Utilizer Agent consumes this memory to refine modality-specific item-item graphs, construct behavior-aware homogeneous graphs, and rerank candidate lists under a frozen evaluation-time memory. This design differs from direct LLM feature augmentation and pure LLM reranking because the generated knowledge is first converted into graph structure and model representations before recommendation. Experiments on three Amazon multimodal recommendation datasets show that AgentMMRec consistently improves Recall and NDCG over recent multimodal baselines, remains effective under sparsity and item cold-start settings, and can transfer its constructed knowledge to existing backbones.

1 Introduction

AgentMMRec addresses the semantic gap between multimodal content and recommendation objectives by converting behavior-aware LLM knowledge into reusable recommendation structure. Its two-agent design improves multimodal recommendation across datasets and challenging settings.

  • Motivation: Multimodal features and similarity graphs can remain misaligned with the recommendation objective because salient content may not explain user interactions.Images and descriptions may emphasize color, style, material, or marketing language without capturing why users engage with items.
  • Motivation: Existing LLM recommendation paradigms use augmentation, fine-tuning, or reranking, but lack reusable semantic knowledge integrated into recommendation structure.The missing mechanism should preserve a standard recommender training and evaluation protocol.
  • Framework: AgentMMRec uses an Integrator Agent and Utilizer Agent with an intermediate knowledge memory rather than replacing the recommender directly.The Integrator extracts behavior- and multimodal-aware user preferences and item properties from training data and content.
  • Framework: The Utilizer converts stored knowledge into refined modality-specific and behavior-aware graphs, representation inputs, and controlled candidate reranking.This links knowledge construction to multiple recommendation stages using the same frozen memory.
  • Results: AgentMMRec improves over recent multimodal baselines across all metrics, with gains persisting under sparsity and item cold-start settings.The study also evaluates transferability, memory updating, backbone choice, self-supervised learning, template robustness, and computational cost.

2 Related Work

Multimodal recommendation has progressed from visual and textual feature enrichment to graph-based modeling, while LLM methods add content understanding, profiling, and reranking. AgentMMRec extends this line by organizing behavior-conditioned multimodal knowledge in reusable memory for graph refinement and recommendation.

  • Multimodal Recommendation: Earlier multimodal recommenders combine visual and textual modalities with user-item connectivity or modality similarity to enrich item representations.Examples include VBPR, MMGCN, DualGNN, LATTICE, and FREEDOM.
  • Multimodal Recommendation: These methods still rely heavily on feature-level similarity, leaving the semantic gap between multimodal data and recommendation tasks unresolved.AgentMMRec interprets multimodal content through behavioral context before constructing graphs and enhanced representations.
  • LLM-based Recommendation: LLMs have supported recommendation through content understanding, user profiling, item attribute extraction, hidden-state embeddings, and list reranking.Representative systems include TALLRec, LEARN, LLMRank, KAR, LLMRec, and UR4Rec.
  • LLM-based Multimodal Recommendation: Recent multimodal LLM approaches refine graphs or build hyper-knowledge graphs, but AgentMMRec differs by storing generated preferences and properties in reusable memory.That memory supports linked graph refinement, representation enhancement, and candidate reranking rather than serving only one downstream role.

3 Methodology

AgentMMRec separates knowledge construction from recommendation use through an Integrator Agent, a reusable knowledge memory, and a Utilizer Agent that builds behavior-aware graphs, enhances representations, and reranks candidates. Its graphs are constructed before training, while the memory-derived signals incorporate user behavior and multimodal content.

  • Framework overview: AgentMMRec defines an Integrator Agent, a Utilizer Agent, and knowledge memory, separating reusable knowledge construction from its use in recommendation.The Integrator interprets multimodal content under behavior context, while the Utilizer applies stored knowledge to graphs, representations, and candidate lists.
  • Problem definition: The problem setting represents items with text and images and user-item interactions, with sparse interactions requiring inference of preferences and item properties.The framework uses MMRec text and visual encoders and a bipartite interaction graph for observed pairs.
  • Knowledge integration: The Integrator extracts behavior- and multimodal-aware user preferences from interacted training items, including textual, visual, and cross-modal preferences, and stores them in key-value memory.For users exceeding the interaction threshold Υ, the method samples items to control context length and inference cost.
  • Behavior-aware graphs: Encoded user preferences and item properties form top-k user-user and item-item graphs based on cosine similarity.These homogeneous graphs are constructed before recommender training, avoiding per-epoch graph-building cost.
  • Knowledge integration: The Integrator derives behavior- and multimodal-aware item properties from items purchased by users connected to each item, then stores them for graph construction.The extracted properties likewise include textual, visual, and cross-modal information, with item neighborhoods sampled when they exceed Υ.
  • Graph refinement: The Utilizer refines modality-specific item-item graphs by combining text and visual neighbors, item properties, and purchaser preferences into a unified graph.It reselects top-k items under the preference-guided procedure, discards invalid returned item ids, and saves the resulting adjacency before training.
  • Representation enhancement: Constructed homogeneous and refined item-item graphs enhance user and item representations through graph propagation, with LightGCN-based modeling and fixed L=3 layers.The model can also incorporate these graph-enhanced representations into its recommendation optimization procedure.

4 Experiment

AgentMMRec is evaluated on three Amazon multimodal datasets using frozen-memory testing, broad baseline comparisons, ablations, sparsity analysis, cold-start evaluation, and compatibility studies. Results consistently favor behavior-aware knowledge integrated into graphs, representations, and reranking, with the strongest gains often appearing under sparse interactions and transferable graph settings.

  • Experimental Setup: Experiments use Baby, Sports, and Clothing with text and image modalities, 5-core filtering, and an 8:1:1 train-validation-test split.Text comes from item descriptions and visual features from product images, following the MMRec preprocessing protocol.
  • Experimental Setup: AgentMMRec is compared with 17 representative multimodal recommendation baselines using Recall@N and NDCG@N at N=10 and N=20.The backbone retrieves 50 candidates, which the Utilizer Agent reranks before evaluation; test metrics use frozen memory and parameters.
  • Overall Performance: AgentMMRec improves over all baselines across datasets and metrics, supporting knowledge injection into graph construction, representation learning, and reranking rather than augmented text alone.The evaluation isolates the agent-memory design by using the same MMRec text and vision encoders as the baselines.
  • Ablation Study: Removing behavior-aware and multimodal graphs consistently degrades performance, with larger drops on sparser Sports and Clothing datasets.Item-item graphs contribute more on Sports and Clothing, while user-user graphs are relatively more useful on Baby.
  • Ablation Study: Replacing the refined unified item-item graph with traditional modality-specific graphs hurts performance, indicating that content-only neighbors can conflict with behavior-aware item relevance.Removing reranking and feedback causes a smaller but consistent drop, suggesting graph and representation enhancement provide the main gains.
  • Compatibility Analysis: Transferred behavior-aware graphs improve all four selected baselines on every dataset; on Sports, Recall@20 rises 4.2% for SMORE, 3.6% for MENTOR, 3.3% for COHESION, and 4.0% for HPMRec.Knowledge-enhanced reranking also helps, but usually less than graph transfer because it changes only final candidate order; all four architectures benefit.
  • Sparsity Analysis: AgentMMRec shows its largest sparsity gains for users with 0–5 training interactions, while positive gains remain for the 20–25 interaction group.Sports and Clothing have larger catalogs and higher sparsity than Baby, and their item-side structure contributes to dataset differences.
  • Cold-Start Analysis: With 20% of items removed from training, AgentMMRec achieves the best performance across Baby, Sports, and Clothing in item cold-start evaluation.Removed items retain text and image fields but lack training-graph interactions.

5 Conclusion

AgentMMRec bridges multimodal content and recommendation objectives by integrating behavior-aware knowledge into graph structure, representations, and candidate reranking. Across three datasets, it delivers consistent gains while preserving evaluation integrity through frozen memory.

  • 5 Conclusion: AgentMMRec uses an Integrator Agent to build behavior-aware user and item knowledge from training interactions.The Utilizer Agent converts this memory into refined graphs, enhanced representations, and candidate reranking.
  • 5 Conclusion: The memory is frozen during validation and testing, preventing ground-truth feedback from entering reported test results.
  • 5 Conclusion: AgentMMRec consistently improves over recent baselines across three multimodal recommendation datasets.Additional analyses cover compatibility, sparsity, item cold-start, memory updating, backbone choice, self-supervised learning, templates, and cost.
  • 5 Conclusion: The results support treating LLM-derived semantics as reusable task-aware structure rather than only augmented features or final-stage ranking text.

A Baseline Details

The baseline suite spans graph-based, self-supervised, denoising, fusion-oriented, and LLM-enhanced multimodal recommenders. These comparisons test whether AgentMMRec contributes beyond established interaction modeling, semantic graphs, feature fusion, and multimodal content processing.

  • A Baseline Details: MMGCN models user-item interactions with modality-specific item features through graph convolution.
  • A Baseline Details: DualGNN adds an auxiliary user-user graph to capture behavioral similarity among users.
  • A Baseline Details: LATTICE constructs item-item semantic graphs from multimodal features and injects learned item relations into recommendation.
  • A Baseline Details: SLMRec applies self-supervised learning through feature perturbation and modality-pattern discovery tasks.
  • A Baseline Details: HPMRec enriches feature diversity with hypercomplex operations, while EVEN evaluates and denoises multimodal content with observed interactions.

B Knowledge Memory Continuous Updating

The knowledge memory is decoupled from the backbone and can be updated before evaluation. Updating helps initially, but additional rounds plateau and relay quality depends on the reliability of the model supplying feedback.

  • B Knowledge Memory Continuous Updating: The decoupled knowledge memory can be updated and reintegrated before evaluation across dataset-level Recall@20 and NDCG@20 analyses.
  • B Knowledge Memory Continuous Updating: Two and three extra AgentMMRec update rounds produce identical results across datasets and metrics.The third round adds no changes because reranked training lists already satisfy the feedback criterion.
  • B Knowledge Memory Continuous Updating: Relay updating is less stable for weaker models because their early candidate lists provide poor feedback.Stronger models such as SMORE, MENTOR, and COHESION provide more useful relay feedback.
  • B Knowledge Memory Continuous Updating: MMSSL and DiffMM relays slightly reduce several dataset-level scores, while LLMRec and LGMRec remain close to the default memory.This pattern is consistent with relay updates requiring sufficiently reliable candidate lists before errors become useful correction signals.

C LLM Backbone Analysis

The backbone analysis compares Qwen2.5-VL-7B, Qwen2.5-VL-32B, and GPT-4o-2024-08-06. Larger backbones improve results, but gains diminish across scaling levels and vary by dataset.

  • C LLM Backbone Analysis: The analysis compares the main Qwen2.5-VL-7B backbone with Qwen2.5-VL-32B and GPT-4o-2024-08-06.
  • C LLM Backbone Analysis: 0.9% Recall@20 and 1.3% NDCG@20 are the average gains of Qwen2.5-VL-32B over Qwen2.5-VL-7B.
  • C LLM Backbone Analysis: GPT-4o achieves the best results, but its average Recall@20 gain over Qwen2.5-VL-32B is smaller than the 32B-over-7B gain.This indicates diminishing returns from backbone scaling in the current design.
  • C LLM Backbone Analysis: Sports benefits most from stronger backbones, whereas Baby and Clothing show smaller gains.The value of LLM capacity depends on item-domain diversity and ambiguity.

D SSL Compatibility Details

AgentMMRec treats self-supervised modality alignment as optional because behavior-aware knowledge extraction already captures much of the encouraged cross-modal signal. InfoNCE and DisAlign provide small, non-dominant gains across datasets.

  • Both InfoNCE and DisAlign improve most metrics, but their gains are small.
  • Behavior-aware knowledge extraction captures much of the cross-modal signal encouraged by auxiliary alignment objectives.
  • InfoNCE has the strongest Baby Recall@20, while DisAlign has the strongest Clothing Recall@20 and ties on Sports Recall@20.
  • The framework therefore treats SSL as a compatible optional component rather than a necessary part.

E Efficiency and LLM Cost Details

AgentMMRec keeps recommender training efficient through precomputed agent-generated graphs and a single graph-convolution layer, while its main added cost comes from offline LLM preprocessing. The number of preprocessing calls scales with users and items, whereas later updating is event-driven.

  • AgentMMRec stays close to LGMRec and SMORE in per-epoch runtime because its graphs are precomputed and graph enhancement uses one convolution layer.
  • HPMRec is substantially more expensive on all three datasets, while COHESION is the fastest compared strong baseline.
  • Offline LLM calls scale with users and items during knowledge extraction and with items during graph refinement.
  • 86,535 calls are required on Baby, 180,222 on Sports, and 210,293 on Clothing.
  • Training-time reranking and preference updating are event-driven every E epochs and can be capped by an update budget.

F Template Dependency Analysis

AgentMMRec is robust to regenerated prompt wording when the analytical plan, output fields, and multimodal integration plan remain fixed. Across datasets, regenerated templates stay close to the original with small variation.

  • The robustness test regenerates each prompt template five times with GPT-5, Claude-Sonnet-4.5, and Gemini-2.5-Pro while preserving the analytical design.
  • Regenerated templates remain close to the original on Recall@20 across all datasets.
  • Standard deviations range from 0.0006 to 0.0015 across the regenerated template sets.
  • The results indicate that performance depends more on the template plan and memory schema than on specific sentence wording.
Loading 2608.29410v1…