Agents as Knowledge Integrator and Utilizer in Multimodal Recommendation

1 Introduction

• Multimodal features and similarity graphs can connect visually similar items while missing behaviorally relevant relations, leaving a semantic gap with the recommendation objective [19] [43].Images and descriptions may emphasize color, style, material, or marketing language that is salient but weakly related to user interactions.Source: §1 · p. 1
• Prior LLM recommendation methods use augmentation, fine-tuned backbones, or list reranking, but lack reusable knowledge structure within standard recommender protocols [1] [27] [34].The paper identifies augmentation as content-side, fine-tuning as costly and data hungry, graph construction as decoupled from reusable memory, and pure reranking as late-stage.Source: §1 · p. 1
• The Integrator stores behavior- and multimodal-aware user preferences and item properties in a key-value memory, which the Utilizer converts into refined graphs, representation input, and candidate reranking.The Utilizer refines modality-specific item-item graphs and constructs behavior-aware homogeneous graphs using the same frozen memory.Source: §1 · pp. 1–2
• AgentMMRec improves over recent multimodal recommendation baselines across all metrics, with gains under sparsity and item cold-start and transfer to existing recommenders.The paper also analyzes memory updating, LLM backbone choice, self-supervised objectives, template robustness, and computational cost.Source: §1 · p. 2

2.1 Multimodal Recommendation

• Multimodal recommendation progresses from content-enriched representations to modality-specific, similarity, and higher-order graphs, self-supervised learning, and cross-modal alignment.Examples include VBPR [11], MMGCN, DualGNN, LATTICE, FREEDOM, MMSSL [33], BM3, MENTOR [42], LGMRec [10], and COHESION [40].Source: §2.1 · p. 2
• Despite modality information's usefulness, feature-level similarity remains common, and the semantic gap between multimodal data and recommendation tasks persists [19] [43].AgentMMRec interprets content through behavioral context before constructing homogeneous graphs and enhancing representations.Source: §2.1 · p. 2

2.2 LLM-based Recommendation

• LLM-based recommendation uses content understanding, profiling, attribute extraction, preference retrieval, and reranking, with methods including TALLRec [1], LEARN, and LLMRank [13].KAR [36] and LLMRec [34] use LLM-derived preferences and attributes for sparse feedback and weak side information; UR4Rec retrieves preferences and item knowledge for candidate-aware reranking.Source: §2.2 · p. 2
• Unlike DOGE and MLLMRec [6], AgentMMRec stores preference and property knowledge in memory for behavior-aware graphs, representation enhancement, and frozen-memory candidate reranking.The same knowledge can benefit standard multimodal recommenders while keeping the recommender backbone and LLM components modular.Source: §2.2 · p. 2

3 Methodology

• The Integrator Agent stores behavior-aware user preferences and item properties in reusable memory, which the Utilizer Agent uses for graph refinement, representation enhancement, and candidate reranking.The two-agent design separates knowledge construction from knowledge use and supports transferring the knowledge to other multimodal recommendation backbones.Source: §3 · p. 2

3.1 Problem Definition

• Because the setting has no temporal behavior sequence, the method infers user preferences and item properties from sparse interactions and multimodal item content.Items have text and image content, and observed interactions induce a user-item bipartite graph.Source: §3.1 · pp. 2–3

3.2.1 Behavior-Aware User-User Graph.

• For each user, the Integrator interprets interacted training items to extract textual, visual, and cross-modal preferences, then uses their encoded representations to form a cosine-similarity top-k user graph.When a user has more than Υ interacted items, Υ are sampled to limit context length and inference cost.Source: §3.2.1 · p. 3

3.2.2 Behavior-Aware Item-Item Graph.

• For each item, the Integrator interprets multimodal data from other training items purchased by users who interacted with it, then builds a cosine-similarity top-k graph from extracted textual, visual, and cross-modal properties.If the other-item set exceeds Υ, the method samples Υ items before extracting and storing properties.Source: §3.2.2 · pp. 3–4

3.3 Refining Traditional Item-Item Graphs

• The Utilizer refines modality-specific item graphs by combining text and visual neighbors with stored item properties and purchasing users’ preferences to select neighbors for a unified graph.This incorporates recommendation-task preferences into graphs that would otherwise reflect item features alone.Source: §3.3 · p. 4

3.4 Representation Enhancement and Model Optimization

• AgentMMRec applies LightGCN to the user-item interaction graph and uses the constructed homogeneous graphs and refined unified item graph to enhance user and item representations.The experiments fix LightGCN at L = 3 layers and use one graph-convolution layer for homogeneous graphs; model optimization uses BPR loss.Source: §3.4 · pp. 4–5

3.5 Reranking Final Item List

• The Utilizer reranks each user’s backbone candidate list using stored user preferences and candidate-item properties, while training feedback can trigger preference updates and another reranking pass.It reranks saved top-50 candidates for top-10 or top-20 evaluation; at validation and test time, memory is frozen and no held-out ground truth updates it or the graphs.Source: §3.5 · p. 5

3.6 Algorithm Overview

• Algorithm 1 summarizes the training and inference protocol.Source: §3.6 · p. 5

4.1 Experiment Setup

• Experiments use 5-core Baby, Sports, and Clothing Amazon data, compare 17 multimodal baselines, and report Recall@N and NDCG@N for N=10 and N=20.The data are split 8:1:1; reranking orders 50 candidates, and memory updates, graph construction, and test-time use are controlled to prevent test-interaction leakage.Source: §4.1.1 · p. 5, §4.1.3 · p. 5, §4.1.2 · p. 5

4.2 Overall Performance

• AgentMMRec improves over all baselines across datasets, using extracted user and item knowledge in graph construction, representation learning, and reranking.The reported comparison uses frozen test rankings.Source: §4.1.2 · p. 5, Table 2 · p. 6, §4.2 · p. 6

4.3 Ablation Study

• Removing behavior-aware graphs or graph refinement degrades performance; graph contributions vary by dataset, while reranking and feedback provide a smaller, consistent gain.Removing both graphs hurts more on Sports and Clothing; item-item graphs contribute more on those datasets, while user-user graphs help more on Baby.Source: Figure 2 · p. 6, §4.3 · p. 7

4.4 Compatibility Analysis

• On Sports, graph transfer increases Recall@20 by 4.2% for SMORE, 3.6% for MENTOR, 3.3% for COHESION, and 4.0% for HPMRec.The +Graph variant improves all four selected baselines across all datasets; +Rerank also improves them, usually by less.Source: Table 3 · p. 7, §4.4 · p. 7

4.5 Sparsity Analysis

• AgentMMRec outperforms baselines across user-sparsity groups, with the largest gains for users with 0–5 interactions and positive gains for those with 20–25.Sports and Clothing have larger catalogs and higher sparsity than Baby, and their item-side graphs contribute more.Source: Figure 3 · p. 7, §4.5 · p. 8

4.6 Cold-Start Analysis

• With 20% of items withheld from training, AgentMMRec achieves the best performance across Baby, Sports, and Clothing in the cold-start setting.The withheld items’ text and image fields remain available, while the items are excluded from the training interaction graph.Source: Table 4 · p. 8, §4.6 · p. 8

4.7 Hyperparameter Analysis; 4.9 Efficiency Study

• The Υ threshold trades representative user histories against LLM inference cost, while the update interval trades update frequency against noisy feedback and additional calls.Because agent-generated graphs are precomputed, recommender training per epoch remains close to standard graph-based multimodal recommenders.Source: §4.7 · p. 8, Figure 4 · p. 8, §4.9 · p. 9

4.8 Compatibility with Extra Modality-alignment SSL Tasks

• InfoNCE and DisAlign consistently improve AgentMMRec, with modest average gains of about 0.3%–0.5% in Recall@20 and NDCG@20.Their gains are limited, and the default configuration omits SSL because it adds augmentation, forward passes, and loss computation.Source: §4.8 · pp. 8–9

5 Conclusion

• AgentMMRec consistently improves over recent baselines on three multimodal recommendation datasets, with supporting compatibility, sparsity, cold-start, memory, backbone, SSL, template, and cost analyses.The analyses also examine item cold-start settings.Source: §5 · p. 9
• The Integrator builds behavior-aware user and item knowledge from training interactions, and the Utilizer converts that memory into refined graphs, enhanced representations, and candidate reranking.The framework aims to bridge the semantic gap between multimodal item content and recommendation objectives.Source: §5 · p. 9
• Memory remains frozen during validation and testing, keeping ground-truth feedback out of the reported test results.The conclusion frames LLM-derived semantics as reusable task-aware structure rather than only augmented features or final-stage ranking text.Source: §5 · p. 9

• MMGCN, DualGNN, LATTICE, LGMRec [10], and COHESION [40] compare interaction-graph propagation, user similarity, multimodal item graphs, hypergraph relations, and composite-graph fusion.LATTICE builds item-item semantic graphs from multimodal features, while DualGNN adds an auxiliary user-user behavioral-similarity graph.Source: A Baseline Details · p. 11
• SLMRec, BM3, MMSSL [33], and MENTOR [42] represent self-supervised perturbation, lightweight dropout regularization, adversarial cross-modal alignment, and multi-level alignment baselines.MMSSL separates modality-common from modality-specific signals; MENTOR tests whether preference and property memory adds information beyond representation alignment.Source: A Baseline Details · p. 11
• FREEDOM, DiffMM [15], SMORE [23], EVEN, and FreRec address graph or modality noise using frozen item graphs, diffusion, spectrum analysis, interaction-aware denoising, or frequency-domain methods.FREEDOM separates graph construction from online training, and SMORE is also used in the compatibility analysis.Source: A Baseline Details · p. 11
• LLMRec [34] augments recommendation graphs with LLM-derived knowledge, while BeFA [7] corrects multimodal features using user behavior; AgentMMRec stores behavior- and content-aware knowledge for graphs, representations, and reranking.LLMRec is the closest baseline in LLM-derived knowledge use; BeFA provides behavior-aware feature correction.Source: A Baseline Details · p. 11
• HPMRec [5] supplies a high-capacity hypercomplex fusion baseline to test whether gains stem from knowledge organization rather than richer feature interaction.Source: A Baseline Details · p. 11

B Knowledge Memory Continuous Updating

• Because knowledge memory is decoupled from the backbone recommender, it can be updated and reintegrated before evaluation.Table 5 reports dataset-level Recall@20 and NDCG@20 and their changes relative to default AgentMMRec memory.Source: B Knowledge Memory Continuous Updating · p. 11
• Stronger multimodal relays improve memory more consistently than one extra AgentMMRec update, while weaker models may provide poor feedback.Two and three extra AgentMMRec rounds yield identical results because reranked training lists already satisfy the feedback criterion.Source: B Knowledge Memory Continuous Updating · p. 11
• Relay updating is not monotonic: MMSSL and DiffMM slightly reduce several scores, whereas LLMRec and LGMRec stay close to default memory.The pattern supports the condition that relay errors help correct memory only when candidate lists are sufficiently reliable.Source: B Knowledge Memory Continuous Updating · p. 11

C LLM Backbone Analysis

• The main experiments use Qwen2.5-VL-7B, and the backbone analysis additionally evaluates Qwen2.5-VL-32B and GPT-4o-2024-08-06.Source: C LLM Backbone Analysis · p. 11
• Qwen2.5-VL-32B improves over Qwen2.5-VL-7B by 0.9% in Recall@20 and 1.3% in NDCG@20 on average.The analysis suggests finer multimodal interpretation can improve the extracted memory.Source: C LLM Backbone Analysis · p. 11
• GPT-4o-2024-08-06 obtains the best results, but its average Recall@20 gain over Qwen2.5-VL-32B is smaller than the 32B-over-7B gain.This trend suggests diminishing returns from backbone scaling in the current design.Source: C LLM Backbone Analysis · p. 11
• Sports benefits most from stronger backbones, while Baby and Clothing show smaller gains.The value of LLM capacity depends on the diversity and ambiguity of the item domain.Source: C LLM Backbone Analysis · p. 11

D SSL Compatibility Details

• InfoNCE and DisAlign improve most metrics only slightly, suggesting behavior-aware knowledge extraction already captures much of their cross-modal signal.Table 7 compares the default AgentMMRec with both SSL variants.Source: D SSL Compatibility Details · pp. 11–12, Table 7 · p. 12
• InfoNCE leads on Baby Recall@20, DisAlign leads on Clothing Recall@20, and they tie on Sports Recall@20, so SSL remains optional rather than necessary.The objectives show no consistent dominance.Source: D SSL Compatibility Details · p. 12

E Efficiency and LLM Cost Details

• AgentMMRec stays close to LGMRec and SMORE in per-epoch runtime, while HPMRec is substantially more expensive and COHESION is fastest among strong baselines.Precomputed agent-generated graphs and a single graph-convolution layer for homogeneous graph enhancement keep AgentMMRec's runtime close to those methods.Source: E Efficiency and LLM Cost Details · p. 12
• LLM calls scale with users and items for knowledge extraction and with items for graph refinement; recommender training remains conventional after preprocessing.Deployment cost depends on the selected LLM backend, batching strategy, and provider pricing.Source: E Efficiency and LLM Cost Details · p. 12
• 86,535 calls on Baby, 180,222 on Sports, and 210,293 on Clothing are the deterministic preprocessing totals.The Integrator Agent uses Cint = 3|U| + 3|I| for user and item preference/property extraction; the Utilizer refines the item graph once per item.Source: E Efficiency and LLM Cost Details · p. 12
• Training-time reranking and preference updating are event-driven, attempted every E epochs, and can be capped by the implementation's update budget.These operations are distinct from deterministic preprocessing costs.Source: E Efficiency and LLM Cost Details · p. 12

F Template Dependency Analysis

• The study regenerates each template five times with GPT-5, Claude-Sonnet-4.5, and Gemini-2.5-Pro while preserving analytical dimensions, output fields, and the multimodal integration plan.The full templates for preference extraction, property extraction, graph refinement, reranking, and preference updating are included in the code’s Prompt/ folder.Source: F Template Dependency Analysis · pp. 12–13
• Across datasets, regenerated templates remain close to the original in Recall@20, with standard deviations of 0.0006–0.0015, suggesting greater dependence on the template plan and memory schema than exact wording.Table 10 reports Recall@20 means and standard deviations over five regenerated template sets.Source: F Template Dependency Analysis · p. 12, Table 10 · p. 13