PaperlayerView exact revision on arXiv

Source-linked AI summary

Information-Guided Selective Modality-Interest Alignment for Multimodal Recommendation

Wenze Ma, Chenyu Sun, Yanmin Zhu, Qiwen Gu, Xuhao Zhao

arXiv:2608.27950v1cs.IR

TL;DR

MMRec must identify which modality signals reflect user interests because indiscriminate multimodal fusion can include weakly aligned information. AMUR uses information-guided graph refinement and selective shared-interest alignment, and experiments on three real-world datasets demonstrate its effectiveness over competitive baselines while preserving complementary modality information.

  • Problem

    Indiscriminately incorporating modality information can weaken preference modeling, while existing alignment methods often lack a clear objective for selecting signals aligned with user interests.

  • Method

    AMUR refines modality graph structures with behavioral signals and selectively aligns shared-interest semantics across modalities.

  • Results

    Experiments on three real-world datasets demonstrate AMUR's effectiveness over competitive baselines.

  • Takeaways & Limitations

    AMUR enhances interest-related modality semantics while preserving useful modality-specific complementary information.

  • Takeaways & Limitations

    Future work will explore more fine-grained preference-aligned modality modeling and broader recommendation scenarios.

Abstract

from arXiv · show

Multimodal recommendation (MMRec) aims to enhance recommendation performance by leveraging rich item content from multiple modalities. However, directly incorporating all modality information does not necessarily lead to better preference modeling, since user interests are often more related to a subset of modality signals, while other signals may be weakly aligned with user preferences or even introduce noise. Although recent MMRec methods improve modality utilization through invariant learning, attention mechanisms, graph refinement, or contrastive learning, their alignment processes are often implicit or heuristic and lack a clear objective for selecting modality signals that better match user interests. In this paper, we propose AMUR, an information-guided selective modality-interest alignment framework for multimodal recommendation. Inspired by an information-theoretic view, AMUR aims to enhance modality information that is more related to user interests while reducing the influence of less aligned signals. Specifically, AMUR first refines modality graph structures towards user behavior, and then selectively aligns shared interest-related semantics across modalities. This enables AMUR to improve modality-interest alignment while preserving useful modality-specific complementary information. Extensive experiments on three real-world datasets demonstrate the effectiveness of AMUR over competitive baselines. The code is available at https://github.com/Wenze1/AMUR.

1 Introduction

MMRec benefits from multimodal item content, but indiscriminate fusion can propagate signals weakly aligned with user interests. AMUR addresses this through information-guided selective refinement and cross-modal alignment that preserves complementary modality-specific information.

  • Multimodal content provides fine-grained item semantics that can alleviate interaction sparsity and improve preference modeling.
  • Users often rely on only part of modality content, so propagating irrelevant visual or textual signals can produce suboptimal recommendations.
  • Preference-aligned modality content is difficult to identify because item-level interactions reveal likes but not the contributing regions, phrases, or attributes.
  • Existing alignment methods are often implicit or heuristic and lack a clear objective for emphasizing or suppressing modality structures.
  • AMUR uses an information-theoretic view to emphasize modality semantics related to user interests while reducing weakly related residual signals.
  • AMUR refines modality graphs with behavioral signals, aligns selected shared-interest semantics across modalities, and preserves modality-specific complementary information.

2 Methodology

The methodology formulates MMRec around predicting user–item preference scores and top-k recommendations. AMUR instead selectively aligns modality information with user interests to preserve relevant semantics and reduce weakly related signals.

  • MMRec predicts a preference score for each user–item pair and recommends a top-k list of potentially interesting items.
  • AMUR preserves preference-relevant modality semantics while reducing the influence of modality signals weakly related to user interests.

2.2 Overview

AMUR uses collaborative interests to guide modality refinement before selectively aligning shared-interest semantics across modalities. It then combines collaborative and refined modality representations for recommendation.

  • AMUR extracts collaborative interests from user–item interactions and uses them to guide modality refinement.
  • Its first alignment stage refines raw modality graphs with behavior-calibrated signals and pulls modality representations toward user interests.
  • Its second stage aligns only the selected shared-interest subspace across visual and textual modalities, avoiding forced alignment of all modality-specific information.
  • AMUR integrates collaborative and refined modality representations for recommendation and jointly optimizes the objectives.

2.3 Collaborative Interest Encoding

AMUR encodes collaborative signals from the user–item interaction graph as preference anchors. The resulting representations summarize high-order behavioral patterns and calibrate modality representations toward user interests.

  • User–item interactions provide direct preference evidence used to construct the collaborative interaction graph.
  • The interaction adjacency is normalized and ID representations are propagated over the interaction graph following LightGCN.
  • Propagation produces user and item collaborative representations that summarize high-order behavioral patterns from observed interactions.
  • These collaborative representations serve as preference anchors for calibrating modality representations toward user interests.

2.4 Selective Modality-Interest Refinement

AMUR frames modality-interest refinement as an information-guided process that preserves preference-relevant modality information while suppressing weakly aligned signals. It calibrates modality graphs with behavioral co-occurrence and then aligns refined modality representations with user collaborative interests.

  • Information-Guided Objective: Existing MMRec approaches use invariant learning, attention, graph refinement, or contrastive learning, but their alignment strategies are often implicit or heuristic.These strategies make it difficult to interpret which modality information should match user interests and which should be suppressed.
  • Information-Guided Objective: AMUR motivates selective refinement by preserving modality information useful for explaining preferences while reducing weakly interest-related details.The framework uses an information-theoretic view in which mutual information captures preference-relevant semantics and conditional entropy represents residual modality information.
  • Information-Guided Objective: AMUR approximates its information-guided objective with behavior-calibrated graph refinement and interest-aware contrastive learning because direct information estimation is intractable at recommendation scale.The two modules respectively suppress noisy modality structures and enhance user-aligned modality semantics.
  • Behavior-Calibrated Graph Refinement: The modality-initialized graph provides local feature-similarity neighborhoods, but AMUR learns which candidate edges to retain under collaborative-behavior guidance.A neural edge-preservation probability uses item ID embeddings and raw modality similarity, while differentiable binary masks form a preference-calibrated subgraph.
  • Behavior-Calibrated Graph Refinement: Behavioral co-occurrence defines a reference distribution over modality-derived candidate edges, and KL regularization moves learned edge probabilities toward that behavior-informed distribution.Edges with stronger collaborative support are more likely to be retained, while weakly supported modality structures are discouraged.
  • Interest-Aware Contrastive Learning: After graph refinement, AMUR pulls representations of interacted items closer to users’ collaborative interests than to other batch items through an interest-aware contrastive objective.The objective is presented as an InfoNCE-based lower-bound surrogate for mutual information that preserves user-aligned modality semantics.

2.5 Selective Cross-Modal Shared-Interest Alignment

AMUR selectively aligns cross-modal semantics by learning a shared-interest subspace rather than aligning complete modality representations. It combines dimension-wise selection with consistency maximization and discrepancy regularization while preserving modality-specific information.

  • Shared-Interest Subspace Selection: The selected shared-interest subspace enhances cross-modal shared semantics while reducing pressure to make modality-specific complementary information identical.Cross-modal objectives are applied only to the selected subspace, while the original representations retain complementary details.
  • Shared-Interest Subspace Selection: AMUR learns a dimension-wise soft selection gate to identify modality dimensions suitable for cross-modal shared-semantic alignment.The gate produces selection weights in (0, 1)^d and softly reweights each modality representation.
  • Information-Guided Alignment Objective: AMUR decomposes alignment into shared semantic consistency and shared-subspace discrepancy, yielding complementary optimization directions.The mutual-information term is enhanced, while conditional entropy is reduced to regularize unnecessary discrepancy.
  • Shared-Subspace Consistency Maximization: The shared-subspace contrastive loss maximizes agreement between different modalities of the same item and treats modalities of other items as negative pairs.Following InfoNCE, this encourages consistent and discriminative shared semantics across selected visual and textual subspaces.
  • Shared-Subspace Discrepancy Regularization: Shared-subspace discrepancy regularization penalizes unnecessary differences only within the selected shared-interest subspace.Under an isotropic Gaussian formulation, minimizing negative log-likelihood becomes minimizing squared distance between selected subspaces.

2.6 Recommendation and Optimization

AMUR combines collaborative and refined modality representations for final recommendation, using BPR training with positive observed items and sampled negative items. Its overall objective jointly optimizes recommendation, modality-interest refinement, and selective shared-interest alignment.

  • Recommendation: AMUR combines each item’s collaborative representation with its refined visual and textual modality representations for final recommendation.The complete modality representations are used at recommendation time, preserving modality-specific complementary information.
  • Recommendation: The selected shared-interest representations are used only for cross-modal alignment, not as the sole representations for final recommendation.This separates selective alignment from the final representation used to score recommendations.
  • Recommendation: AMUR calculates user-item preference scores from the integrated representations and optimizes them with Bayesian Personalized Ranking loss.A training triplet contains an observed positive item and a sampled negative item for each user.
  • Optimization: The final training objective jointly considers recommendation loss, modality-interest refinement loss, and selective shared-interest alignment loss.Trade-off hyperparameters α and β control the auxiliary objectives.

3 Experiments

Experiments on three multimodal recommendation datasets show that AMUR consistently outperforms diverse baselines, while ablations and analyses support its selective refinement and cross-modal alignment designs.

  • Overall Performance: AMUR achieves the best results across all three datasets and evaluation metrics.The evaluation uses Baby, Sports, and Clothing with Recall@K and NDCG@K.
  • Overall Performance: Compared with the strongest baseline, AMUR improves Recall@20 by 1.56%, 3.96%, and 2.26% on Baby, Sports, and Clothing, respectively.It improves NDCG@20 by 2.24%, 7.78%, and 3.65% on the same datasets.
  • Ablation Study: Removing any AMUR component degrades performance across datasets, supporting contributions from BR, CIE, DR, and CE.BR refines modality structures, CIE aligns representations with interests, DR reduces selected-subspace discrepancy, and CE enhances shared semantics.
  • Ablation Study: Removing CE or DR generally causes clear performance drops, indicating the importance of selective cross-modal shared-interest alignment.The alignment stage exploits shared interest-related semantics after modality representations are refined toward user interests.
  • Ablation Study: AMUR outperforms variants using multimodal embedding-based edge estimation or full-space alignment.These results support behavior-guided graph refinement and selective shared-subspace alignment over the corresponding alternatives.
  • In-depth Analysis: Visualization shows more compact user–item modality clusters and aligned representations, while residual representations are more dispersed and less important.Feature-importance analysis reports generally higher importance values for aligned than residual representations.
  • In-depth Analysis: AMUR remains superior under different modality-corruption ratios and achieves its best coefficient setting around 0.01.Further increases may degrade performance, indicating a need to balance recommendation supervision and auxiliary alignment objectives.

4 Related Work

Prior multimodal recommendation methods incorporate modality information through graph propagation, self-supervision, diffusion, and alignment, but often pay insufficient attention to selective user-interest alignment. AMUR instead targets modality signals that match preferences while reducing less aligned information, with experiments showing an effectiveness–efficiency trade-off.

  • Multimodal recommendation research has used graph-based modeling, self-supervised learning, contrastive objectives, and diffusion-based representation refinement to exploit modality information.
  • Existing approaches mainly incorporate modality information while paying less attention to whether introduced signals align with user interests.
  • Many denoising and alignment strategies are implicit or heuristic, lacking a clear objective for emphasizing or weakening specific modality signals.
  • Full-representation consistency can overlook useful modality-specific complementary information, motivating AMUR's selective alignment focus.
  • AMUR achieves higher Recall@20 than strong baselines while requiring less training time, although MGCN is slightly faster.

5 Conclusions and Future Work

AMUR is proposed as an information-guided framework that refines modality information toward user interests and aligns shared semantics across modalities while preserving complementary information. Experiments on three real-world datasets support its effectiveness, while future work targets finer-grained modeling and broader recommendation scenarios.

  • AMUR refines modality information toward user interests and aligns shared interest-related semantics across modalities while preserving modality-specific complementary information.
  • Experiments on three real-world datasets demonstrate AMUR's effectiveness, and further analyses verify contributions from its key components.
  • Future work will explore more fine-grained preference-aligned modality modeling and extend selective alignment to broader recommendation scenarios.

6 GenAI Usage Disclosure

Generative AI tools were used only for language polishing and manuscript clarity; the authors developed and verified the research ideas, methods, experiments, analyses, and conclusions.

  • Generative AI tools were used solely for language polishing and improving manuscript clarity, while the authors developed and verified the research work.
Loading 2608.27950v1…