PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Complementary Signals Bridge Similarity Islands? Manifold-Augmented Graph Embedding for Multimodal Recommendation

Zhenye Yang, Jinpeng Chen, Wenbo Fu, Jun Ma, Huan Li, Qimin Zhou, Tao Wang, Li Sun, Senzhang Wang

arXiv:2610.37409v1cs.IR

TL;DR

Similarity-driven multimodal recommendation can leave behaviorally related but semantically distant complements isolated, making ordinary propagation vulnerable to semantic drift. MAGE combines offline complement-aware geometric augmentation with online asymmetric propagation and cross-modal fusion. Experiments report consistent improvements over strong baselines across four datasets, while the authors note limits in graph-noise robustness and transfer evidence.

  • Problem

    Similarity-driven graph construction and propagation favor semantically similar substitutes, while co-purchase complements may be semantically distant and remain geometrically isolated.

    Source: §1 · p. 2

  • Method

    MAGE augments textual geometry with complement-aware residuals under a local gap-preserving constraint, then uses asymmetric visual and textual propagation with self-calibrated fusion.

    Source: §1 · p. 2

  • Results

    Experiments on real-world datasets show that MAGE consistently improves over strong baselines, especially under sparse and long-tail settings.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Additional analyses indicate that MAGE better organizes functionally related yet semantically distant items, supporting the potential of manifold-aware representation learning for complement-aware multimodal recommendation.

    Source: §6 · p. 10

    The tested 30% graph corruptions do not establish robustness to arbitrary noise, extreme sparsity, or temporal distribution shift, and transfer results are mixed on Baby.

    Source: §6 · p. 10

Abstract

from arXiv · show

Multimodal Recommender Systems (MMRS) leverage rich item features to mitigate data sparsity, yet similarity-driven graph construction and propagation can favor semantically similar substitutes over functionally related complements. When behavioral associations connect semantically distant items, complementary signals can remain geometrically isolated, yielding fragmented "Similarity Islands" and semantic drift when standard propagation is applied. To overcome these limitations, we propose MAGE (Manifold-Augmented Graph Embedding), a framework for personalized multimodal ranking that decouples complementarity modeling into offline geometric warping and online topological propagation. The offline phase enhances item representations by extracting complement-aware residual directions via a Soft Conceptor mechanism and injecting them under a local gap-preserving constraint, incorporating co-purchase evidence while controlling the magnitude of geometric perturbation. Synergistically, the online phase employs an asymmetric dual-stream GNN that treats visual features as stable anchors and augmented textual features as flexible probes, fused through a self-calibrated cross-modal mechanism to reconcile shared semantics and modality-specific residual cues. Extensive experiments on four Amazon datasets show that MAGE yields consistent improvements over strong multimodal recommendation baselines. Together with the supporting analyses in the appendix, these results suggest that geometry-aware augmentation can alleviate the semantic mismatch that arises in complementary recommendation settings. Our code is available at https://github.com/Alex-bupt/mage.

1 Introduction

Similarity-driven multimodal propagation favors semantically similar substitutes, while behaviorally related complements may remain distant and risk semantic drift. MAGE combines offline geometric augmentation with online asymmetric propagation; experiments report consistent improvements over strong baselines on four datasets.

1 Introduction

  • Similarity-based propagation favors substitutes, while co-purchased complements can be semantically distant, geometrically isolated, and vulnerable to semantic drift.The paper calls the resulting separation “Similarity Islands.”Source: §1 · p. 2
  • MAGE combines offline complement-aware geometric augmentation with online asymmetric propagation for personalized user–item ranking.Its offline phase injects residual directions under a local gap-preserving rule; its online phase uses visual anchors and augmented textual probes with a Self-Calibrated Gate.Source: §1 · p. 2
  • Experiments on four benchmark datasets show consistent improvements over strong baselines.The introduction also names hard-complement evaluation, fine-grained ablations, original-text controls, and graph perturbations as analyses of the method’s contributions and sensitivity.Source: §1 · p. 2

2 Related Work

Prior multimodal recommendation work develops feature integration, graph propagation, representation learning, and fusion; complementary recommendation and heterophily research address related but distinct challenges. MAGE targets personalized multimodal ranking by coupling local feature transformation with modality-specific propagation.

Multimodal graph recommendation

  • Multimodal graph recommenders develop multimodal propagation, representation learning, and fusion, while MAGE explicitly transforms textual geometry before assigning similarity and complementary connections to different streams.The cited examples include VBPR, MMGCN, MGCN, LGMRec, MENTOR, and GUME.Source: §2 · pp. 2–3

Complementarity and heterophily

  • Complementary-item methods model relevance with dissimilarity, while MAGE addresses personalized multimodal user–item ranking through local feature transformation and modality-specific propagation.SComGNN uses low- and mid-pass spectral filters; MAGE additionally considers how behavioral topology interacts differently with visual and textual features.Source: §2 · p. 3

3 Problem Definition

The task is personalized user–item ranking using multimodal item features, interaction data, and co-purchase relations treated as imperfect complementarity evidence. The model seeks to augment textual geometry while retaining visual anchors to represent substitute and complementary preferences.

3 Problem Definition

  • Co-purchase edges come from item metadata and serve as imperfect evidence of complementarity, not held-out user–item labels.The task also uses binary user–item interactions and visual and textual item embeddings.Source: §3 · p. 3
  • Similarity-based propagation can drift when co-purchased, functionally related items are heterophilic and semantically distant in raw feature space.Similarity propagation encourages connected items to have similar representations.Source: §3 · p. 3
  • MAGE seeks a locality-preserving transformation of textual features and combines them with stable visual anchors for personalized ranking of substitute and complementary preferences.The target is a user–item preference score, not only complement retrieval for a query item.Source: §3 · p. 3

4 Methodology

MAGE separates complementarity modeling into offline geometric transformation and online asymmetric propagation. It augments text using local co-purchase geometry and gap safeguards, then propagates and fuses modality-specific signals with a collaborative backbone for personalized ranking.

4 Methodology

  • MAGE separates offline resolution of feature conflicts through manifold learning from online complementary-signal propagation through an asymmetric graph neural network.Figure 1 is identified as an overview of the framework.Source: §4 · p. 3, Figure 1 · p. 4

4.1 Modality Initialization: Evidence-Bound Extraction

  • A one-time evidence-bound extraction step removes marketing-heavy content and extracts explicitly supported functional attributes before textual embedding.The processed descriptions are frozen; no LLM is called during training or inference.Source: §4.1 · p. 3

4.2 Phase I: Offline Manifold-based Complementarity Injection

  • Phase I transforms textual features before propagation by extracting a complement-aware residual from local co-purchases while limiting distortion of the substitute–complement margin.The adaptive injection magnitude is constrained using local strong/weak geometric proxies; those proxies do not label individual neighbors semantically.Source: §4.2 · pp. 3–4, §4.2.3 · p. 5

4.2.1 Local Relative Subspace Characterization

  • Local relative-percentile partitions define strong and weak co-purchase sets as geometric proxies, and a Soft Conceptor softly characterizes the dominant substitute subspace.Its weights suppress stronger semantic directions more aggressively during residual extraction.Source: §4.2.1 · p. 4

4.2.2 Complement Residual Extraction

  • Residual extraction removes substitute-dominated components from weak neighbors, aggregates filtered signals into a consensus direction, then orthogonalizes it against the seed embedding.The paper cautions that these residuals are candidate complementary signals, not certified semantic complements.Source: §4.2.2 · pp. 4–5

4.2.3 Gap-Constrained Adaptive Injection

  • Adaptive injection limits over-injection by preserving a target gap between strong and weak geometric proxies, with normalization and backtracking checking the constraint.The constraint controls perturbation magnitude but does not prove semantic correctness.Source: §4.2.3 · p. 5

4.3.1 Asymmetric Encoding Strategy

  • Phase II assigns raw visual embeddings to a stable identity-preserving stream and manifold-augmented text to a flexible complement-exploring stream.Each modality uses a separate encoder; the augmented textual representation carries the offline geometric residuals into online learning.Source: §4.3 · p. 5, §4.3.1 · p. 5

4.3.2 Modal-Asymmetric Topology Construction

  • The visual stream smooths on consensus visual–textual similarity edges, while the textual stream propagates over similarity edges augmented with filtered, down-weighted co-purchase edges.The same k bounds each edge source separately, so their union can contain up to 2k neighbors before symmetrization; a collaborative ID backbone also supports fusion and regularization.Source: §4.3.2 · p. 6

5 Experiments

MAGE is evaluated on four Amazon datasets against collaborative-filtering, multimodal, and graph-structure baselines. The experiments report gains on full-test ranking and a targeted hard-complement subset, alongside ablations and graph-corruption tests.

5.1 Datasets and Experimental Inputs

  • The evaluation uses Baby, Sports, Clothing, and Electronics, with co-purchase metadata kept separate from user–item interaction labels used for evaluation.Text descriptions are encoded as 512-dimensional features and visual features as 4096-dimensional embeddings; dataset and implementation details are also reported in appendices.Source: §5.1 · p. 7

5.2 Compared Methods

  • MAGE is compared with collaborative-filtering, multimodal representation-learning, and graph-structure-learning baselines, including recent methods MENTOR and GUME.The comparison set also includes MF-BPR, LightGCN, VBPR, SLMRec, BM3, MGCN, and LGMRec.Source: §5.2 · p. 7

5.3 Overall Performance

  • 2.77%, 4.81%, 6.25%, and 6.53%: MAGE improves Recall@20 over the strongest baseline on Baby, Sports, Clothing, and Electronics, respectively.MAGE achieves the best reported Recall and NDCG values on all four datasets; improvements over the strongest baseline are significant under a paired t-test over five seeds (p < 0.01).Source: Table 1 · p. 8, §5.3 · p. 8

5.5 Ablation Study

  • Removing either Phase I or complementary topology lowers both reported metrics on all four datasets; symmetric propagation and concatenation also underperform the full model.The ablations support contributions from geometric preprocessing, complementary edges, distinct modality roles, and adaptive visual-textual integration.Source: §5.5 · pp. 8–9

5.4 Performance on the Hard Complement Subset

  • 9.87%: MAGE’s mean improvement over the stronger of GUME and MENTOR spans all 16 dataset–metric combinations on the targeted hard-complement subset.The subset selects co-purchase-connected test positives with low visual and textual similarity to a training-history item; it is a diagnostic, not a full-test benchmark.Source: §5.4 · p. 8, Table 2 · p. 9

5.5 Ablation Study

  • Complete Phase I leads all 40 dataset–variant–metric comparisons; the gap constraint has the largest average contribution among safeguards, while orthogonalization has the smallest.Differences are described as modest, without separate significance or indispensability claims. Original-text controls are mixed, while augmentation improves all 24 reported Sports/Clothing metrics across GUME, MGCN, and MENTOR; transfer results do not establish universal generalization.Source: §5.5 · pp. 9–10

5.6 Robustness and Efficiency

  • Dropping or rewiring 30% of eligible co-purchase edges changes Sports/Clothing metrics by at most 1.96% in magnitude under the tested corruptions.Phase I takes 1.89/4.39/4.36 seconds on Baby/Sports/Clothing after embeddings are available, with peak memory below 0.4 GB; these costs exclude text extraction and embedding generation.Source: §5.6 · p. 10

6 Conclusion

The paper concludes that MAGE incorporates complementary signals through manifold augmentation and asymmetric graph propagation, with experiments showing improvements over strong baselines. It also identifies limits to the tested evidence and cautions that offline accuracy does not establish user benefit or fair exposure.

6 Conclusion

  • Experiments show consistent improvements over strong baselines, especially under sparse and long-tail settings, and analyses indicate better organization of functionally related, semantically distant items.The conclusion presents manifold-aware representation learning as promising for complement-aware multimodal recommendation.Source: §6 · p. 10
  • The tested 30% graph corruptions do not establish robustness to arbitrary graph noise, extreme sparsity, or temporal distribution shift; transfer is mixed on Baby.The authors also note imperfect co-purchase evidence, unresolved simpler alternatives, and the need for broader deployment evaluation.Source: §6 · p. 10
  • Co-purchase signals may reflect popularity and commercial exposure biases, while offline accuracy alone does not establish user benefit or fair exposure.The authors recommend monitoring these effects and the reliability of extracted product claims in deployment.Source: §6 · p. 10

A.1 Gap-Constrained Injection Derivation

Phase I adds locally derived complement-aware directions while constraining distortion of the substitute–complement gap. The appendix also details self-calibrated item and user fusion and auxiliary objectives for stabilizing the multimodal representations.

A.1 Gap-Constrained Injection Derivation

  • Complement-aware residuals are extracted relative to each seed’s local geometry and injected under a gap-preserving rule with post-normalization verification to control semantic drift.The adaptive injection strength is constrained by the local substitute–complement gap rather than set as a fixed global coefficient.Source: §A.1 · p. 13

A.2 Self-Calibrated Fusion Details

  • A shared attention-weighted prototype captures category-level identity, while collaborative representations modulate modality-specific residuals before a bidirectional self-calibrated gate refines them.The refined visual and textual residuals are combined with the coarse prototype to form item representations.Source: §A.2 · p. 14
  • User views are derived by propagating visual and textual item signals over the interaction graph, then fused with the same coarse/residual/gating mechanism used for items.The final user representation adds the fused history-induced view to the collaborative ID representation.Source: §A.2 · p. 14

A.3 Auxiliary Objectives

  • Auxiliary objectives include moment matching to reduce modality-scale mismatch and alignment of fused representations with the collaborative backbone.The appendix also describes perturbed views used in the auxiliary objectives.Source: §A.3 · p. 14

B.1 Dataset Statistics

The experiments use four benchmark datasets and a specified MAGE configuration. Text descriptions are generated and validated before representation learning, while the accompanying motivation examines complement dispersion in raw text embeddings.

B.1 Dataset Statistics

  • Table 5 summarizes users, items, interactions, sparsity, and average co-purchase neighbors per item across the four experimental datasets.The supplied passage does not give the individual dataset values.Source: §B.1 · p. 15, Table 5 · p. 15

B.2 Implementation Details

  • MAGE uses 64-dimensional embeddings, local relative-percentile neighbor partitions, Soft Conceptor aperture γ = 12.0, and adaptively gap-constrained injection capped at αmax = 0.35.The reported configuration also uses three interaction-graph GCN layers, one item-graph layer, and ten neighbors for each edge type.Source: §B.2 · p. 15

B.3 Frozen Text Preprocessing and Failure Handling

  • Generated descriptions are frozen before representation learning, supporting reproducibility of downstream runs but not establishing invariance across serving backends.The preprocessing configuration specifies temperature 0.1, a 900-token output limit, and 48 asynchronous workers.Source: §B.3 · p. 15

C Empirical Evidence for Similarity Islands

  • The empirical motivation compares substitute and complement pairs by cosine similarity and visualizes their distributions in the original text embedding space.The supplied passages specify the comparison design but do not report its findings.Source: C Empirical Evidence for Similarity… · pp. 15–16

C.1 Pairwise Similarity Statistics

Substitute pairs are more similar than complementary pairs in both visual and textual spaces, and the text-space visualization shows complements more dispersed around seed items. Together, these observations characterize fragmented Similarity Islands.

C.1 Pairwise Similarity Statistics

  • On Baby, average text similarity falls from 0.7524 for substitutes to 0.3748 for complements, and average visual similarity falls from 0.5614 to 0.2044.Across all four datasets, substitute pairs are consistently more similar than complementary pairs in both modalities.Source: §C.1 · p. 16, Table 6 · p. 16

C.2 Text-Space Visualization

  • Across datasets, substitutes cluster more locally around seed items, while complementary items are scattered across the raw text embedding space.The pattern is especially visible on Baby and Sports and remains noticeable on Clothing and Electronics.Source: §C.2 · p. 16
  • The observed geometry is organized around semantic similarity, so adding graph edges alone may not recover complementary relations without semantic drift under homophily-based propagation.The paper calls this fragmented geometry “Similarity Islands.”Source: §C.2 · p. 16

D.1 Transferability Across Existing Multimodal Recommenders

Offline manifold augmentation produces consistent gains on Sports and Clothing across three multimodal recommender backbones, while results on Baby are mixed. Controls indicate that text replacement alone is less consistent than geometry-aware augmentation combined with graph propagation.

D.1 Transferability Across Existing Multimodal Recommenders

  • Offline augmentation yields clear, consistent gains on Sports and Clothing across GUME, MGCN, and MENTOR without changing their architectures, graph construction, or training objectives.The reported transfer benefit is not restricted to MAGE’s own architecture.Source: §D.1 · pp. 16–17
  • On Baby, offline augmentation gains are marginal or occasionally negative, so transfer benefits depend on dataset and backbone.The experiment does not isolate neighborhood coverage or co-purchase noise as explanations.Source: §D.1 · p. 17

D.2 LLM Text Refinement Alone Is Not Sufficient

  • LLM-enhanced text replacement alone does not yield consistent gains, whereas offline augmentation provides clearer improvements, especially on Sports and Clothing.The reported control supports geometry-aware augmentation combined with graph propagation over stronger text preprocessing alone.Source: §D.2 · p. 17, Table 7 · p. 18

D.4 Original-Text Dependency Control

  • Adaptive, noise-aware injection that calibrates augmentation strength under extreme sparsity is left for future work.The original-text comparison on Sports uses longer validation patience and is not strictly matched.Source: §D.2 · p. 18, Table 9 · p. 18

D.3 Fine-Grained Phase-I Ablation

  • The Phase-I controls are reported descriptively using matched seeds and validation-selected checkpoints, without separate significance claims for individual safeguards.The original-text control does not establish equivalence between preprocessing choices or isolate convergence effects on Sports.Source: §D.3 · p. 18, §D.4 · p. 18, Table 8 · p. 18

E.1 Hard Complement Subset Evaluation

These appendix analyses probe MAGE on behaviorally connected but semantically distant items, illustrate local embedding changes, and examine ranking across popularity groups. The hard-complement subset is a targeted diagnostic, while the primary evaluation remains the full-ranking test set.

E.1 Hard Complement Subset Evaluation

  • The hard-complement subset retains test positives linked to a user’s training-history item by co-purchase and semantically distant in both modalities.The co-purchase graph uses also_bought and bought_together relations; the subset is a targeted diagnostic, while primary evaluation remains full-ranking.Source: §E.1 · p. 19

E.2 Qualitative Analysis: Visualizing the Functional Manifold

  • In the selected t-SNE visualization, MAGE places several co-purchased complements closer to the target and selected semantic hard negatives farther away than GUME.This qualitative pattern is consistent with local reorganization, but a selected two-dimensional projection does not verify compatibility or establish behavior across the item population.Source: §E.2 · p. 19, Figure 3 · p. 20

E.2.1 Additional Geometric Case

  • For the ILBE Main Pack case, 9 of 10 observed top-10 co-purchase neighbors become closer after Phase I, while median text-neighborhood rank improves from 3,872 to 1,441.Mean cosine similarity rises from 0.3442 to 0.4662; selection by geometric gain makes this an illustration, not an unbiased estimate of average improvement.Source: §E.2.1 · p. 19

E.3 Impact on Long-Tail Items

  • MAGE shows favorable ranking performance across the displayed Clothing popularity groups, with gains in several Tail groups and the Head group.The group-level metrics do not isolate the mechanism or establish that popular-item representations or their relative ordering remain unchanged.Source: §E.3 · p. 20, Figure 4 · p. 20

E.3.1 Matched Popularity-Stratified Diagnostic

  • The matched popularity diagnostic fixes Head and Tail groups using training interaction frequency and reports available Tail Recall separately from the hard-complement subset.The supplied table caption identifies the measurement as Tail Recall under matched popularity-stratified evaluation, without supplying values.Source: §E.3.1 · p. 20, Table 10 · p. 20

F.1 Hyper-Parameter Sensitivity Analysis

The sensitivity analyses find intermediate or relatively sparse settings generally favorable, while the preferred augmentation and regularization strengths vary by parameter and dataset. The sweeps characterize behavior only within tested ranges and do not establish one universal configuration.

Sensitivity to the Soft Conceptor aperture γ

  • Across Baby, Sports, Clothing, and Electronics, γ = 12 gives the best or near-best results, while too little filtering is weak and overly aggressive filtering can remove useful semantics.The reported pattern favors moderate subspace filtering; Figure 5 presents aperture sensitivity, with updated Baby R@20 and Sports N@20 values at γ = 12.Source: §F.1 · p. 21, Figure 5 · p. 21

Sensitivity to the keep ratio ρ

  • The displayed keep-ratio sweeps peak at ρ = 0.4 on Sports and ρ = 0.6 on Baby, Clothing, and Electronics, without a monotonic relation between ρ and ranking quality.A larger ρ tightens gap preservation when the residual would reduce a positive initial gap, but the safety cap and normalized-gap check can constrain the update.Source: §F.1 · pp. 21–22, Figure 6 · p. 22

Sensitivity to auxiliary loss weights

  • The best displayed loss-weight results use λalign = λinfo = 10^-2 on Baby, Sports, and Clothing, but λalign = λinfo = 10^-1 on Electronics.Figure 7 encodes Recall@20 in its cells and colors; the Electronics optimum is at the tested upper boundary, and behavior beyond the tested range remains unresolved.Source: §F.1 · p. 22, Figure 7 · p. 23

Sensitivity to topological sparsity in Phase II

  • Performance peaks at a relatively small complementary neighborhood size k, with k = 10 consistently near-optimal and stable across datasets.Figure 8 describes the sweep as sensitivity to Phase-II topological sparsity; the results characterize settings within the evaluated ranges, not a universal optimum.Source: §F.1 · pp. 22, 24, Figure 8 · p. 23

F.2 Robustness to Co-purchase Graph Corruption

This section reports MAGE’s metric changes under tested co-purchase graph corruption and records checklist disclosures about potential societal risks, LLM-based extraction, and benchmark assets.

F.2 Robustness to Co-purchase Graph Corruption

  • The study independently dropped or rewired 30% of eligible co-purchase edges, preserving source degree and edge count during rewiring.For each corrupted graph, it recomputed Phase-I partitions and residuals, reconstructed complementary topology, and retrained from scratch; the interaction and similarity graphs and full-ranking evaluation remained fixed.Source: §F.2 · p. 24
  • The reported stability applies to the tested corruptions and does not establish robustness to arbitrary noise or extreme sparsity.Source: §F.2 · p. 24
  • Under 30% co-purchase graph corruption, relative metric changes stayed within ±1.96%, supporting stability on the tested corruptions.The comparison is against the matched unperturbed configuration.Source: Table 11 · p. 24, §F.2 · p. 24

NeurIPS Paper Checklist

  • The paper identifies potential harms from improved recommendation, including amplified behavioral biases, filter effects, and commercially skewed exposure without monitoring and safeguards.The stated potential benefit is improved retrieval quality and user experience.Source: NeurIPS Paper Checklist · p. 31
  • The paper credits and states sources and licenses for public datasets, code framework, and pretrained model, and reports no new standalone asset release.The external model is used for preprocessing or feature extraction under its official usage conditions.Source: NeurIPS Paper Checklist · p. 32
  • The experiments use existing public benchmark datasets and involve no crowdsourcing, human-subject research, or required IRB review.The authors state that no direct human-subject research was conducted.Source: NeurIPS Paper Checklist · pp. 32–33
  • The method uses an LLM-based evidence-bound extraction step to normalize structured functional information, not to generate final recommendation outputs.Its role, scope, and prompt design are described in the methodology and appendix.Source: NeurIPS Paper Checklist · p. 33
Loading 2610.37409v1…