Source-linked AI summary
COHESION: Composite Graph Convolutional Network with Dual-Stage Fusion for Multimodal Recommendation
Jinfeng Xu, Zheyu Chen, Wei Wang, Xiping Hu, Sang-Wook Kim, Edith C. H. Ngai
TL;DR
Multimodal fusion strategies may inadequately handle irrelevant information across modalities, while early fusion can lose modality-specific features during training. COHESION combines a composite network using user-item, user-user, and item-item graphs with early behavior-modality refinement, late representation fusion, and adaptive optimization. COHESION improves on the strongest baseline (FREEDOM or LGMRec) in R@20 on Baby and Sports by 6.05% and 4.41%, respectively.
Problem
Multimodal fusion strategies may inadequately handle irrelevant information across modalities, while early fusion can lose modality-specific features during training.
Source: §2.2 · p. 2Method
COHESION combines a composite network using user-item, user-user, and item-item graphs with early behavior-modality refinement, late representation fusion, and adaptive optimization.
Source: §1 · p. 2Results
COHESION improves on the strongest baseline (FREEDOM or LGMRec) in R@20 on Baby and Sports by 6.05% and 4.41%, respectively.
Source: §4.2 · p. 7Takeaways & Limitations
The paper presents modality fusion and representation learning as complementary, with tailored fusion liberating the composite graph network's representation-learning ability.
Source: §1 · p. 2, §5 · p. 9The paper considers behavior, textual, and visual modalities, while noting that the approach can be extended to more modalities.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Recent works in multimodal recommendations, which leverage diverse modal information to address data sparsity and enhance recommendation accuracy, have garnered considerable interest. Two key processes in multimodal recommendations are modality fusion and representation learning. Previous approaches in modality fusion often employ simplistic attentive or pre-defined strategies at early or late stages, failing to effectively handle irrelevant information among modalities. In representation learning, prior research has constructed heterogeneous and homogeneous graph structures encapsulating user-item, user-user, and item-item relationships to better capture user interests and item profiles. Modality fusion and representation learning were considered as two independent processes in previous work. In this paper, we reveal that these two processes are complementary and can support each other. Specifically, powerful representation learning enhances modality fusion, while effective fusion improves representation quality. Stemming from these two processes, we introduce a COmposite grapH convolutional nEtwork with dual-stage fuSION for the multimodal recommendation, named COHESION. Specifically, it introduces a dual-stage fusion strategy to reduce the impact of irrelevant information, refining all modalities using ID embedding in the early stage and fusing their representations at the late stage. It also proposes a composite graph convolutional network that utilizes user-item, user-user, and item-item graphs to extract heterogeneous and homogeneous latent relationships within users and items. Besides, it introduces a novel adaptive optimization to ensure balanced and reasonable representations across modalities. Extensive experiments on three widely used datasets demonstrate the significant superiority of COHESION over various competitive baselines.
1 Introduction
The introduction asks whether combining user-user and item-item graphs improves multimodal recommendation, and how fusion strategy affects that learning. It presents COHESION as a dual-stage fusion approach paired with a composite graph network.
1 Introduction
- Prior work had explored user-user and item-item graphs separately, leaving open whether using both would improve performance and how.The introduction identifies DualGNN with user-user graphs and LATTICE [36] with item-item graphs.Source: §1 · p. 1
- On Baby, early fusion outperforms late fusion with both user-user and item-item graphs, while late fusion wins when either graph is used alone.The comparison uses variants of MMGCN.Source: §1 · p. 2
- Among early-fusion variants, MMGCN_ui outperforms MMGCN_u and MMGCN_i, suggesting a benefit from combining both graphs in this comparison.The variants incorporate only user-user, only item-item, or both graphs.Source: §1 · p. 2
- COHESION combines a composite graph network with early behavior-modality refinement and late fusion of modality representations.The network uses user-item, user-user, and item-item graphs, and adaptive optimization to balance modality representations.Source: §1 · p. 2
2 Related work
Related work develops multimodal recommendation through content, graph, and self-supervised approaches, while modality fusion commonly occurs early or late. The paper motivates a dual-stage alternative to address irrelevant cross-modal information.
2.1 Multimodal Recommendation
- Multimodal recommendation work enriches item representations with visual or textual content, uses user-item and homogeneous graphs, and explores attention, hypergraphs, diffusion, and self-supervision.Examples include DualGNN, FREEDOM [41], LATTICE [36], MONET [12], MARIO, LGMRec [7], and DiffMM [10].Source: §2.1 · p. 2
- Self-supervised tasks are used to uncover cross-modal preference patterns, but their augmented signals can include misclick behavior or popularity bias and potentially reduce accuracy.Source: §2.1 · p. 2
2.2 Multimodal Fusion
- Early fusion combines modality features with ID embeddings before heterogeneous-graph construction, whereas late fusion learns modality representations and combines predicted ratings at the end.The cited early-fusion methods are [8] [36], and late-fusion methods are [41].Source: §2.2 · p. 2
- Early fusion can lose modality-specific features during training, while predefined fusion strategies are described as fragile to irrelevant cross-modal information.The paper frames these issues as motivation for a tailored fusion strategy.Source: §2.2 · p. 2
- The proposed dual-stage strategy refines modalities through the behavior modality early, then fuses their representations late.The paper presents this as a tailored strategy for multimodal recommendation.Source: §2.2 · p. 3
3 Methodology
COHESION combines behavior-guided early modality refinement and attentive late fusion with heterogeneous and homogeneous graph learning, and uses adaptive BPR weighting to focus on weaker modalities.
3.1 Preliminary
- The model defines users, items, and their interaction graph using behavior, textual, and visual modalities, noting that the design can extend to more modalities.The paper uses three modalities in this work: behavior, textual, and visual.Source: §3.1 · p. 3
3.2 Early Fusion
- Early fusion transforms item features to a common dimension and refines each modality embedding using the behavior modality.The authors say this strategy mitigates the gap between modalities.Source: §3.2 · p. 3
3.3 Heterogeneous Graph
- For each modality, COHESION applies a residual-based GCN over a user-item graph, with summed layer representations used as the final modality embedding.The residual design is intended to alleviate over-smoothing.Source: §3.3 · p. 3
3.4 Late Fusion
- COHESION refines modalities with behavior information before heterogeneous-graph learning, then fuses their representations with learned attention weights.The authors frame early and late fusion as complementary stages for improving representation learning.Source: §3.4 · p. 3, §3.2 · p. 3
3.5 Homogeneous Graph; 3.5.1 User-User Graph.
- Homogeneous user-user and item-item graphs enhance user and item representations; the user graph retains each user's top-k similar-preference neighbors and propagates with Softmax weights.Neighbor scores are based on pairwise cosine similarity.Source: §3.5 · p. 3, §3.5.1 · p. 4
3.5.2 Item-Item Graph.
- The item-item graph retains top-k similar items to balance noise from dense graphs against underused modality features in sparse graphs, then applies weighted aggregation.The resulting user and item representations are enhanced by their respective homogeneous graphs.Source: §3.5.2 · p. 4
3.6 Adaptive Optimization
- Adaptive BPR weighting gives more attention to weak modalities by reversing each modality's positive-versus-negative rating gap.The motivation is that standard BPR optimizes overall preferences but not modality-specific preferences directly.Source: §3.6 · p. 5
4 Experiments
The experiments examine COHESION’s comparative performance, key components, fusion behavior, sparse-data scenarios, efficiency, and hyperparameter sensitivity. The learning process transforms and refines modality embeddings, propagates messages across heterogeneous and homogeneous graphs, then computes an adaptive BPR loss.
4 Experiments
- The evaluation asks how COHESION compares with state-of-the-art models and how its components affect recommendation accuracy.These are the first two of six stated research questions.Source: §4 · p. 5
- The experiments also test fusion’s ability to mitigate irrelevant modality information, performance under data sparsity, model efficiency, and sensitivity to hyperparameters.These correspond to RQ3–RQ6.Source: §4 · p. 5
- The learning process transforms modality embeddings, refines them using behavior modality, and propagates messages in the heterogeneous graph before late fusion.It then constructs user-user and item-item graphs, propagates messages within them, obtains enhanced embeddings, and calculates an adaptive BPR loss.Source: §4 · p. 5
- User-user and item-item graph propagation produces final enhanced embeddings, which feed the adaptive BPR loss.The algorithm specifies propagation in both homogeneous graphs, followed by the enhanced embedding and loss steps.Source: §4 · p. 5
4.1 Settings
The experiments use three Amazon datasets with visual and textual item features, compare against traditional and multimodal recommendation baselines, and evaluate top-K recommendation with Recall and NDCG. Models share specified embedding, optimization, and validation settings.
4.1 Settings
- The study filters Baby, Sports, and Clothing interactions with the 5-core setting and randomly splits them 8:1:1 for training, validation, and testing.The datasets are from Amazon and include visual and textual item modalities.Source: §4.1.1 · p. 5
4.1.1 Datasets.
- Textual features use 384-dimensional sentence-transformer embeddings, while the published visual features have 4096 dimensions.These features are used for the item modalities in Baby, Sports, and Clothing.Source: §4.1.1 · p. 5
4.1.2 Baseline.
- Baselines include traditional recommendation models and multimodal methods spanning matrix factorization, graph-based, self-supervised, and hypergraph approaches.The compared models include MF-BPR, LightGCN, SimGCL, LayerGCN, VBPR, MMGCN, DualGNN, LATTICE, FREEDOM, SLMRec, BM3, and LGMRec.Source: §4.1.2 · pp. 5–6
4.1.3 Evaluation Metrics.
- Recall@K and NDCG@K are reported as average test-user scores at K = 10 and K = 20.Table 3 compares baselines and COHESION using Recall@K and NDCG@K; its asterisk marks statistically significant improvement at p < 0.01.Source: §4.1.3 · p. 6, Table 3 · p. 6
4.1.4 Implementation Details.
- All models use 64-dimensional embeddings initialized with Xavier and are optimized with Adam; COHESION’s validation selection uses Recall@20.The reported implementation uses a grid search for hyperparameters, homogeneous-graph depth of one, top-k of 10 for the user-user graph, and early stopping at 20 epochs, with 1000 total epochs.Source: §4.1.4 · p. 6
4.2 Performance Comparison (RQ1)
COHESION outperforms multimodal recommendation baselines across the reported datasets, with gains over the strongest baseline on Baby and Sports. The comparisons also show that multimodal methods generally outperform traditional recommenders, while multimodal-model performance varies by scenario.
4.2 Performance Comparison (RQ1)
- Most multimodal models outperform traditional recommendation models, while multimodal methods show inconsistent performance across scenarios.The authors state that this pattern verifies multimodal information can alleviate data sparsity in traditional recommendation systems.Source: §4.2 · p. 6
- 6.05% and 4.41% R@20 gains over the strongest baseline (FREEDOM or LGMRec) on Baby and Sports, respectively.The comparison reports these gains for COHESION.Source: §4.2 · p. 7
- COHESION significantly outperforms DualGNN and LATTICE, which achieve competitive performance by adding user-user and item-item graphs, respectively.The authors attribute COHESION’s superiority to dual-stage fusion that liberates the representation capacity of its composite graph convolutional network.Source: §4.2 · p. 7
4.3 Ablation Study (RQ2)
The ablations test early fusion, homogeneous graphs, and adaptive optimization as components of COHESION.
4.3.1 Effect of Early Fusion.
- COHESION outperforms all seven early-fusion variants on Recall@20 across three datasets.Refining any modality improves recommendation accuracy, and refining behavior modality also gives slight gains.Source: §4.3.1 · p. 7, Figure 2 · p. 7
4.3.2 Effect of Homogeneous Graphs.
- User-user and item-item graphs both improve recommendation performance, with the item-item graph better representing user preference than the user-user graph.The reported ordering is COHESION-ui < COHESION-i < COHESION-u < COHESION.Source: Table 4 · p. 7, §4.3.2 · p. 7
4.3.3 Effect of Adaptive Optimization.
- Adaptive optimization significantly improves every tested model on all datasets.Figure 3 maps 500 Baby items across ID, textual, and visual modalities using t-SNE.Source: §4.3.3 · p. 7, Figure 3 · p. 8
4.4 Effectiveness of Mitigating Irrelevant Information among Modalities (RQ3)
COHESION’s early fusion mitigates the negative effects of irrelevant cross-modal information, while multimodal inputs do not always outperform single modalities.
4.4 Effectiveness of Mitigating Irrelevant Information among Modalities (RQ3)
- Only FREEDOM and COHESION outperform single-modality inputs when incorporating multiple modalities; COHESION’s reported improvement is 4.99%.Most existing models suffer weaker performance with multimodal inputs than with a single modality.Source: §4.4 · p. 8
- In a t-SNE visualization of 500 Baby items, dual-stage fusion produces more similar modality distributions than late fusion.The authors interpret this as evidence that dual-stage fusion aligns modality representations.Source: §4.4 · p. 8
4.5 Different Sparse Data Scenarios (RQ4)
COHESION is evaluated across interaction-sparsity levels and for efficiency, including resource use and convergence.
4.5 Different Sparse Data Scenarios (RQ4)
- COHESION consistently outperforms LATTICE, FREEDOM, and LGMRec across all three datasets and user groups with different interaction sparsity.Groups are split by users’ training-set interaction counts; the first group has 0–5 interacted items.Source: §4.5 · p. 8
4.6.1 Time Complexity.
- Late fusion before homogeneous-graph enhancement saves |M| times computation and storage versus DualGNN, LATTICE, and FREEDOM while maintaining similar resource requirements to competitive baselines.COHESION does not adopt self-supervised tasks; the analysis also gives separate costs for heterogeneous graphs, homogeneous graphs, and BPR loss.Source: §4.6.1 · pp. 8–9
- COHESION improves recommendation performance over LATTICE, FREEDOM, BM3, and LGMRec while requiring comparable computation resources and training time.Table 7 reports model parameters and total per-iteration time, including training and evaluation.Source: §4.6.2 · p. 9, Table 7 · p. 9
4.6.2 Computational Resource.
- COHESION converges faster than LATTICE, FREEDOM, and LGMRec on all three datasets while maintaining superior recommendation accuracy.The authors attribute the faster convergence and accuracy to dual-stage fusion mitigating irrelevant information across modalities.Source: §4.6.3 · p. 9, Figure 5 · p. 9
4.7 Hyper-parameter Sensitivity Study (RQ6)
The sensitivity study reports dataset-specific preferred settings for heterogeneous-graph depth, learning rate, and regularization weight.
4.7 Hyper-parameter Sensitivity Study (RQ6)
- The optimal heterogeneous-graph layer counts are 1 for Baby and 2 for Sports and Clothing.The paper relates the Baby result to its smaller user-item association degree and says residual-based GCN can alleviate over-smoothing to some extent.Source: §4.7 · p. 9, Figure 6 · p. 9
- Suggested learning rates are 1e-3, 1e-4, and 1e-3, and regularization weights are 1e-4, 1e-3, and 1e-3, for Baby, Sports, and Clothing, respectively.Source: §4.7.2 · p. 9, Figure 7 · p. 9
5 Conclusion
COHESION combines composite graph convolution, dual-stage fusion, and adaptive optimization to improve multimodal recommendation; experiments and ablations demonstrate its effectiveness.
5 Conclusion
- The composite graph convolutional network leverages heterogeneous and homogeneous graphs for representation learning.Source: §5 · p. 9
- Dual-stage fusion mitigates irrelevant information among modalities, further liberating the composite graph convolutional network’s representation learning ability.Source: §5 · p. 9
- Adaptive optimization controls the balance of attention among all modalities.Comprehensive experiments and ablation studies demonstrate COHESION’s effectiveness.Source: §5 · p. 9