PaperlayerView exact revision on arXiv

Source-linked AI summary

D3ER: Supporting Multi-Modal Recommendation via Disentangle and Distillation-based Dynamic Ensemble

Bingnan Wang, Yi Li, Xiongxin Tang, Fanjiang Xu, Jiangmeng Li

arXiv:2608.25737v1cs.IRcs.MM

TL;DR

Jointly learning HOI and HEI can weaken their individual discriminative effectiveness in multi-modal recommendation. D3ER disentangles these information types and alternately optimizes their models with knowledge-distillation-enhanced gradient boosting; it consistently improves performance across datasets and architectures, including a 7.58% Recall@50 gain over MGCN.

  • Problem

    Joint learning of modal-homogeneity discriminative information (HOI) and modal-heterogeneity discriminative information (HEI) tends to weaken their individual effectiveness.

  • Method

    D3ER uses feature component disentanglement and knowledge-distillation-enhanced gradient boosting to alternately learn HOI and HEI.

  • Results

    7.58% improvement in Recall@50 over the backbone MGCN, with consistent and statistically significant gains across three recommendation architectures.

  • Takeaways & Limitations

    D3ER’s disentangling and dynamic ensembling of modal-homogeneity and heterogeneity features improves multi-modal recommendation performance across three popular datasets.

  • Takeaways & Limitations

    The naive ensemble strategy cannot learn comprehensive sample-oriented discriminative information.

Abstract

from arXiv · show

Incorporating items' information shared among multiple modalities into a fused representation, multi-modal recommendation (MR) has demonstrated documented success than canonical unimodal recommendation. Although several attempts have been made to extract the discriminative information unique in each modality, existing methods suffer from a core limitation: the joint learning of modal-homogeneity discriminative information (HOI) and modal-heterogeneity discriminative information (HEI) tends to weaken their individual effectiveness. To remedy this deficiency, we propose a novel method, dubbed Disentangle and Distillation-based Dynamic Ensemble for multi-modal Recommendation (D3ER). We introduce gradient boosting into MR for the first time to formalize the optimization objective for alternately learning HOI and HEI. This design enables models dedicated to each type of information to focus on their proficient samples, thereby promoting specialized optimization. Furthermore, to mitigate the inherent high storage cost and risk of local optima in gradient boosting, we enhance our framework with knowledge distillation and a global correction regularization. Experiments on prevalent real-world datasets confirm the superiority of our proposed method on MR.

1 Introduction

Multi-modal recommendation combines visual and textual item information, but jointly learning shared HOI and modality-specific HEI can weaken their distinct predictive preferences. D3ER addresses this with disentanglement, gradient boosting, distillation, and global correction.

  • Multi-modal recommendation supplements collaborative filtering with rich item semantics such as visual appearance and textual descriptions.
  • Existing pipelines extract modality-specific features, refine them with modality-specific GCNs, and fuse the resulting embeddings.
  • Jointly learning HOI and HEI can dilute sample-oriented prediction preferences and reduce each information type’s discriminative power.Figure 1 reports consistently inferior Recall@20 for joint optimization compared with independent optimization.
  • D3ER disentangles modality-shared and modality-specific features using inter-modal alignment and intra-modal separation in its FCD module.Alignment is performed at both instance and distribution levels.
  • D3ER introduces knowledge distillation-enhanced gradient boosting so models alternately focus on different sample subsets while reducing gradient-boosting storage costs.The framework also includes global correction regularization to address gradient-boosting weaknesses.
  • The paper identifies comprehensive sample-oriented discriminative information learning as the central challenge and reports empirical evaluations on prevalent MR datasets.

2 Related Work

Related work complements user-item graphs with auxiliary structures, self-supervision, disentangled representations, and ensemble learning. D3ER combines these directions through knowledge-distillation-enhanced gradient boosting for HOI and HEI.

  • MR research supplements user-item bipartite graphs with co-occurrence graphs, hypergraphs, and generated graphs to model user profiles.
  • Self-supervised methods commonly align multi-modal representations with contrastive losses to mitigate BPR-loss sparsity.
  • Earlier disentangled MR methods have defects, including potential feature collapse from PAMD’s Euclidean-distance approach.
  • Prior ensemble-learning work includes AdaBoost, GBDT, AdaNet, BoostResNet, AdaGCN, and multi-model vision-language approaches.
  • D3ER extends this line by combining gradient boosting with knowledge distillation to learn discriminative knowledge for HOI and HEI.

3 Methodology

The methodology section introduces the MR task, defines HOI and HEI, and outlines D3ER before detailing its modules.

  • The paper first presents MR preliminaries, HOI and HEI definitions, and the overall D3ER framework.
  • It then explains each designed module separately.

3.1 Preliminaries and Overview

MR learns user and item representations from interactions and modality features, scores items by inner product, and recommends the highest-scoring items. D3ER separates shared HOI from modality-specific HEI before boosting their recommendation models.

  • Task Formulation: MR learns d-dimensional user and item representations from an interaction matrix and raw modality features.
  • Task Formulation: Recommendation scores are computed by the inner product of user and item embeddings, followed by top-n item selection.
  • Task Formulation: The paper mainly considers visual and textual modalities as a mainstream MR configuration.
  • HOI and HEI: HOI denotes task-dependent information shared across multi-modal contents, whereas HEI denotes task-dependent information specific to each unimodal content.
  • Framework Overview: D3ER’s FCD module obtains modal-shared and modal-specific item representations through inter-modal alignment and intra-modal separation.

3.2 Feature Component Disentanglement

The FCD module separates each modality’s features into modality-shared HOI and modality-specific HEI. It combines instance- and distribution-level alignment with intra-modal separation to obtain task-relevant disentangled representations.

  • Each modality’s raw item features are mapped into shared and specific components for learning HOI and HEI.
  • Inter-Modal Alignment: InfoNCE aligns shared components by treating same-item cross-modal representations as positives and different-item representations as negatives.
  • Inter-Modal Alignment: Instance-level alignment can suffer from false negatives and may force semantically identical shared representations into diverse latent distributions.
  • Inter-Modal Alignment: Distribution-level alignment complements instance-level alignment by matching shared-component distributions across modalities with the Wasserstein distance.
  • Inter-Modal Alignment: The Wasserstein distance is estimated through a linear integral method based on the Radon transform because direct calculation is computationally expensive.
  • Intra-Modal Separation: Intra-modal separation makes specific components dissimilar to shared components using disparity loss and an L2 distance threshold.
  • Optimization Objective: The FCD objective combines three terms, while α_c and α_w balance instance-level and distribution-level alignment and d_m limits task-irrelevant information.

3.3 Knowledge Distillation-enhanced Gradient Boosting

D3ER forms separate recommenders for HOI, visual-HEI, and textual-HEI, then alternately optimizes them through residual fitting, distillation, and global correction. This design targets sample-specific discriminative information while reducing gradient boosting’s storage and local-optimum problems.

  • Model construction: D3ER builds separate recommendation models for HOI, visual-HEI, and textual-HEI, whose scores are combined into a cumulative model.Each model contains corresponding feature extractors and embedding encoders, with GCN-based collaborative filtering models using disentangled features as item node embeddings.
  • Naive ensemble: The naive ensemble jointly optimizes the three recommenders with BPR loss, but cannot learn comprehensive sample-oriented discriminative information.The loss uses observed–unobserved item triplets, their score differences, and a sigmoid function.
  • Boosting step: KDBoost alternately trains recommenders to fit residual discriminative information that the previous cumulative model has not learned.Residuals are measured through the negative gradient of the task loss, directing each model toward samples aligned with its strengths rather than uniformly fitting all samples.
  • Distillation step: Distillation updates the cumulative model without explicitly storing all historical model weights, reducing the memory demands of traditional gradient boosting.The previous cumulative model and current model serve as teacher models, allowing the updated model to inherit historical discriminative information.
  • Global correction: Global correction regularization addresses the cumulative model’s tendency toward poor local minima after greedy residual fitting.The updated cumulative model therefore learns both to fit residual information and to improve overall performance.

3.4 Training Pipeline

D3ER trains disentanglement before ensemble optimization, then saves the optimized cumulative model for top-n recommendation. The pipeline uses separate pretraining and ensemble stages with specified epochs and initialization choices.

  • Training stages: D3ER first pre-trains the FCD module for N epochs, then initializes and optimizes the cumulative model for K epochs.The ensemble stage is guided by Eq. (12) and Eq. (15).
  • Initialization and inputs: The training input includes raw multi-modal features, FCD pretraining epochs N, ensemble epochs K, learning rate lr, and method hyperparameters.The algorithm also takes modality-specific dimensions {d_m | m∈M}.
  • Feature disentanglement: The feature-extractor parameters are initialized before training the extractors to acquire HOI and HEI.The algorithm explicitly separates feature extraction for the two information types.
  • Model initialization: Pretrained feature extractors initialize the cumulative model’s feature-extraction component, while embedding encoders are initialized randomly.This combines pretrained disentanglement with newly initialized recommendation encoders.
  • Ensemble updates: At each ensemble step, the current model is trained on residual discriminative information before the cumulative model inherits knowledge from the previous and current models.The final cumulative model is saved for top-n recommendations.

4 Theoretical Analysis

The theoretical analysis studies D3ER’s representation alignment and KDBoost’s generalization behavior. It argues that Wasserstein-barycenter fusion yields coherent HOI representations, while KDBoost reduces generalization error through bias reduction and model diversity.

  • Theoretical scope: The analysis evaluates the proposed modules theoretically, beginning with Wasserstein-barycenter-based HOI and continuing with KDBoost’s generalization error.The HOI analysis uses learned representation and label spaces with a shared true mapping-function assumption.
  • HOI uniformity: Target error measures the expected absolute difference between a hypothesis and the unknown true mapping function under distribution P.It is defined as ξ_P(f, f*) = E_z∈P[|f(z) − f*(z)|].
  • HOI uniformity: Under K-Lipschitz assumptions, the Wasserstein-barycenter analysis provides a theorem concerning HOI uniformity.The theorem is stated for hypotheses f and f* in the hypothesis class F.
  • HOI uniformity: D3ER minimizes the Wasserstein distance between fused and modality representations, whereas archetypal MR approaches cannot minimize this distance.The resulting fusion constructs a coherent unified representation whose post-fusion error is lower than that of any individual modality representation.
  • KDBoost generalization: KDBoost reduces generalization error by lowering average bias and enhancing the diversity term through alternating optimization and residual-information learning.Each model learns discriminative information left unacquired by the other models.

5 Experiments

D3ER is evaluated across three Amazon-based datasets, multiple recommendation backbones, and standard top-n metrics. It consistently outperforms compared methods, while ablations support the contributions of disentanglement, alignment losses, KDBoost, distillation, and global correction.

  • Experimental Setup: D3ER is evaluated on Baby, Sports, and Clothing datasets from Amazon reviews using visual and textual features extracted by pretrained encoders.CLIP-ViT provides visual features for Baby and Sports, VGG-16 for Clothing, and Sentence-BERT provides textual features for all three.
  • Experimental Setup: Performance is measured with Recall@20, Recall@50, NDCG@20, and NDCG@50 against general and multi-modal recommendation baselines.The comparison includes 15 recommendation models, including MF-BPR, LightGCN, MGCN, DiffMM, PGL, and other multi-modal methods.
  • Performance Comparisons: D3ER achieves state-of-the-art recommendation performance across all three datasets, with PGL integration leading on Sports and Clothing and DiffMM integration leading on Baby.With DiffMM as encoder, D3ER obtains the highest Recall and NDCG on Baby; with PGL, it outperforms all compared methods on Sports and Clothing.
  • Performance Comparisons: 7.58% improvement in Recall@50 over MGCN is reported alongside statistically significant gains across three recommendation architectures.The results are attributed to disentangling and ensembling modal-homogeneity and modal-heterogeneity features, whereas contrastive approaches may overlook HEI.
  • Ablation Studies: Removing FCD, either alignment loss, KDBoost losses, or global correction causes performance degradation, supporting the contribution of each design.Using both instance-level and distribution-level alignment gives the best performance, while removing global correction produces the poorest ablation result because of gradient boosting’s local-optimum risk.

5.4 Hyper-parameter and Visualization Analysis

D3ER’s sensitivity and visualization analyses examine loss weights, distance thresholds, disentangled feature separation, and computational cost. Results show dataset-dependent hyper-parameter effects, separated feature representations, and comparable inference efficiency to the backbone.

  • Weighting Hyperparameters: Performance across all three datasets initially increases and then declines as loss weights αc and αw grow.The two loss terms contribute differently across datasets, making weight selection important for instance-level and distribution-level alignment.
  • Weighting Hyperparameters: Appropriately adjusting αc and αw within the provided hyper-parameter range improves recommendation performance.
  • Threshold Hyperparameters: Too-small distance thresholds fail to capture sufficient discriminative information, whereas overly large thresholds introduce feature noise.Optimal {dv,dt} values vary across datasets and can be selected by grid search.
  • Visualization: t-SNE visualization shows clear separation among three types of disentangled features on the Baby dataset.The features zc and zs_t are projected into two dimensions after training, supporting the intended separation of HOI and HEI.
  • Computational Cost: D3ER has comparable inference speed and memory usage to the backbone model.The disentanglement loss adds negligible training overhead, while the primary additional cost comes from distillation, which reduces storage overhead by avoiding historical model-weight storage.

6 Conclusion

D3ER addresses the inability of naive ensembles to capture comprehensive sample-oriented discriminative information from HOI and HEI. It disentangles modal-shared and modal-specific representations and alternately optimizes their recommendation models through boosting and distillation, with consistent improvements across three popular datasets.

  • Naive ensemble strategies cannot capture comprehensive sample-oriented discriminative information from HOI and HEI.
  • The FCD module decouples item representations into modal-shared and modal-specific components using instance-level and distribution-level inter-modal alignment plus intra-modal separation.
  • KDBoost alternately optimizes recommendation models for HOI and HEI through boosting and distillation steps.
  • D3ER demonstrates consistent performance improvement on three popular datasets.
Loading 2608.25737v1…