Source-linked AI summary
DuTri: Dual-Branch Modeling with Triadic Factorization for Multimodal Multi-Scenario Recommendation
Jintao Huang, Chaohua Yang, Kailiang Hao, Dugang Liu, Xing Tang, Zhong Ming
TL;DR
Multimodal multi-scenario recommendation must model shared and scenario-specific preferences while incorporating item semantics, but existing methods may not accommodate the differing requirements of collaborative and multimodal information. DuTri combines triadic collaborative preference learning with preference-guided multimodal fusion and item-level alignment. Experiments on AntM2C and Amazon report that it outperforms evaluated multimodal and multi-scenario baselines.
Problem
Existing multi-scenario methods often reuse parameter-sharing architectures for multimodal features, although collaborative and multimodal information have different modeling requirements.
Source: §1 · p. 1Method
DuTri couples a collaborative branch that learns structured preferences from user–scenario–item interactions with a multimodal branch that uses preferences for adaptive fusion and item-level alignment.
Source: §1 · p. 2Results
Takeaways & Limitations
The reported experiments and ablations support DuTri’s dual-branch design for multimodal multi-scenario recommendation.
Source: §6 · p. 8The formulation assumes pairwise disjoint scenario datasets, with each instance belonging to exactly one scenario.
Source: §3 · p. 3
Abstract
from arXiv · showhide
As recommendation scenarios become increasingly diverse and multimodal information grows more prevalent, effectively modeling cross-scenario user preferences and multimodal semantics has become a critical challenge in recommender systems. When incorporating multimodal information, existing multi-scenario recommendation (MSR) methods typically treat textual and visual representations as generic input features and directly feed them into their original parameter-sharing architectures. However, this extension may not fully accommodate the different modeling requirements of collaborative and multimodal information and may not be the best modeling paradigm. Our controlled experiments show that directly adopting these architectures is not the optimal modeling approach. To this end, we propose Dual-Branch Modeling with Triadic Factorization (DuTri), a framework for multimodal multi-scenario recommendation comprising a collaborative branch and a multimodal branch. Specifically, the collaborative branch learns structured collaborative preference representations through CANDECOMP/PARAFAC (CP) decomposition of user-scenario-item interactions and residual quantization, thereby capturing cross-scenario commonalities and scenario-specific differences. The multimodal branch leverages the learned user and scenario preferences to guide adaptive modality fusion; additionally, item-level contrastive learning aligns collaborative item preference representations with multimodal representations to facilitate effective collaboration between the two branches. Extensive experiments on two real-world datasets, AntM$^2$C and Amazon, demonstrate that DuTri outperforms existing multimodal and multi-scenario recommendation baselines in overall recommendation performance, validating the effectiveness of the proposed framework.
1 Introduction
Multimodal multi-scenario recommendation must model shared and scenario-specific preferences while using item semantics, but conventional parameter-sharing architectures may not suit both collaborative and multimodal information. DuTri addresses this with collaborative and multimodal branches, and controlled experiments show that conventional MSR models’ ID-only advantage does not carry over to multimodal inputs.
1 Introduction
- Multimodal information can provide item semantics beyond sparse IDs, while multi-scenario recommendation jointly models cross-scenario commonalities and scenario-specific characteristics.The paper motivates combining these capabilities for recommendation across increasingly diverse services.Source: §1 · p. 1
- Figure 1 contrasts early fusion into MSR backbones or a Scenario MLP with a dual-branch modeling paradigm.The controlled comparison retains shared ID embeddings and feature-fusion layers for the Scenario MLP.Source: Figure 1 · p. 2, §1 · p. 2
- With multimodal inputs on AntM2C, Scenario MLP achieves higher mean AUC than all three evaluated MSR models across every scenario, reversing the ID-only comparison.Under ID-only inputs, the evaluated MSR models outperform Scenario MLP; a similar multimodal trend is reported on Amazon.Source: §1 · p. 2, Figure 2 · p. 2
- DuTri combines CP decomposition and residual quantization of user–scenario–item interactions with preference-guided adaptive fusion and item-level contrastive alignment.Shared user and item factors, modulated by scenario factors, capture cross-scenario commonalities and scenario-specific variations.Source: §1 · p. 2
- The paper reports that DuTri outperforms evaluated multimodal and multi-scenario baselines on AntM2C and Amazon, with ablations supporting its key components.The introduction identifies direct reuse of parameter-sharing architectures as a limitation when multimodal features are fused.Source: §1 · p. 2
2 Related Work
Related work spans multi-scenario methods that balance commonality and specificity and multimodal methods focused on representation learning and fusion. DuTri connects these strands by using cross-scenario collaborative preferences to guide multimodal modeling.
Multi-scenario Recommendation
- MSR methods model scenario relationships through mechanisms including task-relation modeling, global and scenario-specific branches, meta-attention, latent-domain assignment, feature interactions, and hierarchical correlations.Other approaches select shared instances or optimize embeddings at the instance level; most are designed around ID-based features and do not explicitly address multimodal fusion.Source: §2 · pp. 2–3
Multimodal Recommendation
- Multimodal recommendation work emphasizes cross-modal interaction and alignment, modality fusion, and semantic knowledge transfer, often without explicit cross-scenario collaborative preferences.The passage describes approaches including user-specific fusion weights, semantic pretraining, adapters, and residual quantization.Source: §2 · p. 3
Multi-scenario Recommendation
- DuTri uses shared user and item preference factors with scenario factors to organize cross-scenario sharing and connect separate collaborative and multimodal branches.User and scenario factors condition fusion of ID, textual, and visual representations, while item factors support alignment with modality representations.Source: §2 · p. 3
3 Problem Formulation
The formulation treats recommendation as binary-feedback prediction over multiple scenarios, with each interaction assigned to one scenario. Its goal is a unified predictor that captures shared and scenario-specific user–item preferences while using collaborative and multimodal information.
3 Problem Formulation
- Each interaction belongs to exactly one scenario, and the scenario datasets are pairwise disjoint.Textual and visual item embeddings are extracted offline using pretrained encoders.Source: §3 · p. 3
- The goal of multimodal multi-scenario recommendation is to learn a unified prediction function that captures cross-scenario shared and scenario-specific user–item preferences while effectively leveraging collaborative and multimodal information.Source: §3 · p. 3
- The model uses scenario-shared and scenario-specific parameters and is trained by minimizing average binary cross-entropy over instances.The formulation distinguishes item identifiers, textual information, and visual content as item modalities.Source: §3 · p. 3
4 The Proposed Framework
DuTri couples a collaborative branch that learns user–scenario–item preferences with a multimodal branch that uses those preferences for fusion. Their outputs, alongside the user representation, feed scenario-specific prediction.
4.1 Framework Overview
- DuTri factorizes user–scenario–item interactions, quantizes the resulting preference factors, and uses them to construct collaborative and multimodal representations.The branches are coupled through preference representations and item-level contrastive alignment during training.Source: §4.1 · p. 3, §4.1.2 · p. 4, §4.1.3 · p. 4
4.1.1 Modality Feature Embedding.
- User and item IDs and pretrained text and image features are projected into a common space, while the user representation remains outside item-side fusion.The text and image encoders are frozen and applied offline.Source: §4.1.1 · pp. 3–4
4.1.4 Prediction.
- The user, collaborative, and multimodal representations are concatenated and passed to the prediction head for the current scenario.Prediction heads share an architecture but have separate parameters; training combines recommendation and auxiliary objectives.Source: §4.1.4 · p. 4, §4.4 · p. 5
4.2.1 Triadic Factorization of Preference Factors.
- CP factorization models user–scenario–item interactions with user and item factors shared across scenarios and scenario-specific factors.The factors are parameterized separately from the ID embedding.Source: §4.2.1 · p. 4, §4.1.2 · p. 4
- Residual quantization represents continuous preference factors with discrete codewords and reconstructs them for subsequent modeling.The quantization levels are entity-specific, and the quantizers are shared across scenarios.Source: §4.2.1 · p. 5
4.2.2 Preference-Modulated Item Representation.
- The collaborative representation uses a feature-wise modulation of projected item preference by the reconstructed user, scenario, and item preferences.The user and scenario projections also condition the multimodal branch.Source: §4.1.2 · p. 4, §4.2.2 · p. 5
4.3 Multimodal Branch
- The multimodal branch generates modality weights from projected user and scenario preferences, then normalizes and aggregates item-side modality representations.The gate is shared across scenarios; the fused representation enters prediction alongside the collaborative representation.Source: §4.3 · p. 5, §4.3.1 · p. 5, §4.3.2 · p. 5
4.4 Optimization
- Item-level contrastive alignment treats instances of the same item as positives when aligning collaborative item preferences with modality representations.All batch instances are candidates, and the loss uses a temperature parameter.Source: §4.4 · p. 5
5 Experiments
The experiments compare DuTri with multimodal and multi-scenario baselines, test its components and alternative modeling strategies, and assess efficiency and hyperparameter sensitivity. DuTri leads the evaluated baselines on both datasets, while ablations and strategy comparisons support its design.
5.1.1 Datasets.
- The evaluation uses AntM2C’s five industrial scenarios and three Amazon categories, reporting per-scenario AUC and sample-weighted WAUC across five-seed runs.Amazon retains users active in all three categories, binarizes ratings, and splits chronologically within each user–scenario pair; AntM2C uses a global chronological split.Source: §5.1.1 · p. 6, §5.1.2 · p. 6, §5.1.4 · p. 6
5.1.3 Baseline Methods.
- The baselines include multimodal methods, multi-scenario methods, AMS, and Scenario MLP; methods share data splits, available IDs, multimodal features, and evaluation protocol.Hyperparameters are tuned independently using validation WAUC, and the highest-validation-WAUC checkpoint is evaluated on the test set.Source: §5.1.3 · p. 6, §5.1.4 · p. 6
5.2 RQ1: Performance Comparison
- DuTri achieves higher mean AUC than evaluated multimodal baselines in every scenario on both datasets and higher mean WAUC than evaluated multi-scenario baselines on both datasets.It also outperforms Scenario MLP and AMS; Table 1 reports results on both datasets.Source: §5.2 · p. 6, Table 1 · p. 7
5.3 RQ2: Ablation Study
- Ablations support alignment, triadic CP supervision, quantized preferences, preference-conditioned fusion, and all three prediction representations; removing the multimodal representation causes the largest input-removal drop.Removing either branch with its auxiliary losses causes a larger WAUC drop than removing only that branch’s prediction input; MF Supervision, Pairwise Mod, and USI Gate also underperform DuTri.Source: §5.3 · pp. 7–8, Table 2 · p. 7
5.4 RQ3: Alternative Modeling Strategies
- Compared with direct input, shared-specific and shared-expert quantization achieve higher mean WAUC, while DuTri achieves the highest mean WAUC among evaluated strategies on AntM2C.Quantized Input alone achieves lower mean WAUC than Direct Input.Source: Table 3 · p. 8, §5.4 · p. 8
5.5 RQ4: Hyperparameter Analysis
- On AntM2C, DuTri trains slower than AMS but faster than Diff-MSIN and PLE, has inference time comparable to AMS, and uses fewer parameters than PLE and HierRec.Its additional training-time and storage costs accompany relatively stable performance across tested CP weights; alignment weight and temperature show more pronounced peaks.Source: §5.5 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8
6 Conclusions
The paper concludes that DuTri combines triadic collaborative preferences with multimodal fusion and achieves the best overall performance among evaluated methods. The accompanying Amazon motivational comparison uses AUC by scenario under ID-only and multimodal inputs.
6 Conclusions
- DuTri achieves the best overall performance among evaluated methods on AntM2C and Amazon, with competitive inference efficiency; ablations and alternative strategy comparisons support its design.The conclusion characterizes DuTri as learning triadic collaborative preferences to guide multimodal fusion.Source: §6 · p. 8
A.1 Motivational Comparison on Amazon
- The Amazon comparison reports AUC for scenarios S1–S3 across MSR backbones and Scenario MLP under ID-only and multimodal inputs, averaged over five random seeds.The comparison retains shared ID embeddings and multimodal fusion layers while replacing the post-fusion MSR backbone with independent scenario-specific MLP towers.Source: Figure 6 · p. 10, §A.1 · p. 10
A.2 Additional Efficiency Analysis on Amazon
On Amazon, DuTri trades higher training and storage costs for recommendation performance while keeping inference time comparable to AMS. Figure 7 compares per-batch training and inference times and parameter counts.
A.2 Additional Efficiency Analysis on Amazon
- DuTri achieves higher overall WAUC with additional training and storage costs, reflecting a tradeoff between predictive performance and computational efficiency.Figure 7 compares per-batch training and inference times and parameter counts.Source: §A.2 · p. 10, Figure 7 · p. 10
- DuTri requires less training time than Diff-MSIN, PSRQ, and PLE but more than AMS.The comparison uses the same evaluation protocol on Amazon.Source: §A.2 · p. 10
- DuTri has the largest parameter count among compared methods, while its inference time remains comparable to AMS.Auxiliary alignment losses and projection heads are omitted during inference, so their training costs do not carry over to online prediction.Source: §A.2 · p. 10
B.1 Dataset Statistics and Preprocessing
The experiments use AntM2C, an industrial benchmark with five production scenarios, and Amazon, represented by three product categories. Their preprocessing and temporal splits follow dataset-specific protocols.
B.1 Dataset Statistics and Preprocessing
- AntM2C contains click logs from five production scenarios and uses the official release with multimodal feature extraction settings from prior work.Low-frequency users were already filtered; missing images are zero vectors, and testing uses the last four days after a two-day validation period.Source: §B.1 · p. 10
- Amazon uses Sports and Outdoors, Movies and TV, and Books as its three scenarios, retaining users active in all categories with at least five total interactions.Ratings above 3 count as positive; interactions are chronologically split within each scenario into 80% training, 10% validation, and 10% testing.Source: §B.1 · p. 10
- AntM2C text and visual features use 768- and 512-dimensional representations, while Amazon uses 384-dimensional text and released 4,096-dimensional visual representations.AntM2C text and images use Multilingual BERT and Chinese-CLIP; Amazon text concatenates product title, brand, category, and description, and missing visual features use the available-feature mean.Source: §B.1 · p. 10
B.2 Baseline Adaptation and Implementation
The study adapts multimodal and multi-scenario baselines to a shared input and evaluation pipeline. It also evaluates AMS and a Scenario MLP comparison under the same data splits and input features as DuTri.
B.2 Baseline Adaptation and Implementation
- Multimodal baselines are trained on the union of scenario interactions, then evaluated per scenario with AUCs aggregated into WAUC.This group includes Diff-MSIN, LMF, PSRQ, SimCEN, and DMF, which do not explicitly model scenario-specific interaction distributions.Source: §B.2 · p. 10
- Multi-scenario baselines receive early-fused inputs by concatenating item ID, textual, and visual representations before their original architectures.All baselines use DuTri’s textual and visual features, and hyperparameters are tuned independently on validation data.Source: §B.2 · pp. 10–11
- AMS uses adaptive modality scheduling before multimodal fusion and a downstream multi-scenario backbone; Scenario MLP instead uses independent scenario prediction towers.Scenario MLP retains shared ID embeddings and multimodal fusion layers, and uses the same data splits and input features as DuTri.Source: §B.2 · p. 11
B.3 Implementation Details and Hyperparameter Settings
DuTri uses a common representation space, scenario-specific prediction networks, and CP factor quantization, with shared training and auxiliary-objective settings. The factor quantization section describes jointly training residual-quantized user, scenario, and item factors.
B.3 Implementation Details and Hyperparameter Settings
- All models use Adam, 128-dimensional ID embeddings, and batch size 512; five random-seed runs use checkpoints selected by validation WAUC.Learning rate and ℓ2 regularization are selected from three-value candidate sets, and test evaluation uses the checkpoint with the highest validation WAUC.Source: §B.3 · p. 11
- DuTri maps ID, text, and visual representations to 64 dimensions and uses three-hidden-layer scenario-specific prediction networks; the CP rank is 16 on both datasets.The prediction networks have dimensions [1024, 512, 256] and share architecture but not parameters.Source: §B.3 · p. 11
- User, scenario, and item factors use separate four-level residual quantizers with dimension 32; each level has 256 user/item codewords and 8 scenario codewords.The commitment coefficient β is 0.25.Source: §B.3 · p. 11
- The alignment space is 64-dimensional, and the three modalities share an alignment weight selected from the specified candidate set.The auxiliary-objective settings also tune λCP and contrastive temperature τ from supplied candidate sets.Source: §B.3 · p. 11
C Collaborative Factor Quantization
- Residual quantization is applied to user, scenario, and item CP factors, which are trained jointly with the rest of DuTri.The section specifies the architecture, forward computation, and auxiliary objectives.Source: C Collaborative Factor Quantization · p. 11
C.1 Architecture
The architecture applies entity-specific residual quantizers to CP factors, then uses reconstructed preferences in downstream modeling. Its training objectives combine factor reconstruction and quantization, with batch handling and codebook use specified separately.
C.1 Architecture
- Each entity type maps 16-dimensional CP factors to a 32-dimensional latent space and uses four quantization levels; user/item codebooks have 256 entries per level and scenario codebooks have 8.The commitment coefficient is 0.25.Source: §C.1 · p. 11
C.2 Forward Computation
- The quantizer selects codewords from residuals and sums them across levels; straight-through estimation routes gradients to the encoder output, and the result replaces CP factors downstream.The reconstructed factors are used in preference representations ep, eq, and er.Source: §C.2 · pp. 11–12
C.3 Training Objectives
- The residual quantization objective combines factor reconstruction, which trains the encoder and decoder and pulls CP factors toward reconstructions, with codebook and commitment losses.Losses are averaged over entities or levels as specified, summed over entity types, and LRQ enters the overall objective with unit weight.Source: §C.3 · p. 12
C.4 Batch Handling
- User and item factors are quantized once per distinct mini-batch entity, while the scenario quantizer processes all scenario factors at every step.Duplicated users and items share reconstructions and contribute once to the factor and quantization losses; CTR and CP losses are computed per sample.Source: §C.4 · p. 12
C.5 Codebook Usage and Discrete Codes
- Codeword usage is monitored without dead-code replacement, and full CP factor tables remain in training and inference, so quantization is a forward-pass bottleneck rather than compression.The codebooks do not replace the factor tables with stored indices; no index export or lookup is performed.Source: §C.5 · p. 12