PaperlayerView exact revision on arXiv

Source-linked AI summary

Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited

Zheng Yuan, Fajie Yuan, Yu Song, Youhua Li, Junchen Fu, Fei Yang, Yunzhu Pan, Yongxin Ni

arXiv:2303.13835v4cs.IR

TL;DR

ID-based recommenders have long dominated, while modern text and vision encoders motivate revisiting modality-based recommendation. The paper systematically compares MoRec and IDRec under matched architectures and training settings, finding that end-to-end MoRec can perform on par with or better than IDRec even for non-cold-start items. It also examines how advances in NLP and CV transfer to recommendation and identifies practical limitations and deployment challenges.

  • Problem

    IDRec dominates recommendation but suffers from cold-start, cross-platform transferability, maintenance, and limited use of advances in NLP and CV.

  • Method

    The paper conducts fair empirical comparisons of IDRec and MoRec using matched DSSM and SASRec architectures, text and vision encoders, and two-stage versus end-to-end training.

  • Results

    Modern MoRec with SOTA end-to-end-trained modality encoders performs on par with or better than IDRec in non-cold-start recommendation.

  • Takeaways & Limitations

    MoRec can benefit from technical advances in NLP and CV and may challenge IDRec’s prevailing role when raw modality features are available.

  • Takeaways & Limitations

    The study covers only text and vision, singlemodal encoders, basic fusion, and three medium-sized datasets rather than industrial-scale data.

Abstract

from arXiv · show

Recommendation models that utilize unique identities (IDs) to represent distinct users and items have been state-of-the-art (SOTA) and dominated the recommender systems (RS) literature for over a decade. Meanwhile, the pre-trained modality encoders, such as BERT and ViT, have become increasingly powerful in modeling the raw modality features of an item, such as text and images. Given this, a natural question arises: can a purely modality-based recommendation model (MoRec) outperforms or matches a pure ID-based model (IDRec) by replacing the itemID embedding with a SOTA modality encoder? In fact, this question was answered ten years ago when IDRec beats MoRec by a strong margin in both recommendation accuracy and efficiency. We aim to revisit this `old' question and systematically study MoRec from several aspects. Specifically, we study several sub-questions: (i) which recommendation paradigm, MoRec or IDRec, performs better in practical scenarios, especially in the general setting and warm item scenarios where IDRec has a strong advantage? does this hold for items with different modality features? (ii) can the latest technical advances from other communities (i.e., natural language processing and computer vision) translate into accuracy improvement for MoRec? (iii) how to effectively utilize item modality representation, can we use it directly or do we have to adjust it with new data? (iv) are there some key challenges for MoRec to be solved in practical applications? To answer them, we conduct rigorous experiments for item recommendations with two popular modalities, i.e., text and vision. We provide the first empirical evidence that MoRec is already comparable to its IDRec counterpart with an expensive end-to-end training method, even for warm item recommendation. Our results potentially imply that the dominance of IDRec in the RS field may be greatly challenged in the future.

1 INTRODUCTION

The paper revisits whether modern modality-based recommenders can challenge ID-based recommenders in practical recommendation settings. It evaluates modality encoders, training paradigms, and fair comparisons across text and vision recommendation.

  • Motivation: IDRec dominates recommender systems but depends heavily on interaction data, limiting recommendations in cold-start settings.It is also not transferable across platforms because user and item IDs are generally not shareable.
  • Motivation: MoRec replaces ID embeddings with item modality encoders to address IDRec’s cold-start, transferability, and maintenance weaknesses.Earlier MoRec work mainly targeted cold-start or cross-domain recommendation and was rarely used for popular items.
  • Research questions: The study asks whether modern MoRec can match or surpass IDRec for regular and warm-start recommendations using text and vision modalities.Experiments use DSSM and SASRec with four modality encoders across three large-scale recommendation datasets.
  • Research gap: The authors emphasize that prior work lacked fair comparisons using identical backbones, samplers, loss functions, and experimental settings.They also distinguish their study by examining expensive seq2seq training and both text and vision recommendation.
  • Method: They compare frozen two-stage modality features with joint end-to-end training to test whether foundation-model representations can transfer directly to recommendation.Additional experiments vary encoder size, encoder strength, and pre-training on NLP or CV datasets.
  • Open challenges: The work identifies unresolved practical challenges for MoRec under the end-to-end learning paradigm.The introduction frames these challenges as an additional research focus beyond the main comparison questions.

2 IDREC & MOREC

IDRec and MoRec share the recommendation architecture but differ in how item representations are produced. The paper formalizes this replacement and uses matched training procedures to compare the paradigms fairly.

  • Model formulation: A recommendation model represents users and items and computes their matching score for future interactions.Items may be represented by unique IDs or modality content, while users may be represented by IDs or profiles such as interaction sequences.
  • IDRec: IDRec retrieves each item’s vector from a learnable item ID embedding matrix and feeds it into the recommendation network.The matrix has one vector per item and embedding dimension d.
  • MoRec: MoRec uses a modality encoder to generate an item representation from raw content, replacing the ID embedding in IDRec.Text items can use BERT or RoBERTa, while visual items can use ResNet or ViT.
  • Fair comparison: The models retain identical network architectures except for the item encoder, with modality outputs passed through a dimension-transformation layer.IDRec is fully tuned first, after which MoRec uses the same hyperparameters and searches around the best settings.
  • Architectures: The study evaluates DSSM and SASRec by replacing their item ID embeddings with modality encoders to create corresponding MoRec models.DSSM is two-tower based, whereas SASRec is a sequential model using multi-head self-attention over interacted items.
  • Training: Both IDRec and MoRec use randomly sampled negative interactions and binary cross entropy loss for a fair comparison.For each observed positive interaction, one unobserved negative interaction is sampled during each training epoch.

3 EXPERIMENTAL SETUPS

The experiments use three real-world recommendation datasets, modality-specific item inputs, and matched IDRec/MoRec architectures for fair evaluation. Models are trained and tuned under shared preprocessing, optimization, splitting, and top-N ranking protocols.

  • 3.1 Datasets: The study evaluates MIND news clicks, HM clothing purchases, and Bili video comments as implicit click-signal datasets.Purchases and comments are treated as implicit clicks because users presumably clicked before purchasing or commenting.
  • 3.1 Datasets: Items are represented by news titles for MIND and cover images for HM and Bili, although HM behavior may also depend on price and sales volume.The authors note that no publicly available dataset fully guarantees decisions depend solely on modality content.
  • 3.1 Datasets: The datasets apply modality-specific preprocessing, including 224 × 224 images, news titles capped at 30 tokens, and recent interaction sequences.MIND uses the latest 23 items per user, while HM and Bili use the 13 most recent interactions because image encoding requires more GPU memory.
  • 3.3 Comparison Settings: IDRec and MoRec use exactly the same network architecture except for the item encoder, with modality outputs passed through a dimension-transformation layer.The authors caution that MoRec hyperparameters are not always exhaustively searched because vision experiments can require over 100× the compute and training time of IDRec.
  • 3.4 Evaluations: Table 2 compares regular-setting accuracy for DSSM and SASRec using IDRec and fine-tuned MoRec encoders including BERT, RoBERTa, ResNet, and Swin Transformer.The table reports accuracy percentages and the relative improvement of the best MoRec over the best IDRec.

4 COMPARATIVE STUDIES (Q(I))

The study compares pure MoRec and IDRec under matched recommendation settings, focusing on regular and warm-start item recommendation where IDRec is typically strong. MoRec’s effectiveness depends heavily on the recommendation backbone, modality encoder, and item popularity.

  • Experimental setup: The experiments evaluate DSSM and SASRec with text and vision encoders on regular and warm-start recommendation datasets.IDRec and MoRec use the same backbone and experimental settings, differing in the item encoder.
  • Regular setting: SASRec-based IDRec is around 4.9× better than DSSM-based IDRec in HR@10 for news recommendation, while image-recommendation gaps are around 1.4× on HM and 2.7× on Bili.The training, validation, and testing sets are identical for the news comparison.
  • Regular setting: With DSSM, MoRec performs much worse than IDRec across all three datasets, especially in visual recommendation.This remains true even when MoRec uses state-of-the-art modality encoders.
  • Warm item settings: MoRec is slightly better than IDRec on warm-20 but slightly worse on warm-200, as IDRec strengthens with increasing item popularity.The warm-start datasets remove items with fewer than 20, 50, or 200 interactions.
  • Conclusion: Overall, SASRec-based MoRec can beat IDRec for text recommendation and match it for visual recommendation, including challenging warm-item settings.The paper reports little chance for MoRec to replace IDRec under typical DSSM training, although MoRec remains competitive for very popular items.

5 INHERIT ADVANCES IN NLP & CV? (Q(II))

The paper tests whether advances in NLP and CV transfer to MoRec through encoder scale, encoder strength, and pre-training. In general, stronger or larger encoders and pre-training improve recommendation accuracy, though the pattern is not universal.

  • Study design: The study examines whether NLP and CV advances improve MoRec by comparing encoder size, encoder strength, and pre-training against training from scratch.These experiments assess whether modality encoders transfer their progress from classification and other modality tasks to recommendation.
  • Encoder size: Larger vision encoders consistently improve image recommendation, following ResNet18 < ResNet34 < ResNet50 and Swin-T < Swin-B.For text, BERTtiny < BERTbase < BERTsmall, although BERTbase does not outperform BERTsmall.
  • Encoder strength: MoRec accuracy follows the corresponding NLP and CV strength rankings: RoBERTabase > BERTbase > OPT125M > TextCNN > GloVe, and Swin-T > ResNet50.The comparisons use similarly sized encoder pairs where stated.
  • Pre-training: Pre-trained MoRec improves over training-from-scratch MoRec, with pretrained ResNet and Swin producing around 10% gains on HM and Bili.Pre-training advantages are more obvious on smaller datasets, but BERTbase is worse than its training-from-scratch version on MIND-50K.
  • Conclusion: Overall, MoRec can inherit advances from NLP and CV, leaving opportunities for future improvement as those fields develop.The conclusion concerns MoRec’s ability to connect recommendation with progress in multimedia representation learning.

6 ARE MODALITY REPRESENTATIONS UNIVERSAL FOR RS? (Q(III))

The paper evaluates whether modality representations can be used directly for recommendation or require adaptation. Frozen two-stage features perform poorly, while end-to-end retraining is substantially stronger, especially for visual recommendation.

  • Training paradigms: The study compares a two-stage pipeline using frozen modality features with end-to-end training that jointly optimizes user and item encoders.Two-stage training is attractive in industry because of its high training efficiency.
  • Results: Two-stage MoRec performs poorly relative to IDRec and end-to-end MoRec, achieving only around 60% of end-to-end performance on HM and 25% on Bili with ResNet.Adding dense layers improves two-stage performance but does not close the gap, especially for visual recommendation.
  • Interpretation: The results indicate that modality features learned for NLP and CV tasks are not universal enough for recommendation and perform worse than retraining on recommendation data.This conclusion specifically concerns direct reuse of pre-trained representations in the recommendation setting.
  • Adaptation: TS-DNN adaptation brings two-stage MoRec closer to end-to-end MoRec for text recommendation, with reported values of 16.66 versus 18.23.The paper describes this as potential for future competition rather than parity across settings.
  • Practical implication: The popular two-stage mechanism causes significant performance degradation, especially for image recommendation, and should not be ignored in practice.The authors also state that current foundation-model representations are not yet universal enough for item recommendation.

7 KEY CHALLENGES (Q(IV))

The paper identifies practical challenges in E2E MoRec, including substantial training costs, optimization instability, and dataset-dependent benefits from extra pre-training. Combining ID and modality features does not consistently improve accuracy.

  • Training cost: Larger modality encoders generally improve MoRec performance but also increase training compute, time, and GPU memory, especially for long seq2seq interaction sequences.
  • Practical deployment: E2E-based MoRec had not yet been successfully deployed in real-world recommender systems according to the paper.
  • Training cost: More than 100x higher compute and training time than IDRec makes the best E2E MoRec substantially more expensive to train.The comparison uses SASRec as the user encoder and Swin-B as the item modality encoder; inference remains as fast as IDRec.
  • Extra pre-training: Extra pre-training improves text MoRec on MIND for both TS and E2E models, while vision gains depend on the dataset and are marginal or absent in some settings.MAEbase-MLM improves TS MoRec on HM, gives marginal E2E gains, and yields no improvement on Bili.
  • Combining ID & modality features: Neither additive nor concatenated ID–modality fusion outperforms the best pure IDRec or MoRec result, and adding ID features can worsen E2E MoRec.
  • Scope: The study does not investigate transfer learning because reliable pre-trained models require substantial training data and compute.
  • Model collapse: Improper learning rates can cause MoRec to collapse during training, although careful hyper-parameter searches usually find settings that train successfully.The item modality encoder and remaining modules may require different learning rates.

8 RELATED WORK

Related recommender-system work includes ID-based models built around user and item identities, alongside modality-based models that represent item content such as text, images, video, and audio.

  • ID-based recommender systems: IDRec models use user and item IDs to predict matching scores or model sequential user histories.The literature spans item-to-item collaborative filtering, factorization models, deep neural models, recall systems, CTR models, and sequential recommenders.
  • Modality-based recommender systems: MoRec models focus on item modality features, including text, images, videos, audio, and multimodal text-image pairs.
  • Modality-based recommender systems: Previous MoRec work commonly uses a two-stage mechanism that pre-extracts fixed modality features before incorporating them into the recommendation model.

9 CONCLUSION AND FUTURE WORKS

The study finds that modern E2E MoRec can match or exceed IDRec in non-cold-start recommendation while benefiting from advances in NLP and computer vision. It also identifies substantial limitations and future research needs.

  • Conclusion: Modern MoRec with SOTA, E2E-trained modality encoders can perform on par with or better than Transformer-based IDRec in non-cold-start recommendation.
  • Conclusion: MoRec can benefit from technical advances in NLP and CV, suggesting greater room for future accuracy improvements.
  • Future works: Future work includes more powerful recommendation architectures, expressive item encoders, improved item–user fusion, and methods reducing compute, memory, and training time.
  • Future works: The authors envision that recommendation may eventually shift from IDRec toward MoRec when raw modality features are available.
  • Limitations: The study covers only text and vision, singlemodal encoders, a basic fusion approach, and three medium-sized datasets, leaving broader settings unresolved.It remains unknown whether the findings hold with other modalities, multimodal encoders, improved fusion, or datasets scaled by 100× or 1000×.

A.1 MoRec vs IDRec on cold-start settings

The cold-start evaluation tests whether MoRec’s modality representations help when items have few or no training interactions. Across text and vision settings, MoRec substantially improves over IDRec.

  • Evaluation and results: MoRec consistently and substantially improves IDRec on all three datasets in both cold-item and new-item settings for text and vision modalities.Cold items appeared fewer than 10 times in training; new items did not appear in the training set.
  • Evaluation and results: The reported MoRec advantage is attributed to modality encoders pre-trained on large-scale text and image datasets.
Loading 2303.13835v4…