PaperlayerView exact revision on arXiv

Source-linked AI summary

MISSRec: Pre-training and Transferring Multi-modal Interest-aware Sequence Representation for Recommendation

Jinpeng Wang, Ziyun Zeng, Yunxiao Wang, Yuting Wang, Xingyu Lu, Tianxiang Li, Jun Yuan, Rui Zhang, Hai-Tao Zheng, Shu-Tao Xia

arXiv:2308.11175v2cs.IRcs.AIcs.MM

TL;DR

ID-based sequential recommenders face sparse-ID and cold-start problems, and inconsistent ID mappings hinder transfer. MISSRec combines multimodal interest-aware sequence modeling with user-adaptive item fusion and contrastive pre-training; experiments report improved generalizability and performance, including robustness to incomplete or missing modalities.

  • Problem

    ID-based sequential recommenders struggle with sparse interactions and cold-start items, while inconsistent ID mappings hinder transfer between domains.

    Source: §1 · p. 2

  • Method

    MISSRec uses multimodal pre-training, interest-aware Transformer sequence modeling, and user-adaptive candidate-item fusion for efficient transfer.

    Source: §1 · p. 2

  • Results

    MISSRec demonstrates improved performance and generalizability compared with existing methods, with reported compatibility and robustness to incomplete or missing modalities.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The authors present MISSRec as a practical approach to robust, transferable sequential recommendation across real-world scenarios.

    Source: §1 · p. 2, §5 · p. 8

    Orthogonal regularization is sensitive to the domain and weighting factor, which must be chosen for the specific scenario.

    Source: §4.5.1 · p. 8

Abstract

from arXiv · show

The goal of sequential recommendation (SR) is to predict a user's potential interested items based on her/his historical interaction sequences. Most existing sequential recommenders are developed based on ID features, which, despite their widespread use, often underperform with sparse IDs and struggle with the cold-start problem. Besides, inconsistent ID mappings hinder the model's transferability, isolating similar recommendation domains that could have been co-optimized. This paper aims to address these issues by exploring the potential of multi-modal information in learning robust and generalizable sequence representations. We propose MISSRec, a multi-modal pre-training and transfer learning framework for SR. On the user side, we design a Transformer-based encoder-decoder model, where the contextual encoder learns to capture the sequence-level multi-modal user interests while a novel interest-aware decoder is developed to grasp item-modality-interest relations for better sequence representation. On the candidate item side, we adopt a dynamic fusion module to produce user-adaptive item representation, providing more precise matching between users and items. We pre-train the model with contrastive learning objectives and fine-tune it in an efficient manner. Extensive experiments demonstrate the effectiveness and flexibility of MISSRec, promising a practical solution for real-world recommendation scenarios. Data and code are available on \url{https://github.com/gimpong/MM23-MISSRec}.

1 INTRODUCTION

ID-based sequential recommenders face sparse-ID and cold-start challenges, while inconsistent ID mappings hinder transfer across domains. MISSRec uses multimodal content and interest-aware sequence modeling to learn transferable representations; experiments report better generalizability than ID- and text-based approaches and alleviation of the item cold-start issue.

1 INTRODUCTION

  • ID-based sequential recommenders struggle with sparse interactions and cold-start items, show popularity bias toward popular IDs, and transfer poorly across inconsistent ID mappings.These limitations can isolate similar domains that might otherwise be co-optimized.Source: §1 · p. 2
  • The authors motivate multimodal content as a promising source of robust semantic features for sequential recommendation, while noting that its study in this setting remains fledgling.They pose how multimodal information affects sequential recommendation and how it can be better used in sequence modeling.Source: §1 · p. 2
  • Multimodal sequential recommendation must model user- and context-dependent modality synergy while preventing redundant interactions from overwhelming sparse interests.Treating all interactions equally can overemphasize some item types and underweight others.Source: §1 · p. 2, Figure 1 · p. 2
  • MISSRec combines ID-agnostic multimodal pre-training and transfer with interest-aware sequence modeling and user-adaptive candidate-item fusion.Its adapters distill personalized semantics; interest tokens represent mined multimodal interests, and a Transformer encoder-decoder models their relations to items.Source: §1 · p. 2
  • MISSRec exhibits better generalizability than ID-based and text-based approaches and further alleviates the item cold-start issue.The paper presents it as learning robust and transferable sequence representations.Source: §1 · p. 2

2 RELATED WORKS

Prior work spans ID-based sequence models, multimodal item representations, and pre-training and transfer strategies. MISSRec focuses on transferable multimodal sequence representations, adding interest-aware modeling and user-adaptive candidate fusion.

2.1 Sequential Recommendation

  • Sequential recommendation research includes Markov, deep sequence, interest-modeling, temporal-aware, and contrastive-learning approaches, but most methods use IDs and face cold-start and transferability limits.Earlier multimodal methods include CSAN, MM-Rec, and MMMLP; MISSRec instead targets universal sequence representations from multimodal item content.Source: §2.1 · p. 3

2.2 Pre-training and Transfer Learning in Recommendation

  • ID-based pre-training and transfer can depend on shared ID vocabularies, while modality-based approaches are more flexible; MISSRec adds interest-aware modeling and user-adaptive candidate fusion beyond recent multimodal methods [55] [86].The paper identifies these as two improvements over the prior setting.Source: §2.2 · p. 3

2.3 Multi-modal Recommendation

  • Multimodal recommendation has advanced through matrix-factorization, GNN, self-supervised, and pre-training methods, but existing solutions were predominantly designed for non-sequential scenarios.The authors position their work as addressing challenges specific to multimodal sequential recommendation.Source: §2.3 · p. 3

3.1 Problem Formulation and Method Overview

MISSRec represents each user’s timestamp-ordered interaction sequence and candidate items using multimodal content. Its overview moves from feature extraction and interest-token construction through contextual encoding and interest-aware decoding to contrastive pre-training and transfer fine-tuning.

3.1 Problem Formulation and Method Overview

  • Sequential recommendation predicts a user’s next interacted item from the timestamp-ordered history by learning sequence and candidate-item representations.The paper considers text and image modalities alongside each item's unique ID.Source: §3.1 · p. 3
  • MISSRec adapts multimodal features into input tokens, mines interest tokens, encodes their context, and decodes an interest-aware sequence representation.The overview also includes dynamic fusion for user-adaptive candidate-item representations.Source: §3.1 · p. 3, Figure 2 · p. 4
  • MISSRec pre-trains with sequence-item and sequence-sequence contrastive objectives, then fine-tunes with the sequence-item objective in transductive or inductive settings.Downstream adaptation tunes only modality-specific adapters, as described in the overview figure.Source: Figure 2 · p. 4

3.2 Universal Multi-modal Item Representation

MISSRec builds item representations from text and visual features, then adapts their fusion to each user. Feature augmentation and modality-specific adapters prepare the features for sequence modeling and efficient transfer.

3.2.1 Multi-modal Feature Extraction with Pre-trained Transform-

  • Pre-trained text and visual encoders extract modality features, which modality-specific adapters transform and fuse into item tokens.The paper identifies BERT and ViT as the text and visual encoders, respectively.Source: §3.2.1 · p. 4, §3.2.3 · p. 4

3.2.2 Feature Augmentation.

  • Dropout creates two augmented views of the text features, which serve as positive views for training.The visual features follow an analogous augmentation procedure.Source: §3.2.2 · p. 4

3.2.3 Bridge Domain Gap with Modality-specific Adapters.

  • Adapters with frozen feature extractors reduce pre-training time and memory overhead and support parameter-efficient downstream transfer.The adapters transform augmented modality features before fusion into item tokens.Source: §3.2.3 · p. 4

3.2.4 Candidate Item Representation with Dynamic Fusion.

  • Dynamic fusion weights candidate-item modality embeddings using the user’s sequence representation, producing user-adaptive item representations.The fusion is factorizable for matching-score computation and transitions between mean and max pooling.Source: §3.2.4 · p. 4

3.3 Discover Multi-modal User Interests

MISSRec uses interest modeling to reduce the impact of redundant sequence information and capture personalized semantics. Its multimodal interest discovery module clusters item tokens into interest prototypes and indexes tokens by their nearest prototype.

3.3 Discover Multi-modal User Interests

  • Interest modeling aims to reduce the impact of redundant sequence information and capture personalized semantics for precise representation.The paper introduces multimodal interest discovery to support this goal.Source: §3.3 · p. 4

3.3.1 Excavate and Index Interests via Adaptive Clustering.

  • Before each training epoch, DPC-KNN clusters the item-token set to explore the distribution of user interests.The method computes local density from k-nearest neighbors and minimum distance to a higher-density token, then selects centroids by ρ_i × δ_i.Source: §3.3.1 · p. 5
  • Each item token is assigned to its nearest interest prototype, yielding interest tokens that can be retrieved through an index.The MID module can convert an item into different interests according to its modalities.Source: §3.3.2 · p. 5, §3.3.1 · p. 5, Figure 3 · p. 5

3.4 Multi-modal Interest-aware Sequence Model

MISSRec unifies sequence modeling and multimodal fusion in an encoder-decoder model. Its interest-aware decoder attends from item-token queries to encoded interest tokens to produce the user’s sequence representation.

3.4 Multi-modal Interest-aware Sequence Model

  • A Transformer context encoder processes discovered interest tokens into encoded tokens for next-step decoding.The encoder is part of the model that unifies sequence modeling and multimodal fusion.Source: §3.4.1 · p. 5, §3.4 · p. 5

3.4.2 Transformer-based Interest-aware Decoder.

  • The interest-aware decoder uses encoded interests as cross-attention keys and values and item tokens with positional embeddings as queries to model item-modality-interest relations.The user’s sequence representation is the decoded output of the last item query token.Source: §3.4.2 · p. 5

3.5 Self-supervised Contrastive Pre-training

MISSRec pre-trains sequence representations with contrastive tasks and an orthogonal regularization term. The tasks align sequences with candidate items and encourage consistency across augmented sequence views.

3.5 Self-supervised Contrastive Pre-training

  • Pre-training combines sequence-item and sequence-sequence contrastive learning tasks to pursue universal representations for multimodal interaction sequences.Source: §3.5 · p. 5

3.5.1 Sequence-item Contrastive Learning.

  • Sequence-item contrastive learning captures correspondence between interaction sequences and candidate items.The matching scores use the user-item scoring function, with mini-batch size and temperature specified in the objective description.Source: §3.5.1 · p. 5

3.5.2 Sequence-sequence Contrastive Learning.

  • Sequence-sequence contrastive learning uses semantic invariance between representations of two augmented views of a user’s sequence as a self-supervising signal.Source: §3.5.2 · p. 5

3.5.3 Overall Objectives. We sum up pre-train objectives as follows

  • The overall pre-training objective includes orthogonal regularization to diversify interest-aware decoded results.The loss terms have separate weights.Source: §3.5.3 · p. 5

3.6 Efficient Fine-tuning

Efficient fine-tuning transfers the pre-trained sequence model using sequence-item contrast and orthogonal regularization while tuning only selected parameters. The framework supports inductive transfer for cold-item domains and transductive transfer for domains with mostly warm items.

3.6 Efficient Fine-tuning

  • Fine-tuning uses sequence-item contrast and orthogonal regularization to optimize next-item prediction; only the item-ID table, if included, and modality adapters are tuned while other modules remain frozen.Disabling feature augmentation and sequence-sequence contrast lets the model encode each user once, reducing computation.Source: §3.6 · p. 5

3.6.1 Inductive Transfer.

  • For target domains with many cold items, inductive transfer predicts the next item from the sequence representation without requiring item IDs.The paper describes this as an ID-agnostic solution for extreme sparsity.Source: §3.6.1 · p. 6

3.6.2 Transductive Transfer.

  • For target domains with mostly warm items, transductive transfer uses item IDs alongside the encoded sequence representation for next-item prediction.The item-ID embedding is added to the candidate item representation.Source: §3.6.2 · p. 6

4 EXPERIMENTS

Across downstream comparisons, MISSRec generally outperforms state-of-the-art baselines, while analyses show that results depend on modality combination, transfer setting, and model components. The experiments cover five target domains and examine baseline performance, pre-training, modality use, and design choices.

4.2 Experimental Setup

  • Experiments address baseline performance, modality and pre-training effects, and component design across five Amazon target domains, using nine baselines and Recall and NDCG at K=10 and 50.The 10 Amazon domains are divided into five pre-training and five downstream target domains.Source: §4.1 · p. 6, §4.2.1 · p. 6, §4.2.3 · p. 6

4.3.1 Overall Performance.

  • MISSRec shows the best few-shot performance and maintains improvement as item frequency rises, whereas other modality-based methods’ gains over SASRecID diminish or reverse.Figure 4’s histogram shows test-sample counts by popularity, and its line chart shows relative Recall@10 improvement over SASRecID.Source: Figure 4 · p. 6, §4.3.2 · p. 7
  • MISSRec outperforms state-of-the-art baselines in most cases under both transductive and inductive settings, with considerable margins.The comparison covers five downstream datasets; Table 2 reports results by target dataset and relative improvement over the best baselines.Source: Table 2 · p. 7, §4.3.1 · p. 7

4.4.1 Comparison with Different Input Modalities.

  • On full-modality subsets formed by filtering image-less items from Scientific and Office, visual-only input alone was unsatisfactory, while multi-modal synergy created an effect greater than the sum of its parts.All comparison methods used the same CLIP-B/32 feature extractors.Source: Table 3 · p. 7, §4.4.1 · pp. 7–8
  • Pre-training improves target-domain recommendation under transductive settings but shows negative transfer under inductive settings, especially on Office.The authors suggest that a large domain gap and limited tunable adapters may explain difficulty adapting to target data.Source: Table 4 · p. 8, §4.4.2 · p. 8

4.5.1 Effects of Modality Adapters.

  • Sequence-sequence contrast during pre-training enhances representation robustness and downstream recommendation, while orthogonal regularization helps Scientific slightly but hurts Office.The authors infer that orthogonal regularization is sensitive to the domain or its weighting factor.Source: §4.5.1 · p. 8
  • Removing modality-specific adapters causes significant performance decay, indicating that raw general multi-modal features alone provide insufficient personalization.The authors note that end-to-end optimization of feature extractors can mitigate the semantic gap but uses more computation than adapter tuning.Source: Table 5 · p. 8, §4.5.1 · p. 8

4.5.3 Effects of Interest-aware Decoding and Contextual Encoding.

  • The poor performance of the encoder-only variant and degradation without encoded interest information support collaboration between contextual encoding and interest-aware decoding.The decoder-only variant also degrades when it lacks access to encoded interest information.Source: §4.5.3 · p. 8

5 CONCLUSIONS

MISSRec uses multi-modal pre-training and transfer learning to address ID-feature limitations in sequential recommendation. The paper reports improved performance and generalizability, along with robustness to incomplete or missing modalities.

5 CONCLUSIONS

  • MISSRec combines a Transformer-based contextual encoder, an interest-aware decoder, and lightweight dynamic fusion for sequential recommendation.The paper presents this framework as a way to address limitations of models relying heavily on ID features.Source: §5 · p. 8
  • MISSRec demonstrates improved performance and generalizability compared with existing methods.Source: §5 · p. 8
  • Experiments showcase MISSRec’s compatibility and robustness when modalities are incomplete or missing.Source: §5 · p. 8

Related papers

Loading 2308.11175v2…