PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Universal Sequence Representation Learning for Recommender Systems

Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, Ji-Rong Wen

arXiv:2206.05941v1cs.IRcs.AIcs.LG

TL;DR

Existing sequential recommenders depend on explicit item IDs, limiting transfer to new recommendation scenarios and aggravating cold-start problems. UniSRec instead uses item text with parametric whitening, an MoE-enhanced adaptor, and multi-domain contrastive pre-training to learn transferable representations. Experiments report effective cross-domain transfer, including improved cross-platform recommendation without overlapping users or items.

  • Problem

    Explicit item-ID modeling makes sequential representation learning difficult to transfer across domains and platforms, while cold-start items may have inadequate interactions.

  • Method

    UniSRec uses item text, parametric whitening, an MoE-enhanced adaptor, and sequence-item and sequence-sequence contrastive tasks over multi-domain sequences.

  • Results

    Experiments on real-world datasets show transferable representations across domains and improved cross-platform recommendation without overlapping users or items.

  • Takeaways & Limitations

    UniSRec provides an ID-agnostic sequence representation model that can be transferred to new recommendation scenarios in a parameter-efficient way.

  • Takeaways & Limitations

    The authors leave deeper investigation of universal representations for future work and plan to collect more data and explore additional side information.

Abstract

from arXiv · show

In order to develop effective sequential recommenders, a series of sequence representation learning (SRL) methods are proposed to model historical user behaviors. Most existing SRL methods rely on explicit item IDs for developing the sequence models to better capture user preference. Though effective to some extent, these methods are difficult to be transferred to new recommendation scenarios, due to the limitation by explicitly modeling item IDs. To tackle this issue, we present a novel universal sequence representation learning approach, named UniSRec. The proposed approach utilizes the associated description text of items to learn transferable representations across different recommendation scenarios. For learning universal item representations, we design a lightweight item encoding architecture based on parametric whitening and mixture-of-experts enhanced adaptor. For learning universal sequence representations, we introduce two contrastive pre-training tasks by sampling multi-domain negatives. With the pre-trained universal sequence representation model, our approach can be effectively transferred to new recommendation domains or platforms in a parameter-efficient way, under either inductive or transductive settings. Extensive experiments conducted on real-world datasets demonstrate the effectiveness of the proposed approach. Especially, our approach also leads to a performance improvement in a cross-platform setting, showing the strong transferability of the proposed universal SRL method. The code and pre-trained model are available at: https://github.com/RUCAIBox/UniSRec.

1 INTRODUCTION

Sequential recommendation models user preferences from chronologically ordered interaction histories, but explicit item-ID modeling limits transfer to new domains and platforms. UniSRec addresses this limitation by using item text and multi-domain pre-training to learn transferable sequence representations.

  • Sequential recommendation predicts suitable items from a user’s historical interaction records.
  • Most existing sequence representation learning methods explicitly model item IDs, making them difficult to transfer even when new scenarios share the same data format.
  • Explicit ID modeling also necessitates retraining for new domains and contributes to cold-start difficulties for items with few interactions.
  • UniSRec uses associated item descriptions, titles, or brands to learn transferable representations across recommendation domains.
  • UniSRec combines parametric whitening, a mixture-of-experts adaptor, and two multi-domain contrastive pre-training tasks for universal item and sequence representations.
  • Cross-platform experiments show improved recommendation performance when using a model pre-trained on other platforms without overlapping users or items.

2 METHODOLOGY

UniSRec learns universal item and sequence representations from interaction sequences spanning multiple domains, then transfers them efficiently to new recommendation scenarios.

  • UniSRec learns universal item and sequence representations from mixed-domain behavioral sequences for parameter-efficient transfer to new domains or platforms.

2.1 Overview of the Approach

The methodology represents domain-specific user behavior as separate interaction sequences and learns ID-agnostic representations primarily from item text. It addresses universal recommendation through separate item- and sequence-representation objectives.

  • 2.1 Overview of the Approach: UniSRec represents each user’s behavior as chronological interaction sequences whose items include IDs and associated description text.
  • 2.1 Overview of the Approach: Multiple sequences from different domains or platforms are treated separately without explicitly maintaining user IDs because domains have large semantic gaps.
  • 2.1 Overview of the Approach: Item IDs serve only as auxiliary information, while item text provides the main basis for generalizable ID-agnostic representations.
  • 2.1 Overview of the Approach: The approach learns universal item representations through parametric whitening and an MoE-enhanced adaptor, then learns universal sequence representations with sequence-item and sequence-sequence contrastive tasks.

2.2 Universal Textual Item Representation

UniSRec encodes item text with BERT, transforms the resulting embeddings through learnable whitening, and fuses domain-specific transformations with a gated mixture-of-experts adaptor. This design targets isotropic, transferable item representations while retaining parameter-efficient adaptation.

  • Motivation: Traditional ID embeddings limit transferability because item-ID vocabularies usually differ across recommendation domains.
  • Motivation: UniSRec uses associated item text as a shared natural-language basis for transferable item representations across domains.
  • Textual Item Encoding: BERT encodes each item’s concatenated [CLS] token and text words into a textual representation.
  • Parametric Whitening: A learnable linear whitening transformation maps BERT representations into a more isotropic semantic space suited to universal representation learning.
  • Domain Fusion and Adaptation: The MoE-enhanced adaptor combines multiple whitening transformations through a parameterized router to fuse and adapt information across domains.
  • Domain Fusion and Adaptation: The lightweight adaptor supports parameter-efficient fine-tuning for new domains while using multiple transformations and adaptive gating.

2.3 Universal Sequence Representation

UniSRec learns universal sequence representations by combining self-attentive sequence encoding with two contrastive pre-training tasks that use multi-domain negatives. Joint optimization is intended to improve semantic fusion and adaptation across domains.

  • Universal sequence encoding: UniSRec uses a Transformer-based behavior encoder to derive sequence representations from universal textual item representations rather than item IDs.The final hidden vector at the last sequence position serves as the sequence representation.
  • Sequence-item contrastive task: The sequence-item contrastive task models correlations between observed sequential contexts and potential next items using across-domain items as negatives.This differs from next-item prediction with only in-domain negatives and is designed to enhance semantic fusion and cross-domain adaptation.
  • Sequence-item contrastive task: The sequence-item task encodes each contextual sequence and its positive next item, while randomly constructed in-batch items provide mixed-domain negatives.The contextual sequence representation is normalized, and the loss includes a temperature parameter.
  • Sequence-sequence contrastive task: The sequence-sequence contrastive task compares augmented sequences with multi-domain sequences to discriminate positive augmentations from negative in-batch sequences.Item drop and word drop are introduced as augmentation strategies, with word-drop preprocessing used for efficient pre-training.
  • Multi-task learning: UniSRec jointly optimizes the sequence-item and sequence-sequence contrastive losses, then fine-tunes the pre-trained model for new domains.A hyper-parameter controls the weight of the sequence-sequence loss.

2.4 Parameter-Efficient Fine-tuning

UniSRec adapts to new domains through parameter-efficient fine-tuning of only a small part of its MoE-enhanced adaptor while keeping the main architecture fixed. It supports inductive adaptation with item text and transductive adaptation with item text plus IDs.

  • Parameter-efficient adaptation: UniSRec fixes the major architecture and fine-tunes only a small proportion of MoE-enhanced adaptor parameters for domain adaptation.This is intended to reduce the time and flexibility costs of fine-tuning the whole network.
  • Inductive setting: In the inductive setting, UniSRec represents unseen-domain items from text without relying on item IDs.Next-item prediction uses universal representations for the sequential context and candidate item.
  • Transductive setting: In the transductive setting, UniSRec combines textual and ID embeddings to represent items already observed in the target-domain training data.The remaining sequence-encoder parameters remain fixed.
  • Fine-tuning objective: Both settings fine-tune adaptor parameters with cross-entropy loss and predict next-item probabilities after adaptation.The inductive and transductive settings use the respective prediction formulations described in the paper.

2.5 Discussion

UniSRec is positioned as an ID-agnostic, multi-source transfer approach for sequential recommendation. Its distinguishing scope is pre-training universal item and sequence representations across multiple weakly related or irrelevant domains without requiring overlapping users or items.

  • Sequential recommendation: Conventional sequential recommenders such as GRU4Rec and SASRec rely on explicit item IDs and perform poorly for cold-start items.UniSRec instead models sequential patterns through natural-language item representations.
  • Cross-domain transfer: UniSRec uses an MoE-enhanced adaptor to fuse and adapt domain information through universal textual semantics.This contrasts with cross-domain approaches that commonly require overlapping users or items as anchors.
  • Pre-training scope: Unlike prior pre-training approaches focused on one domain, overlapping users, or closely related domains, UniSRec pre-trains on multiple weakly related or irrelevant domains.The paper presents this as a route to more transferable representations for target domains.

3 EXPERIMENTS

The experiments evaluate UniSRec across cross-domain and cross-platform settings using Amazon and Online Retail data, with text-based and ID-plus-text variants compared against established recommenders. The setup uses next-item ranking metrics, leave-one-out evaluation, and tuned implementations for fair comparison.

  • Experimental settings: The study evaluates UniSRec in both cross-domain and cross-platform recommendation settings.The dataset statistics are summarized after preprocessing.
  • Datasets: Five Amazon categories provide pre-training data, while five different Amazon categories serve as cross-domain target datasets.The source categories include Grocery and Gourmet Food, Home and Kitchen, CDs and Vinyl, Kindle Store, and Movies and TV.
  • Datasets: The cross-platform evaluation uses UK Online Retail transactions without shared users or items with Amazon.The transactions span 01/12/2010 to 09/12/2011.
  • Preprocessing: Preprocessing applies five-core filtering, chronological user-sequence ordering, concatenated item text fields, and a 512-token truncation limit.Amazon text combines title, categories, and brand, while Online Retail uses its Description field.
  • Compared methods: UniSRec_t uses only item text for inductive fine-tuning, whereas UniSRec_t+ID combines item IDs and item text for transductive fine-tuning.Both variants are pre-trained on the five source datasets.
  • Evaluation: Next-item prediction is evaluated with Recall@N and NDCG@N for N=10 and 50 under leave-one-out evaluation.The final, penultimate, and earlier interactions are assigned to test, validation, and training respectively.

3.2 Overall Performance

UniSRec uses universal sequence representations learned through multi-domain pre-training to transfer recommendation knowledge across domains and platforms. Experiments show strong performance, including in inductive, cross-platform, long-tail, and component-ablation analyses.

  • Overall comparison: UniSRec_t+ID achieves the best performance in almost all cases against the baseline methods.Its universal representations are pre-trained on multi-domain datasets and use parametric whitening with an MoE-enhanced adaptor.
  • Overall comparison: UniSRec_t achieves comparable performance with other baselines in the inductive setting without item IDs.This supports the effectiveness of the universal sequence representation approach without explicit item-ID modeling.
  • Universal pre-training: Pre-training on all five datasets outperforms pre-training on any single dataset or training from scratch.The results indicate that multi-domain pre-training captures semantic sequential patterns useful for target domains or platforms.
  • Ablation study: All proposed components improve recommendation performance, while removing the MoE-enhanced adaptor produces poor performance.The adaptor is identified as the key component for representation capacity in domain fusion and adaptation.
  • Long-tail analysis: UniSRec outperforms other baselines in most long-tail groups, especially for unpopular items on Industrial and Scientific and Online Retail.Figure 4 compares Recall@10 improvement ratios with SASRec and shows benefits for long-tail items.
  • Cross-platform transfer: UniSRec achieves good cross-platform performance from Amazon to Online Retail despite no overlapping users or items.The case study illustrates transfer through semantic sequential patterns such as “dog” to “cat,” rather than explicit item IDs.

4 RELATED WORK

Sequential recommendation models user preferences from ordered interaction histories, while transfer-learning methods address sparsity and cold-start challenges across domains or platforms. UniSRec extends this direction by using text-based universal representations that generalize without explicitly shared anchors.

  • Sequential recommendation: Sequential recommendation models chronologically ordered user-item interactions to predict likely next items.The literature has progressed from matrix factorization and Markov-style approaches to neural sequence models.
  • Transfer learning in recommender systems: Transfer-learning methods leverage information across domains or platforms to address data sparsity and cold-start recommendation.Existing approaches commonly rely on shared users, items, social networks, or attributes.
  • Transfer learning in recommender systems: UniSRec represents items through text-based pretrained language models rather than requiring closely related source and target domains.Its universal pre-training uses multiple source domains and supports generalization without explicitly shared anchors.

5 CONCLUSIONS

UniSRec learns transferable sequential representations from item texts instead of explicit item IDs. It combines specialized item encoding with multi-domain contrastive sequence pre-training, while identifying larger behavior models and additional modalities as future directions.

  • Conclusion: UniSRec uses item texts to learn more transferable representations for sequential recommendation than explicit item-ID modeling.The approach is designed as a universal sequence representation learning method.
  • Conclusion: Parametric whitening and a mixture-of-experts-enhanced adaptor learn universal item representations, while two contrastive tasks learn universal sequence representations.The contrastive tasks operate on multi-domain sequences.
  • Future work: Future work will collect more recommendation data for larger behavior models and explore images and videos as additional item information.
Loading 2206.05941v1…