PaperlayerView exact revision on arXiv

Source-linked AI summary

Text Is All You Need: Learning Language Representations for Sequential Recommendation

Jiacheng Li, Ming Wang, Jin Li, Jinmiao Fu, Xin Shen, Jingbo Shang, Julian McAuley

arXiv:2305.13731v2cs.IR

TL;DR

Existing sequential recommendation methods struggle with cold-start items, cross-domain transfer, and mismatches between general language models and item text. Recformer represents items as textual key-value “sentences,” jointly trains language and recommendation representations, and reports improvements in supervised and zero-shot settings.

  • Problem

    ID-based and text-based sequential recommenders have limited support for cold-start items and cross-domain transfer, while pretrained language models differ from item-text domains.

  • Method

    Recformer flattens item attributes into textual sentences and uses a bidirectional Transformer with pretraining and finetuning to recommend from language representations.

  • Results

    15.83% and 39.78% NDCG@10 performance improvements are reported for fully-supervised and zero-shot sequential recommendation, respectively.

  • Takeaways & Limitations

    Recformer learns item and user-sequence representations from words and transfers learned knowledge to cold-start items and new recommendation domains.

Abstract

from arXiv · show

Sequential recommendation aims to model dynamic user behavior from historical interactions. Existing methods rely on either explicit item IDs or general textual features for sequence modeling to understand user preferences. While promising, these approaches still struggle to model cold-start items or transfer knowledge to new datasets. In this paper, we propose to model user preferences and item features as language representations that can be generalized to new items and datasets. To this end, we present a novel framework, named Recformer, which effectively learns language representations for sequential recommendation. Specifically, we propose to formulate an item as a "sentence" (word sequence) by flattening item key-value attributes described by text so that an item sequence for a user becomes a sequence of sentences. For recommendation, Recformer is trained to understand the "sentence" sequence and retrieve the next "sentence". To encode item sequences, we design a bi-directional Transformer similar to the model Longformer but with different embedding layers for sequential recommendation. For effective representation learning, we propose novel pretraining and finetuning methods which combine language understanding and recommendation tasks. Therefore, Recformer can effectively recommend the next item based on language representations. Extensive experiments conducted on six datasets demonstrate the effectiveness of Recformer for sequential recommendation, especially in low-resource and cold-start settings.

1 INTRODUCTION

Sequential recommendation models user interactions over time, but ID-based and text-to-item approaches face cold-start, transfer, and domain-gap challenges. Recformer addresses these issues by representing item attributes as language and jointly learning language understanding with sequential recommendation.

  • Motivation: Traditional sequential recommenders use item IDs and learned embedding tables, limiting understanding of cold-start items and cross-domain transfer.Item-specific IDs prevent representations from transferring directly across recommendation scenarios.
  • Motivation: Cross-domain methods often assume overlapping users, items, or features, although different domains can expose different recommendation attributes.Examples include Laptops and T-shirts, which typically contain different features.
  • Motivation: Text-to-item frameworks transfer language knowledge but face a language-domain gap because pretrained models are trained on general corpora rather than item text.The passage identifies item-text language as a distinct domain from general natural-language corpora.
  • Recformer: Recformer flattens textual key-value attributes such as title, color, and brand into item sentences, replacing item IDs in user sequences.The resulting sequence is processed as text while retaining item-level structure.
  • Recformer: Recformer combines a Longformer-like bidirectional Transformer with pretraining, finetuning, and inference to recommend next items from language representations.The framework is designed to transfer learned knowledge to cold-start items and new recommendation scenarios.
  • Results: 15.83% and 39.78% NDCG@10 performance improvements are reported under fully-supervised and zero-shot sequential recommendation settings, respectively.The experiments use real-world datasets from different domains.

2 METHODOLOGY

Recformer converts each item’s textual attributes into an item sentence and encodes reversed interaction histories with a bidirectional Transformer. It represents both users and items without an item embedding table, then ranks candidates by sequence–item similarity.

  • Model Inputs: Each item has an attribute dictionary of textual key-value pairs, which Recformer flattens into an item sentence.Attributes can include titles, descriptions, colors, and other textual information.
  • Model Inputs: Recformer reverses each interaction sequence so recent items appear first, replaces items with their sentences, and prepends [CLS].The resulting input sequence contains words from all historically interacted items and their attributes.
  • Embedding Layer: The model combines token, token-position, token-type, and item-position embeddings to represent words, attribute roles, and item order.Token types distinguish [CLS], attribute keys, and attribute values, while item positions align tokens with items.
  • Encoder: A Longformer encoder uses local windowed attention for most tokens and global attention for [CLS] to efficiently process long inputs.The first-token representation h_[CLS] serves as the sequence representation.
  • Item Representation: Recformer derives an item representation by encoding an item sentence alone with [CLS], rather than maintaining a separate item embedding table.The resulting h_[CLS] is used as the item representation h_i.
  • Prediction: The model scores every candidate item by cosine similarity between its representation and the user-sequence representation, selecting the highest-scoring item.The relevance score r_i,s measures how likely item i is to be the next item given sequence s.

2.3 Learning Framework

Recformer uses pre-training that jointly supports language understanding and recommendation, followed by two-stage finetuning for downstream scenarios.

  • Recformer’s learning framework combines pre-training with two-stage finetuning for sequential recommendation.The framework is designed to provide language-aware initialization and downstream task adaptation.
  • Pre-training jointly optimizes Masked Language Modeling and item-item contrastive learning.MLM supports language understanding, while the contrastive task targets next-item recommendation.
  • MLM randomly selects 15% of token positions and replaces selected tokens with masked, random, or unchanged tokens.The replacement probabilities are 80%, 10%, and 10%, respectively.
  • The item-item contrastive task uses ground-truth next items as positives and other next items in the batch as negatives.In-batch negatives avoid re-encoding sampled or full item sets for each training batch.

3 Output: 𝑀′, I′

The framework uses contrastive learning for pre-training and finetuning, with two-stage finetuning maintaining item features to provide accurate downstream supervision.

  • Pre-training jointly optimizes Recformer with a multi-task objective combining MLM and item-item contrastive losses.The MLM loss weight is controlled by the hyper-parameter λ, and the resulting model is fine-tuned for new scenarios.
  • Two-stage finetuning maintains an item feature matrix I instead of an independent item embedding table.The matrix is used to provide accurate supervision on downstream datasets.
  • In stage 1, item features are updated each epoch while model parameters are trained; in stage 2, item features are frozen and only model parameters are updated.Items are re-encoded once per epoch rather than in every batch.
  • Finetuning uses item-item contrastive learning with fully softmax rather than in-batch negatives.This addresses the likelihood of false negatives in small downstream datasets.
  • Recformer learns item and user-sequence representations directly from words instead of trainable item embeddings or fixed language-model features.The paper positions this language-representation paradigm as supporting transferability, domain adaptation, and cold-start item understanding.

3 EXPERIMENTS

The experiments evaluate Recformer on preprocessed recommendation datasets and report dataset statistics, including average interaction-sequence length.

  • The experiments empirically evaluate the effectiveness of Recformer and its learning framework.
  • Dataset statistics after preprocessing include the average length of item sequences, denoted as Avg. n.

3.1 Experimental Setup

The experimental setup uses Amazon review categories split across source pre-training and target-domain finetuning, with text-based preprocessing, multiple baselines, and ranking metrics.

  • Recformer is evaluated through pre-training and finetuning on Amazon review datasets from different product categories.
  • Seven categories form source-domain pre-training data, while six different categories form target-domain finetuning data.
  • Interactions are grouped by user and sorted chronologically; items retain title, category, and brand attributes as key-value text.
  • Baselines are grouped into ID-only, ID-plus-text, and text-only sequential recommendation methods.
  • Evaluation reports NDCG@10, Recall@10, and MRR using leave-one-out splitting for finetuning datasets.The most recent interaction is used for testing, the second most recent for validation, and earlier interactions for training.
  • Recformer uses Longformer with local attention windows of 64 and supports up to 1,024 tokens per interaction sequence.The setup also limits each user sequence to 50 items.

3.2 Overall Performance

Recformer achieves the strongest overall performance across the evaluated datasets and improves substantially over prior baselines in both NDCG@10 and MRR. It also learns language representations without pre-trained language models or item IDs.

  • ID-Text methods generally outperform ID-Only and Text-Only baselines by combining item IDs with content features.
  • Recformer achieves the best overall performance on all datasets except Recall@10 on Instruments.
  • 15.83% NDCG@10 and 15.99% MRR are Recformer’s average improvements over the second-best results.
  • Recformer learns language representations for sequential recommendation without pre-trained language models or item IDs.

3.3 Low-Resource Performance

The low-resource and zero-shot evaluations test whether textual representations transfer knowledge when downstream supervision is limited or absent. Recformer performs especially strongly in these settings, including large gains on the Scientific dataset with only 1% and 5% of training data.

  • Zero-Shot: Recformer improves zero-shot recommendation performance over UniSRec and ZESRec across six downstream datasets.
  • Low-Resource: Methods using item text outperform SASRec especially when less training data is available.
  • Low-Resource: Recformer achieves the best performance across training-data ratios from 1% to 100%.
  • Low-Resource: On Scientific, Recformer outperforms other methods by a large margin with 1% and 5% of training data.

3.4 Further Analysis

Further analyses show that Recformer’s language representations help with cold-start items, benefit from pretraining and two-stage finetuning, and reach peak transfer performance after relatively few pretraining steps. Removing structural embeddings can hurt when the pretraining and finetuning domains differ substantially.

  • 3.4.1 Performance w.r.t. Cold-Start Items: Text-Only methods significantly outperform SASRec on cold-start items, while Recformer improves over UniSRec on both in-set and cold-start datasets.
  • 3.4.2 Ablation Study: Two-stage finetuning improves sub-optimal item representations from pretraining and yields a large gain on Instruments.
  • 3.4.2 Ablation Study: Recformer significantly outperforms the no-pretraining variant on Scientific and Instruments, showing the effectiveness of pretraining.
  • 3.4.2 Ablation Study: Removing item position and token type embeddings causes performance decay on Instruments when the pretraining and finetuning gap is large.
  • 3.4.3 Pre-training Steps vs. Performance: Around 4,000 pretraining steps produce the best performance on most datasets, while further pretraining may hurt downstream knowledge transferability.

4 RELATED WORK

Related work develops sequential recommenders for modeling temporally ordered interactions and explores knowledge transfer across domains. This paper positions its approach as a unified Transformer framework for language understanding and sequential recommendation that targets cold-start and transferability challenges.

  • Sequential recommendation models timestamp-ordered user interactions to capture both short-term dynamics and long-term preferences.
  • The paper aims to unify language understanding and sequential recommendation in one Transformer framework for cold-start and cross-scenario transfer.
  • Knowledge-transfer methods commonly rely on shared users, items, or attributes between source and target domains.

5 CONCLUSION

Recformer learns language representations for sequential recommendation by replacing item IDs with textual key-value attributes and combining sequence modeling with language understanding. Experiments show strong performance across supervised and low-resource settings, especially for zero-shot and cold-start recommendation.

  • Recformer represents items as key-value attribute pairs rather than item IDs for language-based next-item recommendation.The framework learns item and sequence representations with a bi-directional Transformer.
  • Its pretraining and finetuning framework jointly supports language understanding, sequential recommendation, and knowledge transfer across recommendation scenarios.
  • Recformer largely outperforms existing methods across full-supervised and low-resource settings, particularly for zero-shot and cold-start item recommendation.
  • An ablation study evaluates the effectiveness of the proposed components.
Loading 2305.13731v2…