PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation

Ruihong Qiu, Zi Huang, Hongzhi Yin, Zijian Wang

arXiv:2110.05730v2cs.IR

TL;DR

Sequential recommendation models can produce anisotropic item embeddings that make semantic representations indiscriminate. DuoRec combines contrastive regularization with Dropout-based model-level augmentation and target-item-based positive sampling, and experiments report superior performance while visualizations show reduced degeneration.

  • Problem

    Sequential models tend to produce anisotropic item-embedding distributions with high semantic similarities, creating a representation degeneration problem.

  • Method

    DuoRec uses contrastive regularization over sequence representations, Dropout-based model-level augmentation, and supervised positive sampling based on target items.

  • Results

    DuoRec achieves superior performance on five benchmark datasets, while visualization results show that it alleviates representation degeneration.

  • Takeaways & Limitations

    More uniform sequence representations can implicitly improve the item-embedding distribution used for dot-product recommendation.

Abstract

from arXiv · show

Recent advancements of sequential deep learning models such as Transformer and BERT have significantly facilitated the sequential recommendation. However, according to our study, the distribution of item embeddings generated by these models tends to degenerate into an anisotropic shape, which may result in high semantic similarities among embeddings. In this paper, both empirical and theoretical investigations of this representation degeneration problem are first provided, based on which a novel recommender model DuoRec is proposed to improve the item embeddings distribution. Specifically, in light of the uniformity property of contrastive learning, a contrastive regularization is designed for DuoRec to reshape the distribution of sequence representations. Given the convention that the recommendation task is performed by measuring the similarity between sequence representations and item embeddings in the same space via dot product, the regularization can be implicitly applied to the item embedding distribution. Existing contrastive learning methods mainly rely on data level augmentation for user-item interaction sequences through item cropping, masking, or reordering and can hardly provide semantically consistent augmentation samples. In DuoRec, a model-level augmentation is proposed based on Dropout to enable better semantic preserving. Furthermore, a novel sampling strategy is developed, where sequences having the same target item are chosen hard positive samples. Extensive experiments conducted on five datasets demonstrate the superior performance of the proposed DuoRec model compared with baseline methods. Visualization results of the learned representations validate that DuoRec can largely alleviate the representation degeneration problem.

1 INTRODUCTION

Sequential recommendation models capture users’ changing preferences from interaction patterns, but their item embeddings can become highly anisotropic and semantically indiscriminate. DuoRec addresses this degeneration with contrastive regularization, model-level Dropout augmentation, and target-item-based positive sampling.

  • Sequential recommendation methods exploit interaction sequences to capture users’ dynamic preferences over time.
  • Item embeddings generated by sequential models can collapse into a narrow cone, producing indiscriminate semantic representations.
  • DuoRec distributes item embeddings more uniformly around the origin, with more slowly decreasing singular values than SASRec.
  • DuoRec uses contrastive regularization over sequence representations to implicitly improve item-embedding uniformity.
  • The model adds Dropout-based model-level augmentation and target-item-supervised positive sampling to address semantic consistency in contrastive samples.
  • Experiments on five benchmark datasets report state-of-the-art performance and validate the effectiveness of DuoRec’s contrastive regularization.

2 REPRESENTATION DEGENERATION PROBLEM

Sequential recommendation encodes historical interactions into context and item vectors for next-item prediction through dot-product similarity. The analysis links training gradients, especially for infrequent items, to anisotropic item embeddings and motivates contrastive regularization to improve their distribution.

  • The task uses historical item interactions to infer user preference and predict the next item through retrieval over the item set.
  • Sequential models encode an interaction context into a d-dimensional vector and represent items in the same latent space.
  • The context–item dot product captures co-occurrence semantics through PMI, item frequency, and a context-related term.
  • For an item absent from the input sequence, cross-entropy optimization produces a gradient influenced by the context vector and non-target item probabilities.
  • Lower-frequency item embeddings receive gradients largely determined by the context vector and consequently move in similar directions within a narrow space.
  • Because recommendation relies on sequence–item dot products, the degenerated item distribution makes it difficult for sequence embeddings to distinguish rare-item similarities.
  • Contrastive regularization relocates sequence representations around the origin, which implicitly improves item-embedding distribution in the shared space.

3 PRELIMINARY: CONTRASTIVE LEARNING

Contrastive learning trains representations by bringing semantically close positive pairs together and separating negative pairs. Its alignment and uniformity perspectives describe closeness within positive pairs and uniform coverage of the representation space.

  • Contrastive learning pulls positive sample pairs closer while pushing negative sample pairs farther apart.
  • Noise Contrastive Estimation: The NCE objective trains an encoder using semantically close positive pairs and independently sampled negative examples, with temperature τ controlling the objective.
  • Noise Contrastive Estimation: The positive-pair distribution p_pos and independent-sample distribution p_data define the sampling sources for contrastive training.
  • Alignment and Uniformity: Alignment measures closeness between representations from positive pairs, while uniformity encourages representations of all data samples to distribute uniformly.

4 METHOD: DUOREC

DuoRec uses a Transformer-based sequential recommender with contrastive regularization over sequence representations. It combines Dropout-based model-level augmentation with target-item-based positive sampling to preserve semantic similarity while improving representation learning.

  • 4 METHOD: DUOREC: DuoRec encodes historical interactions with item embeddings and a multi-head self-attention Transformer to compute user representations.The model represents each input item through an embedding matrix before applying Transformer attention.
  • 4 METHOD: DUOREC: Positional encodings are added to item embeddings before Transformer processing, preserving the time order of interactions.The positional encoding matrix is constructed for the maximum sequence length, and each positional vector is added to the corresponding item embedding.
  • 4 METHOD: DUOREC: The next-item prediction objective classifies over the full item set using similarity between the sequence representation and item embeddings, followed by cross-entropy loss.The predictive score is computed from the sequence representation and item embedding matrix.
  • 4.3 Contrastive Regularization: DuoRec adds contrastive regularization to the recommendation objective to alleviate representation degeneration.The regularizer exploits both unsupervised and supervised contrastive samples, with the overall objective combining recommendation and regularization losses through λ.
  • 4.3.1 Unsupervised Augmentation: Different Dropout masks generate semantically similar sequence representations, replacing data-level cropping, masking, or reordering augmentations that lack guaranteed semantic consistency.The same model weights process the sequence with different Dropout masks; one resulting vector is used as an augmented representation.

4.4 Discussion

The discussion explains how DuoRec’s contrastive objective combines alignment and uniformity to address representation degeneration. It also contrasts model-level Dropout augmentation and supervised target-item sampling with data-level augmentation methods.

  • 4.4 Discussion: The contrastive regularizer aligns representations from augmented views of the same sequence while uniformly distributing sequence representations.Semantic positive pairs are pulled together, whereas the uniformity term pushes sequence representations toward a uniform distribution.
  • 4.4 Discussion: Uniform sequence representations ease the dominant-axis problem and guide item embeddings toward a less degenerate distribution through the recommendation dot product.The discussion links sequence-representation uniformity to the item-embedding distribution and also describes adjustment of rare-word positions relative to the origin.
  • 4.4.2 Connection: DuoRec’s Dropout augmentation avoids the semantic-consistency difficulty of cropping, masking, and reordering discrete sequences.The discussion presents data-level augmentation as unable to guarantee that augmented sequences retain similar semantics.
  • 4.4.2 Connection: UCL consistently outperforms CL4SRec, while SCL can outperform UCL because it uses more appropriate positive samples.These comparisons are attributed to avoiding unsuitable data-level augmentations and incorporating semantically appropriate supervised samples.

5 EXPERIMENT

Experiments on five benchmark datasets compare DuoRec with sequential and contrastive-learning baselines, then examine representation distributions, training dynamics, and hyperparameter sensitivity. DuoRec achieves state-of-the-art performance and produces more uniform representations, while the effects of augmentation and regularization settings vary across datasets.

  • Experimental Setup: Experiments use five benchmark datasets, including Amazon Beauty, Clothing, Sports, MovieLens-1M, and Yelp, with HR@K and NDCG@K evaluated over the full item set.Users and items appearing fewer than five times are removed, and sequences have maximum length 50.
  • Overall Performance: DuoRec achieves the state-of-the-art result among all baseline models and outperforms the baselines by a large margin.The comparison includes BPR-MF, GRU4Rec, Caser, SASRec, and BERT4Rec-related sequential methods.
  • Contrastive Ablation: Adding a contrastive objective generally improves performance over SASRec, while DuoRec performs best when using both unsupervised and supervised positive samples.SCL outperforms the unsupervised methods, whereas directly combining UCL and SCL can harm performance because their losses may be incompatibly aligned.
  • Representation Visualization: DuoRec produces balanced item embeddings with slowly decreasing singular values, unlike SASRec’s narrow-cone distribution and sharply declining singular values.Supervised positive samples yield a more balanced distribution than unsupervised methods, while the combined objectives behave differently across Clothing and Sports.
  • Alignment and Uniformity: During training, DuoRec’s uniformity loss decreases as validation HR@5 increases, whereas CL4SRec’s uniformity loss increases and indicates a more anisotropic distribution.Although alignment loss increases slightly for both methods, the lower uniformity loss is associated with improved recommendation performance in the reported analysis.
  • Parameter Sensitivity: Increasing the Dropout ratio decreases ML-1M performance when augmentation becomes too different from the original input, while the regularization scale λ yields consistent performance across choices.Transformer hidden-layer Dropout has a stronger influence than embedding-layer Dropout, and the best Dropout strategy differs across datasets.

6 RELATED WORK

Related work covers sequential recommendation models and the growing use of contrastive learning across recommendation settings.

  • Sequential recommendation methods use recurrent networks or attention structures to encode interaction sequences.
  • GRU4Rec, SASRec, BERT4Rec, and S3Rec are representative sequential models using recurrent, attention-based, or auxiliary training objectives.
  • Contrastive learning supports self-supervised learning by contrasting representations of related samples across computer vision and recommendation.
  • Recommendation applications include node-level, embedding-level, socially aware, and sequential contrastive learning approaches.

7 CONCLUSION

The conclusion frames DuoRec as a solution to item-embedding representation degeneration in sequential recommendation through contrastive regularization and specialized sample construction.

  • DuoRec addresses item-embedding representation degeneration with theoretical and empirical analysis of the problem.
  • Its contrastive regularization combines Dropout-based model-level augmentation with supervised positive sampling to construct contrastive samples.
  • The regularization properties are analyzed in relation to the representation distribution.

A RESULTS OF DIFFERENT BATCH SIZES

Batch-size experiments evaluate contrastive-learning sample-size sensitivity across choices from 128 to 2048.

  • Performance slightly decreases across the tested batch sizes but remains within an acceptable range.
  • Batch size affects both contrastive learning and the recommendation task under the multi-tasking learning paradigm.

B RESULTS OF DIFFERENT DROPOUT STRATEGIES

Dropout-strategy experiments compare embedding-layer and Transformer-hidden-layer Dropout across ratios from 0.1 to 0.5 and multiple datasets.

  • Embedding-layer and Transformer-hidden-layer Dropout ratios are evaluated at 0.1, 0.2, 0.3, 0.4, and 0.5.
  • Results are reported for Beauty, Clothing, Sports, ML-1M, and Yelp.
  • Transformer hidden-layer Dropout has a stronger influence on overall performance than embedding-layer Dropout.
  • The optimal Dropout strategy differs across datasets.

C RESULTS OF DIFFERENT TEMPERATURES

DuoRec's overall performance remains steady across a reasonable range of temperature values, with slight declines on Beauty and Yelp at small temperatures.

  • DuoRec performance is steady across a reasonable temperature range, with slight drops on Beauty and Yelp when the temperature is small.The evaluated temperatures are 0.1, 0.3, 0.6, 1, 3, and 6.
Loading 2110.05730v2…