PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparse-Interest Network for Sequential Recommendation

Qiaoyu Tan, Jianwei Zhang, Jiangchao Yao, Ninghao Liu, Jingren Zhou, Hongxia Yang, Xia Hu

arXiv:2102.09267v1cs.IRcs.LG

TL;DR

Sequential recommendation must handle users’ multiple, sparse interests rather than relying on a single embedding or a small concept set. SINE adaptively activates concepts, generates multiple interest embeddings, and aggregates them for next-item prediction; experiments report better performance than state-of-the-art baselines and reasonable item clusters.

  • Problem

    Existing methods struggle to represent users’ multiple sparse interests over large, imperfectly clustered concept pools and to select the next active intention.

  • Method

    SINE jointly learns a large conceptual prototype pool, adaptively activates a user-specific subset, generates multiple interest embeddings, and aggregates them for prediction.

  • Results

    SINE outperforms state-of-the-art baselines on challenging datasets while producing reasonable item clusters, including on a billion-scale industrial dataset.

  • Takeaways & Limitations

    SINE provides a sparse-interest framework that models multiple intentions and supports both recommendation accuracy and interpretable item clustering.

  • Takeaways & Limitations

    During inference, the absence of the training-time next-item label creates a train–test gap that may cause performance degeneration.

Abstract

from arXiv · show

Recent methods in sequential recommendation focus on learning an overall embedding vector from a user's behavior sequence for the next-item recommendation. However, from empirical analysis, we discovered that a user's behavior sequence often contains multiple conceptually distinct items, while a unified embedding vector is primarily affected by one's most recent frequent actions. Thus, it may fail to infer the next preferred item if conceptually similar items are not dominant in recent interactions. To this end, an alternative solution is to represent each user with multiple embedding vectors encoding different aspects of the user's intentions. Nevertheless, recent work on multi-interest embedding usually considers a small number of concepts discovered via clustering, which may not be comparable to the large pool of item categories in real systems. It is a non-trivial task to effectively model a large number of diverse conceptual prototypes, as items are often not conceptually well clustered in fine granularity. Besides, an individual usually interacts with only a sparse set of concepts. In light of this, we propose a novel \textbf{S}parse \textbf{I}nterest \textbf{NE}twork (SINE) for sequential recommendation. Our sparse-interest module can adaptively infer a sparse set of concepts for each user from the large concept pool and output multiple embeddings accordingly. Given multiple interest embeddings, we develop an interest aggregation module to actively predict the user's current intention and then use it to explicitly model multiple interests for next-item prediction. Empirical results on several public benchmark datasets and one large-scale industrial dataset demonstrate that SINE can achieve substantial improvement over state-of-the-art methods.

1 INTRODUCTION

Sequential recommendation models commonly compress behavior histories into one embedding, but users often pursue multiple distinct item concepts. SINE addresses this by extracting sparse multi-interests and predicting which interest should guide next-item recommendation.

  • Motivation: Users often interact with conceptually different item types, making a single behavior-sequence embedding a potential bottleneck for next-item prediction.A user’s recent fifty Taobao behaviors contain around 10 item categories.
  • Challenges: Extracting multiple interests is difficult because real items may be poorly clustered, users activate only sparse concepts, and interest selection is discrete.The model must also predict the next intention at inference time without the next-item label available during training.
  • Approach: SINE jointly learns a large interest-group pool, adaptively extracts users’ interacted interests, and produces multiple interest embeddings end to end.Its sparse-interest extraction module is paired with an aggregation module for dynamic next-intention prediction.
  • Approach: The interest aggregation module explicitly models multiple interests for top-N recommendation by selecting the user’s preferred current intention.The framework is evaluated on public benchmarks and an industrial dataset.
  • Contributions: SINE jointly integrates item clustering and sparse-interest extraction, adaptively aggregates interests, and reports state-of-the-art performance with reasonable interest groups.These contributions target sequential top-N recommendation across real-world datasets.

2 RELATED WORK

Related work spans conventional recommendation, deep neural recommendation, sequential models, and attention mechanisms. These lines of research provide the modeling foundations from which sequential recommendation methods and SINE develop.

  • Conventional Recommendation: Conventional recommender systems use collaborative filtering, matrix factorization, and factorization machines to represent users and items in shared latent spaces.These methods estimate user-item similarity or interaction likelihood from historical behavior.
  • Deep Recommendation: Deep recommendation research replaces or augments traditional interaction modeling with neural networks, content features, and autoencoders.Examples include neural collaborative filtering and rating prediction with autoencoders.
  • Sequential Recommendation: Sequential recommendation models item transitions with Markov chains or process behavior sequences using recurrent, convolutional, and attention-based neural modules.These approaches differ in how much history and sequential structure they model.
  • Attention Mechanisms: Self-attention captures long-term sequence semantics while attention-based modules can adapt user representations to particular recommendation contexts.SASRec and DIN illustrate sequence-level and target-specific attention mechanisms.
  • Attention Mechanisms: Attention became a central neural building block through applications in computer vision, machine translation, language modeling, and recommendation.Transformer and BERT exemplify its broader adoption.

3 METHODOLOGY

This section introduces the recommendation problem, details SINE’s framework, and positions it relative to existing methods. It therefore establishes both the formal setup and the model’s methodological context.

  • Section Organization: The methodology section first presents the problem formulation before explaining SINE and comparing it with existing frameworks.The section is organized around formulation, framework details, and methodological differences.
  • Section Organization: The framework discussion is followed by a comparison with existing methods to clarify how SINE differs from prior approaches.The supplied roadmap explicitly separates framework details from related-method analysis.
  • Section Organization: SINE’s methodology is presented as a progression from defining the recommendation task to explaining the proposed architecture and its distinctions.This ordering connects formal setup with model design and comparison.

3.1 Notations and Problem Formulation

The task uses each user’s ordered click sequence as input for matching-stage retrieval. The goal is to retrieve high-quality candidate items from a large item pool based on observed behavior.

  • Notation: A user’s ordered behavior sequence records clicked items, with each sequence element representing the index of a clicked item.The sequence is ordered by the user’s click history.
  • Problem Formulation: SINE targets the matching stage, which retrieves top-N candidate items before ranking sorts them using more precise scores.The matching task uses the observed sequence to retrieve items the user might click.

3.2 Sparse-Interest Framework

SINE explicitly activates a sparse subset of conceptual prototypes for each user, then derives multiple interest embeddings from the behavior sequence. This addresses the difficulty of modeling diverse intentions when conventional sequential encoders produce mixed representations.

  • Design rationale: Unlike implicit multi-interest methods, SINE explicitly detects intentions from latent conceptual prototypes.The paper contrasts prototype-based explicit extraction with neural approaches that implicitly cluster historical behaviors.
  • Concept activation: SINE activates K prototypes from a large concept pool for each user instead of relying only on implicitly extracted multiple representations.The activated prototype matrix C_u is formed from L total concepts, and top-K selection identifies the user-specific subset.
  • Concept activation: A self-attentive layer converts the behavior sequence into a virtual concept vector used to select interested prototypes.Attention weights over item behaviors are aggregated into z_u, which reflects the user’s general intentions.
  • Interest extraction: The selected prototypes assign latent intentions to sequence items and support multiple interest embeddings for next-item prediction.The framework combines conceptual assignment and item-level attention to generate embeddings from the user’s sequence.

3.3 Interest Aggregation Module

SINE avoids selecting interests with the unavailable next-item label during inference by predicting the user’s next intention and using it to aggregate multiple interest embeddings.

  • Motivation: Target-label interest selection creates a training–inference gap because next-item labels are unavailable at inference time.The paper identifies this mismatch as a potential source of performance degeneration.
  • Active prediction: The aggregation module predicts the user’s next intention from the intention sequence reformulated as P_u C_u.The intention distribution matrix assigns sequence items to activated concepts before next-intention prediction.
  • Interest aggregation: Attention weights over diverse interests are computed from the predicted intention and used to combine the multiple interest embeddings.The module calculates aggregation weights after forming the intention-aware sequence.
  • Interest aggregation: With τ=0.1, the aggregator is designed to select the most preferred intention for inference.Large τ produces near-uniform weights, while small τ produces near one-hot weights.

3.4 Model Optimization

SINE is trained with next-click negative log-likelihood, sampled softmax for computational tractability, and covariance regularization to encourage orthogonal conceptual prototypes.

  • Training objective: SINE trains by recovering the user’s next click with a negative log-likelihood objective.The objective uses the user representation and item embedding for each training sample.
  • Training objective: Sampled softmax approximates the intractable denominator of the full next-item likelihood during training.The full denominator is computationally prohibitive because it sums over the item pool.
  • Prototype regularization: A covariance regularizer encourages the learned conceptual prototypes to be orthogonal.The regularizer is applied to the covariance of prototype embeddings and combined with the prediction loss.
  • Prototype regularization: The trade-off parameter λ balances the prediction and covariance-regularization losses.The final loss combines both terms using λ.

3.5 Connections with Existing Models

The paper distinguishes implicit methods, which rely on neural sequence encoders to extract multiple interests, from explicit methods, which use latent prototypes to identify them.

  • Implicit approach: Implicit approaches use neural architectures such as Capsule networks or multi-head self-attention to aggregate behaviors into multiple interest representations.MIND and SASRec are cited as examples of this category.
  • Explicit approach: Explicit approaches maintain conceptual prototypes that determine the intentions associated with items in the behavior sequence.MCPRN and DisenRec are cited as representative prototype-based methods.
  • SINE’s position: SINE belongs to the explicit category because it detects intentions from latent conceptual prototypes.Its prototype-based design contrasts with methods that mix intention detection and embedding in neural sequence encoders.

4 EXPERIMENTS

SINE is evaluated against sequential-recommendation baselines on three public datasets and one billion-scale industrial dataset, using recommendation and prototype-clustering metrics. Results examine overall effectiveness, sensitivity to K and L, industrial performance, learned concepts, and ablations of the proposed modules.

  • Experimental Setup: Experiments cover three benchmark datasets and one billion-scale industrial dataset, comparing single-embedding and multi-embedding recommendation models.The setup includes GRU4Rec and Caser alongside MIND, SASRec, MCPRN, and SINE.
  • Comparisons with SOTA (Q1): SINE achieves comparable performance to baselines across public-dataset evaluation criteria, while multi-embedding methods generally outperform single-embedding methods.The evaluation uses hit rate, NDCG, and NMI; NMI measures the quality of learned conceptual prototypes.
  • Comparisons with SOTA (Q1): SINE consistently outperforms MIND across all evaluation metrics on the three public datasets.The authors attribute this to large prototype pools, sparse inference of preferred intentions, and interest aggregation for prediction.
  • Comparisons with SOTA (Q1): SINE performs best on Taobao when K=8 and L=1000, indicating that its learned concepts can represent virtual categories spanning several item categories.Taobao contains around 9000 categories, while the selected prototype count is 1000.
  • Industrial Results (Q1): On the ULarge industrial dataset, SINE significantly outperforms other baselines, although its advantage over MIND decreases as the number of recalled items increases.The authors connect this behavior to improved capture of diverse interests and ranking of preferred items near the top of the recommendation list.
  • Industrial Results (Q1): Learned prototypes align most closely with Alibaba’s second-level categories rather than its coarser first-level or finer leaf-level categories.This supports interpreting the prototypes as relatively high-level semantics for user-intention modeling.

5 CONCLUSIONS

SINE adaptively activates multiple user intentions from a large prototype pool, aggregates them for top-N recommendation, and outperforms state-of-the-art baselines on challenging datasets.

  • SINE activates multiple intentions from conceptual prototypes to generate multiple interest embeddings for each user.
  • Its interest aggregation module captures multiple interests to obtain the overall top-N recommendation items.
  • SINE performs better than state-of-the-art baselines and remains effective on a billion-scale industrial dataset for recommendation accuracy and item-cluster quality.
Loading 2102.09267v1…