PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Behavior Hypergraph-Enhanced Transformer for Sequential Recommendation

Yuhao Yang, Chao Huang, Lianghao Xia, Yuxuan Liang, Yanwei Yu, Chenliang Li

arXiv:2207.05584v2cs.IRcs.AI

TL;DR

Existing sequential recommenders largely focus on singular interaction types, limiting their treatment of dynamic heterogeneous user-item relationships. MBHT combines a multi-scale low-rank Transformer with customized multi-behavior hypergraph learning to model short- and long-range dependencies. The authors report superiority over state-of-the-art methods across settings, with ablations and case studies supporting the framework’s components.

  • Problem

    Existing sequential recommendation methods focus mainly on singular interaction types, limiting their modeling of dynamic heterogeneous relationships such as views, favorites, and purchases.

  • Method

    MBHT combines behavior-aware multi-scale Transformer encoding with low-rank self-attention and hypergraph learning for global, personalized, long-range multi-behavior dependencies.

  • Results

    MBHT consistently outperforms all types of baselines across different datasets, while ablations and case studies show benefits from its model components.

  • Takeaways & Limitations

    MBHT provides a sequential recommendation framework that explicitly captures both short-term and long-term multi-behavior dependencies across recommendation scenarios.

Abstract

from arXiv · show

Learning dynamic user preference has become an increasingly important component for many online platforms (e.g., video-sharing sites, e-commerce systems) to make sequential recommendations. Previous works have made many efforts to model item-item transitions over user interaction sequences, based on various architectures, e.g., recurrent neural networks and self-attention mechanism. Recently emerged graph neural networks also serve as useful backbone models to capture item dependencies in sequential recommendation scenarios. Despite their effectiveness, existing methods have far focused on item sequence representation with singular type of interactions, and thus are limited to capture dynamic heterogeneous relational structures between users and items (e.g., page view, add-to-favorite, purchase). To tackle this challenge, we design a Multi-Behavior Hypergraph-enhanced Transformer framework (MBHT) to capture both short-term and long-term cross-type behavior dependencies. Specifically, a multi-scale Transformer is equipped with low-rank self-attention to jointly encode behavior-aware sequential patterns from fine-grained and coarse-grained levels. Additionally, we incorporate the global multi-behavior dependency into the hypergraph neural architecture to capture the hierarchical long-range item correlations in a customized manner. Experimental results demonstrate the superiority of our MBHT over various state-of-the-art recommendation solutions across different settings. Further ablation studies validate the effectiveness of our model design and benefits of the new MBHT framework. Our implementation code is released at: https://github.com/yuh-yang/MBHT-KDD22.

1 INTRODUCTION

Sequential recommendation models user preferences from past item sequences, but many existing methods treat interactions as a single behavior type. MBHT addresses dynamic heterogeneous behaviors by combining multi-scale sequence modeling with global dependency learning.

  • 1 INTRODUCTION: Sequential recommendation forecasts future interacted items from users’ past behavior sequences.Existing approaches include recurrent, convolution-based, and self-attention sequence encoders.
  • 1 INTRODUCTION: Most existing methods encode singular interaction types, although practical user behavior combines page views, favorites, and purchases over time.Such interactions can form transitions such as clicking and favoriting products before purchasing them.
  • 1 INTRODUCTION: MBHT uses a multi-scale Transformer to encode behavior-aware item transitions from fine-grained to coarse-grained temporal levels.Low-rank self-attention improves the efficiency of the sequential pattern encoder, while pattern fusion combines multi-grained signals.
  • 1 INTRODUCTION: MBHT proposes multi-behavior hypergraph learning to capture diverse, long-range item dependencies across behavior types with global and customized sequential context.The framework is designed to uncover dynamic multi-behavior user-item interaction patterns.

2 PROBLEM FORMULATION

The paper formulates multi-behavior sequential recommendation around behavior-aware user-item interaction sequences. Interactions are divided into target behaviors to predict and auxiliary behaviors that provide contextual information about user preferences.

  • 2 PROBLEM FORMULATION: Each user is represented by a temporally ordered sequence of item-behavior pairs.The sequence records both the interacted item and its interaction type, such as page view, favorite, cart, or purchase.
  • 2 PROBLEM FORMULATION: The behavior type to be predicted is designated as the target behavior.In e-commerce, purchase may serve as the target because of its relevance to merchandise sales value.
  • 2 PROBLEM FORMULATION: Other interaction types are treated as auxiliary behaviors that provide contextual information about users’ diverse preferences.These auxiliary signals assist prediction of the target interaction type.

3 METHODOLOGY

MBHT combines behavior-aware sequence representations, efficient multi-scale Transformer encoding, and hypergraph-based dependency learning. Its architecture models temporal patterns at multiple granularities while aggregating global and personalized multi-behavior dependencies.

  • 3 METHODOLOGY: MBHT injects item, positional, and behavior-type information into behavior-aware item representations.The context embedding is formed by combining item embedding e_j, positional embedding p_j, and behavior embedding b_j.
  • 3.1 Multi-Scale Transformer: The multi-scale Transformer captures behavior-aware transitional patterns across fine-grained and coarse-grained temporal scales.Sub-sequence aggregation preserves short-term dynamics, and hierarchical scale settings produce scale-specific sequential behavior embeddings.
  • 3.1 Multi-Scale Transformer: Low-rank self-attention approximates pairwise attention through smaller attention operations and reduces the encoder’s computational cost.The method uses trainable projections and low-rank latent representation spaces over the behavior-aware interaction sequence.
  • 3.1.3 Multi-Scale Behaviour Pattern Fusion: Scale-specific behavior patterns are fused into a common latent representation through a projection and concatenation-based fusion layer.The fusion combines the base behavior representation with representations from the two selected scales.
  • 3.1.3 Multi-Scale Behaviour Pattern Fusion: Multi-head encoding projects the sequence into multiple latent spaces and performs head-specific attention operations in parallel.This allows the Transformer to encode multi-dimensional dependencies among items in the behavior-aware sequence.
  • 3.1 Multi-Scale Transformer: The architecture also includes feed-forward nonlinear transformations after multi-scale Transformer representations are generated.The feed-forward module uses two nonlinear transformation layers with GELU activation.

3.2 Customized Hypergraph Learning of Global Multi-Behavior Dependencies

MBHT addresses the difficulty of modeling personalized, long-range dependencies across multiple interaction behaviors by constructing customized hypergraph structures. Its hypergraph component combines semantic item relations with user-specific multi-behavior interactions and supports efficient cross-view representation fusion.

  • Motivation: MBHT targets multi-order, long-range dependencies among heterogeneous user-item behaviors that ordinary pairwise item relations cannot fully represent.The motivating examples include page views, add-to-favorite actions, and purchases when modeling purchase preference.
  • Hypergraph design: The global dependency encoder uses hyperedges to connect multiple items, capturing tetradic or higher-order multi-behavior dependencies over time.Different users receive different hypergraph structures based on their behavior-aware interaction sequences.
  • Semantic dependency hypergraph: The item-semantic hypergraph assigns each unique item a hyperedge connecting its top-k semantically dependent items according to learned item-item relevance scores.The relevance scores are obtained through channel-specific metric learning and mean pooling across representation channels.
  • Multi-behavior dependency hypergraph: A second hypergraph represents personalized item-wise multi-behavior dependencies from typed interactions between each user and items at different timestamps.Its hyperedges correspond to items with multi-typed interactions, so the resulting structure varies across users.
  • Integrated propagation: The semantic and multi-behavior connection matrices are concatenated into one integrated hypergraph, whose convolutional message passing captures global dependencies over time.The two-stage node-hyperedge and hyperedge-node propagation can be approximated using precomputed weights to reduce inference cost.
  • Cross-view aggregation: MBHT adaptively fuses multi-scale behavior-aware sequential representations with personalized global hypergraph representations through an attention layer.The resulting item representation is used by the forecasting layer, while target-behavior items are learned with a Cloze masking objective.

4 EXPERIMENTS

Experiments evaluate MBHT across datasets, baselines, ablations, sequence lengths, and convergence behavior. Results consistently support combining multi-scale behavior-aware sequential modeling with hypergraph learning for short- and long-range dependencies.

  • Experimental Setup: MBHT is evaluated on Taobao, Retailrocket, and IJCAI datasets containing multiple target and auxiliary interaction types.Taobao and IJCAI include purchase, add-to-favorites, add-to-cart, and page-view behaviors; Retailrocket includes purchase, page-view, and add-to-cart behaviors.
  • Experimental Setup: Evaluation uses leave-one-out testing with 100 popularity-based negative instances and HR@N, NDCG@N, and MRR metrics.The last purchase is used as the test sample and earlier purchases as validation samples; larger metric values indicate better performance.
  • Performance Evaluation (RQ1): MBHT consistently outperforms all baseline types across datasets, including general sequential, graph-based, and multi-behavior recommendation methods.The reported improvements are attributed to multi-scale behavior-aware transitions and hypergraph modeling of long-range cross-behavior item correlations.
  • Effects of Key Components: Removing either item-wise semantic or dynamic multi-behavior hypergraph learning reduces performance, while replacing multi-scale attention with vanilla multi-head attention also weakens results.These ablations support both hypergraph dependency-learning components and the multi-scale low-rank self-attention design.
  • Effects of Key Components: The hypergraph view contributes more on Taobao and IJCAI, where longer sequences require modeling long-range multi-relational dependencies.The learned view-specific importance distributions distinguish sequential representations from global multi-behavior dependency representations.

5 RELATED WORK

Related work spans Markov, neural, Transformer, and graph-based sequential recommenders, while most prior methods focus on singular interaction types. Hypergraph methods provide a basis for modeling higher-order recommendation relationships, but diverse user-item behaviors remain a central challenge.

  • Sequential Recommendation: Sequential recommendation has progressed from Markov Chain approaches to recurrent, convolutional, self-attention, and graph neural network models.Examples include GRU4Rec, Caser, SASRec, BERT4Rec, SR-GNN, MTD, MA-GNN, and SURGE.
  • Sequential Recommendation: Most existing sequential methods are designed for singular interaction behaviors and cannot handle diverse user-item relationships.This limitation motivates multi-behavior modeling beyond conventional item-transition representations.
  • Hypergraph Learning for Recommendation: Hypergraph neural networks are used in recommendation because hypergraphs can represent high-order relationships.The passage introduces hypergraph learning as a related direction alongside sequential recommendation methods.

6 CONCLUSION

MBHT explicitly captures short- and long-term multi-behavior dependencies through multi-scale Transformer and hypergraph components, with empirical validation against state-of-the-art methods.

  • MBHT captures both short-term and long-term multi-behavior dependencies in sequential recommendation.
  • A multi-scale Transformer encodes behavior-aware sequential patterns at fine-grained and coarse-grained levels.
  • A multi-behavior hypergraph component captures global cross-type behavior dependencies.
  • Experiments on several real-world datasets validate MBHT against state-of-the-art recommendation methods.

A SUPPLEMENTARY MATERIAL

The supplementary material documents MBHT’s forward propagation, analyzes its time complexity, and describes an efficiency-oriented simplification for hypergraph embedding propagation.

  • Algorithm 1 summarizes the learning process of the MBHT framework.
  • The supplement analyzes MBHT’s model time complexity.
  • The supplement presents a strategy to simplify hypergraph-based embedding propagation for improved model efficiency.

A.1 The Learning Process of MBHT

MBHT’s forward propagation combines multi-scale Transformer processing with behavior and positional signals, semantic dependency modeling, customized hypergraph propagation, and masked-item prediction.

  • The forward pass uses separate Multi-Scale Transformer and hypergraph-learning views.
  • Transformer inputs combine item embeddings, positional signals, and behavior signals, followed by multi-scale attention and point-wise feed-forward processing.
  • The hypergraph view models item-wise semantic dependencies with multi-channel metric learning and constructs customized hyperedges.
  • Hypergraph convolution aggregates information through the constructed graph representation.
  • Attentive cross-view aggregation fuses the Transformer and hypergraph-view representations before prediction.
  • The model calculates the probability that the masked item at position m is item v_t.

A.2 Time Complexity Analysis

Low-rank self-attention reduces the Transformer’s sequence-length cost, while hypergraph components add semantic metric-learning and convolution costs to the overall complexity.

  • Low-rank self-attention reduces the multi-scale Transformer cost to approximate linear complexity O(3LdJ).This assumes the low-rank and resolution scales are much smaller than sequence length J.
  • The hypergraph view has O(J^2d) semantic metric-learning complexity and O(LJd^2) hypergraph-convolution complexity.
  • Overall MBHT complexity is O(3LJd + LJd^2 + J^2d), described as comparable to state-of-the-art baselines.

A.3 Simplifying Hypergraph Message Passing

The paper simplifies hypergraph message passing by approximating the costly two-stage node–hyperedge–node operation with precomputed dependencies and a lightweight matrix. This reduces computational cost while causing only slight performance degradation in the reported Recall@5 evaluation.

  • Approximation design: Hypergraph connections distinguish behavior-aware self-connections, first-order similarities, and second-order similarities between items.The three cases depend on whether items are identical and whether the hyperedge is assigned to one or neither item.
  • Approximation design: The simplified scheme approximates two-stage hypergraph propagation using pre-calculated behavior-aware semantic similarities and a lightweight matrix M′.The approximation replaces costly second-order terms with a hyperparameter-based value while retaining precomputed similarity information.
  • Evaluation: The simplified hypergraph message passing causes only slight performance degradation while improving inference efficiency through lower computational cost.It eliminates the original O((|E_p| + |E_q|) × J^2) calculations and constructs M′ from pre-calculated values and preprocessing.

A.4 Hyperparameter Study (RQ4)

The hyperparameter study examines multi-scale settings, low-rank projection channels, and the number of semantic neighbors connected through item-wise hyperedges. The best settings vary across datasets, while overly broad hypergraph connectivity can introduce noise and reduce performance.

  • Impact of Multi-Scale Settings: The best multi-scale settings differ by dataset: (p1, p2) = (4, 20) for Retailrocket and (8, 40) for Taobao.The reported difference is associated with the datasets’ different average sequence lengths.
  • Impact of Item-wise Semantic Dependency Set: Increasing the number of hypergraph-connected items can initially improve performance but eventually degrades it by introducing noise into embedding propagation.The semantic dependency set is tuned by selecting top-k dependent items.
Loading 2207.05584v2…