Hierarchical Time-Aware Mixture of Experts for Multi-Modal Sequential Recommendation

1 Introduction

• Prior multi-modal sequential recommenders emphasize modality fusion but often overlook redundant information and explicit temporal signals such as timestamps and intervals.Earlier approaches use chronological order as implicit temporal information.Source: §1 · p. 2
• Redundant interest-irrelevant modality content, unstable explicit-time signals, and sparse sequences make user-interest modeling difficult.The examples include irrelevant product text and imagery, changing interests, and short interaction histories.Source: §1 · p. 2
• HM4SR uses Interactive MoE to extract interest-relevant item information and Temporal MoE to incorporate explicit timestamp embeddings into modality modeling.It also uses Transformers to encode modality-specific behavior sequences.Source: §1 · p. 2
• CP, IDCL, and PCL provide auxiliary supervision for category understanding, sequence-to-interest alignment, and integration of temporal information with modalities.PCL promotes consistency between original sequences and augmented sequences containing time-based placeholders.Source: §1 · p. 2
• Experiments on four public datasets are reported to verify HM4SR’s effectiveness.Source: §1 · p. 2

2 Related Work

• Related work is organized around traditional and multi-modal sequential recommendation.Source: §2 · p. 2

2.1 Traditional Sequential Recommendation

• Traditional methods use varied sequence encoders and time-aware designs, but the reviewed approaches do not explore multi-modal item features.Examples include RNN, CNN, self-attention, and time-aware sequence modeling.Source: §2.1 · pp. 2–3

2.2 Mutli-Modal Sequential Recommendation

• Multi-modal methods range from direct fusion to adaptive and MoE-based approaches, while often neglecting explicit temporal data and redundant-modal filtering.The review positions HM4SR as incorporating dynamic temporal information and auxiliary supervision to address these gaps.Source: §2.2 · p. 3

3 Problem Definition

• Each item is represented by its ID, associated text, and associated image content.Source: §3 · p. 3
• A user’s historical behavior is an ordered item sequence, with a corresponding timestamp at each time step.Source: §3 · p. 3
• The objective is to select the item with the highest conditional probability of being the user’s next interaction given the history.Source: §3 · p. 3

4 Methodology

• HM4SR combines initial item representations, two expert levels, modality-specific user-interest learning, and auxiliary multi-task training.The prediction sums logits from three modalities; auxiliary tasks include category prediction, ID contrastive learning, and placeholder contrastive learning.Source: §4 · p. 3

4.1 Initial Item Representation

• Items receive an ID embedding, while text and image features are encoded using pre-trained models.The ID embedding matrix has the item-set size by hidden-dimension shape.Source: §4.1 · p. 3
• BERT encodes item text, and the embedding of the special [CLS] token represents the text features.Source: §4.1 · p. 3
• ViT processes item images by transforming image patches into a sequence and encoding that sequence.Source: §4.1 · p. 3

ViT

• Positional embeddings are added to the ID, text, and image representations.Source: ViT · p. 4

4.2 Hierarchical Mixture of Experts

• The hierarchical MoE pairs Interactive MoE feature extraction with Temporal MoE routing based on explicit time embeddings.Source: §4.2 · p. 4

4.2.1 Interactive MoE.

• Interactive MoE processes item modalities through target-modality experts, semantically enhancing interest-related key information.Source: §4.2.1 · p. 4

4.2.2 Temporal MoE.

• Temporal MoE uses interval and absolute-time embeddings to route experts and model dynamic user interests.Source: §4.2.2 · p. 5

4.3 User Interest Learning and Prediction

• HM4SR feeds modality-specific Temporal MoE sequences into Transformers, then sums their modality scores for the final prediction.Source: §4.3 · p. 5

4.4 Multi-Task Learning

• Sequence-level category prediction uses item categories as labels to supervise modality-based category classification.Source: §4.4 · p. 5
• IDCL aligns user sequence representations with ground-truth item representations while separating them from other items.Source: §4.4 · p. 5
• PCL replaces some sequence items with time placeholders and uses contrastive learning to integrate temporal information with modalities.Source: §4.4 · p. 6

5.1 Experimental Setup

• Evaluation uses four 5-core Amazon datasets, chronological leave-one-out ranking over the full item set, NDCG@K and MRR@K, and traditional, time-aware, and multi-modal baselines.Text combines title, category, and brand; images use each item's first product image.Source: §5.1.1 · p. 6, §5.1.2 · p. 6, §5.1.3 · p. 6

5.2 Overall Comparison

• 2.13%∼22.9% gains over the best multi-modal method and 13.6%∼64.1% over the best time-aware traditional model are reported for HM4SR.Gains are relatively smaller on Games, which the authors suggest has a more unified user-behavior time distribution.Source: §5.2 · pp. 6–7, Table 1 · p. 7

5.3 Ablation Study

• Removing any component or modality reduces performance; Temporal MoE has the largest component effect, while text usually contributes more than images.PCL further amplifies Temporal MoE's enhancement, and Interactive MoE improves extraction of preference-relevant item information.Source: §5.3 · p. 7, Figure 3 · p. 7

5.4 Hyper-Parameter Study

• Sufficient experts improve performance, but further increases may slightly hurt; IDCL/PCL weights and placeholder proportion also work best when reasonably set.Too little placeholder replacement limits time-modality relationship learning, while too much can make positive samples fluctuate.Source: §5.4 · p. 7, Figure 4 · p. 8

5.5 Impact of Temporal Embeddings

• All tested time embeddings facilitate preference learning, but HM4SR's combination of interval and absolute-time embeddings advances more than alternatives.The paper also reports its interval embeddings as more flexible and useful than Cos-based interval embeddings.Source: Figure 5 · p. 8, §5.5 · p. 8

5.6 Case Study

• In a short Toys sequence, HM4SR predicts the sound-related ground-truth toy, while M3SRec favors toys associated with animal figures or the last item's brand.The authors attribute HM4SR's prediction to sound-feature extraction, recent interest in musical toys, and multi-task supervision.Source: §5.6 · p. 8

• Interactive MoE processes item modalities to extract key interest features, while Temporal MoE uses explicit temporal information to model dynamic interests.Source: §6 · p. 8
• Experiments on public datasets demonstrate the effectiveness of HM4SR.Source: §6 · p. 8
• CP, IDCL, and PCL provide auxiliary supervision for item features, sequence context, and temporal-modal correlations.The paper introduces these tasks alongside the main sequential-recommendation task to address information loss from data sparsity.Source: §6 · p. 8

• Table 2 reports statistics for the processed Toys, Games, Beauty, and Home datasets.Source: Table 2 · p. 10, §A.1 · p. 10

• Traditional baselines include non-time-aware sequence models such as GRU4Rec [20], SASRec [27], and LRURec [61], alongside time-aware methods such as TiSASRec, FEARec [16], and TiCoSeRec.Source: §A.2 · p. 10
• Multi-modal baselines include methods for side-information fusion and modality learning, including NOVA, DIF-SR [55], UniSRec [22], MISSRec [48], and M3SRec.Source: §A.2 · p. 10
• IISAN [18] uses parameter-efficient tuning of modality foundation models, while TedRec [57] fuses text and ID at sequence level using frequency-domain processing.Source: §A.2 · p. 11

A.3 Time Distributions of Datasets

• Games has a more uniform interaction-time distribution than the other datasets, which the authors say decreases the difficulty of modeling users’ dynamic interests.The distributions are formed by grouping each dataset’s timestamp range into 30 equal intervals and counting interactions in each group.Source: §A.3 · p. 11

A.4 Additional Results of Hyper-Parameter Study

• Inappropriate hyper-parameter values can hinder HM4SR’s improvement, and MRR@5 sensitivity on Toys and Games follows a tendency similar to NDCG@5.Source: §A.4 · p. 11