PaperlayerView exact revision on arXiv

Source-linked AI summary

Sequence-level Semantic Representation Fusion for Recommender Systems

Lanling Xu, Zhen Tian, Bingqian Li, Junjie Zhang, Jinpeng Wang, Mingchen Cai, Wayne Xin Zhao

arXiv:2402.18166v1cs.IR

TL;DR

Item-level fusion of text and ID features does not let text contribute across a sequence, and pretrained text embeddings can be insufficiently discriminative. TedRec uses MoE-based positional modulation and frequency-domain fusion to integrate the features at the sequence level. Experiments report gains over competitive baselines on ML-1M and OR.

  • Problem

    Existing text–ID fusion often operates at the item level, limiting the use of text in sequence-wide semantics, while pretrained text embeddings can lack fine-grained discriminability among similar items.

    Source: §1 · p. 1

  • Method

    TedRec injects positional information into text embeddings with MoE modulation, then transforms text and ID sequences into the frequency domain for multiplicative sequence-level fusion.

    Source: §1 · p. 2

  • Results

    14% and 38% performance gains over competitive baselines are reported on ML-1M and OR, respectively, across experiments on five public datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach is presented as a general framework applicable with various text encoders and recommender backbones.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

With the rapid development of recommender systems, there is increasing side information that can be employed to improve the recommendation performance. Specially, we focus on the utilization of the associated \emph{textual data} of items (eg product title) and study how text features can be effectively fused with ID features in sequential recommendation. However, there exists distinct data characteristics for the two kinds of item features, making a direct fusion method (eg adding text and ID embeddings as item representation) become less effective. To address this issue, we propose a novel {\ul \emph{Te}}xt-I{\ul \emph{D}} semantic fusion approach for sequential {\ul \emph{Rec}}ommendation, namely \textbf{\our}. The core idea of our approach is to conduct a sequence-level semantic fusion approach by better integrating global contexts. The key strategy lies in that we transform the text embeddings and ID embeddings by Fourier Transform from \emph{time domain} to \emph{frequency domain}. In the frequency domain, the global sequential characteristics of the original sequences are inherently aggregated into the transformed representations, so that we can employ simple multiplicative operations to effectively fuse the two kinds of item features. Our fusion approach can be proved to have the same effects of contextual convolution, so as to achieving sequence-level semantic fusion. In order to further improve the fusion performance, we propose to enhance the discriminability of the text embeddings from the text encoder, by adaptively injecting positional information via a mixture-of-experts~(MoE) modulation method. Our implementation is available at this repository: \textcolor{magenta}{\url{https://github.com/RUCAIBox/TedRec}}.

1 INTRODUCTION

Existing item-level text–ID fusion limits how text can contribute to sequence-wide context, while pretrained text embeddings can blur similar items. TedRec combines text and ID features at the sequence level using frequency-domain fusion and MoE-based positional modulation.

1 INTRODUCTION

  • Item-level fusion uses an item's text embedding to enhance that item, whereas sequence-level fusion lets text at one position contribute to global sequence semantics.The paper identifies this as a limitation of existing fusion approaches.Source: §1 · p. 1
  • TedRec is presented as a general framework for fusing multiple kinds of side information in recommender systems.The approach is described as applicable with various text encoders and recommender backbones.Source: §1 · p. 2
  • TedRec transforms text and ID sequences into the frequency domain and multiplicatively fuses them, yielding sequence-level fusion with contextual-convolution effects.The transformed representations aggregate global sequential characteristics.Source: §1 · p. 2
  • An MoE modulation method adaptively injects positional information to improve the discriminability of text embeddings.This targets the similarity of embeddings for similar or related products.Source: §1 · p. 2
  • TedRec reports 14% and 38% performance gains over competitive baselines on ML-1M and OR, respectively.The experiments cover five public datasets.Source: §1 · p. 2

2 PRELIMINARY

Sequential recommendation predicts a user's next item from an ordered history of prior interactions. The preliminaries introduce DFT and its inverse for moving between time and frequency domains, then describe FFT as a more efficient way to compute DFT.

2 PRELIMINARY

  • Sequential recommendation is defined as predicting the next item from a user's ordered interaction history.The prediction is for the next, or (n + 1)-th, step.Source: §2 · p. 2
  • Discrete Fourier Transform (DFT) converts sampled sequence data from the time domain to the frequency domain.The passage introduces the transform for sequence signal processing.Source: §2 · p. 2
  • The inverse DFT recovers signals in the time domain from representations in the frequency domain.Source: §2 · p. 2
  • FFT computes DFT with complexity O(n log n), reduced from O(n^2) for DFT.It achieves this by factorizing the DFT matrix into a product of sparse factors.Source: §2 · p. 2

3 METHODOLOGY

TedRec combines MoE-based text modulation with frequency-domain text-ID fusion to integrate sequential context. Its theoretical analysis relates the fusion to contextual convolution, and its discussion describes the approach as model-agnostic.

3.1 Sequence-Level Representation Fusion

  • TedRec combines MoE-enhanced text modulation with frequency-domain mutual filtering to fuse textual and ID representations using sequence context.The method is presented alongside theoretical analysis and discussion of its properties.Source: §3 · p. 2, §3.1 · p. 3

3.1.1 Distinguishable Textual Representation Encoding.

  • Multiple MoE experts adapt positional modulation embeddings to interaction contexts, aiming to make text representations more distinguishable before fusion.The text encoder is fixed during training.Source: §3.1.1 · p. 3

3.1.2 Text-ID Semantic Fusion in the Frequency Domain.

  • FFT maps text and ID sequences to the frequency domain, where mutual filtering and a learnable filter attenuate ID noise before inverse FFT and gated combination.The resulting streams are fused text-ID features and filtered ID features.Source: §3.1.2 · p. 4

3.2 Theoretical Analysis

  • Frequency-domain mutual filtering is equivalent to circular convolution across text-ID sequence positions, integrating past and future context before attention.The authors note that shared sequential context can yield shared partial representations, a potential generalization benefit left for future work.Source: §3.2 · pp. 4–5

3.3 Discussion

  • TedRec contrasts with typical item-level fusion through sequence-level frequency-domain fusion and MoE positional modulation, and is designed to work with varied encoders and recommender backbones.Fusion complexity is O(n(G+log n)d); the framework requires no additional PLM training and adds no time cost to backbone self-attention.Source: §3.3 · p. 5

4.1 Experimental Setting

The experiments evaluate TedRec on five public recommendation datasets using item text and compare it with a range of sequential recommenders. Evaluation uses leave-one-out splits, full-sort ranking, and Recall and NDCG at two cutoffs.

4.1.1 Dataset Descriptions.

  • The five datasets are MovieLens-1M, Online Retail, and the Office, Food, and Movies Amazon subsets, with item text assembled from dataset-specific descriptive fields.Movie text uses title, genre, and year; Online Retail uses descriptions; the Amazon subsets use title, categories, and brand.Source: §4.1.1 · pp. 5–6
  • A five-core strategy filters inactive users and unpopular items with fewer than five records.Source: §4.1.1 · p. 6
  • Baselines include ID-based, text-fusion, pre-training, and filtering approaches, including SASRec, FDSA, UniSRec, and DLFS-Rec.UniSRec is trained from scratch without additional pre-training datasets.Source: §4.1.1 · p. 6

4.1.2 Evaluation Metrics.

  • Performance is measured with Recall@10, Recall@20, NDCG@10, and NDCG@20 using leave-one-out splits and full-sort ranking averaged over users.Each user's interaction sequence is split into training, validation, and test sets.Source: §4.1.2 · p. 6

4.1 Experimental Setting

  • Models are implemented in RecBole and trained with Adam, cross-entropy loss, batch size 2,048, and validation-based early stopping.Experiments use an NVIDIA A100; early stopping follows 10 epochs without validation NDCG@10 improvement.Source: §4.1.3 · p. 6

4.2 Performance Comparison

TedRec outperforms the compared baselines on all five datasets. The reported comparisons also show that frequency-based ID models can be competitive on dense data, while textual side information improves several sequential models.

  • On dense MovieLens-1M, ID-only FEARec is comparable to text-enhanced models, while outperforming FMLP-Rec and SASRec in the reported comparisons.Source: §4.2 · p. 6
  • TedRec outperforms all baselines on five public datasets, supporting the effectiveness of its sequence-level text-ID fusion.Source: §4.2 · p. 6
  • Text-fusion baselines show varying improvements over SASRec across datasets, while TedRec outperforms UniSRec and the other compared methods.The authors attribute its advantage over UniSRec to multi-expert modulation and contextual convolution.Source: §4.2 · p. 6

4.3 Further Analysis

Further analyses find that TedRec's components contribute to performance, with sequence-level fusion especially important on sparse Online Retail. The method also shows comparable training efficiency, gains across user activity groups, and improvements across tested backbones.

4.3.1 Ablation Study.

  • TedRec's full framework performs best among the tested variants, and removing sequence-level fusion yields poor results on sparse Online Retail.The ablations remove multi-expert modulation, adaptive gating, item convolution, or contextual convolution.Source: §4.3.1 · p. 6, Figure 3 · p. 7

4.3.2 Efficiency Comparison.

  • TedRec has comparable training efficiency to baseline models while achieving significantly better recommendation performance on two datasets.FEARec and DLFS-Rec have long training times associated with contrastive-learning data reconstruction and Wasserstein-distance calculations.Source: §4.3.2 · p. 7, Figure 4 · p. 7

4.3.3 Impact of User Groups.

  • TedRec improves Recall@10 over baselines with textual features across user activity groups on two datasets, including inactive users.The test-user distributions in both datasets are long-tailed.Source: §4.3.3 · p. 7, Figure 5 · p. 8

4.3.4 Impact of Model Backbones.

  • TedRec consistently improves UniSRec, DLFS-Rec, SASRecF, and DIF-SR, with larger gains on UniSRec and DLFS-Rec than on SASRecF and DIF-SR.The authors suggest the latter backbones' lack of text modeling or expert modulation may limit the effectiveness of TedRec's fusion.Source: §4.3.4 · pp. 7–8, Table 4 · p. 8
  • Among BERT, T5, Flan-T5, and LLaMA2 text representations, pre-trained BERT's CLS embedding performs best; large language models show no significant semantic-modeling advantage.The authors suggest mean pooling of text tokens may be unsuitable for mining semantics.Source: §4.3.5 · p. 8

5 RELATED WORK

Prior work on sequential recommendation uses a range of architectures to model chronological interactions, while side-information research incorporates item attributes through several fusion strategies. Recommender systems also use pre-trained language models for textual representations, task conversion, and data augmentation.

  • Sequential recommenders use CNNs, RNNs, self-attention, graph neural networks, and learnable filters to model interaction sequences and user preferences.Frequency-domain filter methods are also used to attenuate noise and explore periodic preferences.Source: §5 · p. 8
  • Side-information fusion methods include late fusion and pre-training, embedding concatenation, graph aggregation, attention fusion, decoupled attention, and Gaussian-distribution representations.Source: §5 · p. 8
  • Recommender-system work uses discriminative language models for textual representations and generative models for recommendation tasks or data augmentation.Examples include UniSRec for transferable representations and KAR and LLMRG for extracting textual information for fused embeddings.Source: §5 · p. 8

6 CONCLUSION

TedRec fuses text and ID representations at sequence level by integrating global contexts. It combines MoE-based positional modulation with frequency-domain fusion, and experiments on five public datasets demonstrate its effectiveness and efficiency.

  • TedRec transforms text and ID embeddings into the frequency domain and multiplicatively fuses them for sequence-level semantic representation fusion.The fusion has the same effects as contextual convolution in the time domain.Source: §6 · p. 9
  • An MoE-enhanced modulation method adaptively injects positional information to improve the discriminability of text representations from the text encoder.Source: §6 · p. 9
  • Experiments on five public datasets demonstrate the effectiveness and efficiency of TedRec.Source: §6 · p. 9
  • Future work will consider frequency-domain contextual convolution for multimodal and diversified recommendation.Source: §6 · p. 9

Related papers

Loading 2402.18166v1…