Source-linked AI summary
CauseRec: Counterfactual User Sequence Synthesis for Sequential Recommendation
Shengyu Zhang, Dong Yao, Zhou Zhao, Tat-seng Chua, Fei Wu
TL;DR
Sequential recommenders can become brittle when they rely only on noisy and sparse observed interactions. CauseRec synthesizes counterfactual user sequences by modifying indispensable and dispensable concepts, then contrasts their representations with the observational representation. Across public benchmarks, the reported results show improvements over state-of-the-art sequential recommenders, while the authors identify concept-selection and model coverage as future directions.
Problem
Noisy and sparse observed interactions can make user representations brittle, motivating methods that learn representations less sensitive to dispensable behaviors and more reliant on indispensable ones.
Method
CauseRec identifies indispensable and dispensable concepts at item and interest levels, replaces them to synthesize counterfactual sequences, and contrasts counterfactual representations with observational representations.
Results
CauseRec outperforms state-of-the-art sequential recommenders in extensive experiments using a naive matching baseline, with improvements reported across multiple architectures and metrics.
Takeaways & Limitations
Counterfactual contrastive learning provides a simple, effective, and non-intrusive way to learn accurate and robust user representations for sequential matching.
Takeaways & Limitations
The authors use a naive baseline and leave broader model coverage and more effective indispensable/dispensable concept identification for future work.
Abstract
from arXiv · showhide
Learning user representations based on historical behaviors lies at the core of modern recommender systems. Recent advances in sequential recommenders have convincingly demonstrated high capability in extracting effective user representations from the given behavior sequences. Despite significant progress, we argue that solely modeling the observational behaviors sequences may end up with a brittle and unstable system due to the noisy and sparse nature of user interactions logged. In this paper, we propose to learn accurate and robust user representations, which are required to be less sensitive to (attack on) noisy behaviors and trust more on the indispensable ones, by modeling counterfactual data distribution. Specifically, given an observed behavior sequence, the proposed CauseRec framework identifies dispensable and indispensable concepts at both the fine-grained item level and the abstract interest level. CauseRec conditionally samples user concept sequences from the counterfactual data distributions by replacing dispensable and indispensable concepts within the original concept sequence. With user representations obtained from the synthesized user sequences, CauseRec performs contrastive user representation learning by contrasting the counterfactual with the observational. We conduct extensive experiments on real-world public recommendation benchmarks and justify the effectiveness of CauseRec with multi-aspects model analysis. The results demonstrate that the proposed CauseRec outperforms state-of-the-art sequential recommenders by learning accurate and robust user representations.
1 INTRODUCTION
Sequential recommendation relies on historical behavior sequences, but noisy and sparse interactions can produce brittle user representations. CauseRec addresses this by synthesizing counterfactual sequences and contrasting them with observations.
- Motivation: Sequential recommenders model users’ current and recent preferences from time-ordered user-item interactions.They predict the next item a user might interact with based on historical interactions.
- Challenges: Noisy implicit feedback and sparse interactions can make observational-only modeling brittle and less satisfactory.Noise can arise from caption bias, position bias, and sales promotions, while users interact with only a limited subset of a potentially massive item gallery.
- CauseRec: CauseRec models counterfactual data distributions by identifying indispensable and dispensable concepts at item and interest levels.It transforms both fine-grained item-level and abstract interest-level concept sequences.
- CauseRec: CauseRec learns user representations by contrasting observational representations with counterfactually positive and negative representations.Dispensable-concept modifications produce positive representations, whereas indispensable-concept replacements produce negative representations.
- Results: Extensive experiments report that CauseRec outperforms state-of-the-art sequential recommenders using a naive deep candidate-generation baseline.The study also analyzes architectural components and hyperparameters and uses case studies to examine user representations.
2 RELATED WORKS
Related work spans sequential recommendation, contrastive learning for representation learning, and counterfactual reasoning for recommendation. CauseRec combines these directions through model-agnostic, non-intrusive counterfactual contrastive learning.
- Sequential recommendation: Sequential recommendation has progressed from Markov chains and factorization machines to deep models that capture long-term and multi-level dependencies.Examples include RNNs, CNNs, memory networks, and attention mechanisms.
- Contrastive learning: Contrastive learning has been applied across several domains, but its integration into recommendation remains comparatively unexplored.Prior recommendation approaches include noise contrastive estimation and contrastive pretraining, with some requiring fine-tuning.
- Contrastive learning: CauseRec differs from earlier contrastive recommendation methods by using model-agnostic and non-intrusive frameworks in an end-to-end manner.It contrasts original user representations with counterfactually positive and negative samples.
- Counterfactual recommendation: Prior counterfactual recommendation work mainly addresses debiasing learning-to-rank problems, including presentation, position, and clickbait biases.CauseRec instead applies counterfactual reasoning to user representation learning.
3.1 Problem Formulation
Sequential recommendation represents each training example as a user’s historical behavior sequence paired with the next item. Matching systems retrieve a small candidate set before ranking because the item space can be extremely large.
- Data formulation: A sequential recommendation dataset consists of user histories paired with their next chronological behaviors.The history x_u,t contains behaviors before the target y_u,t, and T_u is the user’s number of behaviors.
- Prediction task: The prediction task is to estimate which next item a user will interact with from the user’s historical behaviors.The formulation models probabilities over possible items.
- Serving architecture: Because the item set can reach 100 million items, industrial recommenders typically separate matching from ranking.Matching retrieves Top N candidates, while ranking applies finer-grained features and more complex architectures.
- Serving architecture: Matching focuses on efficient candidate retrieval, whereas ranking further orders the retrieved candidates.This two-phase design addresses system-latency concerns.
3.2 A Naive Matching Baseline
The naive matching baseline encodes historical behavior sequences and items into a shared vector space, then retrieves candidates by similarity. It uses simple embeddings and global average pooling as a deliberately straightforward comparison framework.
- Baseline architecture: A user encoder maps historical behavior sequences to dense interest vectors, while an item encoder maps items into the same vector space.Top-N candidates are retrieved using nearest-neighbor search over item vectors.
- Candidate retrieval: Matching scores are computed using inner products or cosine similarity between user and item representations.The highest-scoring items are returned as Top-N candidates.
- Baseline architecture: The baseline item encoder is a lookup embedding matrix, and the user encoder averages historical item embeddings before an MLP transformation.This produces user and item representations suitable for matching.
- Training objective: The baseline uses sampled softmax because summing over all possible items is infeasible during matching.This replaces the full-item denominator with a computationally tractable objective.
3.3 The CauseRec Architecture
CauseRec synthesizes counterfactual user sequences by identifying indispensable and dispensable concepts at item and interest levels, then contrasts their representations with the observational sequence. Its transformations preserve semantics when dispensable concepts are replaced and create preference deviations when indispensable concepts are replaced.
- Counterfactual transformation: Replacing indispensable concepts produces counterfactually negative sequences with preference deviations, while replacing dispensable concepts produces counterfactually positive sequences with preserved semantics.Negative samples are hard because other dispensable concepts remain unchanged from the original sequence.
- Concept identification: CauseRec identifies indispensable and dispensable concepts at both fine-grained item and abstract interest levels before counterfactual synthesis.Item-level concepts treat individual behavior items as concepts, while interest-level concepts are extracted with attention mechanisms.
- Concept identification: Item-level concepts use individual behavior items, whereas interest-level concepts address redundancy and higher-order relationships through attention-based representations.Interest-level concepts use trainable transformation matrices and a predefined number of concepts.
- Concept identification: Concept scores determine importance, with the top half classified as indispensable and the remaining half as dispensable at each concept level.The top-half rule prevents either category from becoming too small, while more effective identification remains future work.
- Counterfactual transformation: Replacement occurs at rate r_rep using first-in-first-out concept memories, preserving sequence length and the relative positions of remaining concepts.The same sequence terminology covers item-level behavior sequences and interest-level concept sequences.
- Architectures: CauseRec-Item reuses the original user encoder, whereas CauseRec-Interest and CauseRec-H use interest-level inputs and are intrusive designs.CauseRec-H additionally applies item-level counterfactual transformations.
- Contrastive learning: CauseRec contrasts observational and counterfactual representations using objectives designed to make representations robust to dispensable concepts and attentive to indispensable concepts.Additional objectives contrast interest and item representations, while L2 distance and margins are used in the user-representation objective.
- Inference: During serving, only the backbone user-representation model is required; concept identification, transformation, and target-dependent proposal scoring are omitted.The counterfactual machinery is used during training rather than testing.
4 EXPERIMENTS
Experiments on three public recommendation datasets evaluate CauseRec against sequential recommenders, its components, and key hyperparameters. CauseRec improves matching performance and representation quality, with hierarchical concepts and contrastive objectives contributing to the strongest results.
- Experimental Setup: CauseRec is evaluated on Amazon Books, Yelp2018, and Gowalla using an 8:1:1 user split and unseen behavior sequences.Training treats each behavior as a potential target, while validation and testing evaluate users’ representations on later interactions.
- Performance Analysis (RQ1): CauseRec architectures consistently improve over sequential recommender baselines across three reported metrics.The comparison includes CauseRec-Item, CauseRec-Interest, and CauseRec-Hierarchical against state-of-the-art matching-stage recommenders.
- Performance Analysis (RQ1): +.0299 NDCG@50 on Amazon Books and +.0179 Recall@20 on Gowalla are the reported improvements of CauseRec-H over prior SOTA methods.These correspond to relative gains of 22.1% and 8.64%, respectively.
- Performance Analysis (RQ1): CauseRec-Hierarchical achieves the best results by combining interest-level and item-level concepts, retaining fine-grained preference modeling.The paper illustrates that users may prefer one item within a broad interest, such as cake over chocolate among sweets.
- Ablation Studies: Removing contrastive losses reduces performance, indicating complementary roles for contrasting counterfactual users with observations and target items.Removing L_co weakens robustness-oriented alignment, while removing L_ii produces an even larger drop and can permit trivial counterfactual representations.
- Ablation Studies: All tested CauseRec architectures improve over the naive base model, while positive-only and negative-only variants remain weaker than the full contrastive design.The ablations support using both counterfactually positive and negative representations rather than only one type.
- Hyperparameter Analysis: Increasing negative counterfactual representations can help, but increasing positive and negative representations together does not necessarily improve performance.The paper interprets negative counterfactuals as hard negatives because dispensable items remain in the corresponding transformed sequences.
- Hyperparameter Analysis: The replacement ratio performs best at 0.4/0.5 on Yelp and 0.5 on Gowalla, whereas extreme ratios are sub-optimal.Small ratios may weaken counterfactual learning, while large ratios introduce noise from randomly sampled replacement items.
5 CONCLUSION AND FUTURE WORK
CauseRec models counterfactual data distributions and contrasts synthesized counterfactual sequences with observational sequences to learn accurate and robust user representations. Its architectures operate at item and interest levels, while future work targets improved concept identification, broader models, and ranking.
- CauseRec samples counterfactually positive and negative user sequences by transforming dispensable and indispensable concepts at item and interest levels.
- Contrastive objectives compare counterfactual and observational representations to learn accurate and robust user representations.
- CauseRec-Item functions only during training, remains compatible with various user encoders, and does not affect serving efficiency.
- Future work includes better identification of indispensable and dispensable concepts, applying CauseRec to more models and ranking, and using richer features or complex architectures.