Source-linked AI summary
Multi-Pointer Co-Attention Networks for Recommendation
Yi Tay, Luu Anh Tuan, Siu Cheung Hui
TL;DR
Review-based recommenders often compress all reviews into static representations even though only some reviews may matter for a particular user-item match. MPCN selects reviews dynamically with co-dependent pointers, matches selected pairs at the word level, and outperforms strong baselines across 24 Amazon and Yelp datasets, with reported gains up to 19% over TransNet and 71% over DeepCoNN.
Problem
Existing review-based models concatenate many reviews and use static compressed representations, limiting dynamic target-dependent selection and deeper user-item interaction.
Method
MPCN uses multi-hierarchical review-by-review co-attention with Gumbel-Softmax pointers and multiple pointer passes to select and match informative review pairs.
Results
MPCN is the top-performing model on all 24 benchmark datasets, with relative gains up to 71% over DeepCoNN and 19% over TransNet.
Takeaways & Limitations
Pointer behavior reveals different evidence-aggregation patterns across domains, while MPCN dynamically handles these patterns.
Takeaways & Limitations
Different domains exhibit different evidence-aggregation patterns, warranting further investigation.
Abstract
from arXiv · showhide
Many recent state-of-the-art recommender systems such as D-ATT, TransNet and DeepCoNN exploit reviews for representation learning. This paper proposes a new neural architecture for recommendation with reviews. Our model operates on a multi-hierarchical paradigm and is based on the intuition that not all reviews are created equal, i.e., only a select few are important. The importance, however, should be dynamically inferred depending on the current target. To this end, we propose a review-by-review pointer-based learning scheme that extracts important reviews, subsequently matching them in a word-by-word fashion. This enables not only the most informative reviews to be utilized for prediction but also a deeper word-level interaction. Our pointer-based method operates with a novel gumbel-softmax based pointer mechanism that enables the incorporation of discrete vectors within differentiable neural architectures. Our pointer mechanism is co-attentive in nature, learning pointers which are co-dependent on user-item relationships. Finally, we propose a multi-pointer learning scheme that learns to combine multiple views of interactions between user and item. Overall, we demonstrate the effectiveness of our proposed model via extensive experiments on \textbf{24} benchmark datasets from Amazon and Yelp. Empirical results show that our approach significantly outperforms existing state-of-the-art, with up to 19% and 71% relative improvement when compared to TransNet and DeepCoNN respectively. We study the behavior of our multi-pointer learning mechanism, shedding light on evidence aggregation patterns in review-based recommender systems.
1 INTRODUCTION
The paper argues that review-based recommenders should model reviews independently, select them dynamically for each user-item match, and support deeper word-level interactions. It proposes MPCN, a multi-pointer co-attention architecture designed around these requirements.
- Motivation: Existing review-based recommenders concatenate all user or item reviews into one document and compress them into a single latent representation.User and item embeddings are then matched with a parameterized interaction function.
- Motivation: Naive concatenation is noisy because it treats reviews indiscriminately, even when their relevance depends on the target item or user.For example, reviews about unrelated product categories should contribute less to a current match.
- Design Requirements: The paper identifies four needed changes: independent review modeling, unequal review weighting, target-dependent importance, and word-level user-item interaction.These changes address the limitations of document-level compression and static representations.
- Proposed Approach: MPCN matches user and item reviews review by review, applies word-level co-attention to selected pairs, and repeats this process with multiple pointers.The pointer mechanism selects informative reviews for the current user-item pair before deeper interaction modeling.
- Evaluation: Experiments on 24 benchmark datasets compare MPCN with review-based and interaction-only state-of-the-art baselines.The reported comparisons include NeuMF, DeepCoNN, D-ATT, and TransNet.
2 RELATED WORK
Prior recommendation models use factorization, neural encoders, attention, or document-level review representations, while related sequence models motivate finer-grained co-attentional interaction. The paper positions MPCN as an attention-only architecture with pointer-based selection.
- Review-Based Recommendation: Review-based recommendation models use review text to enrich user and item representations beyond interaction data.The literature also motivates review exploitation for mitigating cold-start issues and modeling user and item characteristics semantically.
- Review-Based Recommendation: DeepCoNN encodes all user and item reviews with CNNs before Factorization Machine matching, but test-review leakage was argued to affect its evaluation setting.The cited discussion frames this leakage as making recommendation resemble noisy document-level sentiment analysis.
- Attention Models: D-ATT augments CNN encoders with local and global neural attention that weights document words to produce representations.Its variants model different views of user-item review documents.
- Sequence Pair Modeling: Sequence-pair modeling research shows that co-attention can outperform representation learning without fine-grained interaction modeling.Co-attention learns pairwise attention between two sequences, closely matching the user-item review setting.
- Architectural Relation: MPCN relies solely on attention mechanisms and omits convolutional and recurrent layers.The architecture is inspired by Transformer-style multi-headed attention.
- Pointer Models: Unlike conventional pointer networks that predict sequence tokens at the final layer, MPCN requires pointers within a differentiable recommendation architecture.This motivates its use of a mechanism that supports discrete selection during computation.
- Interaction-Based Recommendation: Factorization-based models remain standard interaction-only baselines, while neural recommendation models increasingly claim state-of-the-art performance.Examples include generalized matrix factorization and neural factorization-machine adaptations.
3 OUR PROPOSED MODEL
MPCN represents users and items as sequences of reviews, filters review information, and uses co-attention to select and combine reviews before deeper interaction modeling. Figure 1 illustrates a one-pointer version of this architecture.
- Input Encoding: The model accepts user and item inputs as sequences of reviews, with each review represented as a sequence of words.The maximum number of reviews is denoted by ℓd, and words are mapped through an embedding matrix.
- Architecture Illustration: Figure 1 depicts the proposed pointer-based architecture in a one-pointer example, omitting review gating and multi-pointer learning for clarity.The figure should therefore be read as a simplified architecture illustration.
- Input Encoding: Unlike document concatenation, the hierarchical representation preserves reviews as separate units built from word sequences.This structure supports review-level selection before later interaction modeling.
- Review Gating Mechanism: The review gating mechanism transforms each review embedding and controls how much information passes to the next level.The paper hypothesizes that this pre-filter improves performance on certain datasets.
- Review-Level Co-Attention: The review-level co-attention layer aims to select the most informative review from each user and item review bank.It computes affinities between user and item review embeddings before applying pooling-based selection.
- Review-Level Co-Attention: Max pooling selects reviews with the greatest affinity to reviews from the partner sequence, yielding an extractive co-attention mechanism.The resulting one-hot pointers identify reviews for subsequent operations rather than serving as final co-attentional representations.
3.3 Review Pointers
MPCN uses Gumbel-based sampling and a straight-through differentiable approximation to select discrete review pointers while retaining end-to-end trainability. Multiple pointer passes provide several review pairs for word-level interaction.
- Gumbel-Softmax: Gumbel-Softmax replaces the non-differentiable arg max with a differentiable softmax so discrete-like variables can be incorporated into neural networks.Its temperature controls how closely the output approaches a one-hot vector.
- Gumbel-Max: Gumbel-Max samples a one-hot vector from a categorical distribution by adding Gumbel noise to unnormalized log probabilities before taking an arg max.The noise perturbs each log-probability term so arg max sampling is weighted by the class probabilities.
- Straight-Through Estimation: The straight-through adaptation discretizes the forward pass while preserving continuous gradients during backpropagation.This enables hard selection of important reviews while retaining end-to-end training.
- Review Pointers: MPCN applies Gumbel-Softmax to compute user and item review pointers for use in subsequent layers.The selected reviews are then passed onward for richer interactions between the paired reviews.
3.4 Word-level Co-Attention
Word-level co-attention compares the selected user and item reviews at word granularity rather than only at the review level. Mean pooling produces stable output representations while retaining fine-grained interactions.
- Word-level interaction: The selected user and item reviews are matched with an affinity matrix computed word-by-word.This enables finer-grained comparison than composing only review embeddings.
- Representation construction: Mean pooling converts the word-level co-attentional information into output representations.The resulting representations are denoted ā′ and b̄′.
- Pooling choice: Average pooling is preferred at word level because max pooling may overemphasize identical-word matches.The authors describe mean pooling as a more stable co-attentional extractor.
- Implementation: The co-attention layers can be implemented using two matrix multiplications, which are efficiently optimized on GPUs.The paper therefore reports that scalability is not a major concern for this implementation.
3.5 Multi-Pointer Learning
Multi-pointer learning addresses the risk that a single selected review pair may provide insufficient information. It repeatedly extracts review pairs, applies word-level co-attention, and combines the resulting representations through alternative composition schemes.
- Pointer extraction: MPCN runs review-level co-attention n_p times to generate distinct user-item review pairs.Each pair is subsequently modeled with word-level co-attention.
- Pointer extraction: Multiple pointers provide additional information when a single hard-selected review pair is insufficient.The number of pointers n_p is treated as a user-defined hyperparameter.
- Output composition: The multi-pointer outputs are formed separately for user and item representations before final composition.The same composition procedure is applied to b̄′, although the text omits the repeated notation.
- Output composition: The model can concatenate all pointer outputs, sum them additively, or pass their concatenation through a ReLU neural network.These three composition schemes are tuned experimentally.
3.6 Prediction Layer
The prediction layer combines the final user and item representations and models their feature interactions with a factorization machine. Its scalar output represents the predicted strength of the user-item interaction.
- Prediction function: The final user and item representations are concatenated and passed to a factorization machine.The factorization machine accepts the resulting real-valued feature vector.
- Prediction function: The factorization machine models pairwise feature interactions using factorized parameters.It also includes a linear regression component and a global bias.
- Training objective: The model outputs a scalar representing the strength of the user-item interaction.Training minimizes standard mean squared error end-to-end.
4 EMPIRICAL EVALUATION
The evaluation tests MPCN against competitive interaction-only and review-based recommenders on 24 Yelp and Amazon datasets. MPCN ranks first across all datasets, with reported gains over major review-based baselines and competitive runtime for small pointer counts.
- Research questions: The experiments address comparative performance, architectural choices, pointer-count effects, and interpretability of the pointer layer.These correspond to four stated research questions.
- Datasets and setup: The evaluation covers 24 benchmark datasets: one Yelp dataset and 23 Amazon product-review datasets.The datasets use time-based train, development, and test splits.
- Compared methods: The comparison includes interaction-only baselines and review-based models such as DeepCoNN, TransNet, and D-ATT.The baselines include MF, FM, MLP, NeuMF, DeepCoNN, and TransNet.
- Results: MPCN is the top-performing model on all 24 benchmark datasets.It consistently and significantly outperforms DeepCoNN, TransNet, and D-ATT.
- Results: Up to 71% improvement is reported over DeepCoNN, 19% over TransNet, and 5% over D-ATT.Average improvements are 16% over DeepCoNN, 3.2% over TransNet, and 2.2% over D-ATT.
- Results: Interaction-only models perform consistently worse than review-based models across the datasets.Among interaction-only models, the strongest model varies between FM and MLP, while MF usually performs worst.
- Runtime: MPCN with one pointer runs at approximately 0.4t, while two and three pointers run at 0.8t and 1.2t relative to DeepCoNN runtime t.TransNet and D-ATT are reported at approximately 2t.
5 HYPERPARAMETER & ABLATION ANALYSIS
The analysis evaluates architectural choices and pointer counts across benchmark datasets. Component effects vary by dataset, while the optimal number of pointers is domain-dependent.
- Ablation Analysis: Removing word-level and review-level co-attention consistently lowered performance across the four ablation datasets.The analysis identifies both co-attention layers as effective components.
- Ablation Analysis: Review gating helped on 3 of 4 datasets, but removing it improved performance on Office and sometimes elsewhere.The gating mechanism generally helps, but its benefit is not universal.
- Ablation Analysis: Removing FM caused huge degradation on Musical Instruments and significant degradation on Instant Video and Office, while slightly improving Beauty.The effect of the prediction layer depends on the dataset.
- Ablation Analysis: The relative ranking of concatenate, additive, and neural-network pointer aggregation functions changes across datasets, so aggregation must be tuned.No aggregation choice is uniformly best.
- Effect of Number of Pointers: The optimal number of pointers varies across datasets and appears domain-dependent rather than correlated with dataset size.Table 3 compares Patio, Automotive, Sports, and Video Games; Video Games is an exception with an optimum of 5 pointers.
6 IN-DEPTH MODEL ANALYSIS
The model analysis examines what pointers select and how multiple pointers aggregate evidence. Pointers track product aspects and domains, producing distinct but sometimes overlapping review-match patterns.
- What are the pointers pointing to?: Pointers often match user and item reviews that share product aspects or sub-categories, such as RPG games, puzzle games, cocoa, and chocolate.The examples indicate that pointer selection surfaces semantically related evidence from both sides of a user-item pair.
- Behavior of Multi-Pointer Learning: With four pointers, all pointers selected different reviews for the same user-item pair without an explicit diversity constraint.The model generated distinct pointers automatically, and Figure 3 visualizes their review-level affinity matrices.
- Behavior of Multi-Pointer Learning: One-to-many pointer patterns were relatively significant alongside the usual one-to-one pattern, with behavior depending on dataset or domain.In a minority of cases, all pointers repeatedly selected the same reviews when the review banks contained insufficient information.
- Behavior of Multi-Pointer Learning: Electronic domains often infer from fewer review subsets, whereas food domains require evidence across multiple reviews.The authors interpret the high one-to-many ratio in electronic domains as one important review being matched with multiple opposing reviews.
7 CONCLUSION
The paper presents MPCN as a review-based recommender evaluated on 24 Amazon and Yelp benchmark datasets. Its analysis finds domain-specific evidence aggregation patterns that the model handles dynamically but that warrant further investigation.
- Conclusion: MPCN outperforms many strong competitors across 24 benchmark datasets from Amazon and Yelp.The conclusion characterizes the proposed model as a state-of-the-art neural model for recommendation with reviews.
- Conclusion: Different domains, including food-related and electronics-related domains, exhibit different evidence aggregation patterns.The authors state that MPCN dynamically handles this domain variation while identifying it as an area for further investigation.