Source-linked AI summary
TransNets: Learning to Transform for Recommendation
Rose Catherine, William Cohen
TL;DR
Review-based recommendation must predict ratings without the target user’s review for the target item, although that review contains substantial predictive information. TransNets extends DeepCoNN with a transform layer that approximates the target review from user and item representations, and TransNets and its extensions substantially improve rating prediction.
Problem
DeepCoNN’s performance depends on access to the target user-item review at test time, although that review is unavailable before the user experiences the item.
Method
TransNets transforms latent user and item representations into an approximation of their pairwise review and can augment it with rating-derived user and item representations.
Results
TransNets and TransNet-Ext perform better than competitive baselines on all datasets, with statistically significant differences reported at p-value ≤0.05.
Takeaways & Limitations
The learned review approximation can support rating prediction without the target review and can help identify reviews similar to what a user might write.
Takeaways & Limitations
DeepCoNN’s strongest performance requires the target user-item review at test time, an assumption considered unreasonable for real-world recommendation.
Abstract
from arXiv · showhide
Recently, deep learning methods have been shown to improve the performance of recommender systems over traditional methods, especially when review text is available. For example, a recent model, DeepCoNN, uses neural nets to learn one latent representation for the text of all reviews written by a target user, and a second latent representation for the text of all reviews for a target item, and then combines these latent representations to obtain state-of-the-art performance on recommendation tasks. We show that (unsurprisingly) much of the predictive value of review text comes from reviews of the target user for the target item. We then introduce a way in which this information can be used in recommendation, even when the target user's review for the target item is not available. Our model, called TransNets, extends the DeepCoNN model by introducing an additional latent layer representing the target user-target item pair. We then regularize this layer, at training time, to be similar to another latent representation of the target user's review of the target item. We show that TransNets and extensions of it improve substantially over the previous state-of-the-art.
1 INTRODUCTION
Review text can improve recommender-system rating prediction, but it describes joint user-item interactions rather than users or items alone. TransNets uses training-time target reviews to approximate unavailable reviews at prediction time.
- Review text has improved recommender-system performance over collaborative filtering methods that use only past ratings.
- DeepCoNN learns separate latent representations from all reviews written by a user and all reviews written about an item, then combines them for rating prediction.
- Much of review text’s predictive value comes from the target user’s review for the target item, which is unavailable at test time.
- TransNets adds a latent user-item layer and regularizes it during training to resemble the latent representation of the actual target review.
- TransNets and its extensions substantially improve rating prediction over previous state-of-the-art methods.
2 PROPOSED METHOD
The method processes review text with CNN-based text processors that convert word sequences into fixed-dimensional representations. Convolution, nonlinearities, max pooling, concatenation, and a fully connected layer form the final representation.
- The CNN Text Processor maps a sequence of words to an n-dimensional vector representation.It is represented as Γ : [w1,w2, ...,wT] → Rn.
- A word-embedding function maps vocabulary words into d-dimensional vectors, which remain fixed during training when pretrained embeddings are used.
- Convolutional filters process t-length windows of embedded word vectors to produce features for each of m neurons.Typical window sizes are 2–5, and the convolution uses a bias and a nonlinearity such as ReLU or tanh.
- Max pooling selects the final feature for each neuron and makes feature detection invariant to location within the text.
- The neuron outputs are concatenated and passed through a fully connected layer with W ∈ Rm×n and bias д ∈ Rn to produce the text representation.
2.2 Te DeepCoNN model
DeepCoNN processes all reviews written by a user and all reviews written about an item with separate CNN Text Processors. It combines their representations in a Factorization Machine for rating prediction.
- DeepCoNN uses two side-by-side CNN Text Processors to compute a user representation xA and an item representation yB.The inputs concatenate the target user’s reviews and the reviews written about the target item.
- Dropout is applied to both representations before they are concatenated into the regression input z = [x̄A, ȳB].
- The regression layer is a Factorization Machine that computes second-order interactions between input dimensions.
- Factorization Machines represent pairwise interactions with inner products of k-dimensional vectors, requiring O(nk) rather than O(n^2) parameters.
- All Factorization Machines in the paper are trained with L1 loss because it performed slightly better than L2 loss in the experiments.
2.3 Limitations of DeepCoNN
DeepCoNN performs best when the target user’s review for the target item is available at test time, an assumption that is unrealistic before the user experiences the item. Omitting that review causes severe degradation.
- DeepCoNN had previously surpassed review-based and rating-only baselines, including HFT, CTR, CDL, MF, and PMF.
- DeepCoNN’s best performance depends on access to the target user-item review at test time, which is unreasonable in real-world recommendation.
- 1.21 MSE with the target review included degraded to 1.89 when it was omitted, worse than Matrix Factorization’s 1.86 MSE on the same dataset.
- Training DeepCoNN while omitting target reviews improved performance to 1.70, but remained worse than training and testing with those reviews available.
- Validation and test reviews are never accessed, during either training or testing, to simulate their unavailability in real-world recommendation.
2.4 TransNets
TransNet uses a Target Network to encode the available target review and a Source Network to infer its latent representation from separate user and item review texts. The inferred representation is then used for rating prediction when the target review is unavailable.
- TransNet separates processing into a Target Network for the actual review and a Source Network for user and item texts excluding that review.The Source Network includes an additional Transform layer.
- The Transform layer maps concatenated user and item latent representations into an approximation of the target review’s latent representation.It uses an L-layer nonlinear network whose first layer accommodates the concatenated 2n-dimensional input.
- TransNet approximates the target review encoding rather than generating its surface text, then uses that latent representation to predict the rating.The Source Network uses the transformed representation for rating prediction.
- During training, the Source Network’s transformed representation is forced to resemble the Target Network’s encoding of the actual review.
2.5 Training TransNets
TransNet training alternates among rating prediction from the actual review, latent-representation matching, and rating prediction from the transformed representation. At test time, only the Source Network is used.
- Training updates the Target Network by minimizing L1 rating loss from the actual target review.The Target Network parameters include its text processor and factorization machine.
- The transformation parameters are updated with L2 loss between the transformed user-item representation and the Target Network’s actual-review representation.These parameters include the user and item text processors and transformation weights and biases.
- The Source Network predictor is trained with L1 loss between the actual rating and the rating predicted from the transformed representation.
- The three sub-steps repeat for each example or batch rather than being trained independently to convergence.
- At test time, TransNet makes predictions using only the Source Network.
2.6 Design Decisions and Other Architectural Choices
The design choices favor simultaneous alternating training and separate feature generators, while extending TransNet with user and item identity embeddings when those identities are available. The cited architectural comparison reports better performance for separate generators.
- Training: Jointly minimizing all losses can produce sub-optimal representations because transformation loss also updates the Target Network.The authors report lower performance when both networks are penalized through a single total loss.
- Training: The Source and Target Networks are trained simultaneously because Target Network convergence cannot be assessed reliably from its unavailable test-time input.Training stops when Source Network test performance is good.
- Feature generators: Separate feature generators allow TransNets to learn transformations beyond generic topic-like features.Reusing the Target Network’s generator would constrain the generated features; separate generators gave better performance in the cited comparison.
Training the Transform without the dropout.
The Source Network’s transformed representation is matched to a dropped-out target representation during training, while TransNet-Ext augments the source representation with user and item identity embeddings.
- Training the Transform without the dropout.: Matching the transformed output to the non-dropped-out target representation leaves the Transform layer unregularized and produces poor performance.
- Extended TransNets: TransNet-Ext addresses known user and item identities by adding separate embedding matrices to the Source Network.
- Extended TransNets: The identity embeddings are dropout-processed, concatenated with the Transform output, and passed to the factorization-machine regression layer.
- Extended TransNets: The Source Network’s parameters now include the user and item embedding matrices when computing its rating loss.
3 EXPERIMENTS AND RESULTS
Experiments evaluate TransNets across four large review-rating datasets against neural and rating-only baselines. TransNet variants outperform competitive baselines across datasets, while qualitative examples suggest the learned representation can retrieve reviews matching users’ original concerns.
- Datasets and evaluation: Experiments use four large review-rating datasets, including Yelp17 and three Amazon review datasets, with random 80:10:10 train-validation-test splits.Models are evaluated by test MSE selected at the validation-MSE minimum.
- Datasets and evaluation: Mean Square Error (MSE) evaluates rating prediction, with lower values indicating better performance.The reported test score is taken when validation MSE is lowest.
- Rating prediction: TransNet and TransNet-Ext outperform competitive baselines on all datasets with p-value ≤0.05.The comparison includes DeepCoNN and other baseline variants.
- Rating prediction: TransNet-Ext improves over vanilla TransNet with p-value ≤0.1, except on AZ-CSJ.The extension learns user and item embeddings from ratings in addition to text.
- Rating prediction: Training DeepCoNN without the target pairwise review substantially helps on Yelp but not on the AZ datasets.This baseline mimics the test setup by excluding the joint review during training.
- Qualitative analysis: TransNets can retrieve candidate reviews whose latent representations are closest to the predicted representation of a user’s missing joint review.Examples show predicted reviews discussing particulars also highlighted by the original reviews.
4 RELATED WORK
Related work spans topic-based, content-based, rating-only, and neural recommender models, including DeepCoNN. TransNets also differ structurally from Student-Teacher models and GANs because their networks use different inputs and compare generated representations with specific target representations.
- Topic and content models: HFT transforms user latent factors into review-topic distributions, while TopicMF aligns rating-based latent factors with topics from joint reviews.Both approaches connect latent factors and review-derived topics through transformation or matching objectives.
- Topic and content models: CTR, CDL, DCF, and ConvMF use content or learned item representations for recommendation rather than modeling reviews as joint user-item context.These methods combine topic models, matrix factorization, autoencoders, or CNNs with recommendation.
- Neural review models: Earlier neural review models include BoWLF and LMLF, while DeepCoNN learns user and item representations from review text for rating prediction.DeepCoNN is identified as the most recent neural model to successfully operate at scale in this comparison.
- Rating-only models: Rating-only research includes CDAE and other collaborative-filtering approaches that learn latent factors without content or review text.CDAE reconstructs user feedback from a corrupted version of that feedback.
- Model comparisons: Unlike Student-Teacher models, TransNets feed different inputs to Source and Target networks: separate user-item review contexts versus the actual pairwise review.TransNets also differ from GANs by starting from user and item reviews and lacking an adversarial classifier.
5 CONCLUSIONS
The paper addresses DeepCoNN’s dependence on unavailable user-item reviews at prediction time by introducing TransNets. Its Transform layer generates an approximate pairwise-review representation, and experiments report substantial improvements over the state of the art.
- Problem: DeepCoNN’s performance depends on access to the user-item pairwise review, which is unavailable in real-world settings.This creates the central deployment gap addressed by the paper.
- Contribution: TransNets add a Transform layer that maps user and item latent representations to an approximate representation of their pairwise review.The generated representation can be used for prediction when the target review is missing.
- Contribution: An extended TransNets version additionally learns user and item representations from ratings and combines them with the generated review representation.The conclusion presents this as an extension of the core model.
- Conclusion: TransNets and their extended version improve substantially over the previous state of the art in the reported experiments.The conclusion states the overall experimental outcome without restricting it to a single dataset or metric.