Source-linked AI summary
Personalized Transformer for Explainable Recommendation
Lei Li, Yongfeng Zhang, Li Chen
TL;DR
Transformer is not personalized because user and item IDs do not share the semantic space of words, limiting their use in natural-language generation. PETER connects IDs to explanation words through context prediction and jointly supports recommendation and explanation generation. The small unpretrained model outperforms fine-tuned BERT on most generation metrics while training faster, and experiments report superiority over state-of-the-art baselines.
Problem
Transformer’s strong language modeling does not directly use user and item IDs for personalized generation because IDs and words occupy different semantic spaces.
Method
PETER uses context prediction to map user and item IDs onto explanation words, while jointly generating explanations and making recommendations.
Results
PETER, a small two-layer unpretrained Transformer, outperforms fine-tuned BERT on most generation metrics and trains faster.
Takeaways & Limitations
PETER provides a unified recommendation-explanation model and suggests a way to exploit Transformer through task design rather than larger model size.
Abstract
from arXiv · showhide
Personalization of natural language generation plays a vital role in a large spectrum of tasks, such as explainable recommendation, review summarization and dialog systems. In these tasks, user and item IDs are important identifiers for personalization. Transformer, which is demonstrated with strong language modeling capability, however, is not personalized and fails to make use of the user and item IDs since the ID tokens are not even in the same semantic space as the words. To address this problem, we present a PErsonalized Transformer for Explainable Recommendation (PETER), on which we design a simple and effective learning objective that utilizes the IDs to predict the words in the target explanation, so as to endow the IDs with linguistic meanings and to achieve personalized Transformer. Besides generating explanations, PETER can also make recommendations, which makes it a unified model for the whole recommendation-explanation pipeline. Extensive experiments show that our small unpretrained model outperforms fine-tuned BERT on the generation task, in terms of both effectiveness and efficiency, which highlights the importance and the nice utility of our design.
1 Introduction
PETER addresses Transformer’s failure to personalize explanations from user and item IDs by connecting IDs to generated words through context prediction. It jointly supports recommendation and explanation generation, incorporates optional item features, and reports stronger, more efficient generation than fine-tuned BERT.
- Motivation: Personalized natural-language tasks use user and item IDs to reflect differing user interests and item characteristics.Explainable recommendation generates an explanation for a user-item pair to justify why the item matches the user’s interests.
- Motivation: Transformer struggles with personalization because IDs and words occupy different semantic spaces, making IDs behave like infrequent out-of-vocabulary tokens.Directly combining them for attention can leave the model insensitive to user and item information.
- PETER: Context prediction maps IDs onto explanation words, connecting user and item representations with the language-generation task.The task is described as drafting words that explanation generation then polishes into a readable sentence.
- PETER: PETER jointly makes recommendations and generates explanations, using information from both the user and item for the same pair.This unifies the recommendation-explanation pipeline within one Transformer model.
- PETER: PETER can incorporate item features to guide targeted explanations, such as discussing an item’s price when requested.The model is a small, two-layer unpretrained Transformer that outperforms fine-tuned BERT on most generation metrics while training faster.
- Evaluation: Experiments evaluate both text quality and explainability, and report that PETER outperforms state-of-the-art baselines on large datasets.Explainability metrics focus on item-feature alignment in generated explanations.
- Implications: The approach also suggests applications to personalized conversational systems and Transformer models handling heterogeneous inputs such as text and images.These broader applications are presented as implications of the solution’s design.
2 Related Work
Prior personalized generation methods commonly encode user and item IDs with MLPs before RNN decoding, while Transformer-based personalization remains difficult because it relies on self-attention. PETER addresses this setting with a three-task architecture that connects IDs and words.
- Explainable Recommendation: Explainable recommendation research spans human-computer interaction and machine-learning approaches, with this work focused on algorithmic explanation generation.Generated natural-language explanations have gained attention alongside advances in language generation and recommendation-platform text data.
- Transformer: Transformer has demonstrated broad language-processing effectiveness, including settings where its encoder or decoder is removed to reduce parameters.The related work frames Transformer as a strong language-modeling architecture rather than a personalized-generation solution.
- Personalized Generation: Previous personalized-generation systems typically encode user and item IDs into a context vector with an MLP, then decode words using an RNN.This pattern appears in review, tip, and explanation generation.
- PETER: PETER contains three tasks and uses context prediction to connect IDs with words.The model is presented as a way to make Transformer handle personalized generation through task design.
3 Problem Formulation
PETER generates a natural-language explanation and estimates a rating for a user-item pair, using user and item IDs at test time. Optional item features can be included to guide the explanation, but then must also be available during testing.
- Tasks: For user u and item i, PETER generates an explanation ˆE_u,i and estimates a rating ˆr_u,i for the user’s preference toward the item.The model therefore supports explanation generation and recommendation for the same user-item pair.
- Inputs: At testing time, user u and item i are sufficient inputs for producing both the explanation and recommendation.Item features are not required in the basic setting.
- Inputs: When available, item features F_u,i are concatenated at the beginning of the explanation to guide generation.If features are used, they are also required during testing.
4 Methodology
PETER adapts Transformer for explainable recommendation by encoding user, item, feature, and explanation tokens, revising attention masking, and jointly learning explanation generation, context prediction, and recommendation. Its context-prediction task connects IDs to explanation words, while shared representations support rating prediction and end-to-end multi-task training.
- Input Representation: PETER represents user IDs, item IDs, features, explanation words, and positions with separate token or positional embeddings.The model does not add users and items to the word vocabulary, avoiding prediction over the large ID space.
- Transformer and Attention Masking: PETER masking lets the user and item positions attend to each other, accommodating context prediction and recommendation alongside left-to-right generation.This differs from standard left-to-right masking, which permits attention only to past tokens and the current token.
- Explanation Generation: Explanation generation applies a vocabulary projection and negative log-likelihood loss to predict explanation words sequentially.At test time, greedy decoding repeatedly selects the highest-probability word and appends it to the input sequence.
- Context Prediction: Context prediction maps user and item representations to words in the explanation, connecting IDs with generated language.Because the first two positions attend to each other, either final representation can absorb both user and item information; PETER uses the second.
- Rating Prediction: Recommendation predicts a user-item rating from the first final representation using a one-hidden-layer MLP and mean squared error.The user and item can attend to each other, so their final representations capture their interaction; recommendation improvement is left for future work.
- Multi-task Learning: The three tasks are integrated into a multi-task objective with regularization weights that balance their learning and enable end-to-end training.The objective jointly trains explanation generation, context prediction, and recommendation.
5 Experimental Setup
The experiments use three explainable-recommendation datasets, evaluate recommendation and explanation quality with multiple metrics, and compare feature-aware and ID-only models against baselines. Models are trained, tuned, and tested under specified data-splitting and implementation settings.
- Datasets: Three datasets—TripAdvisor, Amazon, and Yelp—are randomly split into training, validation, and testing sets at an 8:1:1 ratio for five repetitions.Training retains at least one record for each user and item.
- Datasets: Each dataset record contains a user ID, item ID, rating, explanation, and feature, with explanations extracted from reviews and containing at least one item feature.The datasets cover hotels, movies and TV, and restaurants.
- Evaluation Metrics: Explanation quality is measured by BLEU and ROUGE for text quality, plus FMR, FCR, and DIV for feature-based explainability.FMR checks ground-truth feature matching; FCR measures distinct-feature coverage; DIV measures feature intersection between generated explanations.
- Compared Methods: Recommendation performance is evaluated with RMSE and MAE, while the compared explanation models are divided by whether item features are used.PETER denotes the ID-only model; PETER+ denotes the feature-aware model.
- Compared Methods: The comparison includes Transformer, NRT, and Att2Seq for ID-only explanation generation, and ACMLM and NETE for feature-aware generation.PMF and SVD++ are also included as traditional recommendation baselines.
- Implementation Details: Models are trained on the training set, tuned on validation data, and evaluated on testing data, with results averaged across five splits.PETER uses two layers and two attention heads; training uses batch size 128 and learning rate 1.0.
6 Results and Analysis
PETER produces personalized explanations and recommendations by using context prediction to connect user-item IDs with generated words. Across experiments, it generally improves explanation quality, explainability, diversity, and training efficiency, while smaller datasets can cause underfitting.
- Explanation generation: PETER+ consistently and significantly outperforms ACMLM and NETE on text-quality metrics across three datasets.The comparison uses BLEU and ROUGE for explanation quality.
- Explanation generation: PETER+ also beats ACMLM and NETE on FMR, while remaining competitive on FCR and DIV for explainability.FMR measures whether generated explanations mention the ground-truth feature.
- Explanation generation: PETER produces diverse, high-quality explanations, including the best BLEU-4, whereas Transformer generates nearly identical explanations with near-zero USR.On Yelp, PETER achieves the best performance on most metrics.
- Explanation generation: On Amazon and TripAdvisor, limited training samples may cause PETER to underfit and prevent it from consistently achieving the best performance.NRT and Att2Seq are competitive on these smaller datasets.
- Efficiency: PETER trains in 2.3 minutes per epoch, less than ACMLM, but requires more epochs because it is unpretrained and learned from scratch.The efficiency comparison uses the same machine and TripAdvisor dataset.
- Qualitative analysis: PETER generates distinct context words and explanations for different user-item pairs, while PETER+ can use item features to target specific explanation content.The qualitative examples show PETER+ selecting rooms rather than pool, making its explanation closer to the ground truth.
- Recommendation: On Yelp, PETER performs as well as competitive recommendation baselines, but it may underfit and lose performance on smaller TripAdvisor and Amazon datasets.The comparison reports RMSE and MAE for recommendation performance.
- Ablation study: Disabling context prediction sharply reduces explainability, text quality, and sentence diversity, while removing recommendation learning slightly improves recommendation performance.Replacing PETER masking with left-to-right masking sharply reduces recommendation performance by preventing access to item information.
7 Conclusion
The paper presents PETER as a simple and effective way to personalize Transformer generation for recommender-system explanations. Its experiments support effectiveness and efficiency, while motivating broader applications of personalized generation.
- Conclusion: PETER addresses Transformer’s personalized generation problem by generating recommender explanations from user and item IDs.The paper frames this as exploiting Transformer’s language-modeling power for explainable recommendation.
- Conclusion: Extensive experiments show that the proposed solution is both effective and efficient.The conclusion contrasts designing good tasks with scaling up model size.
- Future work: The authors identify personalized question answering and conversational agents as next applications for their solution.They also plan to incorporate item images in future work.