PaperlayerView exact revision on arXiv

Source-linked AI summary

Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)

Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, Yongfeng Zhang

arXiv:2203.13366v7cs.IRcs.AIcs.CLcs.LG

TL;DR

Recommendation research often uses task-specific architectures and objectives, making transfer across tasks difficult. P5 instead converts recommendation data into prompted natural-language input–target pairs and jointly trains them in a shared text-to-text framework. Across five task families, it matches or surpasses representative approaches and supports zero-shot transfer to new items, domains, and prompts.

  • Problem

    Recommendation tasks commonly use separate architectures and objectives, limiting transfer of learned knowledge and representations across tasks.

  • Method

    P5 converts user–item interactions, descriptions, metadata, and reviews into personalized prompt-based text pairs and jointly trains them with a shared language-modeling framework.

  • Results

    P5 matches or surpasses representative approaches across five task families and demonstrates zero-shot transfer to new items, domains, and personalized prompts.

  • Takeaways & Limitations

    A shared prompt-based language framework can support multiple recommendation tasks and generalization beyond the prompts and items used in training.

Abstract

from arXiv · show

For a long time, different recommendation tasks typically require designing task-specific architectures and training objectives. As a result, it is hard to transfer the learned knowledge and representations from one task to another, thus restricting the generalization ability of existing recommendation approaches, e.g., a sequential recommendation model can hardly be applied or transferred to a review generation method. To deal with such issues, considering that language can describe almost anything and language grounding is a powerful medium to represent various problems or tasks, we present a flexible and unified text-to-text paradigm called "Pretrain, Personalized Prompt, and Predict Paradigm" (P5) for recommendation, which unifies various recommendation tasks in a shared framework. In P5, all data such as user-item interactions, user descriptions, item metadata, and user reviews are converted to a common format -- natural language sequences. The rich information from natural language assists P5 to capture deeper semantics for personalization and recommendation. Specifically, P5 learns different tasks with the same language modeling objective during pretraining. Thus, it serves as the foundation model for various downstream recommendation tasks, allows easy integration with other modalities, and enables instruction-based recommendation based on prompts. P5 advances recommender systems from shallow model to deep model to big model, and will revolutionize the technical form of recommender systems towards universal recommendation engine. With adaptive personalized prompt for different users, P5 is able to make predictions in a zero-shot or few-shot manner and largely reduces the necessity for extensive fine-tuning. On several recommendation benchmarks, we conduct experiments to show the effectiveness of P5. We release the source code at https://github.com/jeykigung/P5.

1 INTRODUCTION

P5 addresses the fragmentation of recommendation tasks by expressing them as personalized natural-language prompts within one sequence-to-sequence framework. It reports promising performance across five task families and zero-shot generalization to novel prompts and items.

  • Motivation: Existing recommendation approaches often address emerging tasks separately, despite shared user–item pools and overlapping contextual features.The paper identifies a gap between task-specific solutions and a comprehensive recommender system supporting diverse features and task types.
  • P5 Paradigm: P5 unifies recommendation tasks by formulating them as prompt-based natural-language problems with a shared sequence-to-sequence framework.User–item information and associated features are integrated into personalized prompt templates.
  • P5 Paradigm: P5 reformulates recommendation as conditional text generation, using one encoder–decoder architecture and language-modeling objective instead of task-specific designs.The approach avoids feature-specific encoders and uses personalized prompts to represent varied recommendation inputs.
  • P5 Paradigm: P5 covers five recommendation task families through a multitask collection of personalized prompts.The introduction presents this prompt collection as a core component of the unified framework.
  • Results: P5 achieves promising performance on five task families and sufficient zero-shot generalization to novel personalized prompts and new items in unseen domains.These outcomes are reported for both seen prompt templates and zero-shot settings.

2 RELATED WORK

The related work situates P5 among unified text-to-text modeling, prompt learning, NLP-based recommendation, and zero-shot or cold-start recommendation. These lines of research motivate shared representations and generalization beyond observed interactions.

  • Unified Frameworks: Unified language models such as T5 and GPT-3 demonstrate knowledge sharing across tasks through common text-to-text or autoregressive language-modeling frameworks.The paper positions shared sequence-to-sequence modeling as a precedent for unifying heterogeneous tasks.
  • P5 Data and Prompts: Figure 2 organizes P5’s raw data into prompt-derived input–target pairs across five task families and three data sources.Rating, review, and explanation prompts share raw data, while sequential recommendation additionally requires interaction history.
  • Prompt Learning: Instruction-based prompts use detailed task descriptions in natural-language formats and support multitask adaptation in pretrained language models.The related work contrasts instruction-based prompts with discrete prompt search and continuous prompt embeddings.
  • NLP for Recommendation: NLP techniques have been applied to recommendation through explainable recommendation, sequential recommendation as language modeling, and text feature extraction.These applications connect recommendation tasks to text generation and language-based representations.
  • Zero-shot and Cold Start Recommendation: Zero-shot and cold-start recommendation evaluates settings where users or items have limited or no previous interaction records.The paper describes content modeling and auxiliary-domain transfer as existing solution strategies.

3 PERSONALIZED PROMPT COLLECTION

P5 constructs personalized prompt templates for five recommendation task families and converts raw records into shared input–target text pairs. The templates vary the requested output while incorporating user, item, and interaction information.

  • Prompt Collection: The prompt collection covers rating, sequential recommendation, explanation, review, and direct recommendation task families.Each family contains multiple personalized prompts to expose different aspects of users and items.
  • Rating: Rating prompts ask P5 to predict scores, answer whether a user will assign a specified score, or classify like versus dislike preference.A star rating of at least 4 is treated as like preference, while lower scores indicate dislike.
  • Sequential Recommendation: Sequential recommendation prompts predict the next item, select it from candidates, or determine whether a given item will be interacted with next.All three prompt types use the user’s interaction history.
  • Text Generation: Explanation prompts generate textual justifications for a user’s preference toward an item, while other families similarly verbalize recommendation-related outputs.The prompt families convert heterogeneous recommendation tasks into natural-language targets.
  • Data Construction: Raw-data fields are substituted into prompt templates to create training input–target pairs or zero-shot testing prompts.Rating, review, and explanation share raw data; sequential and direct recommendation use related data, with sequential recommendation requiring interaction history.

4 THE P5 PARADIGM AND MODEL

P5 unifies recommendation tasks by converting them into natural-language sequence generation with personalized prompts and a shared language-modeling objective. Its encoder–decoder model uses task-specific decoding strategies for generated answers or item lists.

  • Unified paradigm: P5 creates unified pretraining data by expressing diverse recommendation tasks as input–target token sequences with personalized prompts.This shared format is intended to break boundaries among tasks and support understanding of unseen prompts.
  • Model architecture: The model uses a Transformer encoder–decoder that contextualizes input embeddings and predicts future tokens conditioned on prior outputs and encoded input.User and item representations may use multiple sub-word units to avoid excessive tokens for large user and item pools.
  • Training objective: P5 trains all recommendation tasks with one model, one loss, and one data format by sharing the negative log-likelihood objective.The objective minimizes the negative log-likelihood of label tokens conditioned on input text in an end-to-end manner.
  • Task decoding: After pretraining, rating, explanation, and review tasks use greedy decoding, whereas sequential and direct recommendation generate item lists with beam search.Sequential recommendation uses all-item evaluation, while direct recommendation decodes from a candidate set containing one positive item.
  • Task decoding: Beam search represents the output as an item list C with beam size B for sequential and direct recommendation.The decoding expression defines C as the output item list returned by Beam_Search.

5 EXPERIMENTS

The experiments evaluate P5 against task-specific methods across five recommendation task families and test its zero-shot transfer to unseen prompts. Results show strong performance across tasks, including competitive rating prediction, sequential recommendation, explanation generation, review-related tasks, and direct recommendation.

  • Experimental Setup: P5 is evaluated against representative task-specific baselines across five task families using real-world datasets and multiple recommendation metrics.The evaluation covers rating, sequential recommendation, explanation, review-related, and direct recommendation tasks.
  • Zero-Shot Evaluation: The experiments also assess zero-shot transfer by holding out personalized prompts during pretraining and evaluating P5 on unseen prompts and new items.The study reserves prompts for zero-shot evaluation and reports results across the five task families, alongside ablation studies of P5’s design.
  • Rating Prediction: P5-B improves MAE over MF on all three rating-prediction datasets with seen prompts and achieves similar performance with an unseen prompt.With seen Prompt 1-6, P5-B has better MAE and slightly higher RMSE than MF; with unseen Prompt 1-10, its performance is similar to Prompt 1-6.
  • Sequential Recommendation: P5-B surpasses all competitive baselines on sequential recommendation for both seen and unseen prompts under the all-item setting.The comparison uses all items as candidates rather than sampled candidate sets; P5 models interaction history and predicts the next item with beam search.
  • Explanation Generation: P5 achieves the best performance against all baselines for direct explanation generation and outperforms PETER+ in most feature-based cases.The feature-based gains are especially reported for the Beauty and Toys datasets.
  • Review Related: P5-S outperforms T0 and GPT-2 on review summarization for Beauty and Toys despite having a much smaller model size.GPT-2 and T0 have 1.5B and 11B parameters, respectively; P5-S also shows task- and dataset-specific gains in review preference prediction.

Prompts and Items in New Domain (RQ2)

P5 transfers across unseen prompts and product domains, while ablations show that multitask pretraining, prompt coverage, and sub-word personalization affect performance differently across recommendation tasks.

  • Prompts and Items in New Domain (RQ2): P5 models pretrained on seen prompts also achieve strong performance on held-out prompts from the same task families.These results indicate robustness to wording variations in personalized prompts.
  • Prompts and Items in New Domain (RQ2): 741 users shared across all three domains form the evaluation set for transfer to items from a new product domain.P5-B is pretrained on one domain and evaluated with unseen prompts whose item fields come from another domain.
  • Prompts and Items in New Domain (RQ2): The domain-transfer examples cover Toys to Beauty, Beauty to Sports, and Sports to Toys.The examples illustrate zero-shot transfer directions between product domains.
  • 5.6 Ablation on Task Scaling (RQ3): P5-S and P5-SN have different performance profiles on Beauty, enabling comparison between multitask and single-task prompted pretraining.Figure 5 reports the comparison, while the supplied passages do not state its full numerical outcomes.
  • 5.6 Ablation on Task Scaling (RQ3): P5-S achieves comparable or better performance than single-task models on rating, sequential, and direct recommendation, but single-task models perform better on some text-generation tasks.Multitask modeling seeks a balance among tasks and leverages language understanding.
  • 5.8 How to Implement Personalization (RQ4): Figure 6 compares P5-S and P5-I to assess how personalization is implemented.P5-S uses sub-word tokenization with whole-word embeddings, whereas P5-I assigns independent extra tokens to users and items.
  • 5.8 How to Implement Personalization (RQ4): P5-I performs similarly on regression and review summarization, is slightly better on explanation generation, but significantly underperforms P5-S on sequential and direct recommendation.The authors attribute the gap to difficulty training the large number of newly introduced extra-token embeddings.

6 CONCLUSIONS AND FUTURE WORK

The conclusion presents P5 as a unified language-modeling and natural-language-generation framework for recommendation. Experiments report comparable or better performance across five task families and zero-shot transfer to new items, domains, and prompts.

  • 6 CONCLUSIONS AND FUTURE WORK: P5 unifies different recommendation tasks in a shared language modeling and natural language generation framework.Raw interactions, descriptions, metadata, and reviews are converted into input-target text pairs using personalized prompts.
  • 6 CONCLUSIONS AND FUTURE WORK: P5 matches or exceeds representative approaches across all five task families and transfers zero-shot to new items, domains, and personalized prompts.The authors identify larger models, stronger base models, additional modalities, and more tasks as future directions.

APPENDIX

The appendix adds Yelp experiments, reports training and inference time for P5 variants, and provides complete personalized-prompt lists for Amazon and Yelp datasets.

  • APPENDIX: The appendix contains additional Yelp results, training and inference time statistics, and full personalized-prompt lists for Amazon and Yelp.The prompt lists are provided in separate appendix sections for the two datasets.

A EXPERIMENTAL RESULTS ON YELP DATASET

The Yelp evaluation follows the default pretrain–predict setup while reserving selected prompts for zero-shot evaluation and comparing against the same baselines used earlier.

  • A EXPERIMENTAL RESULTS ON YELP DATASET: Yelp evaluation reserves Prompts 1-10, 2-13, 3-10, 4-3, and 5-8 for zero-shot testing while pretraining P5-S on the remaining prompts.The setup uses personalized prompts from the Yelp appendix and supplements the zero-shot prompts with seen prompts for each task family.

B STATISTICS ON TRAINING & INFERENCE TIME (RQ5)

P5 training and inference costs are reported alongside performance tables for multiple recommendation tasks. On Beauty, P5-S trains faster than P5-B, while inference is measured on a single A5000 GPU.

  • Training and inference time: 6.7 hours for P5-S versus 24.3 hours for P5-B were recorded for training on Beauty.The models were trained on 4 A5000 GPUs; the larger P5-B model required more time.
  • Training and inference time: Average inference time for P5-S and P5-B across tasks is reported in Table 15.Inference evaluations used a single A5000 GPU.
  • Task performance: Table 10 reports performance comparison on rating prediction.
  • Task performance: Tables 11–14 report performance on sequential recommendation, explanation generation, review preference prediction, and direct recommendation.
  • Task performance: Table 16 reports direct-recommendation performance under a larger sample size.
  • Training data construction: Direct recommendation training uses one positive item and 99 randomly sampled negative items to form a 100-item candidate list.A prompt is randomly selected from Prompt 5-5 to Prompt 5-8 for each training sentence, and the process is repeated for each interaction.

D FULL LIST OF PERSONALIZED PROMPTS FOR AMAZON DATASETS

The Amazon appendix lists personalized prompt templates spanning rating prediction, sequential recommendation, explanation generation, review-related tasks, direct recommendation, and zero-shot generalization. Templates vary user and item descriptions, histories, candidate lists, ratings, and generated targets.

  • D.1 Task Family 1: Rating Prediction: Rating-prediction prompts ask for 1-to-5 star ratings, yes/no rating judgments, or like/dislike preferences for users and items.Like corresponds to ratings 4 and 5, while dislike corresponds to ratings 1, 2, and 3.
  • Prompt personalization: The prompt collection includes both identifier-based and description-based formulations for the same recommendation tasks.For example, rating and sequential prompts alternate between user identifiers and user descriptions.
  • D.2 Task Family 2: Sequential Recommendation: Sequential-recommendation prompts use purchase history to predict the next item directly, select one item from candidates, or judge whether a candidate will be purchased next.Targets include {{next_item}} or yes/no labels.
  • D.3 Task Family 3: Explanation Generation: Explanation-generation prompts produce textual explanations for a user-item preference, sometimes conditioned on a star rating or feature word.Some prompts first predict the star rating and then generate the explanation.
  • D.4 Task Family 4: Review Related: Review-related prompts summarize product reviews or predict their associated star ratings.Inputs may use the user identifier or a user description.
  • D.5 Task Family 5: Direct Recommendation: Direct-recommendation prompts select the best item from candidate items or decide whether a particular item should be recommended.Targets are either {{target_item}} or yes/no answer choices.
  • D.6 Task Family Z: Zero-Shot Generalization: The zero-shot generalization family supplies new-product facts including title, brand, and price to assess user preference.

E FULL LIST OF PERSONALIZED PROMPTS FOR YELP DATASET

The Yelp appendix adapts personalized prompts to visit histories and businesses while retaining task families for ratings, sequential recommendation, explanations, review prediction, and direct recommendation. Inputs alternate between user identifiers and descriptions, with structured targets such as ratings, items, explanations, and labels.

  • E.1 Task Family 1: Rating Prediction: Rating-prediction prompts ask for 1-to-5 star ratings, yes/no rating judgments, or like/dislike preferences for users and businesses.Like corresponds to ratings 4 and 5, while dislike corresponds to ratings 1, 2, and 3.
  • Prompt personalization: Across Yelp task families, inputs use either user identifiers or user descriptions and outputs use structured fields such as ratings, next items, explanations, and labels.
  • E.2 Task Family 2: Sequential Recommendation: Sequential-recommendation prompts use visit history to predict the next business, select one candidate business, or judge whether a candidate will be visited next.Targets include {{next_item}} or yes/no labels.
  • E.3 Task Family 3: Explanation Generation: Explanation-generation prompts produce explanations about a business, sometimes combining star-rating prediction with explanation generation.Other prompts condition explanations on a feature word or star rating.
  • E.4 Task Family 4: Review Related: Review-related prompts predict the associated star rating from a review written by a user or user description.
  • E.5 Task Family 5: Direct Recommendation: Direct-recommendation prompts select a business from candidates or decide whether a business should be recommended to a user.Targets are either {{target_item}} or yes/no answer choices.
Loading 2203.13366v7…