Source-linked AI summary
Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System
Yunfan Gao, Tao Sheng, Youlin Xiang, Yun Xiong, Haofen Wang, Jiawei Zhang
TL;DR
Traditional recommender systems have limited interactivity and explainability, alongside challenges with cold-start and cross-domain recommendation. Chat-Rec converts user information and interactions into prompts for an untrained, in-context-learning LLM recommender. Across top-k recommendation and zero-shot rating prediction, the reported experiments show improved performance, including gains over LightGCN and Item-KNN on selected metrics.
Problem
Traditional recommender systems face poor interactivity and explainability, as well as cold-start and cross-domain recommendation challenges.
Method
Chat-Rec converts user profiles and user–item interactions into prompts and uses an LLM through in-context learning without explicit training.
Results
Chat-Rec improves top-k recommendation and zero-shot rating prediction, with text-davinci-003 reaching precision 0.3240 and NDCG 0.3802 against LightGCN, and RMSE 0.785 and MAE 0.593 against Item-KNN.
Takeaways & Limitations
Chat-Rec provides a practical route for applying conversational LLMs to interactive, explainable, cross-domain, and cold-start recommendation scenarios.
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated their significant potential to be applied for addressing various application tasks. However, traditional recommender systems continue to face great challenges such as poor interactivity and explainability, which actually also hinder their broad deployment in real-world systems. To address these limitations, this paper proposes a novel paradigm called Chat-Rec (ChatGPT Augmented Recommender System) that innovatively augments LLMs for building conversational recommender systems by converting user profiles and historical interactions into prompts. Chat-Rec is demonstrated to be effective in learning user preferences and establishing connections between users and products through in-context learning, which also makes the recommendation process more interactive and explainable. What's more, within the Chat-Rec framework, user's preferences can transfer to different products for cross-domain recommendations, and prompt-based injection of information into LLMs can also handle the cold-start scenarios with new items. In our experiments, Chat-Rec effectively improve the results of top-k recommendations and performs better in zero-shot rating prediction task. Chat-Rec offers a novel approach to improving recommender systems and presents new practical scenarios for the implementation of AIGC (AI generated content) in recommender system studies.
1 Introduction
Traditional recommender systems face limited interactivity, explainability, cold-start handling, and cross-domain recommendation. Chat-Rec addresses these challenges by using untrained LLMs with in-context learning and prompts, with experiments showing improved recommendation performance.
- Motivation: Existing recommender systems struggle with interactivity, explainability, feedback mechanisms, cold-start cases, and cross-domain recommendations.They may also require external libraries, knowledge graphs, or augmented-data training for background knowledge.
- Prior Work: Prior LLM-based recommendation methods formulate tasks as prompt-based natural-language inputs but still involve LLMs during training.User–item information and features are integrated with personalized prompt templates.
- Chat-Rec: Chat-Rec combines recommender systems with LLMs through prompts and relies solely on in-context learning without additional training.The paradigm converts user information and interactions into prompts and uses LLMs as a recommender-system interface.
- Chat-Rec: Chat-Rec updates candidate recommendations as conversation-based preference information is progressively refined.The framework is designed to provide interactive and explainable recommendations.
- Capabilities and Evaluation: Chat-Rec links preferences between products, supporting cross-domain product recommendations.The paper evaluates recommendation and rating prediction tasks on real-world datasets.
- Capabilities and Evaluation: Experiments demonstrate Chat-Rec’s effectiveness on top-k recommendation and rating prediction tasks.The evaluation uses real-world datasets.
2 Related Work
Related work connects language models with recommendation through augmentation, pretrained representations, and cold-start strategies. These approaches target limitations involving reasoning, explainability, sparsity, generalization, and missing interaction records.
- Augmented Language Models: Augmented Language Models extend traditional language models with reasoning skills and external-tool use to address interpretability, consistency, and scalability limitations.They can use these augmentations separately or together and perform standard language tasks.
- NLP for Recommendation: Pretrained language models provide reusable representations that can improve recommendation performance and explainability.In recommendation, they can help alleviate data sparsity by transferring knowledge from pretrained models.
- Cold-start Recommendation: Cold-start recommendation concerns users or items without prior interaction records.Existing approaches model content features, transfer representations from auxiliary domains, or adapt to new domains.
- Cold-start Recommendation: This paper uses LLM reasoning and background knowledge to enhance recommender-system performance in cold-start scenarios.The stated target is recommendation when interaction data are unavailable.
3 Method
Chat-Rec bridges recommender systems and conversational LLMs by converting user and dialogue information into prompts, then using generated candidates and responses to support personalized, interactive recommendations. Its pipeline compresses candidate sets, explains recommendations, incorporates external information for new items, and transfers preferences across domains.
- Bridge Recommender Systems and LLMs: Chat-Rec combines user-item history, user profiles, queries, and dialogue history with a recommender system through prompts.The module determines whether a query requires recommendation or a direct conversational response.
- Recommendation Based on Candidate Set Compression: The prompt constructor summarizes query and recommendation information, while intermediate answers refine prompts and compress candidates before producing recommendations with explanations.The pipeline can use a recommender’s candidate set and return a smaller final set to the user.
- Recommendation Based on Candidate Set Compression: For action-movie requests, an intermediate top-20 result is reranked to produce a final top-5 recommendation.The second conversational round can instead generate an explanation using the movie title, interaction history, and user profile.
- New Item Recommendation: External information about new items is introduced because ChatGPT does not reliably handle items beyond its September 2021 knowledge timeframe.Textual product descriptions and profile information are used to relate new and existing products for recommendation.
- Cross-Domain Recommendations: Chat-Rec transfers movie preferences to books, TV series, podcasts, and video games for cross-domain recommendations.The cross-domain case demonstrates preference transfer from movies to other item types.
4 Experiment
Experiments on MovieLens 100K evaluate Chat-Rec across top-5 recommendation, rating prediction, and prompt ablations. Results show gains over traditional baselines while revealing sensitivity to candidate ordering and prompt contents.
- Dataset and Experimental Settings: MovieLens 100K supplies 100,000 ratings from 943 users across 1,682 movies, with demographic and movie metadata.The experiment randomly selected 200 users from the benchmark dataset.
- Result and Analysis: 0.3240 precision and 0.3802 NDCG make text-davinci-003 respectively 6.93% and 11.01% higher than LightGCN in top-5 recommendation.Its recall of 0.1404 is 3.51% lower than LightGCN.
- Result and Analysis: 0.785 RMSE and 0.593 MAE are the best reported rating-prediction results for text-davinci-003, exceeding Item-KNN by 15.86% and 19.21%.LightGCN was excluded because it was considered unsuitable for rating prediction.
- Result and Analysis: Chat-Rec primarily reranks movies within the recommender’s candidate set by matching movie knowledge with user preferences and reasoning about their relationship.Additional questioning produced reasons for movies appearing in the recommender’s top 5 but not the LLM’s top 5.
- Ablation Study: At temperature 0.9, shuffling candidates reduced NDCG from 0.3802 to 0.3653, while removing the recommender’s top-1 reduced it to 0.3055.The reported best results were achieved at temperature 0.9; candidate-set design implicitly injects recommender knowledge into the prompt.
5 Conclusion
Chat-Rec bridges recommender systems and LLMs by converting user information and interactions into prompts, improving recommendation capabilities across evaluated tasks.
- Chat-Rec converts user information and user-item interactions into prompts to bridge recommender systems and LLMs.
- The approach was evaluated on top-k recommendation and zero-shot movie rating prediction tasks.
- LLMs offer potential to improve interactivity, explainability, and cross-domain recommendation in recommender systems.
- Experiments indicate that implicitly expressing recommender-system knowledge in prompts can improve recommendation effectiveness.
A.1 Prompts
The appendix lists the prompts used for the top-k recommendation and zero-shot movie rating tasks.
- The appendix lists prompts used for the top-k recommendation task.
- It also lists prompts used for the moving rating task.
A.2 Example Answers
Example answers document failures in producing the expected canonical output format, including missing movie identifiers.
- LLMs do not always produce answers in the expected format, especially at higher temperatures.
- Outputs that do not match the required format are automatically retried during API invocation.
- Table 4 presents cases and explanations of failed canonical-answer generation.
- Some movie outputs lose identifier information when the required [id]:[name] format is used.