Source-linked AI summary
RecMind: Large Language Model Powered Agent For Recommendation
Yancheng Wang, Ziyan Jiang, Zheng Chen, Fan Yang, Yingxue Zhou, Eunah Cho, Xing Fan, Xiaojiang Huang, Yanbin Lu, Yingzhen Yang
TL;DR
Recommendation systems remain limited in cross-task generalization and access to external knowledge. RecMind combines an LLM-powered agent with planning, memory, tools, and Self-Inspiring reasoning. It outperforms existing LLM-based recommendation methods across tasks and achieves comparable performance to fully trained P5.
Problem
Existing recommendation methods are often task-specific and have limited ability to capture textual knowledge or leverage external information for unseen recommendation tasks.
Method
RecMind is an LLM-powered autonomous recommendation agent that combines task-decomposing planning, memory, external tools, and Self-Inspiring planning across previously explored reasoning paths.
Results
RecMind with Self-Inspiring outperforms existing LLM-based recommendation methods across different tasks and achieves comparable performance to fully trained P5.
Takeaways & Limitations
RecMind provides a general recommendation agent evaluated across precision- and explainability-oriented tasks without requiring task-specific fine-tuning.
Takeaways & Limitations
Exploring diverse reasoning paths increases prompt size, exposing the method to long-context and position-bias limitations; the implementation also uses only a small number of external tools.
Abstract
from arXiv · showhide
While the recommendation system (RS) has advanced significantly through deep learning, current RS approaches usually train and fine-tune models on task-specific datasets, limiting their generalizability to new recommendation tasks and their ability to leverage external knowledge due to model scale and data size constraints. Thus, we designed an LLM-powered autonomous recommender agent, RecMind, which is capable of leveraging external knowledge, utilizing tools with careful planning to provide zero-shot personalized recommendations. We propose a Self-Inspiring algorithm to improve the planning ability. At each intermediate step, the LLM self-inspires to consider all previously explored states to plan for the next step. This mechanism greatly improves the model's ability to comprehend and utilize historical information in planning for recommendation. We evaluate RecMind's performance in various recommendation scenarios. Our experiment shows that RecMind outperforms existing zero/few-shot LLM-based recommendation baseline methods in various tasks and achieves comparable performance to a fully trained recommendation model P5.
1 Introduction
Existing recommender systems struggle to generalize across unseen tasks and to use textual or external knowledge effectively. RecMind addresses these limitations with an LLM-powered agent that plans, retains information, uses tools, and applies Self-Inspiring planning.
- Existing DNN-based and pretrained-language-model methods inadequately capture textual knowledge about users and items because of model-scale and data-size limitations.
- Most recommendation methods are task-specific and inadequate at generalizing to unseen recommendation tasks.
- Existing LLM recommendation studies mainly rely on knowledge stored in model weights, neglect external tools for real-time information, and underuse reasoning for recommendation.
- RecMind combines planning, personalized and world memory, and external tools around an LLM-powered API to decompose tasks, retain information, and retrieve relevant knowledge.
- Self-Inspiring retains all previously explored states across reasoning paths when generating the next state, unlike CoT and ToT, which discard prior-path states.
- Across five recommendation tasks and two datasets, RecMind outperforms non-fine-tuned LLM-based baselines and achieves competitive performance with fully pretrained P5.
2 Related Work
LLM-based agents augment language models with actions and API calls, while LLM recommendation systems use natural-language preferences and interaction histories. RecMind extends these directions by combining agentic reasoning with external tools and memory for recommendation.
- LLM-as-Agent: LLM agents generate text outputs and actions that perform API calls and operations within specific environments.
- LLM for Recommendation: LLM recommendation research leverages language models to understand users’ preferences and past interactions in natural language.
- LLM for Recommendation: Existing recommendation studies use prompts with interaction and optional profile data under no-, full-, or parameter-efficient fine-tuning settings.
- LLM for Recommendation: Unlike existing studies, RecMind harnesses LLM reasoning together with tools and memory to obtain relevant information beyond model weights.
3 Architecture
RecMind combines LLM-driven planning, memory, and external tools to decompose recommendation tasks and access information beyond model parameters. Its Self-Inspiring planner retains historical states across reasoning paths to support alternative planning branches.
- Architecture: RecMind uses an LLM-powered API, planning, memory, and tools to handle complex recommendation tasks step by step.Planning decomposes tasks into sub-tasks, while memory and tools provide information beyond the LLM’s parameters.
- Planning: Planning represents each step as a thought, action, and observation, producing a final result after traversing a latent reasoning tree.This extends the traditional input-output formulation by conditioning generation on a planning sequence.
- Planning: CoT follows one reasoning path, whereas ToT explores multiple paths through breadth-first candidate generation or depth-first branching with pruning and backtracking.ToT-BFS selects among sampled next-state candidates by majority vote; ToT-DFS explores one branch at a time before backtracking.
- Self-Inspiring Planning: Self-Inspiring retains states from previously explored paths when generating the next planning state and can initiate alternative reasoning branches.At the m-th path and step t, the next state is generated using all previous paths rather than only the current final path.
- Tools: RecMind’s tools access database knowledge, real-time search information, and summaries of lengthy item-review texts.The database tool translates natural-language questions into SQL, the search tool retrieves external information, and the summarization tool condenses multiple reviews.
4 Experiments
RecMind is evaluated across multiple recommendation tasks, datasets, baselines, planning methods, and domain-transfer settings. Results show strong performance in rating prediction, sequential recommendation, explanation generation, and general reasoning, while direct recommendation remains constrained by large candidate sets.
- Evaluation Setup: The evaluation covers rating prediction, sequential recommendation, direct recommendation, explanation generation, and review summarization across Amazon Reviews and Yelp.Metrics include RMSE, MAE, HR@k, NDCG@k, and BLEU-based measures.
- Compared Methods: RecMind is compared with fine-tuned P5, ChatGPT prompting baselines, and RecMind variants using CoT, ToT, and Self-Inspiring planning.The compared methods include zero-shot and few-shot prompting settings where specified.
- Precision-oriented Recommendation Tasks: RecMind with different planning mechanisms usually outperforms fully trained models on rating prediction, supported by access to users’ and items’ rating histories.The authors attribute the improvement mainly to database access to interaction histories.
- Precision-oriented Recommendation Tasks: In direct recommendation, fully trained models such as P5 usually perform better, although ToT and Self-Inspiring alleviate positional bias by gradually filtering candidate items.Large candidate sets and prompt context-length limitations remain difficult for LLM agents.
- Precision-oriented Recommendation Tasks: Self-Inspiring achieves comparable performance to fully trained P5 and S3-Rec in sequential recommendation, while explicit reasoning and domain knowledge help explore cross-user item connections.Without diverse planning, LLMs tend to prefer items whose names resemble preceding items.
- Explainability and Domain Transfer: RecMind with Self-Inspiring achieves explanation-generation performance comparable to fully trained P5 and shows better domain-transfer performance than compared approaches.Personalized memory, including other users’ reviews of the same item, supports explanation generation; transfer is evaluated with MAE, HR@5, and BLEU2.
5 Conclusions
RecMind is an LLM-powered autonomous agent for recommendation that combines planning, memory, external tools, and Self-Inspiring to support varied recommendation tasks. Across evaluations, RecMind with Self-Inspiring outperforms existing LLM-based recommendation methods and performs comparably to the fully trained P5 model.
- RecMind combines planning, memory, and external tools to support various recommendation tasks.Planning decomposes tasks, memory retains and recalls information, and external tools provide information missing from model weights.
- Self-Inspiring integrates multiple reasoning paths to improve planning.
- RecMind with Self-Inspiring outperforms existing LLM-based recommendation methods across different recommendation tasks.
- RecMind achieves comparable performance to P5, a model fully trained for recommendation.
Limitations
Exploring more diverse reasoning paths increases prompt size, exposing RecMind to long-context and position-bias limitations. The implementation also adopts only a small number of external tools.
- More diverse reasoning paths greatly increase prompt size, leading to long-context and position-bias limitations in LLMs.The paper suggests summarizing historical paths to condense context and potentially remove noise.
- The current implementation adopts only a small number of external tools.
Ethical Concerns and Broader Impacts
The experiments use anonymous-user Amazon Reviews and Yelp datasets for academic purposes, while noting that recommendation systems may influence consumer behavior and preferences.
- All experiments use the Amazon Reviews and Yelp recommendation datasets with anonymous user IDs.
- The datasets are used only for academic purposes under their terms of use.
- The proposed recommendation system has the potential to influence consumer behavior and preferences.
A Additional Implementation Details
The implementation prompts an LLM agent to interleave thoughts, actions, and observations while using SQL, summarization, search, and finish tools. Search and SQL outputs are converted into meaningful text responses, and Self-Inspiring prompts the agent to consider alternative thoughts using previously explored steps.
- The agent prompt structures recommendation as interleaved Thought, Action, and Observation steps.Actions include SQL, text summarization, search, and finishing with an answer.
- The search tool converts structured Google search results into meaningful sentences responding to the question.
- The SQL tool converts questions into SQL queries, executes them, and converts query results into meaningful text responses.
- Self-Inspiring asks the same LLM to determine whether an alternative thought could help complete the task given previous and current steps.
- CoT and ToT prompts respectively generate interleaved reasoning steps or candidate thoughts followed by a choice of the most promising option.
B.1 Results on Review Summarization
RecMind is evaluated on shortening review comments into review titles in Amazon Reviews’ beauty domain. It outperforms ChatGPT but does not surpass the fully trained P5 model.
- The review summarization task converts review comments into shorter review titles.Automatically generated review titles were filtered from the test data.
- RecMind performs better than ChatGPT on review summarization in the Amazon Reviews beauty domain.The comparison results are reported in Table 8.
- RecMind does not outperform P5 on review summarization.The passage attributes P5’s advantage to full training for the summarization task.
B.2 Human Evaluation
The study uses human evaluation to assess the quality and rationality of RecMind-generated explanations. Three evaluators rank outputs from P5, ChatGPT, two RecMind reasoning variants, and ground truth.
- Three human evaluators assess the quality and rationality of explanations generated by competing recommendation methods.The evaluation uses 100 test examples.
- The comparison includes P5, few-shot ChatGPT, few-shot RecMind with tree-of-thoughts, few-shot RecMind with self-inspiring, and ground truth.Results are reported using top-1 ratios for each evaluator.
B.3 Running Time Analysis
The running-time analysis compares RecMind using Chain-of-Thoughts, Tree-of-Thoughts, and Self-Inspiring reasoning. On sampled Beauty-domain data with GPT-3.5, Self-Inspiring is reported to improve performance while requiring less inference time.
- RecMind’s Chain-of-Thoughts, Tree-of-Thoughts, and Self-Inspiring reasoning methods are compared by average running time.The comparison uses 100 randomly sampled Beauty-domain Amazon Reviews test examples and GPT-3.5 as the base model.
- Self-Inspiring is reported to improve the LLM-powered agent’s performance while taking less inference time.The comparison results are presented in Table 10.
B.4 Results on Sports and Toys Domains in Amazon Reviews
Additional experiments evaluate RecMind across rating prediction, direct and sequential recommendation, review summarization, and explanation generation in Sports and Toys Amazon Reviews domains. The reported results indicate that RecMind performs well across these tasks and domains.
- Rating prediction: The additional experiments cover rating prediction in the Sports and Toys domains.The corresponding results are reported in Table 11.
- Direct and sequential recommendation: Direct and sequential recommendation are evaluated in both the Sports and Toys domains.These comparisons are reported in Tables 12 and 13.
- Summarization and explanation generation: Review summarization and explanation generation are evaluated in the Sports and Toys domains.The corresponding comparisons are reported in Tables 14 and 15.
- Overall result: RecMind also performs well across different recommendation tasks on other Amazon Reviews domains.This conclusion follows the reported Sports- and Toys-domain experimental results.