Source-linked AI summary
Recommender AI Agent: Integrating Large Language Models for Interactive Recommendations
Xu Huang, Jianxun Lian, Yuxuan Lei, Jing Yao, Defu Lian, Xing Xie
TL;DR
Recommender models provide domain-specific recommendations but lack versatile conversational abilities, while LLMs lack detailed catalog and behavioral knowledge. InteRecAgent addresses this gap by using LLMs as the brain and recommender models as tools, with memory, planning, and reflection mechanisms. Experiments show effective conversational recommendation and advantages over general-purpose LLMs, particularly in private or less world-knowledge-covered domains.
Problem
Recommender models struggle with explanations and conversation, whereas LLMs lack domain-specific catalogs and behavioral patterns, making their combination important for interactive recommendation.
Method
InteRecAgent integrates LLMs with traditional recommendation models through querying, retrieval, and ranking tools, shared candidate memory, user profiles, dynamic plan-first planning, and reflection.
Results
InteRecAgent outperforms general-purpose LLMs across public datasets, with particularly significant advantages in less world-knowledge-covered domains; RecLlama also surpasses several larger models.
Takeaways & Limitations
The framework enables traditional recommender systems to become interactive systems with natural-language interfaces by combining recommender models’ domain strengths with LLM capabilities.
Abstract
from arXiv · showhide
Recommender models excel at providing domain-specific item recommendations by leveraging extensive user behavior data. Despite their ability to act as lightweight domain experts, they struggle to perform versatile tasks such as providing explanations and engaging in conversations. On the other hand, large language models (LLMs) represent a significant step towards artificial general intelligence, showcasing remarkable capabilities in instruction comprehension, commonsense reasoning, and human interaction. However, LLMs lack the knowledge of domain-specific item catalogs and behavioral patterns, particularly in areas that diverge from general world knowledge, such as online e-commerce. Finetuning LLMs for each domain is neither economic nor efficient. In this paper, we bridge the gap between recommender models and LLMs, combining their respective strengths to create a versatile and interactive recommender system. We introduce an efficient framework called \textbf{InteRecAgent}, which employs LLMs as the brain and recommender models as tools. We first outline a minimal set of essential tools required to transform LLMs into InteRecAgent. We then propose an efficient workflow within InteRecAgent for task execution, incorporating key components such as memory components, dynamic demonstration-augmented task planning, and reflection. InteRecAgent enables traditional recommender systems, such as those ID-based matrix factorization models, to become interactive systems with a natural language interface through the integration of LLMs. Experimental results on several public datasets show that InteRecAgent achieves satisfying performance as a conversational recommender system, outperforming general-purpose LLMs. The source code of InteRecAgent is released at https://aka.ms/recagent.
1 Introduction
InteRecAgent combines LLMs’ interaction and reasoning abilities with traditional recommender models’ domain-specific knowledge to create interactive recommendation systems. Its workflow integrates specialized tools, memory, planning, demonstrations, and reflection.
- Conversational interfaces create a need for recommendation systems that understand and respond to diverse user inquiries and intents.
- LLMs support contextual, engaging interactions but lack fine-grained domain-specific behavior patterns, private-domain knowledge, and knowledge of newly released items.
- InteRecAgent connects LLMs with traditional recommendation models through querying, retrieval, and ranking tools for interactive recommendation tasks.A shared candidate bus stores intermediate item states and enables communication between tools.
- Long-term and short-term user profiles track preferences and history, supplying the ranking tool with information for improved personalization.
- The framework uses plan-first execution with dynamically retrieved demonstrations to reduce inference costs and improve tool-calling plan generation.It generates all tool-calling steps at once and follows the resulting execution plan.
- RecLlama is a 7-billion-parameter model fine-tuned on GPT-4-derived tool plans, and it surpasses several larger models as the recommender agent’s core.Experimental results from three public datasets demonstrate InteRecAgent’s effectiveness, especially in domains less covered by world knowledge.
2 Related Work
Related work covers conversational recommendation, expanding LLM capabilities, and LLM-based agents augmented with memory, planning, reflection, and external tools. These directions motivate using LLMs as general-purpose agents with domain-specific tool support.
- Conversational recommender systems primarily address attribute-based question answering or open-ended conversation.Attribute-based systems aim to recommend suitable items within as few interaction rounds as possible.
- Scaling model parameters and data has advanced LLMs’ in-context learning, instruction following, planning, and reasoning capabilities.
- Research has explored LLMs as autonomous agents equipped with memory modules, planning abilities, and tool use.
- Agent research includes step-wise and multipath reasoning, reflection on errors, and external tools such as search engines, mathematical tools, code interpreters, and visual models.
3 Methodologies
InteRecAgent combines an LLM brain with recommendation-model tools for natural-language interaction, using memory, planned tool execution, and reflection to support domain-specific recommendations.
- The Overall Framework: InteRecAgent uses an LLM as the brain and recommendation models as tools that provide domain-specific knowledge during natural-language interaction.The LLM interprets user intent, decides whether tools are needed, and generates responses from tool outputs.
- The Overall Framework: Its minimum tool set includes information query, item retrieval, and item ranking for answering inquiries and producing personalized recommendations.Information query retrieves backend item details; retrieval narrows candidates; ranking predicts preferences using user profiles.
- The Overall Framework: Retrieval handles hard conditions with SQL and soft semantic demands with item-to-item latent-embedding matching.Hard conditions include explicit attributes such as popularity, genre, or price, while soft conditions require semantic similarity.
- Memory Mechanism: The Candidate Bus stores current candidates and tool records separately, allowing candidates to stream through tools while avoiding their repeated inclusion in prompts.The data bus is refreshed after each tool execution, and the tracker records tool names, inputs, outputs, and execution information.
- Plan-first Execution: Plan-first execution generates all tool calls before execution, reducing inference costs and avoiding the long observations and inter-tool dependencies of step-by-step execution.The strategy separates planning from execution and can be combined with dynamic demonstrations to improve plan generation.
- Reflection: An actor-critic reflection mechanism evaluates tool-use behavior to reduce errors such as invalid tools, omitted tools, or improperly formatted inputs.The actor makes a plan, obtains tool outputs, and generates a response; the critic evaluates the actor’s behavioral decisions.
4 Experiments
Experiments evaluate InteRecAgent through simulated conversations, one-turn recommendation, lifelong-memory settings, and ablations. Across these settings, the tool-augmented framework improves recommendation accuracy, efficiency, domain handling, and long-context performance relative to baselines.
- Experimental Setup: The evaluation uses user simulators, one-turn recommendation, and lifelong conversation settings across Steam4, MovieLens, and Amazon Beauty datasets.Metrics include Hit@k, average turns, Recall@k, and NDCG@k; testing uses sampled instances because of budget constraints.
- User Simulator: InteRecAgent surpasses other LLMs in hit rate and average turns across all three datasets in session-wise conversations.Larger LLMs generally perform better, while LlaMA2-7B and LlaMA2-13B perform extremely poorly on Amazon Beauty.
- User Simulator: Amazon Beauty shows larger gains because its private, complex, and newer items are poorly covered by general world knowledge, causing severe hallucination for many LLMs.Chat-Rec improves over GPT-3.5 and GPT-4 through embedding retrieval but remains below InteRecAgent and cannot handle multifaceted queries.
- Lifelong Conversation: Adding long-term memory further improves InteRecAgent under both LONG-CHAT and LONG-CONTEXT configurations.Without long-term memory, InteRecAgent already consistently outperforms GPT-4 across datasets.
- One-Turn Recommendation: In one-turn recommendation, InteRecAgent outperforms all baselines and maintains high accuracy on Amazon Beauty, while some LLMs perform below random and popularity ranking methods.The authors attribute poor ranking performance mainly to instruction nonadherence, uncertainty, and out-of-scope items, especially for smaller models.
- Ablation Study: Removing any ablated mechanism reduces performance; removing reflection has the largest impact, while removing plan-first execution raises Steam API calls from 2.78 to 4.51 per turn.The increased API calls produce approximately 10–20 seconds of additional latency; reflection corrects tool-input errors and misuse.
- Case Study: InteRecAgent preserves GPT-4’s conversational capabilities while adding domain-specific retrieval and ranking that improve factuality for games and e-commerce products.In the case studies, it retrieves product information and avoids nonexistent Amazon products produced by GPT-4.
5 Conclusion
InteRecAgent combines LLMs with traditional recommender models to create interactive systems for personalized conversational recommendations.
- InteRecAgent transforms traditional recommender models into interactive systems by combining LLM capabilities with recommendation tools.Its framework includes information query, retrieval, and ranking tools, alongside enhanced memory, planning, and tool-learning components.
A Dataset
The evaluation uses Steam, MovieLens, and Amazon Beauty datasets prepared with a conventional k-core filtering strategy.
- Experiments evaluate the methods on Steam, MovieLens, and Amazon Beauty datasets.
- The in-domain soft condition item retrieval and ranking tools are trained after filtering users and items with fewer than 5 interactions.The filtering uses the conventional k-core strategy.
- Table A1 reports the statistical information for the filtered datasets.
B Prompts
The appendix documents task descriptions, tool prompts, reflection, demonstration generation, user simulation, and one-turn conversation generation.
- B.2 Tool Descriptions: InteRecAgent uses SQL querying, two item retrieval tools, item ranking, memory initialization, and final-item fetching tools.
- B.3 Reflection: The reflection mechanism uses a dedicated critic prompt.
- B.4 Demonstration Generation: Demonstrations are generated with input-first and output-first intent strategies, while plans use a dedicated plan-generation prompt.
- B.5 User Simulator: The user simulator is instructed through a dedicated prompt to act as a user seeking recommendations.
- B.6 One-Turn Conversation Generation: One-turn recommendation separates retrieval and ranking into independently generated conversations.Separate prompts support retrieval and ranking tasks.
- B.6 One-Turn Conversation Generation: The assistant mines user interests through conversation and selects tools according to chit-chat, item information, or recommendation requests.Recommendations lacking preference information trigger further conversation, while informed requests use available tools.
- B.1 Task Descriptions: The task description uses bracketed texts as placeholders for variables.