Source-linked AI summary
On Generative Agents in Recommendation
An Zhang, Yuxin Chen, Leheng Sheng, Xiang Wang, Tat-Seng Chua
TL;DR
Agent4Rec addresses the gap between offline recommendation metrics and online behavior by using LLM-empowered generative agents as simulated users. It equips agents with dataset-initialized profiles, factual and emotional memory, reflection, and recommendation-specific actions, then evaluates alignment, recommendation behavior, filter bubbles, and causal relationships. The reported results show both alignment and deviation between agents and personalized user preferences, while the authors identify offline-data and action-space constraints.
Problem
A substantial gap between offline metrics and online performance motivates a simulator that can better represent personalized human behavior in recommender systems.
Method
Agent4Rec combines LLM-empowered generative agents with profile, memory, and action modules and a recommendation environment initialized from real-world datasets.
Results
Evaluations find both alignment and deviation between simulated agents and user-personalized preferences while examining recommendation outcomes, filter bubbles, and causal relationships.
Takeaways & Limitations
Agent4Rec provides an experimental platform for evaluating recommender systems and investigating filter-bubble and causal phenomena through simulated user interactions.
Takeaways & Limitations
Agent4Rec relies on offline datasets and has a limited action space that omits influences such as social networks, advertising, and word-of-mouth marketing.
Abstract
from arXiv · showhide
Recommender systems are the cornerstone of today's information dissemination, yet a disconnect between offline metrics and online performance greatly hinders their development. Addressing this challenge, we envision a recommendation simulator, capitalizing on recent breakthroughs in human-level intelligence exhibited by Large Language Models (LLMs). We propose Agent4Rec, a user simulator in recommendation, leveraging LLM-empowered generative agents equipped with user profile, memory, and actions modules specifically tailored for the recommender system. In particular, these agents' profile modules are initialized using real-world datasets (e.g. MovieLens, Steam, Amazon-Book), capturing users' unique tastes and social traits; memory modules log both factual and emotional memories and are integrated with an emotion-driven reflection mechanism; action modules support a wide variety of behaviors, spanning both taste-driven and emotion-driven actions. Each agent interacts with personalized recommender models in a page-by-page manner, relying on a pre-implemented collaborative filtering-based recommendation algorithm. We delve into both the capabilities and limitations of Agent4Rec, aiming to explore an essential research question: ``To what extent can LLM-empowered generative agents faithfully simulate the behavior of real, autonomous humans in recommender systems?'' Extensive and multi-faceted evaluations of Agent4Rec highlight both the alignment and deviation between agents and user-personalized preferences. Beyond mere performance comparison, we explore insightful experiments, such as emulating the filter bubble effect and discovering the underlying causal relationships in recommendation tasks. Our codes are available at https://github.com/LehengTHU/Agent4Rec.
1 INTRODUCTION
Agent4Rec addresses the gap between offline recommendation metrics and online performance by using LLM-empowered generative agents to simulate personalized user behavior. The simulator evaluates agent alignment and recommender-system behavior while probing filter bubbles and causal relationships.
- Offline recommendation metrics can diverge substantially from online performance, hindering progress toward real-world recommender-system deployment.
- Agent4Rec combines LLM-empowered generative agents with a recommendation environment to emulate personalized user preferences and behavior.
- Each agent uses profile, memory, and action modules to represent social traits, preferences, interactions, emotions, and recommendation behaviors.
- Evaluations measure user alignment and recommendation outcomes including watched items, ratings, engagement time, and satisfaction across recommender configurations.
- The simulator is also used to emulate filter bubbles and collect data for discovering latent causal relationships in recommendation systems.
- The paper presents Agent4Rec as an experimental platform intended to support further research on generative-agent simulation in recommender systems.
2 AGENT4REC
Agent4Rec models recommendation users with dataset-initialized profiles, factual and emotional memories, reflection, and recommendation-specific actions. Its environment generates item profiles and presents recommendations page by page, enabling agents to interact through taste- and emotion-driven behaviors.
- Agent4Rec represents users and items with real-world data while combining generative-agent architecture with a configurable recommendation environment.
- Profile Module: User profiles encode social traits such as activity, conformity, and diversity alongside personalized tastes inferred from historical viewing and rating patterns.
- Profile Module: Personal identifiers such as name, gender, age, and occupation are obscured, while preferences are inferred from interaction histories and ratings.
- Memory Module: The memory module stores factual interactions and emotional states, including watched items, feedback, fatigue, and satisfaction.
- Recommendation Environment: The recommendation environment generates item profiles and presents algorithmic recommendations page by page for agent interaction.
- Memory Module: Memory operations retrieve relevant information, write interactions and emotions, and trigger emotion-driven reflection after a predefined action count.
- Action Module: The action module supports taste-driven viewing, rating, and post-viewing feelings alongside emotion-driven exiting, system evaluation, and comments.
3 AGENT ALIGNMENT EVALUATION.
Agent4Rec evaluates whether LLM-empowered agents reproduce user behavior and recommendation responses across taste, social traits, rating distributions, and interactive feedback. Results show strong alignment in several dimensions, while also exposing limitations in low-rating behavior and diversity-trait differentiation.
- Evaluation scope: Agent4Rec frames agent alignment as reproducing genuine human behavior across user preferences, social traits, and recommendation interactions.The evaluation examines both user-aligned behavior and the recommendation environment.
- Rating distribution alignment: Agents closely match the overall MovieLens rating distribution, with ratings of 4 dominant and ratings of 1–2 uncommon.The scarcity of low ratings differs from genuine behavior because agents often avoid low-quality films before watching them.
- Social traits alignment: Social traits significantly influence agent behavior, although diversity may produce minimal differentiation when movie categories strongly overlap.The profile-module evaluation combines distribution analysis, ablations, and statistical tests across activity, conformity, and diversity.
- Recommendation strategy evaluation: Agents report higher satisfaction for algorithm-based recommendations than random or popularity-based strategies, with LightGCN outperforming MF and MultVAE across evaluation criteria.The simulator evaluates strategies through multi-facet satisfaction measures in a page-by-page setting.
- Page-by-page recommendation enhancement: Adding viewed movies to retraining improves both offline metrics and simulated satisfaction, whereas adding unviewed movies typically worsens user experience.These results support agent movie choices as consistent indicators of unique user preferences in feedback-driven recommendation enhancement.
- Case study of feeling interview: Post-exit interviews expose how agents combine personal tastes, social traits, and emotional memory when explaining satisfaction with recommendations.One MovieLens case shows recognition of preference alignment alongside dissatisfaction with recommendations dominated by popular movies.
4 INSIGHTS AND EXPLORATION
Agent4Rec is used to explore whether LLM-based agents can illuminate unresolved recommendation problems, including filter bubbles and causal relationships. Simulations show increasingly centralized movie recommendations and recover several causal patterns involving movie quality, popularity, exposure, viewing, and ratings.
- Agent4Rec is used to investigate whether generative-agent simulation can provide insights into unresolved recommendation problems.
- Filter Bubble Effect: As simulation rounds increase, recommended movies become more centralized: Ngenres decreases while Ptop1-genre intensifies.These metrics represent genre diversity and primary-genre dominance, respectively.
- Filter Bubble Effect: The observed concentration validates Agent4Rec’s ability to reflect the filter bubble effect commonly observed in recommender systems.
- Discovering Causal Relationships: The causal-discovery experiment collects movie quality, popularity, exposure rate, view number, and simulated movie ratings, then applies DirectLiNGAM to learn a weighted DAG.
- Discovering Causal Relationships: The learned graph identifies movie quality and popularity as causes of movie ratings, with quality contributing most and popularity exerting a smaller influence.The passage also connects the popularity effect with humans’ tendency to rate popular movies highly.
- Discovering Causal Relationships: The learned graph also reveals a feedback loop in which highly popular movies receive increased exposure, amplifying popularity bias.
5 RELATED WORKS
Related work covers LLM-empowered agents and recommendation simulators. It distinguishes agents by whether they execute predefined tasks or simulate human behavior, while positioning recommendation simulators as alternatives to online environments with applications including reinforcement learning and recommendation research challenges.
- LLM-empowered Generative Agents: Prior work divides LLM-empowered agents into task-oriented agents and simulation-oriented agents.Task-oriented agents execute predefined human-established tasks, whereas simulation-oriented agents replicate human behavior in specific scenarios.
- LLM-empowered Generative Agents: Simulation-oriented agents support behavioral replication, data acquisition, and exploration of social issues in specific scenarios.
- Recommendation Simulator: Recommendation simulators replicate user interactions without relying directly on real-world user data, offering a cost-effective alternative to online environments.
- Recommendation Simulator: Earlier recommendation simulators primarily supplied simulated interaction data for downstream applications such as reinforcement learning.Examples include Virtual Taobao, RecSim, RecoGym, and MINDSim.
- Recommendation Simulator: Recommendation simulators also offer a potential route for studying challenges such as causal discovery and filter bubbles.
- Recommendation Simulator: Traditional simulators are described as relying on relatively simple rules, which limits their flexibility and validity.
6 LIMITATIONS AND FUTURE WORK
Agent4Rec’s limitations concern its dependence on offline data, a restricted action space, and occasional LLM hallucinations. The authors frame broader data access and richer behavioral factors as directions for further investigation.
- Datasource Constraints: Agent4Rec is constrained by its exclusive reliance on offline datasets, while suitable online data would be more natural but difficult to acquire.LLMs require prior knowledge about recommended items, making datasets with only item IDs or sparse descriptions unsuitable.
- Limited Action Space: Its action space omits influential factors such as social networks, advertising, and word-of-mouth marketing.This simplification supports reliable evaluation in simple scenarios but creates a gap from real-world decision-making.
- Hallucination in LLM: The authors identify occasional hallucinations in LLM outputs as an additional limitation requiring further exploration.
- Future Work: Future work should incorporate a wider range of influential factors to make simulations more representative of recommendation scenarios.