Source-linked AI summary
Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation
Se-eun Yoon, Zhankui He, Jessica Maria Echterhoff, Julian McAuley
TL;DR
The paper asks whether LLM-based synthetic users can represent diverse human behavior in conversational recommendation, an important question because real-user evaluation is costly. It introduces a five-task protocol comparing simulator outputs with human data and finds systematic deviations that prompting and model selection can help reduce. The protocol is a benchmark of necessary conditions rather than a sufficient guarantee of realistic simulation, and its evaluation is limited to movie-domain data.
Problem
The paper addresses the lack of an automatic protocol for evaluating whether LLM-based user simulators represent diverse human behavior in conversational recommendation.
Method
The protocol decomposes evaluation into five independent tasks and compares populations of simulator outputs with curated human data from four platforms.
Results
The tasks reveal that baseline simulators differ from humans in item diversity, preference alignment, request personalization, and feedback coherence.
Takeaways & Limitations
Prompting and model selection can reduce simulator–human gaps, while the tasks provide an automatic benchmark for improving simulator realism.
Takeaways & Limitations
The tasks provide necessary but not sufficient conditions for representing real users, and the study's datasets are limited to movies.
Abstract
from arXiv · showhide
Synthetic users are cost-effective proxies for real users in the evaluation of conversational recommender systems. Large language models show promise in simulating human-like behavior, raising the question of their ability to represent a diverse population of users. We introduce a new protocol to measure the degree to which language models can accurately emulate human behavior in conversational recommendation. This protocol is comprised of five tasks, each designed to evaluate a key property that a synthetic user should exhibit: choosing which items to talk about, expressing binary preferences, expressing open-ended preferences, requesting recommendations, and giving feedback. Through evaluation of baseline simulators, we demonstrate these tasks effectively reveal deviations of language models from human behavior, and offer insights on how to reduce the deviations with model selection and prompting strategies.
1 Introduction
The paper proposes an automatic protocol for evaluating whether LLM-based user simulators represent diverse human behavior in conversational recommendation. Five tasks expose systematic simulator–human discrepancies, while prompting and model selection can reduce some gaps.
- Motivation: Existing synthetic users are often restricted to binary responses, target-item guessing, or predetermined rules and templates.These designs provide simplified, static representations of human users.
- Motivation: Conversational recommendation requires simulators to capture both individually diverse preferences and population-level preference patterns.Users differ in the items they mention, preferences they express, and recommendation requests they make.
- Protocol design: The protocol addresses missing input–ground-truth mappings, free-form textual outcomes, and ambiguous ground truth across conversational trajectories.These challenges make direct behavioral comparison difficult.
- Scope: The tasks reveal simulator distortions but do not guarantee that simulators perfectly represent human users.They measure systematic differences from humans rather than establishing complete realism.
- Findings: Baseline simulators favor popular items, correlate weakly with human preferences, personalize requests poorly, and sometimes produce incoherent feedback.The study also finds that prompting and model selection can reduce these deviations.
- Contribution: The protocol evaluates LLM user simulators through five tasks and comparisons with curated human data.Each task targets a key property of realistic simulation, and outcomes are compared against human data from four platforms.
2 Evaluation Tasks
The evaluation decomposes conversational user simulation into five independent tasks covering item choice, binary and open-ended preferences, recommendation requests, and feedback. Each task compares simulator behavior with corresponding human behavior using task-appropriate representations.
- ItemsTalk: ItemsTalk compares the distribution of items mentioned by simulators with the distribution in real-user conversations.The task covers items discussed to request similar items, express preferences, or chat about an item.
- BinPref: BinPref fixes responses to binary while testing whether simulators reflect human preferences in response to binary questions.The task uses conversational questions such as whether a user enjoyed a movie.
- OpenPref: OpenPref analyzes whether simulators mention the same item aspects and sentiments as real users in open-ended responses.Responses are compared through aspect and sentiment distributions after aspect-based sentiment analysis.
- RecRequest: RecRequest evaluates recommendation requests ranging from general demands to personalized constraints based on preferences and immediate circumstances.The task distinguishes requests from preferences because requests express an immediate recommendation need.
- Feedback: Feedback tests whether simulators generate coherent final judgments about recommendation success.Relevant recommendations and explanations should make acceptance more likely, although real users may not provide explicit feedback.
3 Methods
The protocol evaluates zero-shot LLM user simulators by comparing task-specific outputs from simulator populations with human data, while treating simulator design as a black box. It covers item discussion, preferences, recommendation requests, and feedback through real-world datasets and prompt-based baselines.
- Protocol setup: The protocol treats simulator design as a black box and requires free-form natural-language input, zero-shot task execution, and generated language output.Simulators are evaluated as populations rather than as replicas of a fixed user pool, and are not informed about the tasks or metrics.
- Datasets: Real-world datasets provide human outputs for comparison, with ReDial, Reddit, MovieLens, and IMDB assigned to tasks according to their differing data types.The datasets cover conversations, recommendation requests and comments, ratings, and user-aggregated movie reviews.
- Baselines: Prompt-based baselines use gpt-3.5-turbo, gpt-4, and text-davinci-003, with variants adding demographic information, pickiness personality, or interaction history.Vanilla LLM relies on inherent output variability, while the other variants explicitly condition simulator behavior.
- ItemsTalk: ItemsTalk compares simulator and human item distributions after removing prompted items, summarizing distributional diversity with entropy.The number of mentioned items and, for the interaction-history baseline, the history are determined from dataset entries.
- BinPref: BinPref samples frequent and infrequent movies, collects binary preferences from 100 simulators per movie, and correlates positive rates with average ratings.The design tests whether simulator preferences reflect human preference patterns across movies with different rating frequencies.
- OpenPref, RecRequest, and Feedback: OpenPref compares human and simulator aspect-sentiment distributions, while RecRequest compares request diversity and granularity using lexical and embedding-based measures.Feedback evaluates whether simulators reject negative recommendations and prefer positive recommendations over negative ones.
4 Experiments
Across five tasks, baseline simulators diverge from human behavior in item diversity, preference alignment, preference expression, request personalization, and feedback coherence, while prompting and model selection reduce some gaps.
- ItemsTalk: Simulators mention less diverse items than humans, with distributions skewed toward popular items across baselines and datasets.Interaction-history prompting substantially improves item diversity and can slightly exceed human diversity in some settings.
- BinPref: Most simulators poorly reflect human preferences, but gpt-4 with pickiness is an exception whose positive rates decline with lower human ratings.Higher item frequency does not necessarily improve preference alignment; adding pickiness can produce strong correlations.
- OpenPref: Simulators express more explicit sentiment-associated aspects, repeat predictable aspects, and favor positive sentiment compared with humans.gpt-4 with demographic information and pickiness has aspect and sentiment statistics closest to humans.
- RecRequest: Simulator requests are less diverse than human requests across entropy levels, with gpt-4 producing 23% less diverse requests than humans.Repeated generic expressions make simulator requests less specific, whereas humans express finer-grained criteria.
- Feedback: Feedback is usually coherent, but incoherence ranges from 3% for gpt-3.5-turbo to 35% for text-davinci-003 in the accept/reject task.In comparison tasks, coherence ranges from 65% for text-davinci-003 to 90% for gpt-4; explanations can slightly reduce coherence.
- Feedback: Feedback errors often reflect missed request nuances, such as confusing a movie being about a loner with featuring a loner character.This pattern shows that compelling factual explanations can still accompany semantically misaligned feedback.
5 Related Work
Prior conversational-recommendation simulators used binary responses, target items, fixed rules, or templates, limiting their realism. This work instead positions LLM-based simulators as generative human proxies while addressing the lack of recommendation-specific evaluation protocols.
- Earlier simulators relied on binary responses, predefined target items, or fixed action rules and utterance templates, rather than flexible user behavior.
- LLMs have been explored as human proxies in recommendation and conversational domains, including broader environments with browsing, search, and social-media behavior.
- Existing LLM recommendation work primarily studies LLMs as recommenders, whereas this paper studies them as recommendation seekers.
6 Conclusion
The paper introduces a protocol for evaluating LLM-based user simulators in conversational recommendation through five tasks targeting essential properties of realistic user proxies. Applying the tasks reveals discrepancies from real users and supports a benchmark for automatic simulator evaluation.
- The protocol contains five evaluation tasks, each addressing an essential property required for realistic conversational-recommendation user proxies.
- Running the tasks exposes discrepancies between simulator behavior and real users, making them useful for automatic simulator benchmarking.
- The benchmark is intended to support future efforts to improve simulator realism.
Limitations
The protocol evaluates necessary but not sufficient conditions for representing real users, and its empirical scope is constrained by movie-only datasets and simple OpenAI prompt-based baselines.
- The tasks provide necessary conditions, not sufficient conditions, for simulators to represent a group of real users.Additional tasks could assess recommendation questions or interactions with evolving items outside LLM training corpora.
- Although the approach is domain-agnostic, the datasets cover only movies, so other domains may require domain-specific tasks and produce different results.
- The baseline observations may not generalize to all simulators because the study uses OpenAI models, default temperatures, and simple prompt-based baselines.Further analysis with open-source models, varied hyperparameters, and advanced simulators is left open.
Ethics Statement
LLM simulators may support pre-deployment testing and broader alignment research, but they introduce risks and cannot fully replace human interactions. The study also documents the licensing and usage conditions of its scientific artifacts.
- More realistic simulators may introduce risks of misuse, deception, and overreliance, motivating clear disclosure that they are not human.
- Simulators are valuable for pre-deployment testing but cannot fully replace human interactions, so final testing requires real user experiments.
- The study reports usage conditions for ReDial, MovieLens, Reddit, IMDB, PyABSA, Word2Vec, and SBERT artifacts.ReDial is CC BY 4.0; MovieLens permits research use under conditions; Reddit was used with direct author permission; IMDB is non-commercial; PyABSA, Word2Vec, and SBERT use Apache or MIT licensing.
A.1 Dataset statistics
The study compares simulator behavior with human data across ReDial, Reddit, MovieLens, and IMDB, using dataset-specific samples and processing choices.
- ReDial contains 11,348 conversations and 6,925 movies, with 1,309 movies used for ItemsTalk.
- Reddit is processed into 23,167 movie-recommendation requests, each represented by one sampled head comment, with 9,974 movies used for ItemsTalk.
- MovieLens provides ratings from 162,000 users across 62,000 movies, including samples of 200 frequent and 200 infrequent movies.
- An additional 300 randomly sampled MovieLens movies have very low frequencies, with median frequency 5 and mode 1 appearing 49 times.
- IMDB contributes 22,918 reviews from 1,083 users, including 928 users with at least 11 reviews and 8,138 ItemsTalk movies.
A.2 Prompts
The appendix specifies prompts for simulator variants and evaluation tasks, then illustrates ItemsTalk, preference, and feedback analyses across datasets and examples.
- A.2.1 ItemsTalk: ItemsTalk prompts simulators to generate dataset-conditioned movie mentions, using demographic information, interaction histories, or both as context.
- A.2.1 ItemsTalk: ItemsTalk target length equals the entry’s total movies minus the movies supplied as interaction history, with prompts requiring only formatted movie lists.
- A.2.2 BinPref: BinPref asks whether a simulator liked a watched movie, comparing demographic-only prompts with prompts that randomly assign one of three pickiness levels.
- A.2.3 OpenPref: OpenPref uses analogous demographic and pickiness prompts to elicit bounded free-text opinions, with target review length derived from processed IMDB reviews.
- A.2.4 RecRequest: RecRequest generates recommendation requests containing specified movies and approximately matching real Reddit request lengths.
- A.2.5 Feedback: Feedback evaluates accept/reject judgments and pairwise recommendation choices, randomizing agent assignment to reduce position bias and sometimes requesting reasons.
- A.3.1 Results from ItemsTalk: ItemsTalk results compare simulator and human item distributions across ReDial, Reddit, and IMDB, while BinPref compares average ratings and positive rates using Tables 8 and Figures 12–14.
- A.3.3 More Feedback examples: Feedback examples include a simulator rejecting a show when movies or shows were requested, rejecting a recommendation containing an already watched movie, and choosing an unclear negative recommendation.