Source-linked AI summary
InterviewPlayground: A Simulation Environment for Evaluating AI Interviewers
Jonathan Ivey, Aimee Liang, Arthur Y. S. Wang, Madeline Mandell, Ziang Xiao, Anjalie Field
TL;DR
Evaluating AI interviewers in extended, adaptive interviews is difficult, and human studies can be costly and narrow. InterviewPlayground evaluates them through socially grounded simulated participants and validated measures; its simulated performance predicts human-study evaluations across 12 measures, with an average Pearson correlation of 0.86.
Problem
Method
Results
An average Pearson correlation of 0.86 across 12 measures shows that simulated evaluations predict quantitative AI-interviewer performance in human studies.
Source: §1 · p. 2Takeaways & Limitations
The findings support using the three InterviewPlayground settings to test and evaluate AI interviewers and offer a roadmap for validated simulation-based evaluations.
Source: §1 · p. 2, §6 · p. 9For sensitive topics, simulated participants expressed discomfort more directly than human participants, who often reported it outside the interview.
Source: §5.3 · p. 9
Abstract
from arXiv · showhide
Increasingly, AI interviewers are being developed to elicit open-ended responses in applications like market research, public polling, preference elicitation, and social science research. However, evaluating AI interviewers is challenging because they function in extended, multi-turn interactions where they must adapt to participant behaviors. To address this need, we develop InterviewPlayground, a simulation environment for evaluating AI interviewers using simulated study participants whose behaviors are grounded in social theory. Simulated studies in InterviewPlayground produce an InterviewReportCard, which assesses the performance of AI interviewers using a suite of validated measures. To test whether our simulation-based evaluations predict performance with human participants, we conduct 15 real qualitative studies with five AI interviewers, three interview topics, and 450 human participants and compare them to simulated studies in InterviewPlayground. We find that AI interviewer performance in InterviewPlayground predicts performance in human studies with an average Pearson correlation of 0.86 across 12 measures, and the simulated interactions from InterviewPlayground reproduce key findings from behavioral analysis of AI interviewers in the human studies. Together, these findings support the validity of InterviewPlayground in assessing AI interviewer performance and examining potential failure modes. Our work contributes a simulation environment for AI interviewers supported with empirical validation, and more broadly, a roadmap for future work to develop validated, simulation-based evaluations of conversational AI systems.
1 INTRODUCTION
Evaluating AI interviewers is difficult because extended, adaptive interviews are costly to study with humans, often limiting evaluation to small, single-domain pilots. InterviewPlayground uses socially grounded simulated participants and validated measures, and its evaluations predict human-study performance across 12 measures.
1 INTRODUCTION
- Human-participant evaluation is expensive and time-consuming, so many AI-interviewer studies use small, single-domain pilots that limit generalizability and system comparison.Existing simulations also commonly lack validation with real users, leaving a Sim2Real gap.Source: §1 · p. 2
- InterviewPlayground combines socially grounded simulated participants, three simulation settings, and InterviewReportCard measures of responses, interviewer behavior, and participant experience.The authors compare simulations with 15 real studies involving five AI interviewers, three topics, and 450 human participants.Source: §1 · p. 2
- An average Pearson correlation of 0.86 across 12 measures shows that InterviewPlayground simulations predict quantitative AI-interviewer evaluations in human studies.The authors also report reproducing key behavioral-analysis findings from human studies.Source: §1 · p. 2
2 RELATED WORK
Prior AI-interviewer evaluations often rely on costly, narrow human studies, while existing simulations lack research-grounded participant behavior or validation against human performance. InterviewPlayground instead focuses on evaluating interviewers’ ability to elicit information in a socially grounded setting.
2 RELATED WORK
- Human-based AI-interviewer evaluations are often costly, single-domain studies, limiting generalizability and comparisons across systems.Some basic simulations also lack research grounding and evidence that their evaluations correlate with human-study performance.Source: §2 · pp. 2–3
- InterviewPlayground grounds simulated participant behavior in social theory and compares its evaluations with 15 real studies involving five AI interviewers, three topics, and 450 people.This comparison provides evidence about whether simulation-based performance tracks human-study performance.Source: §2 · p. 3
- Prior participant simulations often predict what information people might provide, whereas InterviewPlayground evaluates interviewers’ ability to elicit existing study information.Its design emphasizes factors in the literature that influence interview process and outcomes.Source: §2 · p. 3
3 INTERVIEWPLAYGROUND
InterviewPlayground models participants using behavior identified in social theory and qualitative-interview research, then lets AI interviewers interact with simulated samples in study-based settings. Its design includes a cognitive participant model, an interview environment, and three settings derived from published studies.
3.1 THEORETICAL GROUNDING
- The literature review identified 20 participant characteristics and consolidated them into six traits: knowledge, understanding, reflexivity, memory, verbosity, and disclosure.The traits describe factors shaping participants’ responses and behavior.Source: §3.1 · p. 3
- The framework leaves hostile non-cooperation and mental and physical impairment unsimulated because these traits are complex, high-stakes, and relatively rare.Source: §3.1 · p. 3
3.2 COGNITIVE MODEL
- Each simulated participant has a persona, trait values, and 200 generated memories; five relevant memories are retrieved for each question, with memory-level noise modeling recall failures.Of the memories, 194 are generated independently of the interview topic and six are conditioned on desired study insights.Source: §3.2 · p. 4
- Experts rated simulated participants as realistic as human participants: 3.51 versus 3.34 on a four-point scale, a difference that was insignificant (p = 0.28, t = 1.08).Experts identified five of six behavioral traits but struggled to identify memory, without access to the participants’ memory banks.Source: §3.2 · p. 4
3.3 SIMULATED INTERVIEW ENVIRONMENT
- The interview environment combines research questions, simulated participants with varied traits, and an interview guide; each participant is interviewed separately until a time limit.Completed transcripts are analyzed with InterviewReportCard.Source: §3.3 · p. 4
3.4 SIMULATION SETTINGS
- The three simulation settings draw on published studies of obesity and weight management, Asian American identity and politics, and generative AI in knowledge work.Each setting uses 30 simulated participants, estimated target-population trait distributions, and insights distributed among participants’ memories; the samples represent populations rather than individuals.Source: §3.4 · p. 5
4 INTERVIEWREPORTCARD
InterviewReportCard assesses interview quality across participant responses, interviewer behavior, and participant experience. Its measures combine response-level and transcript-level judgments, with validation against expert ratings for several measures and human self-reports for experience measures.
4 INTERVIEWREPORTCARD
- InterviewReportCard groups its measures into participant responses, interviewer behavior, and participant experience.The suite draws on interview-quality research and human–computer interaction evaluations.Source: §4 · p. 5, Table 1 · p. 6
Participant Responses
- Relevant Response Volume weights each participant response’s length by its research-question relevance; guide coverage and novel responses measure topic coverage and relevant material outside planned topics.The relevance score is adapted to avoid positively scoring completely irrelevant responses.Source: §4 · p. 5
Interviewer Behavior
- Interviewer-behavior measures score coherence, adaptiveness, leading and unclear questions, and support or rapport, characterizing process rather than prescribing one ideal style.Some behaviors, such as leading questions, may be appropriate under particular interview styles.Source: §4 · p. 6
Participant Experience
- Experience measures assess overall experience and comfort through human self-reports or LLM judgments of simulated-interview excerpts; several other judges show moderate agreement with experts.The experience measures are not validated against third-party judgments; they are compared with participant self-reports.Source: §4 · p. 6
5 EXPERIMENTS
The experiments compare InterviewPlayground with human studies, finding strong quantitative agreement on most measures and similar behavioral patterns, while exposing differences in comfort-related experiences.
5.1 STUDY DESIGN
- Five AI interviewers replicated three published topics in human and simulated studies; 450 people joined human studies, while simulations reused 30 participants per topic.The human studies used samples of 30 participants for each interviewer-topic combination.Source: §5 · p. 7, §5.1 · p. 7
5.2 EXPERIMENT 1: PREDICTING QUANTITATIVE EVALUATIONS OF AI INTERVIEWERS
- 10 of 12 measures correlated strongly (r = 0.74–1); comfort level and overall experience correlated moderately (r = 0.60 and r = 0.67), with simulations underpredicting both.The correlations compare performance averaged across five AI interviewers and three topics in simulated and human studies.Source: Figure 3 · p. 8, §5.2 · p. 8
- Models with as few as 4B parameters achieved an average correlation of 0.83 across all measures, and results were generally robust to simulator-model choice.The study reran 15 simulations using seven simulator LLMs; the authors suggest smaller, less expensive models for initial testing and development.Source: §5.2 · p. 8
5.3 EXPERIMENT 2: REPRODUCING BEHAVIORAL ANALYSIS OF AI INTERVIEWERS
- Repetitive questioning was the most common failure in low-experience human and simulated interviews, with both participant types expressing frustration at InterviewGPT’s repeated questions.The analysis examined the eight lowest-rated human and simulation transcripts for each interviewer.Source: §5.3 · p. 9
- LLMRoleplay achieved the best guide coverage and relevant response volume in both settings, using specific questions, probes for detail, and clear transitions.The authors found similar interaction patterns in samples of 12 human and 12 simulated transcripts.Source: §5.3 · p. 9
- Low-disclosure participants were uncomfortable discussing sensitive topics in detail; simulated participants voiced discomfort more directly, prompting interviewers to move on rather than probe further.Human participants more often reported uncomfortable questions in post-surveys instead of telling the interviewer.Source: §5.3 · p. 9
6 CONCLUSIONS
The conclusion presents InterviewPlayground as an empirically compared environment for evaluating AI interviewers and a roadmap for validated simulation-based evaluations. It also describes the study’s use of generative AI and its human-subject safeguards and data limits.
6 CONCLUSIONS
- InterviewPlayground reflects quantitative and qualitative findings from human studies, and the authors offer its development process as a roadmap for validated simulation-based evaluations.They suggest the findings may support future evaluations of information-elicitation systems in software requirements engineering and psychiatric intake.Source: §6 · p. 9
- The authors used generative AI tools for literature discovery, software development, figures, writing feedback, and persona generation, with authors reviewing identified papers and verifying software and figures.The authors state that they take responsibility for the final content, including work produced with generative AI assistance.Source: AI USE STATEMENT · p. 10
- The human-subject studies involved sensitive topics and included consent, a report button, the option to leave, and IRB approval to mitigate participant risks.Participants were compensated at $15 per hour; transcripts and identifying data are not publicly released to reduce privacy risks.Source: ETHICS STATEMENT · p. 10
- InterviewPlayground and its three simulation settings, along with experiment-reproduction code and procedural details, are released; human interviews can be replicated with new participants, but transcripts are unavailable.The authors cite sensitive human-subject data as the reason transcripts cannot be publicly released.Source: REPRODUCIBILITY STATEMENT · p. 10
A QUALITATIVE-RESEARCH LITERATURE REVIEW
The review grounds the simulated participants in interview research, identifying influential participant behaviors and translating them into prompts for generating memories and interview responses.
A QUALITATIVE-RESEARCH LITERATURE REVIEW
- The review consolidated 20 influential characteristics into six traits—knowledge, understanding, reflexivity, memory, verbosity, and disclosure—while excluding hostile non-cooperation and impairment.The authors cite their complexity, high stakes, and relative rarity as reasons for excluding those two traits.Source: A QUALITATIVE-RESEARCH LITERATURE REVIEW · p. 14
B.1 MEMORY GENERATION PROMPTS
- Each participant receives two memory-generation prompts: one for 194 unrelated background memories and one for six memories directly informative to the study target.The study-related prompt covers one memory per topic; memories are labeled for reflexivity and sensitivity.Source: §B.1 · p. 15
B.2 PARTICIPANT RESPONSE GENERATION
- For each interviewer question, the simulated participant responds using the five most relevant retrieved memories and trait-conditioned guidance for realistic, disfluent speech.The response guidance discourages polished summaries and allows hesitations, incomplete thoughts, and occasional drift to minor details.Source: §B.2 · pp. 16–17
B.3 EXPERT RATINGS OF REALISM AND TRAIT PRESENCE
Expert ratings suggest that participants’ assigned traits are generally recognizable, while memory is harder to identify; simulated participants receive realism ratings comparable to human participants.
B.3 EXPERT RATINGS OF REALISM AND TRAIT PRESENCE
- Three qualitative-research annotators rated excerpts from 50 human and 50 simulated interviews, including trait levels and realism.The interviews were randomly sampled, and annotators saw unlabeled transcripts in random order.Source: §B.3 · p. 18
- Experts identified five traits—knowledge, understanding, reflexivity, verbosity, and disclosure—but struggled to identify memory; Table 3 reports agreement measures.Annotators did not have access to participants’ memory banks.Source: §B.3 · p. 18, Table 3 · p. 18
B.4 ESTIMATING INTERVIEW TIME
InterviewPlayground estimates elapsed interview time from interviewer generation time and estimated participant speaking time, with speaking rates and delays adjusted by verbosity.
B.4 ESTIMATING INTERVIEW TIME
- Elapsed time combines the interviewer’s wall-clock question-generation time with estimated participant response time based on syllables and a pre-speech delay.Source: §B.4 · p. 19
- For low-, medium-, and high-verbosity responses, estimated speaking rates are 167.7, 189.5, and 216.1 syllables per minute, with delays of 11.3, 16, and 26.6 seconds.Source: §B.4 · p. 19
- The estimates use 16,676 human responses, grouped into three length categories with k-means, followed by Theil-Sen regression for each group.Source: §B.4 · p. 19
B.5 ESTIMATING TRAIT PROPORTIONS
This material describes how simulated participants’ trait proportions are assigned and how LLM-judge prompts operationalize interview coverage, interviewer technique, and participant experience ratings.
B.5 ESTIMATING TRAIT PROPORTIONS
- Each setting uses 30 simulated participants whose trait distributions are estimated from self-reports for the target population, with verbosity estimated by response-length clusters; traits are assigned independently.Constrained random assignment makes the simulated sample proportional to the estimated trait values.Source: §B.5 · p. 19
C.1 LLM JUDGE PROMPTS
- The coverage judge labels which interview-guide subtopics each participant response actually addresses, including none when no subtopic is addressed.Source: §C.1 · p. 20
- Technique prompts rate coherence and adaptiveness, distinguishing logical question flow from follow-up that engages with participants’ specific answers.Source: §C.1 · p. 20
- Other prompts classify leading, supportive or rapport-building, and unclear interviewer utterances using wording- and structure-based criteria.Source: §C.1 · p. 21
- Participant-perspective prompts rate comfort on a 1–4 scale and overall interview experience on a 1–5 scale.Source: §C.1 · pp. 21–22
C.2 HUMAN VALIDATION
Human expert ratings check InterviewReportCard’s LLM judges, while interview-guide coverage is compared with human-identified topics. Overall experience and comfort are instead checked against participant self-reports.
C.2 HUMAN VALIDATION
- The validation used 1,800 expert judgments from three qualitative-research annotators rating excerpts from both human and simulated interviews.Each annotator rated 50 excerpts from each interview type.Source: §C.2 · p. 22
- Human annotators showed moderate agreement, and median-human versus Gemini 3.1 Pro ratings showed comparable agreement across five interview-behavior measures.The measures were coherence, adaptiveness, leading questions, support and rapport, and unclear questions.Source: Table 4 · p. 22, §C.2 · p. 22
- LLM-identified interview-guide topics had an average Jaccard similarity of 0.69 with topics identified by human annotators.Source: §C.2 · p. 22
- Overall experience and comfort were not validated against third-party judgments; instead, they were compared with participant self-reports.Source: §C.2 · p. 22
D EXPERIMENT 1 DETAILS
The experiment details include correlations between simulated and human studies matched by interviewer and topic, and correlations across seven simulator models.
D EXPERIMENT 1 DETAILS
- Table 5 reports Pearson correlations between InterviewPlayground simulations and human studies matched by AI interviewer and interview topic.Source: Table 5 · p. 23
- Table 5 also reports Spearman correlations for the same matched simulated and human studies.Source: Table 5 · p. 23
- Table 6 reports correlations across the 12 measures using seven different LLMs as simulator models.Source: Table 6 · p. 23