PaperlayerView exact revision on arXiv

Source-linked AI summary

Can AI interview? Comparing AI-led to Human-led Chat Interviews

Semra Guven, Tobias Gårdhus, Andreas Bjerre-Nielsen, Anna Rogers, Hjalmar Bang Carlsen

arXiv:2610.23149v1cs.CYcs.CLcs.HC

TL;DR

Qualitative interviews offer depth, but collection costs have traditionally limited sample sizes, and direct empirical comparison of AI-led and human-led interview quality was lacking. The authors compare the approaches in chat interviews and find AI-interviewers produce consistently on-topic responses and collect more material and answered questions per minute. These findings support AI-interviewing for large-N qualitative research in the studied non-expert, chat-based setting.

  • Problem

    Qualitative data collection costs have traditionally limited interview sample sizes, and no systematic empirical study had directly compared data quality in AI-led and human-led interviews.

    Source: Introduction · p. 3

  • Method

    The study directly compared AI-led and human-led interviews in a non-expert, chat-based setting and assessed material yield per unit of interview time.

    Source: Measures of Data Quality · p. 12, Conclusion · p. 23

  • Results

    AI-interviewers produced responses as consistently on-topic and of specificity indistinguishable from trained human interviewers, while collecting more material and answered questions per minute.

    Source: Conclusion · p. 23

  • Takeaways & Limitations

    The findings support the viability of AI-interviewing for large-N qualitative data collection in sociological research.

    Source: Conclusion · p. 23

    Five participants skipped questions in the AI condition, and the study could not distinguish why they did so.

    Source: Scope and Interpretation · p. 20

Abstract

from arXiv · show

Large language model chatbots have been proposed as AI-interviewers for collecting qualitative interview data at scale, potentially bridging social science's longstanding divide between the breadth of surveys and the depth of qualitative interviews. This paper provides the first direct empirical comparison of data quality in AI-led and human-led interviews. In a randomized experiment (N = 41), university students were interviewed via synchronous text chat by either trained non-expert interviewers or a locally hosted AI system; we evaluate exposure, specificity, and relevance. Median interview duration was similar across conditions, but AI-interviewers asked more than twice as many questions per interview and elicited roughly 65 percent more respondent text per interview. Per-question responses were longer with human interviewers, though not after multiple-testing correction; only AI-condition participants skipped questions. Responses were almost uniformly on-topic in both conditions; for specificity we detect no difference, but the design cannot exclude one of up to three quarters of a point. These results were obtained with a small, locally hosted open-weight model (7B parameters), which addresses concerns about data security, GDPR compliance, and scientific reproducibility. Our findings support the viability of AI-interviewing for large-N qualitative data collection.

Introduction

Qualitative interviews provide depth, but their cost has traditionally limited sample sizes, and direct empirical evidence comparing AI-led with human-led interview quality was lacking. This study compares the two in randomized chat interviews and finds AI interviews maintain on-topic responses while collecting more material per minute.

Introduction

  • A critical prerequisite for AI-interviewing’s promise of large-scale qualitative data collection is that its data quality be sufficient relative to human interviews.Traditional collection costs have limited qualitative sample sizes, hindering population-level inference.Source: Introduction · p. 3
  • The randomized experiment assigned 41 participants to human-led or AI-led synchronous chat interviews using the same interface, topic, and guide.Data quality was evaluated across exposure, specificity, and relevance.Source: Introduction · pp. 3–4
  • More than twice as many questions per minute and substantially more respondent text per minute were collected by AI-interviewers, while responses were overwhelmingly on-topic and specificity showed no detectable difference.Median interview duration was similar across conditions.Source: Introduction · p. 4
  • The study used a locally hosted, open-weight 7B-parameter model and reports acceptable interview performance while addressing concerns about data security, GDPR compliance, and reproducibility.The authors describe this as their first AI-interviewer solution based on a relatively small open-weight model.Source: Introduction · p. 4

System Design: Local, Open, and Restricted

The system uses a smaller, locally hosted open-weight model and limits the AI-interviewer to follow-up probes on researcher-defined questions. Its design separates interview-flow decisions, question reformulation, and probing into distinct agents.

System Design: Local, Open, and Restricted

  • The system differs from prior implementations by using a smaller, locally hosted open-weight model and deliberately constraining the AI-interviewer’s degrees of freedom.The section describes the rationale for these choices and the resulting system.Source: System Design: Local, Open, and… · p. 4

The Case for Local, Open-Weight Models

  • Locally hosted models keep sensitive interview data under researcher control, ease GDPR compliance, and support reproducibility by avoiding opaque, changing commercial APIs.A stable local model can be shared and reused to replicate interview rounds and study designs.Source: The Case for Local, Open-Weight Models · p. 5

System Architecture

  • Three agents divide interview tasks: classification controls flow and avoids repeated questions, reformulation incorporates prior-answer context, and probing generates responsive follow-ups.The probing agent is identified as the central component and the key advantage over static survey instruments.Source: System Architecture · p. 6, Figure 1 · p. 7
  • The AI-interviewer generates follow-up probes only in response to predefined open-ended main questions, keeping the interview anchored in researcher-justified questions.The agents ran locally on OpenHermes 2.5, an instruction-tuned version of Mistral 7B.Source: System Architecture · p. 6

Experimental Design

The study randomized participants to AI- or human-led synchronous chat interviews under a shared interface and guide, then operationalized data quality across exposure, specificity, and relevance.

Treatment: AI or Human-Led Chat Interviews

  • The shared interface and identical instructions held the interview mode constant, leaving interviewer identity as the primary difference between conditions.Participants were told their assignment before the interview, which may have affected disclosure, elaboration, register, or effort.Source: Treatment: AI or Human-Led Chat… · p. 8

Interview Guide and Instructions

  • The guide moved from concrete help-seeking experiences to feelings and reflections, using screening questions and six open-ended questions per section; the third case was excluded from analyses.The third case was dropped after the first three days because human interviewers rarely reached it in the half-hour slots.Source: Interview Guide and Instructions · pp. 9–10

Interviewers, Setting, and Recruitment

Measures of Data Quality

  • Exposure included response length, total text, answered questions, and per-minute rates; specificity and relevance were coded on 1–5 scales.The rate measures capture material yielded per unit of researcher time and do not require equal interview lengths.Source: Measures of Data Quality · p. 12
  • Specificity coding had ordinal Krippendorff’s α of 0.65, below the 0.667 threshold; relevance’s α of 0.37 reflected little variance from near-universal top scores.The authors interpret relevance as establishing that responses were overwhelmingly on-topic, not as distinguishing between conditions.Source: Measures of Data Quality · p. 13

Statistical Analysis

  • For similarity claims, the authors report Hodges-Lehmann shifts and distribution-free confidence intervals; they also use MWU tests with Holm adjustment and OLS checks.The interval bound farthest from zero identifies the smallest difference the design can exclude.Source: Statistical Analysis · pp. 13–14

Results

AI-led interviews generated more questions and respondent text, especially per minute, while the conditions showed no detected difference in specificity or relevance. The specificity result remains uncertain, and relevance was measured with a coarse on/off-topic distinction.

Exposure

  • Human-led interviews elicited 150 characters per answer versus 112 for AI-led interviews, a 34 percent difference that was not significant after correction.Human-led answers also varied more in length.Source: Exposure · p. 14
  • AI-led interviews averaged 3,244 characters and 31 questions, versus 1,968 characters and 14 questions for human-led interviews; seven AI interviews reached the guide’s 44-question limit.The total-character difference was 65 percent, and the question count partly reflected the researcher-set probe budget.Source: Exposure · p. 15
  • Median interview duration was 28.9 minutes for AI-led and 28.3 minutes for human-led interviews, but the interval allows an AI advantage of up to 12.8 minutes.The authors therefore report subsequent exposure measures as rates normalized by interview duration.Source: Exposure · p. 15
  • AI-interviewers obtained 1.18 answers per minute versus 0.54 and 106 respondent-text characters per minute versus 72, yielding roughly 48 percent more material per minute.They also produced more than twice as many answered questions per minute.Source: Exposure · p. 15
  • AI-condition participants used the skip button 22 times across five interviews, while no human-condition participant skipped; counting skips as zero-length answers did not change conclusions.The reported skip comparison was uncorrected and outside the family of nine comparisons.Source: Exposure · pp. 15–16

Specificity and Relevance

  • Mean relevance was 4.9 in both conditions, while specificity averaged 3.8 for human-led and 3.6 for AI-led interviews; neither difference was significant after correction.Relevance scores were near the scale ceiling, and the measure registered only the coarse distinction between on- and off-topic answers.Source: Specificity and Relevance · pp. 16–17
  • For specificity, the estimated human advantage was 0.16 points, but the interval still permits a difference of about half a point on the five-point scale.Both conditions had difficulty eliciting developed, experientially grounded accounts, potentially limiting the measure’s sensitivity.Source: Specificity and Relevance · pp. 16–17

Discussion

In structured, chat-based interviews with non-expert interviewers, AI responses were as on-topic and not detectably less specific, while offering greater scalability. The comparison has limits: skipped-question explanations are unresolved, qualitative quality measures are difficult to standardize, and open-weight models retain transparency constraints.

Scope and Interpretation

  • In large-scale structured-guide settings, AI responses were as on-topic and not detectably less specific than non-expert human responses, while offering greater scalability.The authors do not present this comparison as evidence that AI can replace expert qualitative interviewers.Source: Scope and Interpretation · pp. 19–20
  • The chat setting lacked social cues and rapport-building introductions, which likely disadvantaged human interviewers; the authors call for face-to-face comparisons.They hypothesize that human rapport advantages may be substantial face-to-face.Source: Scope and Interpretation · p. 20
  • With five participants skipping questions, the study could not distinguish among plausible explanations for why skips concentrated in the AI condition.Possible explanations include lower social cost, treating the interview more like a survey, or repetitive later probes.Source: Scope and Interpretation · p. 20

Challenges in Measuring Qualitative Data Quality

  • Character count is only a proxy for depth, and question-answer coding omits conversational context, complicating quantitative comparison of qualitative quality.The paper also notes that qualitative standards lack consensus and that existing criteria have been contested.Source: Challenges in Measuring Qualitative… · pp. 20–21
  • AI-interviewing creates opportunities for experimental qualitative-methods research through large transcript corpora and systematic manipulation of interview parameters.These directions were previously impractical, according to the authors.Source: Challenges in Measuring Qualitative… · p. 21

Open-Weight Models and the Future of AI-Interviewing

  • Locally hosted open-weight models keep data under researcher control, support reproducibility, reduce environmental impact, and avoid dependence on commercial providers.The paper presents them as a viable alternative for research groups facing GDPR constraints and commercial-API infrastructure costs.Source: Open-Weight Models and the Future of… · pp. 21–22
  • Open-weight models are not fully transparent when the base model’s training data are undisclosed; the authors advocate shared development resources and task-specific fine-tuning.They used OpenHermes 2.5 because it performed adequately and its fine-tuning data are published, although the underlying Mistral 7B training data are not; they released prompts, the interview guide, and analysis code.Source: Open-Weight Models and the Future of… · p. 22

Conclusion

The study offers a direct comparison of AI-led and human-led chat interviews and supports AI-interviewing for large-N qualitative research. It also emphasizes locally hosted models and shared quality measures as priorities for responsible, cumulative development.

Conclusion

  • Within non-expert, chat-based interviews, AI responses were as consistently on-topic and not detectably less specific, while collecting more material and more answered questions per minute.The authors conclude that these findings support AI-interviewing for large-N qualitative data collection in sociology.Source: Conclusion · p. 23
  • Locally hosted open-weight models address concerns associated with commercial APIs, while shared qualitative data-quality measures are needed for cumulative system improvement.The authors call for AI-interview systems to be evaluated on ethical and epistemological grounds and for collaboration between qualitative methodologists and computational researchers.Source: Conclusion · p. 23

Data Availability

Study materials are available through the project repository, with raw interview transcripts and respondent-level demographic characteristics withheld to protect confidentiality.

Data Availability

  • The repository provides aggregated behavioral metrics and coded relevance and specificity scores, but not raw interview transcripts or respondent-level demographics.The authors withhold those materials because of confidentiality commitments and re-identification risk in the small sample.Source: Data Availability · pp. 23–24
  • Coding data include independent coder scores and pair-level scores linked to interview identifiers, without response text.The repository is available at the project’s GitHub repository.Source: Data Availability · pp. 23–24
  • The repository includes the interview guide, AI system prompts, design data, analysis code, and derivation and verification scripts.The scripts reproduce reported comparisons and check manuscript quantities against the source.Source: Data Availability · p. 24

Ethics Declaration

The study received institutional review board approval, followed applicable institutional guidelines, and obtained informed consent from all participants.

Ethics Declaration

  • Participants were informed of the study purpose, voluntary participation, and confidential handling of responses with personally identifiable information removed from publications.The authors report institutional review board approval and compliance with relevant institutional guidelines and regulations.Source: Ethics Declaration · p. 25

A Coding Guidelines

The coding guidelines define separate five-point assessments for response specificity and relevance. Specificity concerns detail sufficient for understanding and analytical insight, while relevance concerns alignment with the question.

A Coding Guidelines

  • A first author and student research assistant independently coded a random sample of 400 question-answer pairs, averaging their scores before calculating interview-level means.After excluding the third case, 395 pairs entered the analysis.Source: A Coding Guidelines · p. 30

Specificity

  • Specificity was assessed by whether responses offered enough detail for understanding and in-depth analytical insights.The scale runs from 1 (very vague) to 5 (very specific and comprehensive).Source: Specificity · p. 30

Relevance

  • Relevance was assessed by how well each response aligned with the question and engaged with its substance.The scale runs from 1 (completely off-topic) to 5 (completely relevant).Source: Relevance · pp. 30–31

B Sample Composition and Regression Results

The regression analyses corroborate the non-parametric findings: AI-led interviews were longer and included more questions, while individual answers were shorter, with no significant differences in specificity or relevance. The estimates account for day-level randomization and the measured sample characteristics were balanced across conditions.

Sample Composition

Regression Results

  • Model 2 adds interview-day fixed effects and is the design-supported estimate because randomization occurred within days.Days with only one condition, including the first data-collection day, do not contribute to its coefficient; it is identified from 35 interviews on seven days.Source: B Sample Composition and Regression… · pp. 32–33
  • Neither specificity nor relevance differed significantly; individual answers were shorter with AI interviewers, but response-length findings were not significant after multiple-testing correction.The day-fixed-effects response-length coefficient was −40.25 (p = 0.037), versus −38.14 (p = 0.061) without day fixed effects; these p-values are unadjusted.Source: B Sample Composition and Regression… · p. 33, Table 5 · p. 34
  • Respondent-characteristic adjustments were omitted because randomization did not require them for identification and the small sample would leave roughly a dozen residual degrees of freedom.The proposed adjustment included age, gender, region, programme, degree level, and prior ChatGPT use.Source: B Sample Composition and Regression… · p. 33

C Robustness to the Day Structure

Robustness checks preserving day structure do not change the main conclusions: exposure differences remain significant, while duration, response length, specificity, and relevance remain non-significant. Excluding the first day makes the specificity equivalence bounds less favorable to AI interviewers.

Robustness to the Day Structure

  • The first day could confound pooled comparisons because it included only human-led interviews and those respondents had shorter answers and lower specificity than later human-led participants.First-day versus later-day human interviews averaged 98 versus 162 characters per answer and 3.21 versus 3.88 in specificity.Source: C Robustness to the Day Structure · p. 34
  • The checks compare conditions within days using van Elteren tests and repeat Mann–Whitney tests after excluding day 1; p-values are Holm-adjusted across nine comparisons.The within-day test uses the 35 interviews on days with both conditions; the day-1-excluded analysis uses 37 interviews.Source: C Robustness to the Day Structure · pp. 34–35
  • Exposure differences remained significant under both robustness checks, while duration, response length, specificity, and relevance remained non-significant.The specificity interval without day 1 was −0.94 to 0.21, so a human advantage of almost a full scale point could no longer be excluded.Source: C Robustness to the Day Structure · p. 35, Table 6 · p. 35

D Interview Guide

The interview guide investigates university students’ academic help-seeking through experiences of receiving help and of needing help without asking. It uses open-ended questions and bounded probes, then closes by inviting participants to add perspectives.

Framing (for model use only)

  • The guide frames help-seeking around two cases: receiving needed help and needing help without asking, with attention to social processes, context, feelings, attitudes, and behavior.The full machine-readable guide is available in the data repository.Source: D Interview Guide · p. 35, Framing (for model use only) · p. 36

Introduction (shown to participants)

  • Participants were told the study concerned academic help-seeking, that there were no right or wrong answers, and that answers should be sent in one message.The introduction also stated that responses would remain confidential and identifying information would be removed from publications.Source: Introduction (shown to participants) · p. 36

Section 1: Receiving Help

  • The receiving-help section asks about a specific episode, the help needed and obtained, whom the student approached or considered, feelings afterward, and general help-seeking patterns.Questions include bounded probes about relationships, decision factors, and reasons for seeking help.Source: Section 1: Receiving Help · p. 36

Section 2: Not Asking for Help

  • The not-asking section explores a situation where needed help was not requested, why the student refrained, how the issue was resolved, and feelings about the situation.Probes cover whom the student considered asking and what conditions or university changes might support help-seeking.Source: Section 2: Not Asking for Help · p. 37

Closing

  • The guide also asks generally when or from whom students refrain from seeking academic help and ends by inviting anything they would like to add or perspectives missed.The final open invitation is presented before the interview concludes.Source: Section 2: Not Asking for Help · p. 37, Closing · p. 37
Loading 2610.23149v1…