The Interviewer's Perspective: Unpacking the Impact of Real-Time AI Interviewing Assistance on Social Dynamics

1 Introduction

• Semi-structured interviewing is cognitively demanding, and prior work left unclear how interviewers negotiate AI’s social and professional tensions during live interviews.Interviewers must interpret responses, formulate follow-up questions, and manage conversational flow; prior research had focused on task support and risks rather than the triadic interaction.Source: §1 · p. 2
• Eighteen participants each completed three simulated online interviews: one without AI and two using ProbeAssist variants that differed in initiation and suggestion granularity.The restrained variant provided on-demand brief keywords, while the expressive variant proactively suggested fully formulated questions.Source: §1 · p. 2
• Interviewers valued AI assistance for probing but had diverse, context-dependent preferences, while navigating tensions around agency, creativity, ownership, cognitive costs, and interpersonal communication.The study derives implications around mitigating social pressure, balancing idea alignment with novel inspiration, and preserving interpersonal presence.Source: §1 · pp. 2–3
• The paper presents ProbeAssist as a high-fidelity, open-source prototype for real-time AI assistance in online semi-structured interviews.The study uses comparative structured observation, a qualitative-first approach intended to deepen understanding across design variants rather than evaluate particular prototypes.Source: §1 · pp. 2–3

2 Related Work

• AI for qualitative research spans study design, post-hoc analysis, and data collection, with this study situated in emerging real-time interview assistance.The section frames the flexible and socially situated demands of semi-structured interviews as an open design challenge.Source: §2 · p. 3, §2.2.2 · p. 5

2.1.1 AI for Qualitative Study Design and Analysis.

• AI supports qualitative study design and post-hoc analysis, but these systems often assume time for deliberation, limiting suitability for real-time interviewing.Reported risks include misinterpretation, over-trust, and researcher deskilling.Source: §2.1.1 · p. 3

2.1.2 Synchronous AI support during data collection.

• Live AI assistance must fit interviewers’ attention to interviewees and conversational flow; processing delays can make suggestions obsolete or disrupt the rhythm.Prior work found cognitive-load benefits from task support but also limitations in the actionability of proactive suggestions.Source: §2.1.2 · p. 3

2.2.1 Design factors in human-AI collaboration.

• Initiation and suggestion granularity offer complementary design choices: on-demand versus proactive support, and brief keywords versus fully formulated output.More detailed suggestions can reduce user effort, while less detailed ones leave more room for human interpretation and formulation.Source: §2.2.1 · p. 4

2.2.2 Social dynamics of human-AI collaboration.

• AI in interviews joins an existing interviewer–interviewee relationship, and disclosure can be contested because revealing AI may alter the interaction and collected data.This differs from triadic contexts that assume both human parties know AI is involved.Source: §2.2.2 · p. 5
• How interviewers experience AI-assisted versus unassisted semi-structured interviews, negotiate agency and ownership, and perceive AI’s social role in the triad remains unclear.The authors designed two assistance variants informed by initiation and granularity to surface these experiences.Source: §2.2.2 · p. 5

3 Designing ProbeAssist

• ProbeAssist is a high-fidelity prototype for real-time AI assistance in semi-structured interviews, developed through a user-centered process and implemented in restrained and expressive variants.The process involved brainstorming and design critiques informed by four design considerations.Source: §3 · p. 5

3.1 Design Considerations

• The design considerations call for in-the-moment probing, contextual relevance, low cognitive and information load, and interviewer control.The considerations were derived from literature on qualitative methods and human-AI collaboration.Source: §3.1 · pp. 5–6

3.2 Prototype Design

• Brainstorming with 7 researchers and critique with 30 researchers and students helped prioritize features for exploration rather than select one optimal design.The team then clustered and merged features into two variants differentiated by initiation and assistance granularity.Source: §3.2 · p. 6

3.2.1 Core Features.

• The core features combine a dual-panel display, on-demand or proactive initiation, and transcript-grounded keyword or phrase suggestions.Keywords summarize responses and offer probing directions; phrases provide ready-to-use follow-up questions.Source: §3.2.1 · p. 6

3.2.2 Two Variants of AI Assistance.

• The restrained variant pairs on-demand keyword-level assistance with interviewer interpretation, while the expressive variant proactively offers ready-to-use follow-up questions.Other combinations, such as on-demand phrases or proactive keywords, remain possible for future exploration.Source: Table 1 · p. 6, §3.2.2 · p. 7

3.2.3 Interaction Flow.

• Before an interview, the interviewer uploads the guide and research objectives, selects a variant, and activates AI support; during the interview, suggestions can be accepted, adapted, or dismissed.Afterward, the prototype generates a downloadable package with the recording, transcript, and AI usage log.Source: §3.2.3 · p. 7

3.3 Prototype Implementation

• ProbeAssist uses WebRTC for video, Firebase Firestore for session management, and OpenAI’s Realtime API for transcription and analysis of transcripts against a structured interview guide.The authors state they will release the prototype code as open source.Source: §3.3 · pp. 7–8

4 Methods

• A qualitative-first comparative structured observation study examined nuanced experiences with unassisted and AI-assisted interviews, prioritizing debriefing interviews over prototype performance evaluation.Survey and log measures complemented the qualitative analysis.Source: §4 · p. 9

4.1.1 Simulated interviews.

• Participants conducted three simulated online interviews using similarly structured guides on time management, team projects, and supervisory relationships with one trained research assistant as interviewee.Using one role-player helped keep interactional context consistent and reduced risks of processing real interviewees’ sensitive information with an LLM.Source: §4.1.1 · p. 9

4.1.2 Procedure.

• Each approximately 90-minute session included onboarding, three 10–15 minute interviews, and a 30-minute debrief; the unassisted interview came first and the two AI variants were counterbalanced.Participants completed brief surveys after interviews and reflected in the debrief on experience, data richness, cognitive load, control, and future design requirements.Source: §4.1.2 · pp. 9–10, Figure 4 · p. 10

4.2 Participants

• The sample comprised 18 participants with prior semi-structured interviewing experience, recruited from a university and the broader qualitative research community.The authors did not aim for full representativeness, though the sample varied in gender, research background, interviewing expertise, and AI experience.Source: §4.2 · p. 10

4.3 Data Collection and Analysis

• The first author conducted inductive reflexive thematic analysis of cleaned interview transcripts, discussing emerging codes and themes with co-authors.Recordings and post-interview surveys supplied supporting behavioral and quantitative measures.Source: §4.3 · pp. 11–12

4.4 Positionality Statement

• The authors describe disciplinary perspectives and the first author’s positive outlook on AI as positional factors that may influence the research process.They state that the authors’ varied backgrounds brought multiple disciplinary perspectives to study design and analysis.Source: §4.4 · p. 12

5.1.1 Usage and perceptions.

• Participants asked more total follow-up questions with both AI variants than without AI (M_R=9.83, M_E=8.72 vs M_N=5.83; p=.005 for both comparisons).The increase was additive: participants incorporated or adapted about three AI-assisted questions per session alongside self-formulated questions. Interviews were also longer with the restrained (13.76 min) and expressive (12.65 min) variants than without AI (10.12 min).Source: §5.1.1 · p. 12, Table 3 · p. 13
• Participants rated interview quality higher with both AI variants (M_R=6.04, M_E=5.90) than without AI (M_N=5.56), and reported high control with both variants.AI impact ratings were above 5.14 out of 7; self-reported cognitive load did not significantly differ across the three interviews.Source: §5.1.1 · p. 12

5.1.2 Preferences.

• Preferences depended on context: on-demand assistance offered control, proactive assistance convenience, while keywords preserved personal style and ready-to-use phrases reduced formulation effort.Some participants wanted to combine keywords with full questions, so they could quickly choose whether to read the suggested phrasing.Source: §5.1.2 · p. 13

5.2.1 Defining boundaries between wanted and unwanted AI roles.

• Participants wanted AI as a supportive tool or subordinate assistant, but rejected it as an assessor, competitor, or equal co-interviewer that could undermine their judgment and control.Assessment could prompt distraction and self-doubt; strong competing suggestions could threaten professional identity, while AI directing the interview or addressing the interviewee was seen as violating interviewer responsibility.Source: §5.2 · p. 13, §5.2.1 · pp. 14–15

5.2.2 Building trust through transparency and customization.

• Participants sought transparency into why suggestions were made and customization that aligned AI behavior with their individual interviewing practices.They said inspecting reasoning during a live interview was difficult, and envisioned training or feedback mechanisms that would gradually teach AI their preferences and build trust.Source: §5.2.2 · pp. 15–16

5.3 Theme 2: Balancing AI’s Benefits Against its Cognitive and Social Costs

• AI offered a cognitive safety net, reassurance, and occasional novel insights, but useful suggestions could demand effort to evaluate and compete with interviewers’ ongoing thinking.Novel suggestions were valued particularly in unfamiliar domains, yet participants still had to filter them using their own judgment; high-quality alternatives could create context switching and lingering uncertainty.Source: §5.3.1 · pp. 16–17, §5.3.2 · p. 17

5.3.3 The magnified social cost of latency in an interpersonal context.

• Even short AI delays could become socially awkward, dividing attention between the system and interviewee, disrupting rapport, or making a suggestion unusable after the conversation moved on.System generation time averaged 3.3 s (SD=6.7, range 0.24–5.5); participants described awkward silence and pressure to manage pauses while maintaining the interviewee-facing interaction.Source: §5.3.3 · pp. 17–18

5.4.1 The agency paradox: gaining more control by giving away some.

• Participants viewed control over the interview as non-negotiable, while the section frames an agency paradox in which delegating operational tasks could increase strategic control.They linked control to accountability for data quality and interviewee well-being, and said they would prioritize their own judgment regardless of how much work AI removed.Source: §5.4 · p. 18, §5.4.1 · p. 18
• Delegating operational tasks could free attention for strategic and interpersonal flow, but interviewers who saw those tasks as integral instead felt less in control or more scattered.The tension depended on how interviewers understood the tasks, not simply on who performed them.Source: §5.4.1 · p. 18
• Participants wanted to retain the final call and choose when to apply AI assistance, keeping themselves at the center of the interaction.P17i likened AI to a slight nudge while the interviewer kept the steering wheel.Source: §5.4.1 · p. 19

5.4.2 Creativity and ownership: protecting professional identity while using AI assistance.

• AI suggestions could spark new directions and questions, yet also foster dependency, constrain departures from protocol, and make interviews feel less fun.Some participants valued AI as an externalized creative brain, while others felt more creative without it.Source: §5.4.2 · p. 19
• Ready-made suggestions could obscure interviewers’ emerging ideas and raise concerns about losing professional identity, ownership of “my work,” and homogenizing interviewing.Participants worried that AI could dictate interviews and displace the interviewer as the primary researcher.Source: §5.4.2 · p. 19
• Interviewers addressed ownership tensions by asserting responsibility for chosen questions, framing AI use as collaboration, or seeking customization that made AI think more like them.AI involvement could also diffuse some psychological responsibility when a suggested question backfired.Source: §5.4.2 · p. 19

6 Discussion

• AI’s cognitive offloading is necessary but insufficient: successful integration also requires navigating tensions around agency, ownership, and interpersonal communication.Source: §6 · p. 20

6.1 Mitigating Social Pressure

• AI can shift from assistant to competitor or assessor, while visible friction may lead interviewees to perceive interviewers as inattentive or less competent.Source: §6.1 · p. 20
• In triadic interactions, risk includes damage to professional identity or rapport, even when AI assistance succeeds at the task level.Source: §6.1 · p. 20

6.2 Navigating the Spectrum from AI as a Second Self to a Second Mind

• Interviewers value both AI that follows their thinking and AI that introduces novelty, but alignment can reinforce assumptions while divergent ideas can displace their own.Source: §6.2 · p. 21
• Because interview needs shift in real time, assistance should allow low-interruption adjustments between closely aligned support and novel ideas.Source: §6.2 · p. 21

6.3 Enhancing Interpersonal Presence; 6.4 Limitations and Future Work

• AI’s value depends on whether it frees attention for interviewees; this study focused on online interviews and interviewer perspectives, using one role-playing interviewee across sessions.Source: §6.3 · p. 22, §6.4 · p. 22

• Participants used AI as support but resisted it as an assessor or competitor, highlighting tensions around agency, ownership, creativity, and interpersonal communication.Source: §7 · p. 23
• The study compared no-AI interviewing with two ProbeAssist variants in a comparative structured observation study.Source: §7 · p. 23
• The proposed implications are to mitigate social pressure, enable shifts between aligned and exploratory assistance, and preserve interpersonal presence.Source: §7 · p. 23

• The interview aims to understand graduate students’ time-management strategies, challenges, and adaptations across time.Source: §A.1 · p. 26
• Questions ask students to describe their time-management experience and the methods or tools they typically use.Source: §A.1 · p. 26
• The guide asks about a particularly difficult time-management experience and how students handled it, as well as how their approach has changed since entering graduate school.Source: §A.1 · p. 26

• The interview aims to understand teamwork practices, challenges, and adaptations across stages of students’ academic journeys.Source: §A.2 · p. 26
• Questions cover students’ group-project experiences and how they coordinate tasks, meetings, and team check-ins.Source: §A.2 · p. 26
• The guide asks about a particularly difficult project experience and how students’ collaboration strategies have changed over time.Source: §A.2 · p. 26

A.3 Topic 3: Supervisory Relationship

• The supervisory-relationship topic seeks accounts of working relationships, communication, difficult moments, and how collaboration changes over time.Its objective is to inform understanding of communication practices, challenges, and supervisory dynamics.Source: §A.3 · pp. 26–27

B.1 System Prompt

• The system prompt frames the AI as a qualitative research assistant that analyzes interviewee answers, never answers questions itself, and returns task-specific JSON.Its outputs include summaries, probe opportunities, and follow-up questions.Source: §B.1 · p. 27

B.2 Prompt for Restrained Variant

• The restrained variant summarizes new, target-relevant information while excluding what is already known, and identifies protocol-aligned opportunities to probe.It evaluates what is missing, unclear, or worth exploring further.Source: §B.2 · p. 27

B.3 Prompt for Expressive Variant

• The expressive variant filters out incomplete or non-interview turns, then offers an aligned follow-up of at most 15 words or proceeds when the question is fully answered.It analyzes only a completed interviewee response and checks for probe opportunities.Source: §B.3 · p. 28

C Interviewee Role-Playing Protocol

• The research assistant reviews interview guides, then practices responding from real experiences without signaling prior knowledge during role-play.Responses should be spontaneous and flexible to follow-up questions.Source: C Interviewee Role-Playing Protocol · p. 28
• The assistant avoids answering comprehensively by default, leaving room for the interviewer to probe further.This creates space for follow-up on selected questions.Source: C Interviewee Role-Playing Protocol · p. 28
• Response depth adapts to the interviewer’s approach: effective prompts elicit elaboration, while poorly phrased or closed questions may receive shorter, more reserved answers.Source: C Interviewee Role-Playing Protocol · p. 28

D Self-Report Result Visualization

• Figure 5 covers subjective ratings of interview quality, cognitive load, perceived AI impact, control, and whether AI was seen as a teammate.Source: Figure 5 · p. 29