PaperlayerView exact revision on arXiv

Source-linked AI summary

Evidence-Traceable Dynamic Interviewer Architecture for Expertise-Adaptive Qualitative Interviews Using Local LLMs

Aisvarya Adeseye, Jouni Isoaho, Adeyemi Adeseye, Seppo Virtanen, Mohammad Tahir

arXiv:2610.11651v1cs.AI

TL;DR

Existing automated interviewers can rely on fixed questions and limited personalization, while context-sensitive generation alone does not ensure expertise-appropriate, non-repetitive follow-ups. This paper presents a locally hosted, five-module interviewer that tracks interview state and adapts question complexity to demonstrated expertise. In the 246-participant evaluation, expertise profiling had 78.9% exact agreement with participant self-assessment and weighted Cohen’s κ of 0.80.

  • Problem

    Many existing interviewers rely on fixed question sequences or limited context-based personalization without considering participants’ knowledge.

    Source: §1 · p. 2

  • Method

    A locally hosted, five-module interviewer uses participant responses and a persistent interview-state record to profile expertise, adapt questions, and validate question uniqueness.

    Source: §1 · p. 3, §3 · p. 5, §6 · p. 22

  • Results

    M3 achieved 78.9% exact agreement with participant self-assessment and a linearly weighted Cohen’s κ of 0.80.

    Source: §5.2 · p. 19

  • Takeaways & Limitations

    Expertise profiling influences the depth and cognitive demand of subsequent questions, while M5 filters potentially repetitive questions before presentation.

    Source: §6 · p. 22

    Evaluation used only Llama 3.2 3B, so model-independent performance and consistency across model families and parameter sizes remain unestablished.

    Source: §6 · p. 22

Abstract

from arXiv · show

Automated interviewers and conversational agents are increasingly used in research, recruitment, customer service, and education. However, many existing systems rely on fixed question sequences and provide limited context-based personalization without considering participants' knowledge, which can lead to repetitive or irrelevant follow-up questions. Therefore, there is a need for an adaptive interviewing system that can adjust question depth while maintaining conversational continuity and semantic progression. To address this, an Evidence-Traceable Dynamic Interviewer Architecture is presented using a locally hosted Large Language Model (LLM), with the interview continuously adapted throughout the entire conversation based on the participant's responses and evolving context. The interviewer profiles participants' expertise in real time to generate knowledge-appropriate questions, well-articulated responses, and smooth transition messages that support conversational continuity. A five-module prompt-driven architecture and persistent interview-state record support these functions. The interviewer was evaluated with 246 participants. Expertise Profiling module (M3) showed 78.9% exact agreement with independently reported participant expertise, with a weighted Cohen's K of 0.80. Generate Iterative Questions module (M4) showed a strong expertise-complexity association (p=.79, p<.001), and participants reported high relevance (mean 4.41), engagement (mean 4.32), and satisfaction (mean 4.38), providing evidence that the architecture's adaptive components operated consistently with their intended functions while participants reported a positive interview experience.

1 Introduction

Existing automated interviewers often rely on fixed or limited dialogue mechanisms, while context-sensitive generation alone does not ensure expertise-appropriate, non-repetitive questions. The study proposes a locally hosted, evidence-traceable architecture that adapts questions to demonstrated expertise and evaluates its operation and participant experience.

1 Introduction

  • Many existing systems use predefined questions, rule-based dialogue, or limited intent recognition, constraining responsive follow-up, exploration of emerging topics, and adaptation to participants’ knowledge and experience.LLM context-sensitive generation adds flexibility but does not by itself ensure expertise-appropriate questions or prevent semantic repetition.Source: §1 · p. 2
  • Local LLM deployment can reduce the need to transmit raw participant responses to external services, addressing privacy concerns around sensitive interview data.The study identifies structured prompting, controlled adaptation, output validation, and local deployment as supports for effective adaptive interviewing.Source: §1 · pp. 2–3
  • The study proposes a locally hosted, evidence-traceable interviewer that links participant responses, expertise estimates, generated questions, and validation decisions throughout the interview.Its research questions address expertise inference and complexity adaptation, relevant and non-repetitive questioning, and participants’ evaluations of the interview experience.Source: §1 · p. 3
  • The study evaluates a modular, prompt-driven interviewer with 246 participant interviews, examining expertise agreement, expertise-driven question adaptation, semantic uniqueness, and participant perceptions.The architecture is intended to adjust question complexity according to demonstrated expertise and produce contextually relevant, non-redundant, semantically coherent dialogue.Source: §1 · p. 3

2 Related Work

Prior work advances conversational interviewing, expertise modeling, semantic similarity, and structured prompting, but these capabilities are generally studied separately. The remaining gap is an integrated interview process that adapts question complexity to demonstrated expertise while preserving continuity and semantic uniqueness.

2 Related Work

  • Conversational interfaces can improve engagement, response quality, social presence, and self-disclosure, but interview systems rarely maintain an explicit, continuously updated representation of demonstrated expertise.LLM interviewers support scalable interviewing and useful probes, yet reported issues include inappropriate follow-ups, weak acknowledgment, and limited strategic progression.Source: §2 · pp. 3–4
  • Adaptive-system research supports inferring expertise from observable behavior and representing it as an evolving user state, with question complexity guided by reasoning evidence rather than technical vocabulary alone.Expert–novice research distinguishes deeper principle-based reasoning from novices’ reliance on surface characteristics, while expertise-reversal findings indicate that interaction suitable for inexperienced users may become ineffective for more knowledgeable users.Source: §2 · p. 4
  • Separating interviewer output into a Response Message, Transition Message, and Next Question makes acknowledgment, conversational flow, and information elicitation distinct but connected functions.The rationale draws on active-listening and adaptive-interviewing work emphasizing contextual awareness, relevance, openness, and natural transitions.Source: §2 · p. 4
  • Semantic-similarity methods such as Sentence-BERT, SimCSE, and sentence meta-embeddings identify conceptual similarity beyond lexical overlap, but explicit generation gates for rejecting redundant interview questions are less common.Structured prompting can control task instructions, memory, reasoning constraints, feedback, and output formats, while local deployment offers greater control over processing sensitive responses.Source: §2 · p. 4
  • Existing work advances conversational surveys, LLM interviewing, expertise modeling, semantic similarity, and structured prompting, but these capabilities are generally studied separately.The stated gap is integrating them into an interview process that links participant responses to expertise estimation, adapts subsequent question complexity, and maintains continuity and semantic uniqueness.Source: §2 · pp. 4–5

3 Evidence-Traceable Dynamic Interviewer Architecture

The architecture divides adaptive interviewing into five configurable modules, with a persistent interview-state record supporting traceability across turns. Together, these modules establish constraints, begin with a calibration question, adapt follow-ups to demonstrated expertise, and screen for semantic repetition.

3.1 Module 1 (M1): Generate System Prompt

  • M1 turns interview configuration into a reusable system prompt that sets consistent behavioral constraints, while linking important prompt constraints to their purpose and rationale.The configuration covers research objectives and priorities, interview style, language, ethics, and knowledge handling.Source: §3.1 · pp. 5–6

3.2 Module 2 (M2): Generate Initial Question

  • M2 opens with an accessible, low-complexity question that lets participants demonstrate topic familiarity for M3’s initial expertise profile.It selects a high-priority research area; M2 itself does not determine expertise.Source: §3.2 · pp. 7–8

3.3 Module 3 (M3): Expertise Profiling

  • M3 classifies demonstrated expertise across four observable factors, supplies supporting evidence, and reassesses the profile after each response.The factors are terminology and specificity, conceptual correctness and relevance, reasoning depth, and application or evaluative reasoning; personal experiences and opinions are not assessed for factual correctness.Source: §3.3 · pp. 8–9

3.4 Module 4 (M4): Generate Iterative Question

  • M4 generates an acknowledgment, transition, and next question, adjusting question depth to expertise while also considering interview history and underexplored research areas.Lower demonstrated expertise leads to clearer, less technically demanding questions; higher expertise supports deeper analytical, comparative, scenario-based, or evaluative questions.Source: §3.4 · pp. 10–11

3.5 Module 5 (M5): Validate Question Uniqueness

  • M5 compares each candidate question with question history and accepts distinct lines of inquiry or returns substantially overlapping questions to M4 for regeneration.The validation decision and its internal justification are stored in the persistent interview-state record.Source: §3.5 · pp. 11–12

4 Experiment and Verification Setup

The workplace-oriented case study evaluated a five-module interviewer through expertise profiling, adaptive question generation, semantic-uniqueness checks, and participant ratings. Its setup combined independent expertise self-reports with question-complexity assessments and post-interview evaluations.

4 Experiment and Verification Setup

  • The workplace-oriented experiment examined expertise profiling, expertise-adaptive question generation, semantic question uniqueness, and the resulting interview experience.The five research areas and their priorities shaped expected coverage and initial and subsequent question selection.Source: §4 · p. 13, §4.1 · p. 13

4.1 Case Study and Interview Configuration

  • The study evaluated a workplace LLM-use interview with five research areas covering knowledge, organizational applications, skills, privacy, and adoption guidelines.Configuration set 16 participant-facing questions: M2 generated the initial question and M4 generated 15 adaptive follow-ups per interview.Source: §4.1 · p. 13

4.2 Technical Implementation

  • The interview was implemented with a locally hosted Llama 3.2 3B model, with M1–M5 handling configuration, initial questioning, expertise profiling, iterative questioning, and duplication checks.A persistent state record retained responses, questions, coverage, expertise estimates, and validation decisions for subsequent adaptation and post-interview verification.Source: §4.2 · p. 14

4.3 Participants, Recruitment, and Experimental Procedure

  • The online English study included 246 consenting adults with workplace experience, and each interview followed the same general flow while question sequence and complexity could adapt.Participants also reported their perceived expertise independently of M3 and rated their interview experience afterward.Source: §4.3 · pp. 14–15

4.4 Expertise-Profiling Verification

  • M3 expertise estimates were compared with participants’ independently collected self-reported categories, treated as a verification reference rather than absolute ground truth.The comparison used exact agreement, percentage agreement, a confusion matrix, and linearly weighted Cohen’s κ to account for ordered categories.Source: §4.3 · pp. 14–15, §4.4 · p. 15

4.5 Adaptive Question Verification

  • The adaptive-question evaluation related turn-level M3 expertise to M4 question complexity, using participant-level mean complexity and ordinal analysis across 3,690 follow-up questions.Two researchers independently classified questions; disagreements were resolved by consensus, and M5 validation histories were separately used to assess acceptance and regeneration.Source: §4.5 · pp. 15–16

4.6 Participant-Based Verification

  • Participant-based verification measured question relevance and coherence, cognitive and emotional engagement, and overall satisfaction after each interview.The setup paired these experience ratings with architecture-level checks of expertise correspondence, adaptive complexity, and semantic-overlap decisions.Source: §4.6 · pp. 16–17

5 Results

Across 246 interviews, inferred expertise tracked subsequent question complexity, while the system screened overlapping questions and participants rated the interview experience positively. Results also describe the sample, agreement with self-reported expertise, and examples of how question depth progressed.

5 Results

  • The results section reports participant characteristics, M3 and M4 verification, M5 validation, adaptive-question examples, and participant experience ratings.Table 10 presents participant demographic and background characteristics.Source: §5 · p. 17, Table 10 · p. 18

5.1 Participant Characteristics

  • The 246 participants varied in age, education, employment, professional experience, and LLM familiarity; 64.2% used LLMs daily or weekly, while 19.1% rarely or never did.Self-reported expertise spanned all four categories, from 21.1% Novice to 18.3% Expert.Source: §5.1 · pp. 17–18

5.2 Expertise-Profiling Results

  • M3 matched participants’ self-reported expertise for 194 of 246 people (78.9%); agreement within one adjacent level was 98.4%, with weighted Cohen’s κ = 0.80.Table 11 presents the classification comparison; self-reports served as a verification reference, not absolute ground truth.Source: Table 11 · p. 18, §5.2 · p. 19

5.3 Expertise-Adaptive Question Generation

  • M3 expertise level was strongly associated with subsequent question complexity (Spearman’s ρ = .79, p < .001), which increased from descriptive prompts for Novices to evaluative questions for Experts.Source: §5.3 · p. 19

5.4 Examples of Adaptive Question Progression

  • Representative questions progressed from personal experience for novices to explaining problems, assessing organizational risks, and proposing technical or governance measures for experts.The example sequence also maintained a response, transition, and next-question structure and moved from recognizing a privacy risk to considering mitigation.Source: §5.4 · pp. 20–21

5.5 Semantic Question-Uniqueness Validation

  • Across 3,690 first-pass M4 follow-ups, M5 accepted 3,469 (94.0%) and returned 221 (6.0%) for semantic overlap; 207 returned questions were accepted after one regeneration.Most rejected questions sought the same information using different wording; 14 required two or more regeneration cycles.Source: §5.5 · p. 21

5.6 Participant Evaluation

  • In participant evaluations, question relevance and coherence averaged M=4.41, engagement M=4.32, and overall satisfaction M=4.38; participants found interviews relevant, engaging, and satisfactory while questioning adapted to demonstrated expertise.Source: §5.6 · pp. 21–22

6 Conclusion

The study presents a five-module interviewer with a persistent record linking responses, expertise estimates, question-generation decisions, and validation outcomes. In a 246-participant workplace case study, the evaluated components showed the reported agreement, expertise-complexity association, and positive participant ratings; evaluation used only one local LLM.

6 Conclusion

  • M3 achieved 78.9% exact agreement with independently reported expertise, M4 expertise and question complexity were strongly associated (ρ = .79, p < .001), and participants rated relevance, engagement, and satisfaction positively.M5 accepted 94.0% of first-pass questions and returned 6.0% for regeneration after identifying semantic overlap.Source: §6 · p. 22
  • The five-module architecture and persistent interview-state record connect participant responses to expertise inference, adaptive questions, and validation decisions.The paper characterizes this as a structured, inspectable process for selecting an appropriate next question.Source: §6 · pp. 22–23
  • The evaluation used a single local model, Llama 3.2 3B, so it does not establish model-independent performance; cross-model evaluation is needed to assess robustness and consistency.The architecture is described as conceptually model-independent, but different local models may respond differently to the same prompts.Source: §6 · p. 22

Ethics Approval and Informed Consent

The study followed Finnish research ethics principles and obtained electronic informed consent from all participants.

Ethics Approval and Informed Consent

Related papers

Loading 2610.11651v1…