If I Hear You Correctly: Building and Evaluating Interview Chatbots with Active Listening Skills

1 INTRODUCTION

• Open-ended interview responses are difficult to handle because chatbot platforms may lack free-text support or require designers to enumerate possible user intents.The paper also identifies digressions, misunderstood questions, and irrelevant or uncooperative replies as challenges.Source: §1 · p. 2
• Effective interview chatbots must return to their agenda after nonlinear conversations, while most chatbots instead support scripted dialog trees.Users may digress, ask for clarification, dodge questions, or give responses that are ambiguous or irrelevant.Source: §1 · p. 2
• Training-data requirements and limited AI expertise or resources make it difficult for designers to use advanced conversational AI.The paper notes that interview data are hard to acquire.Source: §1 · p. 2
• The study asks whether publicly available AI technologies can build effective interview chatbots with active listening, and how such chatbots affect input handling, user experience, and interview quality.The authors created chatbots with and without active listening and evaluated them live with 206 Amazon Mechanical Turk participants.Source: §1 · p. 2
• The work presents practical implementations for powering interview chatbots with a subset of active listening skills.The paper describes these skills as comprehending user input and responding properly.Source: §1 · p. 2
• The proposed hybrid framework uses rules to bootstrap a chatbot and data-driven models to improve its conversation capabilities.The authors report that their investigation demonstrates the feasibility and effectiveness of using publicly available AI technologies.Source: §1 · p. 2
• The paper offers design considerations for empathetic chatbots in applications beyond interviews, including counseling and training.It connects this extension to active listening’s role in effective communication beyond interviews.Source: §1 · p. 2

2.1 Conversational Agents for Information Elicitation

• Information-elicitation agents include survey bots, AI-powered chatbots, and embodied AI interviewers; the paper is most closely related to AI-powered chatbots.Existing AI-powered chatbots are described as limited in handling complex and highly diverse user input.Source: §2.1 · pp. 2–3

2.2 Task-Oriented vs. Social Conversational Agents

• Interview chatbots must combine task-oriented and social dialogue while completing their own information-elicitation task, including handling uncooperative or irrelevant replies.This distinguishes them from systems that support both dialogue types to help users complete tasks such as restaurant reservations.Source: §2.2 · p. 3

2.3 Recent Advances in Conversational Agents

• Retrieval and generative models address diverse input, but require extensive training data; interview data can be difficult to obtain, and limited interpretability and control pose risks in high-stakes settings.The paper notes hybrid approaches as inspiration for its prototype.Source: §2.3 · p. 3

2.4 Chatbot Platforms

• Existing chatbot platforms either offer little AI/NLP capability or have a steep learning curve and are often designed for task-oriented bots.The authors chose to extend Juji, which supports task-oriented and social dialogues and allows extensions.Source: §2.4 · p. 3

3 PROTOTYPE OVERVIEW

• The prototype combines a rule-based system with extensions supporting data-driven models.The authors regard a hybrid system as promising because rules can bootstrap a chatbot while data-driven approaches can handle complex and diverse conversations.Source: §3 · pp. 3–4
• Juji serves as the rule-based layer, where designers create an ordered conversation agenda and a dialog manager activates topics in sequence.Users can interact through a website texting interface or Facebook Messenger.Source: §3 · p. 4, Figure 2 · p. 4
• The intent discoverer and sentence ranker identify user intents and label training data, with human review available to correct mislabeled examples.The sentence encoder produces fixed-length dense vectors, which the model builder uses to train models such as text classifiers for intent prediction.Source: §3 · p. 4
• Trained models can predict how semantically relevant a user response is to an interview question, informing the chatbot’s response.The paper frames these extensions as a way to handle free-text input.Source: §3 · p. 4

4.1 Prototype Scope and Technical Focus

• The prototype’s technical focus is interpreting user free-text semantics, which is needed before a chatbot can respond properly using active-listening techniques.Understanding semantic relevance can help encourage a better response, while understanding response meaning is needed for summarization.Source: §4.1 · p. 4
• The initial prototype supports only a subset of active-listening techniques, excluding impromptu techniques such as balancing because it targets structured interviews with predefined questions.Table 1 is identified as the list of active listening supported by the prototype.Source: §4.1 · p. 4, Table 1 · p. 4

4.2 Prototype Design Criteria

• The design criteria were reproducibility, practicality, and adoption by non-AI experts.The team selected publicly available technologies, low-resource options requiring little training or customization, and tools accessible to non-AI experts building custom interview chatbots.Source: §4.2 · p. 5

4.3 The Juji Base System

• Juji was selected for public availability, rule-based chatbot bootstrapping without training data, and demonstrated use in interview applications.Its rule-based system and adoption for job candidate, market research, and student interviews informed the choice.Source: §4.3 · p. 5

4.3.1 Ability to Manage Nonlinear Conversations

• Juji tracks conversation context and brings users back on track from side talking, including after repeated interruptions to an interview question.Table 2 gives an example of this nonlinear-conversation handling.Source: §4.3.1 · p. 5, Table 2 · p. 5

4.3.2 Extensibility

• Juji rules can call third-party functions or APIs, while its topic-based model supports targeted improvements with less training data and fewer cross-topic effects.The authors contrast this with end-to-end dialog changes, which may improve one part while adversely affecting others.Source: §4.3.2 · p. 5

4.4 The Extensions: Data-Driven User Intent Prediction

• The prototype focuses on identifying users’ high-level implied intent and adds text-classification support for scalable, topic-specific intent prediction.Its components support preparing training data, encoding it, and training text-classification models.Source: §4.4 · p. 5

4.4.1 Preparing Training Data: Machine-Aided Labeling

• LDA discovers intents, centroid-based ranking finds representative positive and negative responses, and humans validate the resulting labels.Labels are challenging because responses are diverse and may convey multiple intents simultaneously; human validators can adjust parameters and rerun the analysis.Source: §4.4.1 · pp. 5–6

4.4.2 Encoding Training Data: Sentence Embedding

• The prototype uses Universal Sentence Encoder embeddings, chosen for longer-text representation, generalization, fine-tuning availability, and a balance of performance and resource needs.BERT-Large produced comparative F1 scores but required twice the storage for its smallest encoding; the prototype did not fine-tune encoders.Source: §4.4.2 · p. 6

4.4.3 Training Text Classification Models

• The team trained binary classification models that output probability scores, selecting them for interpretability, performance, and training-data requirements.Source: §4.4.3 · p. 6

4.5 Enabling Active Listening

• Topic rules use classification results to guide responses, with a default response when no intent reaches the required confidence.In the top-challenge example, four models assess relevance and three possible intents; the response follows the best-detected intent.Source: §4.5 · pp. 6–7
• For a response classified as coping with changes, the chatbot generates candidate replies using a summarizing technique.The example response candidates acknowledge the user’s difficulty with change or new settings.Source: §4.5 · pp. 6–7

5.1 Evaluating User Intent Prediction Models

• Across 17 intents, logistic regression and linear SVM performed best overall; logistic regression favored heterogeneous data, while linear SVM favored homogeneous data.The evaluation trained 68 models using four classifiers and stratified 10-fold cross-validation.Source: §5.1 · p. 7

5.2.1 Study Design

• The live study compared a baseline chatbot with a full version that added intent-classification models, using the same questions in the same order.The interview covered six topics, including the four topics used for active listening.Source: §5.2 · p. 7, §5.2.1 · p. 8

5.2.3 Measures

• Measures covered chatbot comprehension, user interest and chat experience, engagement duration, response length, informativeness, and response quality.Response Quality Index combined specificity, relevance, and clarity.Source: §5.2.3 · p. 8

5.2.2 Participants

• The study collected 206 completed interviews: 108 participants used the full chatbot and 98 used the baseline.Participants were adults recruited on Mechanical Turk from the U.S. or Canada.Source: §5.2.4 · p. 8, §5.2.2 · p. 8

5.2.4 Results

• The full chatbot significantly outperformed the baseline across all measures, including comprehension, interest, chat experience, engagement, response length, informativeness, and response quality.The comparison used ANCOVA analyses controlled for participant demographics; some outcomes were also controlled for engagement duration.Source: §5.2.4 · pp. 8–9, Table 6 · p. 9
• Real-world intent prediction matched cross-validation results for Q1 and Q2 but was worse for Q3 and Q4; Q4's training data missed many financial challenges, while new responses can support incremental model updates.Q2 intent models performed best and Q4 worst overall.Source: §5.2.4 · p. 9

5.3 Summary of Findings

• The authors conclude that existing practical AI technologies can build effective interview chatbots with active listening skills.They also conclude these chatbots engage users and elicit higher-quality responses than chatbots without those skills.Source: §5.3 · p. 9

6.1 Effect of Individual Active Listening Techniques

• The evaluation compared chatbots with and without active listening but did not isolate each technique’s effect; technique choice is random and may need tailoring to users and topics.The authors call for studying effects across different users and interview topics.Source: §6.1 · p. 9

6.2 Interpreting Deeper User Intents

• The prototype captures high-level intent, but vague, shallow replies motivate extracting deeper semantic concepts and relationships.Knowledge-graph approaches are suggested as a possible route.Source: §6.2 · p. 9

6.3 Interrelating Interview Topics

• Treating topics independently can make users feel unheard when they repeat information; follow-on conversation should use remembered input, though diverse responses complicate knowledge extraction.A knowledge graph is one proposed way to relate topics, but requires parsing user input into entities that can be retrieved and reasoned over.Source: §6.3 · pp. 9–10

6.4 Active Listening by Asking

• The prototype lacks impromptu follow-up questions, which could help pursue interesting ideas and discover unexpected insights in goal-oriented interviews.The authors are exploring question generation for dynamic, goal-oriented conversations.Source: §6.4 · p. 10

6.5 Experimenting with Alternatives

• Alternative intent-discovery and sentence-encoding methods remain untested; comparing them requires deeper expertise and more resources.Options include correlation explanation, fine-tuning sentence encodings, and jointly training sentence embeddings and text classification.Source: §6.5 · p. 10

6.6 Evaluating Prototype Usability

• System usability was not evaluated, and the Juji GUI plus command-line workflow for labeling and training is considered cumbersome.The authors say a more integrated UI is needed as the prototype advances.Source: §6.6 · p. 10

6.7 Improving Specific Interview Topics

• The approach performed better on general topics, leaving its scope on domain-specific topics and its generalizability with actual interviewees to be examined.The authors call for evaluation across more topics and in real-world interviews.Source: §6.7 · p. 10

7.1 Practical Approaches to Effective Interview Chatbots

• The evaluation supports a hybrid approach for engaging users and eliciting quality input, with topic-specific models trained on small datasets as a practical alternative to large end-to-end dialog datasets.The methodology is platform-agnostic despite implementation on Juji.Source: §7.1 · p. 10

7.2 Hybrid and Progressive Chatbot Platforms

• A progressive platform can start with rules, collect training data, and improve with trained models; LDA-based intent discovery can help designers label that data.The framework supports incremental chatbot improvement.Source: §7.2 · p. 10

7.3 Building Empathetic Chatbots Beyond Interviewing

• Active listening also applies to counseling and training, and the work points toward non-AI experts creating customizable empathetic agents beyond interview tasks.The authors describe this work as a stepping-stone toward that direction.Source: §7.3 · p. 10

• The prototype combines a rule-based chatbot builder with data-driven models for active listening, which handles open-ended responses and is reported to improve engagement and response quality.Source: §8 · p. 10