Source-linked AI summary
Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactions
Zahra Abbasiantaeb, Yifei Yuan, Evangelos Kanoulas, Mohammad Aliannejadi
TL;DR
The paper addresses the costly, inconsistent, and non-scalable use of human annotators for simulating human-to-human CQA. It replaces both student and teacher with zero-shot prompted GPT-4 models and evaluates their conversations against human-generated data. The LLM teacher produces longer and generally more accurate, complete answers, while the LLM student asks more diverse questions covering more aspects of a topic.
Problem
Human annotation for CQA simulation is time-consuming, resource-intensive, expensive, and variable in quality and consistency.
Method
A zero-shot LLM-to-LLM framework uses one GPT-4 model as a proactive student and another as a teacher answering from topic text.
Results
LLM teachers generate longer, more comprehensive, consistent, fluent, and more accurate answers, while LLM students generate more diverse questions with broader topical coverage.
Takeaways & Limitations
LLMs show potential for facilitating interactive and informative retrieval experiences through simulated human-to-human CQA conversations.
Takeaways & Limitations
The pipeline is constrained because only GPT-4 consistently follows instructions to generate reasonable conversations, while simulated data may contain biases and prompting remains time-consuming to construct manually.
Abstract
from arXiv · showhide
Conversational question-answering (CQA) systems aim to create interactive search systems that effectively retrieve information by interacting with users. To replicate human-to-human conversations, existing work uses human annotators to play the roles of the questioner (student) and the answerer (teacher). Despite its effectiveness, challenges exist as human annotation is time-consuming, inconsistent, and not scalable. To address this issue and investigate the applicability of large language models (LLMs) in CQA simulation, we propose a simulation framework that employs zero-shot learner LLMs for simulating teacher-student interactions. Our framework involves two LLMs interacting on a specific topic, with the first LLM acting as a student, generating questions to explore a given search topic. The second LLM plays the role of a teacher by answering questions and is equipped with additional information, including a text on the given topic. We implement both the student and teacher by zero-shot prompting the GPT-4 model. To assess the effectiveness of LLMs in simulating CQA interactions and understand the disparities between LLM- and human-generated conversations, we evaluate the simulated data from various perspectives. We begin by evaluating the teacher's performance through both automatic and human assessment. Next, we evaluate the performance of the student, analyzing and comparing the disparities between questions generated by the LLM and those generated by humans. Furthermore, we conduct extensive analyses to thoroughly examine the LLM performance by benchmarking state-of-the-art reading comprehension models on both datasets. Our results reveal that the teacher LLM generates lengthier answers that tend to be more accurate and complete. The student LLM generates more diverse questions, covering more aspects of a given topic.
1 INTRODUCTION
The paper addresses the cost, inconsistency, and limited scalability of human-annotated CQA simulation by replacing both conversation roles with zero-shot prompted LLMs. It evaluates the resulting teacher-student conversations against human-generated data, finding broader topical coverage in LLM-generated questions and stronger answer quality from the LLM teacher.
- Motivation: Human annotation is time-consuming, expensive, inconsistent, and difficult to scale for generating substantial CQA conversation data.Human students may also struggle to explore topics outside their background knowledge.
- Research aim and framework: Zero-shot prompted LLMs simulate both student and teacher roles in proactive CQA conversations, producing the SimQuAC dataset.The student explores a predefined topic through questions, while the teacher provides complete and correct answers using topic text.
- Evaluation design: The study evaluates the teacher with automatic and human assessment, the student through behavioral comparison, and both datasets with reading-comprehension benchmarks.The authors also conduct linguistic and broader comparative analyses of LLM- and human-generated conversations.
- Evaluation and findings: LLM-generated questions provide more topical coverage than human-generated questions during student-level evaluation.The comparison examines question-asking patterns and behavior from multiple perspectives.
- Evaluation and findings: LLM-generated answers are generally lengthier, more comprehensive, consistent, and fluent than human-generated answers.Human evaluation also finds the LLM teacher more accurate in providing correct answers.
2 METHODOLOGY
The methodology simulates QuAc!-style information-seeking conversations with zero-shot LLM student and teacher agents, supported by validation and prompt-selection components. The teacher answers from section text while the student explores a topic using limited background information.
- The framework simulates QuAc!-style conversations by having a student generate questions and a teacher answer them using Wikipedia article information.The student receives the topic, section header, and article background, while the teacher additionally accesses the full section text.
- Teacher simulation: The teacher is prompted to copy exact spans from the section text and return “I cannot find the answer” when the answer is unavailable.Teacher instructions include the article title, background, and section text, with a maximum answer-length mechanism.
- Teacher simulation: Answer validation checks whether responses contain exact section-text spans and do not copy from the background, regenerating answers when either criterion fails.The system normalizes section text by removing extra spaces and bracketed text before searching for matching answers.
- Student simulation: Student question generation uses zero-shot prompting, then filters questions exceeding 25 words or containing newlines or enumerated items.The student is instructed to explore the topic from the section header and background without access to the full section text.
- Student simulation: When questions cannot be answered from the section, prompt selection adjusts subsequent student guidance toward more general and answerable questions.The selector randomly chooses guiding prompts, including instructions not to ask overly specific questions.
3 TEACHER EVALUATION
TeacherSim is evaluated against human-generated QuAC answers through automatic comparison and pairwise human judgments. Across these evaluations, teacherSim answers are generally preferred for their accuracy, completeness, naturalness, and conversation-level quality.
- Experimental setup: The study samples 50 QuAC training conversations, reuses their topics and questions, and has at least three crowd-workers independently assess each conversation.Examples are randomized between System A and System B to reduce position bias; workers also complete onboarding and inconsistent justifications are removed.
- Experimental setup: GPT-4 serves as both teacher and student, while other tested models failed to copy exact answer segments in zero-shot prompting.The teacher validation loop uses a maximum patience of four iterations.
- Experimental setup: The evaluation compares teacherSim and QuAC answers for correctness, completeness, and naturalness, plus annotator preference for interacting with each system.Annotators viewed the shared background and section text while comparing the two answers side by side.
- Experimental results: 49.0% of 359 compared answers had no overlap, while 21.4% were identical and 29.5% overlapped as substrings.TeacherSim also returned multiple text segments for 41 questions, supporting more complete answers when needed.
- Experimental results: TeacherSim outperformed the QuAC human teacher by a large margin on all question-based metrics.The evaluation reported substantial inter-annotator agreement, with Fleiss’ κ=0.4365.
- Experimental results: 87.7% of topics favored teacherSim answers over QuAC answers in pairwise human evaluation.Annotators compared correctness, completeness, naturalness, and interaction preference; Table 4 reports the evaluation outcomes.
4 SIMULATION EVALUATION
The evaluation introduces SimQuAC and compares LLM-simulated conversations with QuAC across dataset characteristics, student behavior, and reading-comprehension performance. SimQuAC produces broader, less sequential questioning, while most tested models perform better overall on it despite lower exact match.
- 4.1 SimQuAC dataset: SimQuAC contains 4,005 questions from 334 simulated conversations generated with GPT-4 studentSim and teacherSim.The conversations were simulated from unique topics sampled from 342 QuAC training conversations.
- 4.2 Student evaluation: GPT-4 tends to generate longer, more detailed questions, and its simulated student continues asking questions after the human student stops.The comparison uses conversations on the same topic and contrasts QuAC with SimQuAC.
- 4.2 Student evaluation: SimQuAC questions cover more of the source text than QuAC questions, with means of 0.365 and 0.238 respectively (p-value < 0.001).The corresponding standard deviations are 0.163 for SimQuAC and 0.122 for QuAC.
- 4.2 Student evaluation: The average Kendall rank correlation is lower for SimQuAC than QuAC, indicating that SimQuAC conversations are less sequential relative to answer order in the source document.Lower KRCC represents greater distance between the conversation order and document order.
- 4.3 Reading comprehension benchmarking: Most reading-comprehension models perform better overall on SimQuAC than QuAC, although SimQuAC has lower exact match because answers span longer text and include more unanswered questions.Generative methods such as T5 outperform discriminative methods such as BERT in this evaluation.
5 RELATED WORK
Related work positions conversational question answering within interactive retrieval and distinguishes proactive user simulation from systems that mainly react to system actions. The paper focuses on conversational machine reading comprehension and uses LLMs to support more realistic simulation.
- 5.1 Conversational question answering: Conversational question answering requires interpreting each question in the context of previous conversation turns and includes conversational machine reading comprehension systems.The paper distinguishes these systems from sequential knowledge-based question-answering agents.
- 5.2 User simulation: Existing user simulators often react to system actions, whereas real users also proactively initiate and guide conversations through questions.The paper presents LLMs as an opportunity to make user simulation more realistic through coherent, contextually appropriate language.
6 CONCLUSIONS AND FUTURE WORK
The paper concludes that zero-shot LLM-to-LLM simulation can model human-to-human question-answering conversations and may support interactive, informative retrieval. It also identifies instruction-following, bias, and manual prompting as limitations requiring further work.
- 6 CONCLUSIONS AND FUTURE WORK: Two GPT-4 models simulate a student asking topic-based questions and a teacher answering from a supplied text, with evaluation at both teacher and student levels.The evaluation combines automated and human assessment with comparisons against human-generated conversations.
- 6 CONCLUSIONS AND FUTURE WORK: The approach highlights the potential of LLMs for facilitating interactive and informative retrieval experiences.This conclusion is stated within the paper’s simulated conversational question-answering setting.
- 6 CONCLUSIONS AND FUTURE WORK: Only GPT-4 consistently follows instructions to generate reasonable conversations, while simulation can inherit model biases and still depends on time-consuming manual prompt construction.The paper proposes better bias mitigation and more efficient automatic prompting as future directions.
7 ETHICAL CONSIDERATIONS
The paper identifies ethical risks in using LLMs to simulate users, including inherited bias, misrepresentation, limited transparency, and environmental impact.
- 7 ETHICAL CONSIDERATIONS: LLM-simulated users may reproduce training-data biases, misrepresent user types, obscure decision-making, and incur environmental costs.The paper frames these issues as ethical considerations requiring attention in this research area.