PaperlayerView exact revision on arXiv

Source-linked AI summary

Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data

Jing Wei, Sungdong Kim, Hyunhoon Jung, Young-Ho Kim

arXiv:2301.05843v2cs.HCcs.AIcs.CL

TL;DR

Prompts offer a way to build chatbots that collect self-reports through natural conversation, but how to design them for that task remains unclear. The paper tests four prompt designs in an online study and finds that they cover 79% of desired information slots, with prompt designs and topics influencing conversation flows and data collection. It discusses opportunities and challenges for LLM-driven chatbots.

  • Problem

    It remains unclear how to design prompts that let LLM chatbots conduct naturalistic conversations while collecting specified self-report data.

    Source: §1 · p. 2, §2.4 · p. 6

  • Method

    The authors tested four prompt designs varying information format and personality modifier in an online study with 48 participants conversing with GPT-3 chatbots across four health topics.

    Source: §1 · p. 3, §3.2.1 · p. 8

  • Results

    79% of desired information slots were covered, and prompt design and conversation topic influenced conversation flows and data collection performance.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings suggest that prompting can support LLM chatbots in asking health-related questions through natural conversations, while the paper also discusses problematic messages and ways to improve future chatbots.

    Source: §5 · p. 19, §6 · p. 25

    The study used GPT-3, and its prompts may not perform similarly on other LLMs because models differ in training corpora and architecture.

    Source: §5.5 · p. 24

Abstract

from arXiv · show

Large language models (LLMs) provide a new way to build chatbots by accepting natural language prompts. Yet, it is unclear how to design prompts to power chatbots to carry on naturalistic conversations while pursuing a given goal, such as collecting self-report data from users. We explore what design factors of prompts can help steer chatbots to talk naturally and collect data reliably. To this aim, we formulated four prompt designs with different structures and personas. Through an online study (N = 48) where participants conversed with chatbots driven by different designs of prompts, we assessed how prompt designs and conversation topics affected the conversation flows and users' perceptions of chatbots. Our chatbots covered 79% of the desired information slots during conversations, and the designs of prompts and topics significantly influenced the conversation flows and the data collection performance. We discuss the opportunities and challenges of building chatbots with LLMs.

1 INTRODUCTION

The introduction frames a need for chatbots that collect self-reports through natural conversation and asks how prompts can steer task-oriented LLM chatbots. It presents a study of prompt designs and their effects on data collection and conversational behavior.

1 INTRODUCTION

  • Existing scripted frameworks can feel rigid, while more dynamic chatbots often require large domain datasets; low-effort ways to combine task performance with natural conversation remain underexplored.The introduction also notes that robust prompts matter because LLM chatbots may err or digress from their tasks.Source: §1 · p. 2
  • The study examined 16 GPT-3 chatbots across four health topics, varying information-specification format and personality modifier.The topics were sleep, food intake, work and productivity, and exercise.Source: §1 · pp. 2–3
  • 79% of desired information slots were covered by the zero-shot prompts, and prompt format and personality modifier affected slot filling and conversation style.The prompts used neither example dialogues nor fine-tuning.Source: §1 · p. 3
  • The work reports feasibility in a 48-person online study and examines chatbots’ context maintenance, state tracking, off-topic suggestions, and dialogue errors.Participants also described some chatbots as understanding their messages and giving accurate, detailed responses.Source: §1 · p. 3

2 RELATED WORK

Related work situates the study among chatbot-based self-report collection, chatbot design and platform choices, and emerging work on LLM prompting. It identifies a gap in prompts for chatbots that proactively ask questions and guide conversations toward specified information.

2 RELATED WORK

  • The related-work section spans self-report collection, chatbot design, chatbot platforms, and LLM prompt design.The table caption identifies the specified information slots and job identities for each conversation topic.Source: §2 · p. 3, Table 1 · p. 6

2.1 Self-Report Data Collection through Chatbots

  • Prior work has used chatbots to collect self-reports, while this study explores LLMs for collecting them across sleep, food, work, and exercise without predefined logic or domain-specific retrieval training.Earlier work also links conversational self-report interfaces with engagement and richer responses.Source: §2.1 · p. 4

2.2 Design Considerations for Chatbots

  • Chatbot design research addresses breakdown repair, empathy, playfulness, personality, and context maintenance as facets of user experience.The cited work connects human-like traits with enjoyment and engagement, and recommends individualized repair strategies.Source: §2.2 · pp. 4–5

2.3 Chatbot Platforms

  • Rule-based platforms can provide predictable flows but may feel rigid, while more dynamic systems can demand programming skills or large datasets.Natural-language prompts offer a route to customizing chatbot behavior, but how to design prompts for effective question asking remains unclear.Source: §2.3 · p. 5

2.4 Designing LLM Prompts for Conversations

  • Prompting research has studied factors such as identity, phrasing, style, and randomness, but prior work has not optimized prompts for domain-specific chatbots that proactively ask predefined questions.This study investigates prompts for self-report chatbots with job identities and examines social attributes, empathy, and breakdown handling.Source: §2.4 · pp. 5–6

3 METHOD

The study tested GPT-3 chatbots for collecting self-reports across four health topics, varying prompt format and personality modifier. Its methods combined an online study with dialogue coding, slot-filling assessment, and statistical analysis.

3.1 The LLM-driven Chatbot Framework

  • Chatbots combined a prompt prefix describing persona and desired information slots with the ongoing conversation history to generate each response.The study covered sleep, food intake, work and productivity, and exercise, with behavioral data defined as topic-specific slots.Source: §3.1 · p. 6, §3.1.1 · p. 7

3.1.1 Prompt Designs for Chatbots.

  • Prompt development progressed from a generic assistant prompt to topic-specific job identities, then compared literal slot descriptions with structured placeholders.The researchers selected sleep expert, dietitian, life coach, and fitness coach identities after internal trials; both slot formats performed similarly in those limited tests.Source: §3.1.1 · p. 7
  • The prompt designs varied information format and an empathy-and-engagement modifier, with the modifier intended to elicit more empathic, responsive conversation.Figure 2 illustrates the two factors for the Food intake topic; the team also instructed the chatbot to ask one question at a time.Source: §3.1.1 · pp. 7–8, Figure 2 · p. 8

3.1.2 Model and Parameters.

  • GPT-3 received the prompt and dialogue history through a web interface, using davinci-text-002 and uniform generation settings across chatbots.The model accepted up to 4,000 BPE tokens; initial prompts used about 120 tokens, and settings were temperature 0.9, presence penalty 0.6, and frequency penalty 0.5.Source: §3.1.3 · p. 8, §3.1.2 · p. 8

3.2 Online Study

  • Participants received one of four prompt designs and completed conversations on all four topics, each with positive and negative paths.Topic order was counterbalanced between two sequences; paths provided general guidance for imagined open-ended conversations.Source: §3.2.1 · pp. 8–9

3.2.1 Experimental Conditions.

  • Each participant completed eight conversations and advanced when they judged a conversation naturally over or repetitive, then rated chatbot understanding, responsiveness, and empathy.The study had no ending-detection algorithm; the post-session survey also included an open-ended feedback question.Source: §3.2.2 · p. 9, §3.2.3 · p. 9

3.3 Participants

  • Recruitment yielded 54 completed sessions, of which six were excluded, leaving 48 participants with 12 assigned to each prompt condition.Participants were aged 19 to 56; 18 were female, 14 were native or bilingual English speakers, and 22 had never heard of or used LLMs.Source: §3.3 · p. 9

3.4 Data Analysis

  • Researchers assessed slot filling from dialogue transcripts and coded dialogue acts and chatbot behaviors, then modeled study factors and compared subjective ratings.A slot counted as filled only under specified criteria, including a numerical response for sleep quality or productivity rate; coding agreement reached Cohen’s Kappa 0.96 for essential acts and 0.935 for empathy and engagement behaviors.Source: §3.4 · pp. 9–11

4 RESULTS

Slot filling varied with prompt design, topic, and conversation path, while prompt choices also shaped dialogue patterns and chatbot behaviors. Participants generally rated understanding and acknowledgment favorably, though the study also documented errors and mixed perceptions of empathy.

4.1 Descriptive Statistics

  • About 20 turns occurred per dialogue; descriptive conditions had more turns and words than structured conditions, with DP producing the most words.One DP dialogue reached 75 turns.Source: Table 3 · p. 11, §4.1 · p. 12

4.2 Slot Filling Rate

  • 93% slot filling in SP-Exercise was highest, while SN-Work was lowest at 64%; average slot filling exceeded 70% across chatbot conditions.The study reported significant effects involving prompt condition, topic, and conversation path.Source: Table 4 · p. 12, §4.2 · p. 12
  • Slot filling varied significantly with study condition, topic, and conversation path: DP trailed SP and DN, Exercise reached 88.4%, and Positive exceeded Negative.The DP condition was significantly lower than SP (p = .01) and DN (p = .008); the Positive path had a higher rate than the Negative path (p = .01).Source: Figure 3 · p. 12, §4.2 · pp. 12–13
  • Earlier-listed slots were generally more likely to be covered; Sleep and Work's last slots were omitted in around 40% of dialogues, unlike Food's final slot.For Food, chatbots often asked about feelings after each meal rather than once at the end.Source: Figure 4 · p. 13, §4.2 · p. 13

4.3.1 Required and Secondary Questions.

  • Structured prompts increased required-question turn ratios and reduced secondary-question ratios; required-question ratios correlated positively with slot filling, while secondary-question ratios correlated negatively.Question-answering acts accounted for 4,879 of 7,442 turns (64.72%); the Positive path also had higher required-question and lower secondary-question ratios overall.Source: §4.3.1 · p. 14, Figure 5 · p. 15

4.4 Empathy & Engagement Behaviors

  • Descriptive prompts increased Acknowledging and Appreciating, the personality modifier increased Sympathizing, and conversation-path effects varied with the behavior category.Acknowledging was the most common empathy behavior; Positive paths had more Appreciating, while Negative paths had more Acknowledging and Sympathizing.Source: §4.4 · pp. 14–16, Table 6 · p. 15, Figure 6 · p. 16

4.5 Problematic Chatbot Turns and User Responses

  • Errors affected 6.7% of chatbot turns; structured prompts exposed slot notation in 16 turns, self-talk was below 1.1%, and repetitive turns accounted for 3.8%.75.7% of conversations ended organically; identical or linguistic repetition each accounted for 19.8% of prematurely ended conversations.Source: Table 7 · p. 16, §4.5 · pp. 16–18

4.6 Subjective Evaluation

  • Ratings showed no significant condition differences for understanding, acknowledgment, or empathy; understanding and acknowledgment were rated favorably, while empathy ratings were mixed (median 3).Some participants also praised chatbot accuracy, responsiveness, and consideration of their answers.Source: Figure 7 · p. 19, §4.6 · p. 19

5 DISCUSSION

The discussion finds prompt-driven GPT-3 chatbots can collect health self-reports, but performance and behavior vary with prompt design, topic, and conversation path. It weighs flexible interaction against reliability and safety concerns, then offers design guidance and study boundaries.

5.1 Designing Effective Prompts for Chatbots that Collect Self-Reports

  • Structured prompts elicit more direct questions, while descriptive prompts elicit context; combining format and persona changes slot filling, and SP and DN have comparable rates.The authors consider structured prompts with a personality modifier more desirable for data collection, while descriptive prompts without one may support acknowledgment.Source: §5.1 · pp. 19–20

5.2 Opportunities of LLM-driven Chatbots

  • GPT-3 produced varied scale questions, on-topic follow-ups, and clarifications, including interpreting a meal as a traditional Korean meal.These follow-up questions could supplement the topic and show a level of understanding.Source: §5.2 · p. 20
  • Chatbots sustained some contexts across multiple turns without dedicated context-management mechanisms, including intimidation and summaries extending beyond five turns.The paper contrasts this with existing frameworks' limited context handling.Source: §5.2 · p. 22
  • In-context learning avoids collecting training data or configuring dialogue flows; the chatbots also handled chit-chat, suggestions, and off-topic requests.The authors suggest robust prompts may let people without AI backgrounds personalize chatbots in natural language.Source: §5.2 · pp. 21–22

5.3 Drawbacks of LLM-driven Chatbots

  • Stochastic outputs can produce inconsistent or erroneous replies; accumulated errors persist, while repetitive turns can trap chatbots in dead loops.Proposed mitigations include filters for repetitive, self-talk, and system-offline errors, alongside intensive testing.Source: §5.3 · pp. 22–23

5.4 Ethical consideration

  • Although the study observed no harmful messages, chatbots sometimes gave potentially dangerous advice to teasing prompts, so outputs are not assured medically sound.The authors recommend caution about treating chatbot messages as medically or professionally sound.Source: §5.4 · pp. 23–24

5.5 Study Limitation

  • Comparisons are constrained by imperfect counterbalancing, differing slot compositions, and uncontrolled user language; GPT-3 prompt performance may not transfer to other LLMs.User inputs become part of prompts and may affect both generated responses and slot-filling performance.Source: §5.5 · p. 24

5.6 Future Work

  • Future work could test few-shot and multi-stage prompting for larger slot sets, and examine user perceptions against other chatbot frameworks.The authors also identify voice-based chatbot perceptions as an area for further study.Source: §5.6 · pp. 24–25

6 CONCLUSION

The conclusion presents GPT-3 prompting as a way to ask health questions conversationally and collect self-reports, while emphasizing prompt design and problematic messages.

6 CONCLUSION

  • The study finds that prompting GPT-3 can support conversational health questions and self-report collection, and offers prompt-design insights alongside discussion of problematic messages.The authors hope the work informs research on LLM-powered chatbots for robust data collection.Source: §6 · p. 25

A.2 Hint Texts for Conversation Paths

Hint texts for conversation paths are color-coded by polarity.

A.2 Hint Texts for Conversation Paths

  • Blue marks positive-path hint texts, while red marks negative-path hint texts.The colors appear in the instruction.Source: §A.2 · p. 32

A.3 Least-Squared Mean Statistics for Slot Filling Rate

The appendix reports a significant linear model for slot filling rate and describes estimated means by variable.

A.3 Least-Squared Mean Statistics for Slot Filling Rate

  • The significant slot-filling-rate model includes personality, format, topic, path, and a personality-by-format interaction: F(7, 366) = 6.493, p < .0001.The passage labels this as an estimate based on a significant linear model formula.Source: §A.3 · p. 33
  • Table 8 reports estimated least-squared means with 95% confidence intervals for slot filling rate by variable.The caption gives the table's scope but does not provide its cell values.Source: Table 8 · p. 33

A.4 Least-Squared Mean Statistics for Required and Secondary Question Turn Ratios

The RQ turn-ratio model includes personality, format, topic, path, and specified interactions, with a significant participant random effect.

A.4 Least-Squared Mean Statistics for Required and Secondary Question Turn Ratios

  • The RQ turn-ratio model includes personality, format, topic, path, Personality × Format, and Topic × Path, with a significant participant random effect (p < .0001).The passage reports the fitted mixed-effect model but gives no coefficient estimates or direction of effects.Source: §A.4.1 · p. 33

A.4.1 Turn Ratio of Required Questions.

The supplied material identifies a table of estimated RQ turn ratios by variables and reports a significant model for SQ turn ratios.

A.4.1 Turn Ratio of Required Questions.

  • Table 9 reports estimated least-squared means with 95% confidence intervals for RQ turn ratio by variables.The supplied caption does not include the estimates or identify differences between variables.Source: Table 9 · p. 33
  • The SQ turn-ratio linear model includes Personality, Format, Topic, Path, and Topic × Path (F(9, 364) = 10.04, p < .0001).The supplied model passage concerns SQ turn ratio, not RQ turn ratio.Source: §A.4.2 · p. 34

A.4.2 Turn Ratio of Secondary Questions.

The supplied passages give table scopes for SQ and three chatbot behavioral turn ratios, plus significant models for acknowledging, appreciating, and sympathizing turns.

A.4.2 Turn Ratio of Secondary Questions.

  • Table 10 reports estimated least-squared means with 95% confidence intervals for SQ turn ratio by variables.The supplied caption does not include the estimates or identify differences between variables.Source: Table 10 · p. 34

A.5 Least-Squared Mean Statistics for Chatbot Behavioral Turn Ratios

  • The acknowledging turn-ratio model includes Personality, Format, Topic, Path, and Personality × Format (F(7, 366) = 6.715, p < .0001).Table 11 reports estimated least-squared means with 95% confidence intervals for acknowledging turn ratio by variables.Source: §A.5.1 · p. 34, Table 11 · p. 34

A.5.2 Chatbot’s Appreciating Turn Ratio.

  • The appreciating turn-ratio model includes Personality, Format, Topic, Path, and Topic × Format (F(9, 364) = 8.903, p < .0001).Table 12 reports estimated least-squared means with 95% confidence intervals for appreciating turn ratio by variables.Source: §A.5.2 · p. 35, Table 12 · p. 35
  • The sympathizing turn-ratio model includes Personality, Format, Topic, Path, and Topic × Path (F(9, 364) = 8.219, p < .0001).Table 13 reports estimated least-squared means with 95% confidence intervals for sympathizing turn ratio by variables.Source: §A.5.3 · p. 35, Table 13 · p. 35
Loading 2301.05843v2…