PaperlayerView exact revision on arXiv

Source-linked AI summary

BianQue: Balancing the Questioning and Suggestion Ability of Health LLMs with Multi-turn Health Conversations Polished by ChatGPT

Yirong Chen, Zhenyu Wang, Xiaofen Xing, huimin zheng, Zhipei Xu, Kai Fang, Junhong Wang, Sihang Li, Jieling Wu, Qi Liu, Xiangmin Xu

arXiv:2310.15896v2cs.CLcs.HC

TL;DR

Health LLMs often offer suggestions without asking the multi-turn questions that can clarify a user’s situation. BianQue combines a ChatGLM-based model with a multi-turn health conversation dataset, and it outperforms existing models and ChatGPT on multi-turn health conversations, especially in CoQ.

  • Problem

    Health LLMs generally provide suggestions from single-turn instructions and lack the multi-turn questioning ability used by doctors to understand patients’ situations.

    Source: §1 · pp. 1–2

  • Method

    The paper constructs BianQueCorpus from multi-turn health conversations and fine-tunes a ChatGLM-6B-based health LLM with it.

    Source: §2.1 · p. 3, §2.2 · p. 3

  • Results

    BianQue outperforms existing models and ChatGPT on multi-turn health conversations, especially in its ability to conduct CoQ.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper identifies the conversion mechanism between questioning and suggestion as an area for future work.

    Source: §4 · p. 4

    BianQue is an early research model; its health recommendations lack rigorous examination and proofreading and cannot substitute for real-world doctors.

    Source: Ethics Statement · p. 5

Abstract

from arXiv · show

Large language models (LLMs) have performed well in providing general and extensive health suggestions in single-turn conversations, exemplified by systems such as ChatGPT, ChatGLM, ChatDoctor, DoctorGLM, and etc. However, the limited information provided by users during single turn results in inadequate personalization and targeting of the generated suggestions, which requires users to independently select the useful part. It is mainly caused by the missing ability to engage in multi-turn questioning. In real-world medical consultations, doctors usually employ a series of iterative inquiries to comprehend the patient's condition thoroughly, enabling them to provide effective and personalized suggestions subsequently, which can be defined as chain of questioning (CoQ) for LLMs. To improve the CoQ of LLMs, we propose BianQue, a ChatGLM-based LLM finetuned with the self-constructed health conversation dataset BianQueCorpus that is consist of multiple turns of questioning and health suggestions polished by ChatGPT. Experimental results demonstrate that the proposed BianQue can simultaneously balance the capabilities of both questioning and health suggestions, which will help promote the research and application of LLMs in the field of proactive health.

1 Introduction

Health LLMs can provide suggestions but often do not ask the questions needed to understand users’ situations, while doctors use iterative inquiries to inform targeted advice. BianQue pairs a multi-turn health conversation corpus with a model that outperforms existing models and ChatGPT in multi-turn conversations, especially in CoQ.

1 Introduction

  • Health LLMs often provide general suggestions without multiple rounds of questioning, although doctors use iterative inquiries to understand patients and give targeted advice.The paper defines this process as Chain of Questioning (CoQ) and attributes the gap partly to insufficient multi-turn training data and answers that favor suggestions over questions.Source: §1 · p. 1
  • Existing health LLMs were built on the assumption that users can clearly describe their situation, so their construction did not account for asking users questions.The paper notes that these models can perform well on medical QA but lack the ability to ask users questions.Source: §1 · p. 2
  • 46.2% questions and 53.8% suggestions in BianQueCorpus accompany BianQue's stronger multi-turn results versus existing models and ChatGPT, especially for CoQ.The corpus is designed to balance questioning and suggestion targets.Source: §1 · p. 2, Figure 2 · p. 3

2 Methodology

BianQueCorpus is constructed from multi-turn health conversations that are cleaned and polished to improve doctors’ suggestions. BianQue uses ChatGLM-6B as its base model and represents dialogue input across turns.

2.1 BianQueCorpus: Balancing Questioning and Suggestion

  • Real-world multi-turn health conversations were collected through data outsourcing, cleaned to address noise, and polished with ChatGPT to expand doctors’ brief suggestions.The construction process uses two stages: regular-expression-based automatic cleaning and a polishing prompt for doctors’ suggestions.Source: §2.1 · p. 3
  • Questions account for 46.2% of doctors’ answers in BianQueCorpus.Source: §2.1 · p. 3

2.2 BianQue Model

  • BianQue uses ChatGLM-6B as its base architecture, selected for being open source and for its Chinese understanding and generation performance.The input is defined in terms of dialogue turns, with N denoting the number of turns.Source: §2.2 · p. 3

3 Experiments

BianQue is compared with three baseline models on four multi-turn health datasets, using text-overlap and proactive-questioning measures. It scores better than the other models across all metrics on all four datasets.

3.1 Baselines and Benchmarks

  • The comparisons use ChatGLM-6B, ChatGPT (gpt-3.5-turbo), and DoctorGLM on test sets from four multi-turn conversation datasets with questions and suggestions in their targets.Source: §3.1 · p. 3

3.2 Implementation details

  • BianQue is finetuned with WarmupDecayLR for 25,000 training steps, using a maximum input length of 1,536 and target length of 512.Training uses batch size 80; inference uses top-p sampling with p = 0.75 and temperature τ = 0.95.Source: §3.2 · p. 3

3.3 Results and Analysis

  • Evaluation uses BLEU-1/2/3/4, ROUGE-1/2/L, and a new Proactive Questioning ability metric (PQA).PQA distinguishes samples where target and prediction are both questions, question targets predicted as suggestions, and both target and prediction are suggestions.Source: §3.3 · pp. 3–4
  • Figure 3 presents the BianQueCorpus and BianQue construction process, while Figure 4 presents the prompt for polishing doctors’ suggestions using real-world multi-turn context.Source: Figure 3 · p. 4, Figure 4 · p. 4
  • BianQue achieves better scores than the other models across all metrics on MedDialog-CN, IMCS-V2, CHIP-MDCFNPC, and MedDG.Source: §3.3 · p. 4

4 Conclusion and Future Work

The conclusion presents BianQue as a health LLM balancing questioning and suggestion, finetuned on BianQueCorpus. It reports superior multi-turn questioning ability and identifies the questioning-to-suggestion conversion mechanism as future work.

4 Conclusion and Future Work

  • BianQue is a health LLM designed to balance questioning and suggestion, finetuned on the large-scale multi-turn health conversation dataset BianQueCorpus.Source: §4 · p. 4
  • BianQueCorpus targets comprise 46.2% questions and 53.8% suggestions.Source: §4 · p. 4
  • The empirical results highlight BianQue’s superior multi-turn questioning ability, while future work should further examine the conversion mechanism between questioning and suggestion.Source: §4 · p. 4

Limitations

The authors describe BianQue as early-stage research and limit it to academic research, citing unresolved safety and mechanism concerns.

Limitations

  • BianQue is limited to academic research and cannot be used for real-world deployment.Source: Limitations · p. 4
  • Generated health suggestions may be inaccurate, so inspection and error-correction mechanisms are needed.Source: Limitations · p. 4
  • Proactive questioning can raise privacy risks by eliciting personal information, and the questioning and suggestion mechanisms remain insufficiently clear.Source: Limitations · p. 4

Ethics Statement

BianQue prioritizes proactive questioning over professional diagnosis or advice. Its dataset is cleaned to remove private information and dirty text, but the model’s questions and recommendations retain safety limitations.

Ethics Statement

  • BianQue aims to improve proactive questioning rather than provide professional medical diagnosis or advice.Source: Ethics Statement · p. 4
  • The conversation data undergoes cleansing to remove private information and dirty text; the reported excellent rate rises from 82% to 93%.Source: Ethics Statement · pp. 4–5
  • The model may involve user privacy in its questions, and its health recommendations have not undergone rigorous examination and proofreading, so they cannot replace doctors.Source: Ethics Statement · p. 5

A Reproducibility Checklist

The checklist records BianQue’s hardware, software, model configuration, and training setup. The authors say the model and corpus will be released upon the paper’s decision.

A Reproducibility Checklist

B Sample Conversations of LLMs

The examples describe three Chinese LLMs that lack multi-turn chain-of-questioning ability. They generally give universal suggestions from single-turn instructions, whereas doctors often ask multiple questions to tailor advice.

B Sample Conversations of LLMs

C Sample Conversations of BianQue

Figure 8 presents a BianQue example in which the model proactively asks questions.

C Sample Conversations of BianQue

Loading 2310.15896v2…