PaperlayerView exact revision on arXiv

Source-linked AI summary

Conversational Health Agents: A Personalized LLM-Powered Agent Framework

Mahyar Abbasian, Iman Azimi, Amir M. Rahmani, Ramesh Jain

arXiv:2310.02374v5cs.CL

TL;DR

Current conversational health agents have limited access to personalized health information and lack robust multi-step problem-solving. The paper presents openCHA, an LLM-powered orchestrator framework that uses external sources to gather information and execute actions; it demonstrates the framework through two demos and four use cases.

  • Problem

    Current CHAs have limited access to individualized health data, hindering personalized responses to users’ healthcare queries.

    Source: Existing Research Gaps and Challenges · pp. 5–6

  • Method

    openCHA uses an orchestrator to analyze queries, gather information from external sources, execute actions, and formulate personalized responses.

    Source: Orchestrator · p. 9

  • Results

    The authors demonstrate openCHA through two demos and four use cases addressing healthcare-related queries.

    Source: CONCLUSION · p. 22

  • Takeaways & Limitations

    The framework is presented as an approach for addressing healthcare queries through analysis, data collection, action execution, and personalized responses.

    Source: CONCLUSION · p. 22

    Evaluation is application-specific and requires assessing both external-source accuracy and the behavior of the configured CHA.

    Source: Study Limitations and Future Work · p. 22

Abstract

from arXiv · show

Conversational Health Agents (CHAs) are interactive systems that provide healthcare services, such as assistance and diagnosis. Current CHAs, especially those utilizing Large Language Models (LLMs), primarily focus on conversation aspects. However, they offer limited agent capabilities, specifically lacking multi-step problem-solving, personalized conversations, and multimodal data analysis. Our aim is to overcome these limitations. We propose openCHA, an open-source LLM-powered framework, to empower conversational agents to generate a personalized response for users' healthcare queries. This framework enables developers to integrate external sources including data sources, knowledge bases, and analysis models, into their LLM-based solutions. openCHA includes an orchestrator to plan and execute actions for gathering information from external sources, essential for formulating responses to user inquiries. It facilitates knowledge acquisition, problem-solving capabilities, multilingual and multimodal conversations, and fosters interaction with various AI platforms. We illustrate the framework's proficiency in handling complex healthcare tasks via two demonstrations and four use cases. Moreover, we release openCHA as open source available to the community via GitHub.

INTRODUCTION

LLM conversational systems make health information more accessible, but current healthcare agents remain limited in personalization, multi-step problem-solving, and multimodal analysis. The paper introduces an initial framework direction for combining conversation with agent capabilities and diverse health information sources.

INTRODUCTION

  • Dynamic health management requires agents to capture evolving health states through conversation and access to personal data.These data must be collected and analyzed using knowledge from research and practitioners’ experience.Source: INTRODUCTION · p. 2
  • CHAs should identify needed data, computations, and knowledge sources from multimodal interactions, then translate them into actionable health-management insights.The discussion includes text and speech, and potentially other modalities.Source: INTRODUCTION · pp. 2–3
  • Personalized conversations matter in healthcare because empathy and companionship require interactions tailored to users.Studies cited in the passage report efficacy, usability, and satisfaction for conversational CHA features.Source: INTRODUCTION · p. 3
  • Current medical LLM chatbots mainly provide basic text-based conversation, lacking multi-step problem-solving and access to users’ longitudinal data and EHRs.Their responses may therefore be generic, insufficiently tailored to individual circumstances, and potentially outdated.Source: INTRODUCTION · pp. 3–4
  • The paper presents an initial direction for combining tools and knowledge from diverse sources to develop CHAs intended to be trustworthy, understandable, and actionable.The stated aim includes a user-friendly and culturally attuned environment for a global audience.Source: INTRODUCTION · p. 4

Related Work

Prior CHA research includes conversational, specialized-health, and multimodal LLM approaches. The paper identifies gaps in personalized data access, current knowledge, analytical-tool integration, and multi-step problem-solving.

Related Work

  • Related work groups LLM-based healthcare efforts into chatbot evaluations, specialized health LLMs, and multimodal health LLMs.Examples include models combining modalities such as images, signals, and text, as well as tabular health data.Source: Related Work · p. 5

Existing Research Gaps and Challenges

  • Existing CHAs have limited access to individualized data streams, constraining their ability to provide more than generic guidance or potentially inaccurate answers.Relevant data include wearable and mHealth measurements, images, time-series, and tabular records.Source: Existing Research Gaps and Challenges · p. 6
  • Conventional LLMs rely on limited or outdated training and Internet-derived knowledge, while healthcare web content can include false information and miss reliable updates.The passage links outdated or less pertinent data to difficulty identifying hallucinations and reduced response trustworthiness.Source: Existing Research Gaps and Challenges · p. 6
  • Current agents often fail to integrate established multimodal analysis tools and predictive models that require external execution.The passage says generative AI capabilities may be overestimated, leaving established tools underused despite their ability to manage diverse data types.Source: Existing Research Gaps and Challenges · p. 6
  • Existing CHAs often lack robust data analysis or multi-step problem-solving, and may specialize in one task while ignoring other relevant health information.The cited X-ray reporting example relies solely on X-ray images.Source: Existing Research Gaps and Challenges · pp. 6–7
  • The paper argues that LLMs alone are insufficient and that practical CHAs need auxiliary components and external resources.The stated requirements include sequential reasoning, personalized health-history analysis, and data fusion.Source: Existing Research Gaps and Challenges · pp. 5, 7

Key Contributions

The paper proposes openCHA, an LLM-powered framework with an orchestrator and external resources for personalized healthcare responses. It also describes multimodal and multilingual capabilities, demos and use cases, and an open-source release.

Key Contributions

  • The proposed framework targets personalized responses by using current Internet resources and multimodal healthcare analysis tools, including machine-learning methods.The paper presents it as a holistic framework for addressing the limitations discussed earlier.Source: Key Contributions · p. 7
  • The LLM-powered Orchestrator analyzes queries, gathers required information, performs actions, and provides personalized responses..Source: Key Contributions · p. 7
  • External healthcare data sources, knowledge bases, and AI and analysis models are included to support reliable, trustworthy, and up-to-date responses..Source: Key Contributions · p. 7
  • The framework incorporates multimodal and multilingual capabilities to increase usability..Source: Key Contributions · p. 7
  • The authors report demonstrating the framework with two demos and four use cases, and release it as open source with documentation on GitHub..Source: Key Contributions · p. 7

MATERIAL AND METHODS

  • The user-facing framework uses a central agent to manage external-resource access through APIs or function calls, with bidirectional interaction for ongoing conversations.The overview identifies Interface, Orchestrator, and External Sources as its three major components.Source: MATERIAL AND METHODS · p. 8, Figure 2 · p. 8

Interface

The Interface passes user queries and multimodal metadata to openCHA’s Orchestrator, which plans and executes tasks using external sources before a separate Response Generator prepares the answer.

Interface

  • The Interface transmits user queries to the Orchestrator and supports text and audio communication, with images, audio, and gestures also usable as metadata.A meal image, for example, can accompany a question about nutritional values or calorie content.Source: Interface · pp. 8–9

Orchestrator

  • The Orchestrator aggregates and structures input, plans and executes actions with external sources, and uses the gathered information to formulate a response.The Task Planner identifies necessary steps; Tree of Thought generates three task-sequence strategies, weighs their pros and cons, and selects one, while ReAct is also available.Source: Orchestrator · p. 9
  • The Task Executor converts inputs, invokes planned tasks through external sources, and returns results to the Task Planner for further planning when needed.Planner–executor communication is iterative until the planner has sufficient information to answer; intermediate results may inform subsequent actions.Source: Orchestrator · p. 10
  • The Data Pipe stores metadata and acquired intermediate data for later retrieval, including data that may be too large or difficult for the LLMs to use directly.It can range from in-memory key/value storage to a database; stored intermediate results can be represented to the Task Planner by unique keys.Source: Orchestrator · pp. 10–11
  • The Promptist adapts query text or external-source outcomes into prompts, while a separate LLM-based Response Generator turns gathered information into an understandable answer.Separating planning from response generation allows different models and prompting techniques, with the Response Generator handling empathy and companionship and the planner focusing on personalization and up-to-date conversations.Source: Orchestrator · p. 11

External Sources

openCHA’s External Sources connect the Orchestrator to healthcare data, current healthcare knowledge, analytic models, and translation services. The framework can handle multiple data formats and support diverse-language communication.

External Sources

  • Healthcare Data Sources integrate user-consented data from EHRs, smartphones, and smartwatches, spanning biosignals, images, videos, and tabular data.Examples of platforms with third-party integration APIs include ZotCare and ilumivu.Source: External Sources · p. 12
  • Knowledge Bases retrieve current, pertinent information from healthcare literature, reputable websites, or knowledge graphs through search or retrieval models.openCHA configures knowledge bases as tasks.Source: External Sources · p. 12
  • AI and Analysis Models extract information, associations, and insights through tasks such as denoising, abstraction, classification, and event detection.The passage notes that generative models cannot effectively perform extensive computations or machine-learning inference on data.Source: External Sources · p. 12
  • Translator platforms convert languages such as users’ languages into widely spoken languages, supporting communication with diverse communities.The paper connects this integration to accessibility challenges faced by underserved communities, including educational disparities, financial constraints, and biases favoring developed nations.Source: External Sources · p. 13
  • The Orchestrator supports external sources that return text, JSON-formatted data, or unstructured data such as images and audio.The sources are selected based on the information they provide and how they interact with the Orchestrator.Source: External Sources · pp. 12–13

DEMONSTRATION

The demonstrations show openCHA planning and sequencing tasks to answer healthcare queries using linked data and analysis tools. They illustrate patient-record reporting and stress estimation, including multilingual interaction.

DEMONSTRATION

Demo 1: Patient health record reporting

  • The patient-record demo retrieves sleep and physical-activity data, performs basic statistical analysis, and can use Google Search and Extract Text tasks for internet information.Example requests ask for sleep summaries, REM sleep assessment, step counts, and activity summaries; the data come from a longitudinal college-student mental-health study.Source: Demo 1: Patient health record reporting · pp. 13–14
  • The patient-record demo shows the Task Planner selecting tasks, their execution order, and inputs for sample queries and responses.Table 1 presents a strategy chosen using Tree of Thought prompting for the first question.Source: Demo 1: Patient health record reporting · p. 14, Table 1 · p. 16

Demo 2: Objective stress level estimation with multilanguage interaction

  • The stress-estimation model achieves 86% accuracy on a test set.The demo extracts 32 HRV features, reduces them to 12 with an autoencoder, then uses a four-layer neural network to classify five stress levels.Source: Demo 2: Objective stress level… · p. 16
  • The stress-estimation demo retrieves PPG data, extracts HRV metrics, and runs stress analysis; the estimated stress level and an explanation are returned to the user.The example also demonstrates answering in multiple languages, with the question shown in English and Spanish.Source: Demo 2: Objective stress level… · pp. 14, 16, 17, Figure 5 · p. 17

openCHA Use cases

The use cases apply openCHA to personalized nutrition, diabetes recommendations, mental-health chatbot evaluation, and empathy-aware dialogue. They illustrate how external models, knowledge, and analysis tools support different healthcare applications.

openCHA Use cases

  • ChatDiet introduced a personalized, nutrition-oriented food recommendation agent using openCHA, integrating personal and population models as external sources to provide tailored food suggestions; in a case study, effectiveness was 92%, outperforming ChatGPT.Source: openCHA Use cases · p. 17
  • The diabetes agent integrated American Diabetes Association dietary guidelines and nutritional-intake tools, outperforming GPT-4 on an evaluation of 100 diabetes-related questions.The tools calculated nutritional intake to support tailored dietary recommendations.Source: openCHA Use cases · p. 18
  • The mental-health chatbot evaluator achieved the lowest mean absolute error against expert scores and provided unbiased evaluation scores.It used expert guidelines, benchmarks, and Internet search as external sources; the passage reports a maximum MAE of 10.Source: openCHA Use cases · p. 18
  • The Empathy-enhanced CHA used multimodal dialogue to interpret and respond to users’ emotional states, connecting speech-to-text, text-to-speech, and speech-emotion models.These speech models were connected to openCHA as external sources.Source: openCHA Use cases · pp. 18–19

DISCUSSION

The discussion describes openCHA’s flexibility, explainability, personalization, and reliability, while noting limits tied to planning, evaluation, ambiguity, and scalability. It concludes with future work on those challenges.

openCHA Potentials and Limitations

  • openCHA supports replaceable tasks, LLMs, and planning techniques, while its conversation history can reveal which tasks were used and how.Figure 6 illustrates an interaction in which the system identifies PPG and HRV data used to determine stress levels.Source: openCHA Potentials and Limitations · p. 19, Figure 6 · p. 21
  • Personalization and reliability depend on external sources: their quality affects recommendation effectiveness, and validated information and computations can support more reliable answers.The discussion also flags possible latency as tasks and steps increase, LLM token limits, and privacy and security requirements for external data.Source: openCHA Potentials and Limitations · p. 20

Study Limitations and Future Work

  • LLM-based planning and response generation retain risks of bias and trustworthiness problems that external sources may reduce but cannot completely eliminate.Accuracy also depends on researchers’ choices of external sources, LLM, and planning technique, as well as source quality.Source: Study Limitations and Future Work · pp. 21–22
  • Evaluation is application-specific and requires assessing both external-source accuracy and whether the configured CHA behaves as expected.The discussion also identifies ambiguous queries and scalability under larger data volumes and more simultaneous users as areas for future work.Source: Study Limitations and Future Work · p. 22

CONCLUSION

  • The paper presents openCHA as an LLM-powered framework for healthcare queries, with demonstrations spanning health-record reporting and objective stress estimation.Future work includes planning robustness, accuracy and evaluation, query ambiguity, and scalability.Source: CONCLUSION · pp. 22–23

APPENDIX 1 - SAMPLE TASK IMPLEMENTATION

Appendix 1 explains how developers add tasks as external sources to openCHA, from configuration through execution, and illustrates the process with a Google search task.

Implementation options

  • Tasks can be implemented locally or hosted as services; service providers maintain their APIs and document registration, API-key acquisition, and privacy policies.Local implementations may include a Python library, GitHub repository, or complete task implementation.Source: APPENDIX 1 - SAMPLE TASK IMPLEMENTATION · pp. 27, 29

Sample task implementation

Task configuration

  • A task’s name identifies it in the system, while its chat name supports user-facing references and explainability.The name is generally lowercase underscore_case; the chat name is typically descriptive CamelCase.Source: APPENDIX 1 - SAMPLE TASK IMPLEMENTATION · p. 29
  • Dependencies identify tasks or services needed by a task, and inputs describe what the Planner should supply to its execution function.The conventions specify dependency identifiers and input descriptions as arrays of strings.Source: APPENDIX 1 - SAMPLE TASK IMPLEMENTATION · pp. 29–30
  • Outputs describe the return format, while output_type determines whether results enter the Datapipe for later tasks or return directly to the Task Planner.The configuration uses a Boolean value for output_type.Source: APPENDIX 1 - SAMPLE TASK IMPLEMENTATION · p. 30

Execution

  • The Google search task’s _execute function receives the query supplied by the Planner, searches online, and returns the URL specified by its output configuration.The example uses the google_engine Python library.Source: APPENDIX 1 - SAMPLE TASK IMPLEMENTATION · p. 30

APPENDIX 2 - TASK PLANNER SAMPLE IMPLEMENTATION

Appendix 2 describes the Task Planner’s Tree of Thought implementation: it selects a task strategy using query and conversation context, then translates the choice into executable task calls.

Planning technique

  • The appendix identifies Tree of Thought as the planning technique used in the Task Planner.The paper also describes ReAct as an alternative prompting technique elsewhere.Source: APPENDIX 2 - TASK PLANNER SAMPLE… · p. 31

Planning prompt

  • The first planning stage supplies available tasks, metadata, previous task records, conversation history, and user input to the planner.This context is intended to support task selection, continuity, and avoidance of redundant planning.Source: APPENDIX 2 - TASK PLANNER SAMPLE… · p. 31

First planning stage

Second planning stage

  • The second stage translates the selected decision into task-function sequences, with the Orchestrator’s execute_task function retrieving and running tasks with appropriate inputs.The second prompt instructs the LLM to invoke execute_task; Figure 9 shows its structure.Source: APPENDIX 2 - TASK PLANNER SAMPLE… · p. 31, Figure 9 · p. 32

APPENDIX 3 - RESPONSE GENERATOR IMPLEMENTATION

Appendix 3 explains how openCHA builds a query-specific prompt from task planning and execution results, then directs the Response Generator to answer from those results.

Prompt construction

  • The Task Planner determines which sequence of tasks is needed for the current question, enabling openCHA to generate a mega prompt dynamically.The stated objective is to give the LLM information needed to generate more precise answers.Source: APPENDIX 3 - RESPONSE GENERATOR… · p. 33

Task results

  • The Orchestrator compiles task results into the mega prompt, omitting outputs stored in the Datapipe and providing descriptions rather than metadata files such as images.For image outputs, descriptive information can include plot details and relevant data insights.Source: APPENDIX 3 - RESPONSE GENERATOR… · p. 33

Response generation

  • The Response Generator receives formatted task-result answers called the “thinker” prompt and is instructed to rely on it rather than internal knowledge.The prompt may also include implementation guidance, and developers can customize it with a prefix for task requirements.Source: APPENDIX 3 - RESPONSE GENERATOR… · pp. 33–34

APPENDIX 4 - SAMPLE PROMPTS

Appendix 4 illustrates openCHA prompts and generated code for a sleep-improvement query, plus a separate example of passing results between tasks through the Data Pipe.

Sleep-query example

  • For “How to improve my sleep?”, the example uses a Google search followed by webpage text extraction, with the generated search query “tips to improve sleep.”The sample response prompt presents task results as its trusted source and instructs the generator not to answer beyond them.Source: APPENDIX 4 - SAMPLE PROMPTS · p. 34, Table 6 · p. 37, Table 7 · p. 38

Data Pipe example

Planning prompt

  • The first-stage prompt presents available tools and prior context, asks for three strategies with pros and cons, and requires a final selected strategy.Table 3 shows the prompt’s search and webpage-extraction tools and its example user query.Source: Table 3

Task execution

  • The second-stage planning examples show prompts for generating executable task code, followed by sample generated code for task execution.Tables 4 and 5 are identified as parts of the second-stage code-generation prompt; Table 6 shows sample code.Source: Table 4 · p. 36, Table 5 · p. 37, Table 6 · p. 37
Loading 2310.02374v5…